Gitlab Upgrade to v14.9.1

We discussed about gitlab-deployment using docker before, Now we will discuss about how to upgrade the gitlab from v13.0.12 to v14.9.1

I checked for an update path but none of them worked. Then I deployed each version and found a working upgrade path. I hope it helps someone.

Upgrade path:-

13.0.12 — -> 14.0.12 —- →14.1.5— -→14.2.3 — -→14.6.5 — -→14.9.1

— — — — — — — — — — — — — — — Loop Start — — — — — — — — — — — — — — — — —

Take backup from git-lab v13.0.12,

docker exec <container-name> gitlab-rake gitlab:backup:create

Backup will be stored inside the container /var/opt/gitlab/backups or backup will be also available from the mounted path.

Deploy the gitlab v14.0.12,

Copy the backup to new gitlab container,

docker cp <backup-path/backup> <container-name>:/var/opt/gitlab/backup

Check if any git-lab services are running:-

docker exec -it <container-name> gitlab-ctl status

If any services are running please stop it,

docker exec -it <name of container> gitlab-ctl stop puma
docker exec -it <name of container> gitlab-ctl stop sidekiq

Then restore the backup,

docker exec -it <name of container> gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce

# OR

docker exec -it <name of container> gitlab-backup restore BACKUP=/var/opt/gitlab/backup/11493107454_2018_04_25_10.6.4-ce

Check gitlab status,

docker exec -it <name of container> gitlab-rake gitlab:check SANITIZE=true

Check if any background migration job is running,

# Exec into the conatiner
docker exec -it <container-name> /bin/bash

# Wait untill the background migration finishes
gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending.count'

*****Important*******
## Go to the next step if the above commands returns zero value, If you don't wait to finish the backgound migration, It will throw error.

Now migrate the Database,

# Exec into the conatiner
docker exec -it <conatiner-name> /bin/bash

# Migrate the gitlab database
gitlab-rake db:migrate

Check the status of git-lab,

# Exec into the conatiner
docker exec -it <conatiner-name> /bin/bash

# Recongiure gitlab
gitlab-ctl reconfigure

# To check whether there is a issue
gitlab-rake gitlab:check and gitlab-rake gitlab:check SANITIZE=true 

# To check whether there is issue with nginx
gitlab-ctl tail

# Restart gitlab
gitlab-ctl restart

If no issues, then repeat this loop till you reach git-lab v14.9.1 or your target version.

— — — — — — — — — — — — — — — Loop Ends— — — — — — — — — — — — — — — — —

Possible issues:

You will get 500 error when trying to configure webhook, disable the inbuilt CI/CD …etc

Fix:

# Exec into the conatiner
docker exec -it <conatiner-name> /bin/bash

# Get into the gitlab postgres database
gitlab-psql -d gitlabhq_production

# Check if any value is there
SELECT * FROM public."ci_group_variables";
SELECT * FROM public."ci_variables";

# Delete the data in the table
DELETE FROM ci_group_variables;
DELETE FROM ci_variables;

# OR

DELETE FROM ci_group_variables WHERE group_id = <GROUPID>;
DELETE FROM ci_variables WHERE project_id = <PROJECTID>;

# Set webhook and ci/cd values as zero
UPDATE projects SET runners_token = null, runners_token_encrypted = null;
UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
UPDATE application_settings SET runners_registration_token_encrypted = null;
UPDATE application_settings SET encrypted_ci_jwt_signing_key = null;
UPDATE ci_runners SET token = null, token_encrypted = null;
UPDATE ci_builds SET token = null, token_encrypted = null;
TRUNCATE web_hooks CASCADE;

After this, all the web-hooks and saved settings will be gone and you should reconfigure them.

Hope this helps…