opendev/system-config
Code Issues Proposed changes

Add docs on restoring a gitea repository

Browse Source 
We have discovered that it is possible for a gitea repository to be come
corrupted. Since gitea is not the source of truth the easiest way to
handle this is to replace the repo with a new empty repository and have
Gerrit replicate back to it. This adds documentation that walks through
the process of doing this.

Change-Id: Ief990adaaf3cbb3c748bc9ee6ceb466a1104915a
This commit is contained in:
Clark Boylan 2022-03-02 11:16:36 -08:00
parent 93e2b84df0
commit 6f178c2737
1 changed files with 40 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

41
doc/source/gitea.rst
View File

@ -31,6 +31,8 @@ The OpenDev Git repositories are hosted on a pool of servers. They
are served via https using Gitea behind HAProxy which handles load
balancing across the nodes.
.. _gitea-maintenance:
Backend Maintenance
===================
@ -68,7 +70,7 @@ repo rename redirects) so there is a small amount of process required
to deploy a new Gitea backend.
To deploy a new Gitea backend we add it to the ansible inventory, but
exclude it from the manage-projects.yaml playbook. This will full
exclude it from the manage-projects.yaml playbook. This will fully
provision an empty Gitea server with running Gitea and database processes.
Then we can manually restore the database from another node, create all of
the bare git repos, and replicate all of the repo content.
@ -139,3 +141,40 @@ You can monitor the progress of this with::
Once this is complete, add the server back into the haproxy as
discussed above.
Reset a Corrupted Git Repo
==========================
It is possible for the repos Gitea hosts to become corrupted. Since Gerrit
is our source of truth the easiest way to handle this is have Gerrit replicate
the data back to Gitea. Unfortunately, replication will fail with a corrupted
repo on the destination. To work around this we replace the repo with a new
empty bare repository and then replicate.
First thing to do is :ref:`remove the backend from the haproxy rotation<gitea-maintenance>`.
Next we need to stop gitea on the backend::
cd /etc/gitea-docker
docker-compose down
Then move aside the old repo and replace it with a new empty bare repo::
cd /root/corrupted_repos
mv /var/gitea/data/git/repositories/org/example.git ./example.git.bak
git init --bare example.git
chown -R 1000:1000 example.git
mv ./example.git /var/gitea/data/git/repositories/org/example.git
Now start the gitea service back up again::
cd /etc/gitea-docker
docker-compose up -d mariadb gitea-web
# Wait for web to be responseive
docker-compose up -d gitea-ssh
Finally trigger a Gerrit replication::
ssh -p 29418 user.admin@review.opendev.org replication start --url $giteabackendname org/example
Once replication is complete you can add the backend to the haproxy rotation.