From 6f178c273710e937bd9fec6a5be2a1413c7b514d Mon Sep 17 00:00:00 2001
From: Clark Boylan <clark.boylan@gmail.com>
Date: Wed, 2 Mar 2022 11:16:36 -0800
Subject: [PATCH] Add docs on restoring a gitea repository

We have discovered that it is possible for a gitea repository to be come
corrupted. Since gitea is not the source of truth the easiest way to
handle this is to replace the repo with a new empty repository and have
Gerrit replicate back to it. This adds documentation that walks through
the process of doing this.

Change-Id: Ief990adaaf3cbb3c748bc9ee6ceb466a1104915a
---
 doc/source/gitea.rst | 41 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 40 insertions(+), 1 deletion(-)

diff --git a/doc/source/gitea.rst b/doc/source/gitea.rst
index a060031da5..367cdd369b 100644
--- a/doc/source/gitea.rst
+++ b/doc/source/gitea.rst
@@ -31,6 +31,8 @@ The OpenDev Git repositories are hosted on a pool of servers.  They
 are served via https using Gitea behind HAProxy which handles load
 balancing across the nodes.
 
+.. _gitea-maintenance:
+
 Backend Maintenance
 ===================
 
@@ -68,7 +70,7 @@ repo rename redirects) so there is a small amount of process required
 to deploy a new Gitea backend.
 
 To deploy a new Gitea backend we add it to the ansible inventory, but
-exclude it from the manage-projects.yaml playbook. This will full
+exclude it from the manage-projects.yaml playbook. This will fully
 provision an empty Gitea server with running Gitea and database processes.
 Then we can manually restore the database from another node, create all of
 the bare git repos, and replicate all of the repo content.
@@ -139,3 +141,40 @@ You can monitor the progress of this with::
 
 Once this is complete, add the server back into the haproxy as
 discussed above.
+
+Reset a Corrupted Git Repo
+==========================
+
+It is possible for the repos Gitea hosts to become corrupted. Since Gerrit
+is our source of truth the easiest way to handle this is have Gerrit replicate
+the data back to Gitea. Unfortunately, replication will fail with a corrupted
+repo on the destination. To work around this we replace the repo with a new
+empty bare repository and then replicate.
+
+First thing to do is :ref:`remove the backend from the haproxy rotation<gitea-maintenance>`.
+
+Next we need to stop gitea on the backend::
+
+  cd /etc/gitea-docker
+  docker-compose down
+
+Then move aside the old repo and replace it with a new empty bare repo::
+
+  cd /root/corrupted_repos
+  mv /var/gitea/data/git/repositories/org/example.git ./example.git.bak
+  git init --bare example.git
+  chown -R 1000:1000 example.git
+  mv ./example.git /var/gitea/data/git/repositories/org/example.git
+
+Now start the gitea service back up again::
+
+  cd /etc/gitea-docker
+  docker-compose up -d mariadb gitea-web
+  # Wait for web to be responseive
+  docker-compose up -d gitea-ssh
+
+Finally trigger a Gerrit replication::
+
+  ssh -p 29418 user.admin@review.opendev.org replication start --url $giteabackendname org/example
+
+Once replication is complete you can add the backend to the haproxy rotation.