borg-backup: add fuse

Add the FUSE dependencies for our hosts backed up with borg, along
with a small script to make mounting the backups easier.  This is the
best way to recover something quickly in what is sure to be a
stressful situation.

Documentation and testing is updated.

Change-Id: I1f409b2df952281deedff2ff8f09e3132a2aff08

doc/source/sysadmin.rst

@@ -183,8 +183,13 @@ key setup just for backup communication (see ``/root/.ssh/config``).
 Restore from Backup
 -------------------
 
-``borg`` has many options for restoring but a basic way to dump a host
-at a particular time is to
+Hosts have ``/usr/local/bin/borg-mount`` (specify one of the backup
+servers as an argument) that will mount the backups to
+``/opt/backups`` via FUSE.
+
+``borg`` has other options for restoring.  If you need to extract on
+the backup server itself, a basic way to dump a host at a particular
+time is to
 
 * log into the backup server
 * sudo ``su -`` to switch to the backup user for the host to be restored
@@ -194,6 +199,7 @@ at a particular time is to
 * move to working directory
 * extract one of the appropriate archives with ``/opt/borg/bin/borg extract ~/backup <archive-tag>``
 
+
 Rotating backup storage
 -----------------------
 

playbooks/roles/borg-backup/tasks/main.yaml

@@ -16,6 +16,12 @@
     dest: /usr/local/bin/borg-backup
     mode: 0755
 
+- name: Install mount script
+  template:
+    src: borg-mount.j2
+    dest: /usr/local/bin/borg-mount
+    mode: 0755
+
 - name: Generate keypair for backups
   openssh_keypair:
     path: /root/.ssh/id_borg_backup_ed25519

playbooks/roles/borg-backup/templates/borg-mount.j2

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+if [ -z "$1" ]; then
+    echo "Must specify backup host"
+    exit 1
+fi
+
+BORG="/opt/borg/bin/borg"
+
+# Setting this, so the repo does not need to be given on the commandline:
+export BORG_REPO="ssh://{{ borg_username}}@${1}/opt/backups/{{ borg_username }}/backup"
+
+# some helpers and error handling:
+info() { printf "\n%s %s\n\n" "$( date )" "$*" >&2; }
+
+info "Mounting backup to /opt/backups"
+
+mkdir -p /opt/backups
+$BORG mount $BORG_REPO /opt/backups
+
+info "To unmount: $BORG umount /opt/backups"

playbooks/roles/install-borg/tasks/main.yaml

@@ -13,6 +13,9 @@
       - libacl1-dev
       - libacl1
       - build-essential
+      - libfuse-dev
+      - fuse
+      - pkg-config
 
 - name: Install borg
   pip:
@@ -20,6 +23,6 @@
     # but the requirements don't bring it in.
     name:
       - cython
-      - 'borgbackup=={{ borg_version }}'
+      - 'borgbackup[fuse]=={{ borg_version }}'
     virtualenv: /opt/borg
     virtualenv_command: /usr/bin/python3 -m venv

testinfra/test_borg_backups.py

@@ -75,3 +75,16 @@ def test_borg_backup(host):
         '/usr/local/bin/borg-backup borg-backup01.region.provider.opendev.org 2>> '
         '/var/log/borg-backup-borg-backup01.region.provider.opendev.org.log')
     assert cmd.succeeded
+
+    cmd = host.run(
+        '/usr/local/bin/borg-mount borg-backup01.region.provider.opendev.org')
+    assert cmd.succeeded
+
+    cmd = host.run('ls /opt/backups')
+    # this directory should now have a directory
+    #  borg-backup-test0X-YYYY-MM-DDT...
+    assert 'borg-backup-test' in cmd.stdout
+
+    # unmount it for sanity
+    cmd = host.run('umount /opt/backups')
+    assert cmd.succeeded