Backup and Restore

This guide describes the StarlingX backup and restore functionality.

Overview

This feature provides a last resort disaster recovery option for situations where the StarlingX software and/or data are compromised. The provided backup utility creates a deployment state snapshot, which can be used to restore the deployment to a previously good working state.

There are two main options for backup and restore:

Platform restore, where the platform data is re-initialized, but the applications are preserved – including OpenStack, if previously installed. During this process, you can choose to keep the Ceph cluster (Default option: wipe_ceph_osds=false) or to wipe it and restore Ceph data from off-box copies (wipe_ceph_osds=true).
OpenStack application backup and restore, where only the OpenStack application is restored. This scenario deletes the OpenStack application, re-applies the OpenStack application, and restores data from off-box copies (Glance, Ceph volumes, database).

This guide describes both restore options, including the backup procedure.

Note

Ceph application data is not backed up. It is preserved by the restore process by default (wipe_ceph_osds=false), but it is not restored if wipe_ceph_osds=true is used. You can protect against Ceph cluster failures by using off-box custom backups.
During restore, images for applications that are integrated with StarlingX are automatically downloaded to the local registry from external sources. If your system has custom Kubernetes pods that use the local registry and are not integrated with StarlingX, after restore you should confirm that the correct images are present, so the applications can restart automatically.

Backing up

There are two methods for backing up: local play method and remote play method.

Local play method

Run the following command:

ansible-playbook /usr/share/ansible/stx-ansible/playbooks/backup.yml -e "ansible_become_pass=<sysadmin password> admin_password=<sysadmin password>"

The <admin_password> and <ansible_become_pass> must be set correctly by one of the following methods:

The -e option on the command line
An override file
In the Ansible secret file

If you deploy the system with rook instead of ceph backend, you must add the rook_enabled=true variable.

ansible-playbook /usr/share/ansible/stx-ansible/playbooks/backup.yml -e "ansible_become_pass=<sysadmin password> admin_password=<sysadmin password> [ rook_enabled=true ]"

The output of the command is a file named in this format: <inventory_hostname>_platform_backup_<timestamp>.tgz

The prefixes <platform_backup_filename_prefix> and <openstack_backup_filename_prefix> can be overridden via the -e option on the command line or an override file.

The generated backup tar files will look like this: localhost_platform_backup_2019_08_08_15_25_36.tgz and localhost_openstack_backup_2019_08_08_15_25_36.tgz. They are located in the /opt/backups directory on controller-0.

Remote play method

Log in to the host where Ansible is installed and clone the playbook code from opendev at https://opendev.org/starlingx/ansible-playbooks.git
Provide an inventory file, either a customized one that is specified via the -i option or the default one which resides in the Ansible configuration directory (/etc/ansible/hosts). You must specify the IP of the controller host. For example, if the host-name is my_vbox, the inventory-file should have an entry called my_vbox as shown in the example below:
```
all:
  hosts:
    wc68:
      ansible_host: 128.222.100.02
   my_vbox:
      ansible_host: 128.224.141.74
```
Run Ansible with the command:
```
ansible-playbook <path-to-backup-playbook-entry-file> --limit host-name -i <inventory-file> -e <optional-extra-vars>
```
The generated backup tar files can be found in <host_backup_dir> which is $HOME by default. It can be overridden by the -e option on the command line or in an override file.

The generated backup tar file has the same naming convention as the local play method.

Example:

ansible-playbook /localdisk/designer/repo/cgcs-root/stx/stx-ansible-playbooks/playbookconfig/src/playbooks/backup-restore/backup.yml --limit my_vbox -i $HOME/br_test/hosts -e "host_backup_dir=$HOME/br_test ansible_become_pass=Li69nux* admin_password=Li69nux* ansible_ssh_pass=Li69nux* ansible_ssh_pass=Li69nux*"

If you deploy the system with rook instead of ceph backend, you must add the rook_enabled=true variable.

Backup content details

The backup contains the following:

Postgresql config: Backup roles, table spaces and schemas for databases
Postgresql data:
- template1, sysinv, barbican db data, fm db data,
- keystone db for primary region,
- dcmanager db for dc controller,
- dcorch db for dc controller
ETCD database
LDAP db
Ceph crushmap
DNS server list
System Inventory network overrides. These are needed at restore to correctly set up the OS configuration:
- addrpool
- pxeboot_subnet
- management_subnet
- management_start_address
- cluster_host_subnet
- cluster_pod_subnet
- cluster_service_subnet
- external_oam_subnet
- external_oam_gateway_address
- external_oam_floating_address
Docker registries on controller
Docker proxy (See docker_proxy_config for details.)
Backup data:
- OS configuration
  
  ok: [localhost] => (item=/etc) Note: Although everything here is backed up, not all of the content will be restored.
- Home directory ‘sysadmin’ user and all LDAP user accounts
  
  ok: [localhost] => (item=/home)
- Generated platform configuration
  
  ok: [localhost] => (item=/opt/platform/config/<SW_VERSION>)
  
  ok: [localhost] => (item=/opt/platform/puppet/<SW_VERSION>/hieradata) - All the hieradata in this folder is backed up. However, only the static hieradata (static.yaml and secure_static.yaml) will be restored to bootstrap controller-0.
- Keyring
  
  ok: [localhost] => (item=/opt/platform/.keyring/<SW_VERSION>)
- Patching and package repositories
  
  ok: [localhost] => (item=/opt/patching)
  
  ok: [localhost] => (item=/www/pages/updates)
- Extension filesystem
  
  ok: [localhost] => (item=/opt/extension)
- atch-vault filesystem for distributed cloud system-controller
  
  ok: [localhost] => (item=/opt/patch-vault)
- Armada manifests
  
  ok: [localhost] => (item=/opt/platform/armada/<SW_VERSION>)
- Helm charts
  
  ok: [localhost] => (item=/opt/platform/helm_charts)

Restoring

This section describes the platform restore and OpenStack restore processes.

Platform restore

In the platform restore process, the etcd and system inventory databases are preserved by default. You can choose to preserve the Ceph data or to wipe it.

To preserve Ceph cluster data, use wipe_ceph_osds=false.
To start with an empty Ceph cluster, use wipe_ceph_osds=true. After the restore procedure is complete and before you restart the applications, you must restore the Ceph data from off-box copies.

Steps:

Backup: Run the backup.yml playbook, whose output is a platform backup tarball. Move the backup tarball outside of the cluster for safekeeping.
Restore:
1. If using wipe_ceph_osds=true, then power down all the nodes.
  
  Do not power down storage nodes if using wipe_ceph_osds=false.
  
  Important
  
  It is mandatory for the storage cluster to remain functional during restore when wipe_ceph_osds=false, otherwise data loss will occur. Power down storage nodes only when wipe_ceph_osds=true.
2. Reinstall controller-0.
3. Run the Ansible restore_platform.yml playbook to restore a full system from the platform tarball archive. For this step, similar to the backup procedure, we have two options: local and remote play.
  
  Local play
  1. Download the backup to the controller. You can also use an external storage device, for example, a USB drive.
  2. Run the command:
```
ansible-playbook /usr/share/ansible/stx-ansible/playbooks/restore_platform.yml -e "initial_backup_dir=<location_of_tarball> ansible_become_pass=<admin_password> admin_password=<admin_password> backup_filename=<backup_filename>"
```
4. If you deploy the system with rook instead of ceph backend, you must add the rook_enabled=true variable in above command.
  
  Remote play
  1. Log in to the host where Ansible is installed and clone the playbook code from OpenDev at https://opendev.org/starlingx/ansible-playbooks.git
  2. Provide an inventory file, either a customized one that is specified via the -i option or the default one that resides in the Ansible configuration directory (/etc/ansible/hosts). You must specify the IP of the controller host. For example, if the host-name is my_vbox, the inventory-file should have an entry called my_vbox as shown in the example below.
```
all:
hosts:
    wc68:
    ansible_host: 128.222.100.02
my_vbox:
    ansible_host: 128.224.141.74
```
  3. Run Ansible:
```
ansible-playbook <path-to-backup-playbook-entry-file> --limit host-name -i <inventory-file> -e <optional-extra-vars>
```
    Where optional-extra-vars include:
    - <wipe_ceph_osds> is set to either wipe_ceph_osds=false (Default: Keep Ceph data intact) or wipe_ceph_osds=true (Start with an empty Ceph cluster).
    - <backup_filename> is the platform backup tar file. It must be provided via the -e option on the command line. For example, -e “backup_filename=localhost_platform_backup_2019_07_15_14_46_37.tgz”
    - <initial_backup_dir> is the location on the Ansible control machine where the platform backup tar file is placed to restore the platform. It must be provided via the -e option on the command line.
    - <admin_password>, <ansible_become_pass> and <ansible_ssh_pass> must be set correctly via the -e option on the command line or in the Ansible secret file. <ansible_ssh_pass> is the password for the sysadmin user on controller-0.
    - <ansible_remote_tmp> should be set to a new directory (no need to create it ahead of time) under /home/sysadmin on controller-0 via the -e option on the command line.
    Example command:
```
ansible-playbook /localdisk/designer/jenkins/tis-stx-dev/cgcs-root/stx/ansible-playbooks/playbookconfig/src/playbooks/restore_platform.yml --limit my_vbox -i $HOME/br_test/hosts -e "ansible_become_pass=Li69nux* admin_password=Li69nux* ansible_ssh_pass=Li69nux* initial_backup_dir=$HOME/br_test backup_filename=my_vbox_system_backup_2019_08_08_15_25_36.tgz ansible_remote_tmp=/home/sysadmin/ansible-restore"
```
5. If you deploy the system with rook instead of ceph backend, you must add the rook_enabled=true variable in above command.
6. After Ansible is executed, perform the following steps based on your deployment mode:
  
  AIO-SX
  1. Unlock controller-0 and wait for it to boot.
  2. Applications should transition from restore-requested to applying and make a final transition to applied state. If applications transition from applying to restore-requested state, ensure there is network access and access to the Docker registry. The process is repeated once per minute until all applications are transitioned to the applied state.
  AIO-DX
  1. Unlock controller-0 and wait for it to boot.
  2. Applications should transition from restore-requested to applying and make a final transition to applied state. If applications transition from applying to restore-requested state, ensure there is network access and access to the Docker registry. The process is repeated once per minute until all applications are transitioned to the applied state.
  3. Reinstall controller-1 (boot it from PXE, wait for it to become online).
  4. Unlock controller-1.
  Standard (with controller storage)
  1. Unlock controller-0 and wait for it to boot. After unlock, you will see all nodes, including storage nodes, as offline.
  2. Applications should transition from restore-requested to applying and make a final transition to applied state. If applications transition from applying to restore-requested state, ensure there is network access and access to the Docker registry. The process is repeated once per minute until all applications are transitioned to the applied state.
  3. Reinstall controller-1 and compute nodes (boot them from PXE, wait for them to become online).
  4. Unlock controller-1 and wait for it to be available.
  5. Unlock compute nodes and wait for them to be available.
  Standard (without controller storage)
  1. Unlock controller-0 and wait for it to boot. After unlock, you will see all nodes, except storage nodes, as offline. If wipe_ceph_osds=false is used, storage nodes must be powered on and in the available state throughout the procedure. Otherwise, storage nodes must be powered off.
  2. Applications should transition from restore-requested to applying and make a final transition to applied state. If applications transition from applying to restore-requested state, ensure there is network access and access to the Docker registry. The process is repeated once per minute until all applications are transitioned to the applied state.
  3. Reinstall controller-1 and compute nodes (boot them from PXE, wait for them to become online).
  4. Unlock controller-1 and wait for it to be available.
  5. If wipe_ceph_osds=true is used, then reinstall storage nodes.
  6. Unlock compute nodes and wait for them to be available.
  7. (Optional) Reinstall storage nodes.
7. Wait for Calico and Coredns pods to start. Run the system restore-complete command. Type 750.006 alarms will disappear one at a time, as the applications are being auto-applied.

OpenStack application backup and restore

In this procedure, only the OpenStack application will be restored.