diff --git a/doc/source/backup/kubernetes/backing-up-starlingx-system-data.rst b/doc/source/backup/kubernetes/backing-up-starlingx-system-data.rst index 2d7c66087..dc7252ff0 100644 --- a/doc/source/backup/kubernetes/backing-up-starlingx-system-data.rst +++ b/doc/source/backup/kubernetes/backing-up-starlingx-system-data.rst @@ -32,7 +32,9 @@ Contents of System Backup .. _backing-up-starlingx-system-data-ul-s3t-bz4-kjb: -The following content is included in the backup: +The following content is included in the backup list: + +- Hashicorp Vault contents (when ``backup_hc_vault`` ansible option is selected) - All platform configuration data required to fully restore the system to a working state following the platform restore procedure. @@ -76,6 +78,9 @@ System Backup Size Consider the following for backup size: +- The size of the backup for the Vault application depends on the amount of data + stored in the Vault. + - The base size of a platform system backup sizes range from 10MB to 30MB, depending on the size of the system and deployment. |AIO-SX| systems are typically 20MB or less. diff --git a/doc/source/backup/kubernetes/index-backup-kub-699e0d16c076.rst b/doc/source/backup/kubernetes/index-backup-kub-699e0d16c076.rst index 0500d7411..254ec4762 100644 --- a/doc/source/backup/kubernetes/index-backup-kub-699e0d16c076.rst +++ b/doc/source/backup/kubernetes/index-backup-kub-699e0d16c076.rst @@ -22,7 +22,9 @@ System and storage restore restoring-starlingx-system-data-and-storage running-restore-playbook-locally-on-the-controller + run-hashicorp-vault-restore-playbook-locally-on-the-controller-10daacd4abdc system-backup-running-ansible-restore-playbook-remotely + run-hashicorp-vault-restore-playbook-remotely-436250ea3ed7 node-replacement-for-aiominussx-using-optimized-backup-and-restore-6603c650c80d diff --git a/doc/source/backup/kubernetes/restoring-starlingx-system-data-and-storage.rst b/doc/source/backup/kubernetes/restoring-starlingx-system-data-and-storage.rst index 29a2e8ac4..4c3423982 100644 --- a/doc/source/backup/kubernetes/restoring-starlingx-system-data-and-storage.rst +++ b/doc/source/backup/kubernetes/restoring-starlingx-system-data-and-storage.rst @@ -83,6 +83,9 @@ conditions are in place: accessible remotely, if restore is done by running Ansible Restore playbook remotely. +- If ``backup encryption`` is enabled during platform backup, you have the + encryption passphrase needed to decrypt the archive file. + - You have the original |prod| ISO installation image available on a USB flash drive. It is mandatory that you use the exact same version of the software used during the original installation, otherwise the restore @@ -395,6 +398,14 @@ conditions are in place: $ system application-upload |prefix|-openstack-20.12-0.tgz $ system application-apply |prefix|-openstack +#. If Hashicorp Vault data was also backed up, it can be restored using the + vault_restore playbook. For more information on Hashicorp Vault restore + see: + + - :ref:`Run Hashicorp Vault Restore Playbook Locally on the Controller ` + + - :ref:`Run Hashicorp Vault Restore Playbook Remotely `. + #. Run the :command:`system restore-complete` command. .. code-block:: none diff --git a/doc/source/backup/kubernetes/run-hashicorp-vault-restore-playbook-locally-on-the-controller-10daacd4abdc.rst b/doc/source/backup/kubernetes/run-hashicorp-vault-restore-playbook-locally-on-the-controller-10daacd4abdc.rst new file mode 100644 index 000000000..012febe8c --- /dev/null +++ b/doc/source/backup/kubernetes/run-hashicorp-vault-restore-playbook-locally-on-the-controller-10daacd4abdc.rst @@ -0,0 +1,77 @@ +.. WARNING: Add no lines of text between the label immediately following +.. and the title. + +.. _run-hashicorp-vault-restore-playbook-locally-on-the-controller-10daacd4abdc: + +============================================================== +Run Hashicorp Vault Restore Playbook Locally on the Controller +============================================================== + +Similar to platform restore, Hashicorp Vault restore backup files must be on +the active controller. + +You can use an external storage device, for example, a USB drive. Use the +following command to run the Hashicorp Vault Restore playbook: + +.. code-block:: none + + ~(keystone_admin)]$ ansible-playbook /usr/share/ansible/stx-ansible/playbooks/vault_restore.yml -e "initial_backup_dir= backup_filename= on_box_data=" + +Below you can find other ``-e`` command line options: + +**Common** + +``-e "initial_backup_dir=/home/sysadmin"`` + Where the backup .tgz files are located on the box. + +``-e backup_filename=localhost_hc_vault_backup.tgz`` + The basename of the platform backup tgz. The full path will be a + combination ``{{initial_backup_dir}/{backup_filename}}}`` + +``-e backup_encryption_enabled=false`` + If ``backup encryption`` is selected during platform backup, then this option + is also required when you run the Vault restore procedure. The archive file + of Vault backup contains metadata which indicates whether the snapshot was + encrypted with a user supplied passphrase, for example: + + .. code-block:: none + + [sysadmin@controller-0 ~(keystone_admin)]$ VAULT_ARCHIVE=/opt/platform-backup/localhost_hc_vault_backup_2024_07_17_18_00_24.tgz + + [sysadmin@controller-0 ~(keystone_admin)]$ METAF="$( tar tf $VAULT_ARCHIVE | grep "tar.metadata$" )" + + [sysadmin@controller-0 ~(keystone_admin)]$ tar xf "$VAULT_ARCHIVE" -O "$METAF" | jsonpath.py 'user_encrypted' + + Select the ``backup_encryption_enabled`` option if user_encrypted is 'true' + in the metadata. + +``-e backup_encryption_passphrase=`` + When ``backup_encryption_enabled`` is set to true, then the + ``backup_encryption_passphrase`` option is also required. Consider using + ansible-vault to store and specify passwords. + +(Optional): You can set these options to specify the backup file location: + +- To define a convenient place to store the backup files, defined by + ``initial-backup_dir``, on the system (such as the home folder for sysadmin, + or /tmp, or even a mounted USB device), use the ``on_box_data=true/false`` + parameter. + + If this parameter is set to true, Hashicorp Vault playbook will look + for the backup file provided on the target server. The parameter + ``initial_backup_dir`` can be omitted from the command line. In this + case, the backup file will be under ``/opt/platform-backup`` directory. + + If this parameter is set to false, the Hashicorp Vault playbook will + look for backup files provided on the Ansible controller. In this + case, both the ``initial_backup_dir`` and ``backup_filename`` must be + specified in the command. + + Example of a backup file in ``/home/sysadmin``: + + .. code-block:: none + + ~(keystone_admin)]$ ansible-playbook /usr/share/ansible/stx-ansible/playbooks/vault_restore.yml -e "initial_backup_dir=/home/sysadmin backup_filename=localhost_hc_vault_backup_2020_07_27_07_48_48.tgz" + + + diff --git a/doc/source/backup/kubernetes/run-hashicorp-vault-restore-playbook-remotely-436250ea3ed7.rst b/doc/source/backup/kubernetes/run-hashicorp-vault-restore-playbook-remotely-436250ea3ed7.rst new file mode 100644 index 000000000..8ad86a374 --- /dev/null +++ b/doc/source/backup/kubernetes/run-hashicorp-vault-restore-playbook-remotely-436250ea3ed7.rst @@ -0,0 +1,114 @@ +.. WARNING: Add no lines of text between the label immediately following +.. and the title. + +.. _run-hashicorp-vault-restore-playbook-remotely-436250ea3ed7: + +============================================= +Run Hashicorp Vault Restore Playbook Remotely +============================================= + +In this method you can run Ansible Hashicorp Vault playbook and point to controller-0. + +.. rubric:: |prereq| + +.. _system-backup-running-ansible-restore-playbook-remotely-ul-ylm-g44-bkb: + +- It is recommended that you have Ansible version 2.7.5 or higher installed + on your remote workstation. Copy the Ansible Backup/Restore playbooks + from directory ``/usr/share/ansible/stx-ansible/playbooks/``. + +- Your network has IPv6 connectivity before running Ansible Playbook, if + the system configuration is IPv6. + +.. rubric:: |proc| + +.. _system-backup-running-ansible-restore-playbook-remotely-steps-sgp-jjc-ljb: + +#. Log in to the remote workstation. + + You can log in directly on the console or remotely using :command:`ssh`. + +#. Provide an inventory file, either a customized one that is specified + using the ``-i`` option, or the default one that is in the Ansible + configuration directory (that is, /etc/ansible/hosts). You must + specify the floating |OAM| IP of the controller host. For example, if the + host name is |prefix|\_Cluster, the inventory file should have an entry + called |prefix|\_Cluster. + + .. parsed-literal:: + + --- + all: + hosts: + wc68: + ansible_host: 128.222.100.02 + |prefix|\_Cluster: + ansible_host: 128.224.141.74 + +#. Run the Ansible Hashicorp Vault playbook. + + .. code-block:: none + + ~(keystone_admin)]$ ansible-playbook path-to-hashicorp-vault-restore-playbook-entry-file --limit host-name -i inventory-file -e optional-extra-vars + + where ``optional-extra-vars`` can be: + + - To set a convenient place to store the backup files defined by + initial-backup_dir on the system (such as the home folder for sysadmin, + or /tmp, or even a mounted USB device), use the following parameter: + + .. code-block:: + + on_box_data=true/false + + If this parameter is set to true, Ansible Hashicorp Vault playbook will look + for the backup file provided on the target server. The parameter + ``initial_backup_dir`` can be omitted from the command line. In this + case, the backup file will be under ``/opt/platform-backup`` directory. + + If this parameter is set to false, the Ansible Hashicorp Vault playbook will + look for a backup file provided on the Ansible controller. In this + case, both the ``initial_backup_dir`` and ``backup_filename`` must be + specified in the command. + + - ``backup_filename`` is the platform backup tar file. It must be + provided using the ``-e`` option on the command line, for example: + + .. code-block:: none + + -e backup_filename= localhost_hc_vault_backup_2019_07_15_14_46_37.tgz + + - The ``initial_backup_dir`` is the location where the platform backup + tar file is placed to restore the platform. It must be provided using + ``-e`` option on the command line. + + .. note:: + + When ``on_box_data=false``, ``initial_backup_dir`` must be defined. + + - The :command:`admin_password`, :command:`ansible_become_pass`, + and :command:`ansible_ssh_pass` need to be set correctly using + the ``-e`` option on the command line or in the Ansible secret file. + :command:`ansible_ssh_pass` is the password to the sysadmin user + on controller-0. + + - If ``backup encryption`` was enabled when the Vault backup was taken + then the options ``backup_encryption_enabled=true`` and + ``backup_encryption_passphrase="`` are also + required when restoring the Vault snapshot. Consider storing the + ``backup_encryption_passphrase`` in the Ansible secret file. + + The archive file of the Vault backup contains metadata which indicates + whether the snapshot was encrypted with user supplied passphrase, + for example: + + .. code-block:: none + + [sysadmin@controller-0 ~(keystone_admin)]$ VAULT_ARCHIVE=/opt/platform-backup/localhost_hc_vault_backup_2024_07_17_18_00_24.tgz + + [sysadmin@controller-0 ~(keystone_admin)]$ METAF="$( tar tf $VAULT_ARCHIVE | grep "tar.metadata$" )" + + [sysadmin@controller-0 ~(keystone_admin)]$ tar xf "$VAULT_ARCHIVE" -O "$METAF" | jsonpath.py 'user_encrypted' + + Select the ``backup_encryption_enabled`` option if ``user_encrypted`` + is 'true' in the metadata. diff --git a/doc/source/backup/kubernetes/running-ansible-backup-playbook-locally-on-the-controller.rst b/doc/source/backup/kubernetes/running-ansible-backup-playbook-locally-on-the-controller.rst index f12bcad00..9c0c628d0 100644 --- a/doc/source/backup/kubernetes/running-ansible-backup-playbook-locally-on-the-controller.rst +++ b/doc/source/backup/kubernetes/running-ansible-backup-playbook-locally-on-the-controller.rst @@ -56,7 +56,7 @@ Following are the ``-e`` command line options: Used in conjunction with optimized restore. This will create a backup of ``registry.local``. The file generated by this flag will be named according to the pattern ``_image_registry_backup_.tgz``. - + **Common** ``-e backup_dir=/opt/backups`` @@ -66,9 +66,31 @@ Following are the ``-e`` command line options: When set to true, the backup playbook is allowed to be run on an unhealthy system. This is needed in some extreme cases. - .. warning:: +.. warning:: - Restoring a backup from an unhealthy system will lead to undefined behavior and/or failure. + Restoring a backup from an unhealthy system will lead to undefined behavior + and/or failure. + + +``-e backup_encryption_enabled=false`` + When set to true, the platform backup file and Hashicorp Vault backup will + be encrypted. When this option is true then ``backup_encryption_passphrase`` + is also requierd. The platform backup file will have a ".gpg" extension when + encrypted. The Hashicorp Vault snapshot is encrypted before it is written + to disk, so that the backup file remains a tar file. + +``-e backup_encryption_passphrase=`` + When ``backup_encryption_enabled`` is set to true, then the + ``backup_encryption_passphrase`` option is also required. + +.. note:: + + It is also recommended to put the ``backup_encryption_passphrase`` into + ansible-vault. + +``-e backup_hc_vault=false`` + When set to true, the backup playbook will also backup the Hashicorp Vault + application, if the application is applied in the system. To exclude a directory and all the files in it like ``/var/home*`` you can use the optional parameter ``-e "exclude_dirs=/var/home/**,/var/home"``. @@ -109,6 +131,7 @@ command if the file needs to be edited after it is created. admin_password: "" backup_registry_filesystem: "true" exclude_dirs: /var/home/**,/var/home" + backup_encryption_passphrase: "" ... EOF @@ -130,15 +153,17 @@ system configuration. .. _running-ansible-backup-playbook-locally-on-the-controller-ul-wj1-vxh-pmb: -- ``inventory_hostname_platform_backup_timestamp.tgz`` +- inventory_hostname_platform_backup_timestamp.tgz -- ``inventory_hostname_wr-openstack_backup_timestamp.tgz`` +- inventory_hostname_wr-openstack_backup_timestamp.tgz -- ``inventory_hostname_user_images_backup_timestamp.tgz`` +- inventory_hostname_user_images_backup_timestamp.tgz -- ``inventory_hostname_dc_vault_backup_timestamp.tgz`` +- inventory_hostname_dc_vault_backup_timestamp.tgz -- ``inventory_hostname_image_registry_backup_timestamp.tgz`` +- inventory_hostname_image_registry_backup_timestamp.tgz + +- inventory_hostname_hc_vault_backup_timestamp.tgz The output files' prefixes can be overridden with the following variables using the ``-e`` option on the command line or by using an override file. @@ -157,20 +182,27 @@ using the ``-e`` option on the command line or by using an override file. - registry_filesystem_backup_filename_prefix +- hc_vault_backup_filename_prefix + The generated backup tar files will be displayed in the following format, when custom prefixes are not specified, for example: .. _running-ansible-backup-playbook-locally-on-the-controller-ul-p3b-f13-pmb: -- ``localhost_docker_local_registry_backup_2020_07_15_21_24_22.tgz`` +- localhost_docker_local_registry_backup_2024_07_15_21_24_22.tgz -- ``localhost_platform_backup_2020_07_15_21_24_22.tgz`` +- localhost_platform_backup_2024_07_15_21_24_22.tgz -- ``localhost_openstack_backup_2020_07_15_21_24_22.tgz`` +- localhost_openstack_backup_2024_07_15_21_24_22.tgz -- ``localhost_dc_vault_backup_2020_07_15_21_24_22.tgz`` +- localhost_dc_vault_backup_2024_07_15_21_24_22.tgz -- ``localhost_image_registry_backup_2020_07_15_21_24_22.tgz`` +- localhost_image_registry_backup_2024_07_15_21_24_22.tgz + +- localhost_hc_vault_backup_2024_07_15_21_24_22.tgz + +When encryption is enabled, the platform backup file will have a ".gpg" extension, +for example, "localhost_platform_backup_2024_07_15_21_24_22.tgz.gpg" These files are located by default in the ``/opt/backups`` directory on controller-0, and contains the complete system backup. diff --git a/doc/source/backup/kubernetes/running-restore-playbook-locally-on-the-controller.rst b/doc/source/backup/kubernetes/running-restore-playbook-locally-on-the-controller.rst index 961204a10..d41ba31f5 100644 --- a/doc/source/backup/kubernetes/running-restore-playbook-locally-on-the-controller.rst +++ b/doc/source/backup/kubernetes/running-restore-playbook-locally-on-the-controller.rst @@ -44,6 +44,16 @@ Below you can find other ``-e`` command line options: The basename of the platform backup tgz. The full path will be a combination ``{initial_backup_dir}/{backup_filename}`` +``-e backup_encryption_enabled=false`` + If ``backup encryption`` is selected during platform backup, then this option + is also required when you run the platform restore procedure. Select this + option if the platform backup archive file includes the '.gpg' extension. + +``-e backup_encryption_passphrase=`` + When ``backup_encryption_enabled`` is set to true, then the + ``backup_encryption_passphrase`` option is also required. Consider using + ansible-vault to store and specify passwords. + .. _running-restore-playbook-locally-on-the-controller-steps-usl-2c3-pmb: (Optional): You can select one of the following restore modes: diff --git a/doc/source/backup/kubernetes/system-backup-running-ansible-restore-playbook-remotely.rst b/doc/source/backup/kubernetes/system-backup-running-ansible-restore-playbook-remotely.rst index 23de8ea87..63081484b 100644 --- a/doc/source/backup/kubernetes/system-backup-running-ansible-restore-playbook-remotely.rst +++ b/doc/source/backup/kubernetes/system-backup-running-ansible-restore-playbook-remotely.rst @@ -71,7 +71,7 @@ In this method you can run Ansible Restore playbook and point to controller-0. wipe_ceph_osds=true - - To set a convinient place to store the backup files defined by + - To set a convenient place to store the backup files defined by ``initial-backup_dir`` on the system (such as the home folder for sysadmin, or /tmp, or even a mounted USB device), use the following parameter: @@ -111,6 +111,12 @@ In this method you can run Ansible Restore playbook and point to controller-0. :command:`ansible_ssh_pass` is the password to the sysadmin user on controller-0. + - If ``backup encryption`` was enabled during a platform backup then the + options ``backup_encryption_enabled=true`` and + ``backup_encryption_passphrase=""`` are also + required when restoring the platform. Consider storing the + ``backup_encryption_passphrase`` in the Ansible secret file. + - The :command:`ansible_remote_tmp` should be set to a new directory (not required to create it ahead of time) under ``/home/sysadmin`` on controller-0 using the ``-e`` option on the command line.