From 562829e2ecf3b58f8a72d3ba57181a618e8faf99 Mon Sep 17 00:00:00 2001 From: Ngairangbam Mili Date: Mon, 18 Nov 2024 15:08:43 +0000 Subject: [PATCH] Subcloud Remote Install, Upgrade and Prestaging Adaptation Story: 2010676 Task: 51372 Patchset 7: Removed the occurences of upgrades as confirmed by the dev team. Change-Id: I57bc08833abd643fe1007e06a8b88cc2c49b4271 Signed-off-by: Ngairangbam Mili --- ...-subcloud-using-dcmanager-df756866163f.rst | 278 ++++++++++++------ ...ge-subcloud-orchestration-eb516473582f.rst | 78 +++-- 2 files changed, 225 insertions(+), 131 deletions(-) diff --git a/doc/source/dist_cloud/kubernetes/prestage-a-subcloud-using-dcmanager-df756866163f.rst b/doc/source/dist_cloud/kubernetes/prestage-a-subcloud-using-dcmanager-df756866163f.rst index 80e0ae5d7..c8fdbc8ff 100644 --- a/doc/source/dist_cloud/kubernetes/prestage-a-subcloud-using-dcmanager-df756866163f.rst +++ b/doc/source/dist_cloud/kubernetes/prestage-a-subcloud-using-dcmanager-df756866163f.rst @@ -4,18 +4,11 @@ Prestage a Subcloud =================== -Before you start an |AIO-SX| subcloud upgrade or reinstall for the purpose of -restoring the subcloud; the subcloud can be prestaged with OSTree repo -(software updates) and container image archives outside the maintenance window -using the dcmanager CLI. The prestaged data is stored in the subcloud -persistent file system ``/opt/platform-backup/``. This data will be -used when the subcloud is reinstalled next. - -Where, the `` number is the active load of the System Controller. - -.. note:: - - Only |AIO-SX| subclouds can be prestaged using the dcmanager CLI. +Before you deploy platform software updates on the subcloud or reinstall it to +restore the subcloud, the subcloud can be prestaged with OSTree repo (software +updates) and container image archives outside the maintenance window. This can +be done using the dcmanager CLI. The prestaging reduces the duration of the +subsequent software update or reinstall in the maintenance window. For information on prestaging a batch of subclouds, see, :ref:`prestage-subcloud-orchestration-eb516473582f`. @@ -27,8 +20,9 @@ The main steps of this task are: #. Ensure prestaging prerequisites are met, see :ref:`prestaging-prereqs`. #. Upload the list of container images to prestage. This step is relevant to - upgrade and must be performed after the System Controller - has been upgraded. See :ref:`Upload Prestage Image List `. + major release software deployment and must be performed after the system + controller has been updated with the new major release. See :ref:`Upload + Prestage Image List `. #. Use dcmanager commands to prestage the subcloud(s). @@ -48,19 +42,26 @@ orchestration. See :ref:`prestage-subcloud-orchestration-eb516473582f`. There are two types of subcloud prestage: -- **Prestage for upgrade**: when the subcloud is running a different (older) - load than the System Controller at the time of prestaging. +- **Prestage for software deployment** occurs when the subcloud is running an + older software than the system controller during prestaging. This prestage + transfers software update(s) and container images to the subcloud while + preparing for new software deployment. The prestaged data is stored in + various locations on the subcloud. For example, metadata is stored in + ``/opt/software/`` and new images are added to the local docker + registry. -- **Prestage for reinstall**: When the subcloud is running the same load as the - System Controller at the time of prestaging. - - .. note:: - Only |AIO-SX| subclouds can be prestaged using the dcmanager CLI. +- **Prestage for install** occurs when the subcloud is running the same or older + software than the system controller during prestaging. This prestage + transfers the entire platform software and container images of the + specified release while preparing for the subcloud redeployment. This is + used if the subcloud needs to be reinstalled and restored to a specific + release. The prestaged data is stored in the subcloud persistent + filesystem ``/opt/platform-backup/``. **Pre-conditions common to both types of prestage**: -- Subclouds to be prestaged must be |AIO-SX|, online, managed and free - of any management affecting alarms. +- Subclouds to be prestaged must be online, managed, and free of any management + affecting alarms. .. note:: @@ -75,31 +76,31 @@ There are two types of subcloud prestage: image pulls and archive file generation. If the total size of prestage images is N GB, available Docker space should be N*2 GB. +**Pre-conditions specific to prestage for install**: + +- The active controller must be controller-0. + +- Subcloud ``/opt/platform-backup`` must have enough available disk space for + prestaged data. + .. note:: - For the prestaged remote subcloud installations, the ISO imported via - ``load-import --inactive`` must be at the same patch level as the system - controller. If the system controller is patched after prestaging of - subclouds, you need to repeat the prestaging of each subcloud. This - ensures that the subcloud boot image aligns with the patch level of the - load that needs to be installed on the subcloud. + In case of prestage for-install, the ISO to be uploaded using the + :command:`system software upload` command must be at the same patch level + as the system controller. If the system controller is patched after + prestaging of subclouds, you need to repeat the prestaging of each + subcloud. This ensures that the subcloud boot image aligns with the patch + level of the load that needs to be installed on the subcloud. -.. warning:: +**Pre-conditions specific to prestage for the deployment of major release**: - If the available docker space is inadequate, some application pods can get - evicted due to temporary disk pressure during the prestaging process. The - cert-manager application will fail subcloud upgrade if its evicted pods are - not cleaned up. +.. The total size of prestage images and custom images restored over upgrade +.. must not exceed docker-distribution capacity. -**Pre-conditions specific to prestage for upgrade**: - -- The total size of prestage images and custom images restored over upgrade - must not exceed docker-distribution capacity. - -- Prestage images must already exist in the configured source(s) prior to - subcloud prestaging. For example, if the subcloud is configured to - download images from the central registry; the specified images must - already exist in the registry on the System Controller. +Prestage images must already exist in the configured source(s) prior to +subcloud prestaging. For example, if the subcloud is configured to download +images from the central registry, the specified images must already exist in +the registry on the system controller. .. _prestaging-image-list: @@ -108,18 +109,23 @@ Upload Prestage Image List -------------------------- The prestage image list specifies what container images are to be pulled from -the configured sources and included in the image archive files during prestaging. -This list is only used if the prestage is intended for subcloud upgrade i.e. -the System Controller and subclouds are running different loads at the time of -prestaging. +the configured sources and included in the image archive files during +prestaging. This list is only used if the prestage is intended for deployment +of a major release, that is, subclouds are running different loads at the time +of prestaging. The prestage image list must contain: -- Images required for subcloud platform upgrade. +- Images required for the deployment of major platform update. - Images required for the restore and update or |prod-long| applications, - currently applied on the subcloud, for example, cert-manager, |OIDC|, and - metrics-server. + currently applied on the subcloud. For example: + + - cert-manager + + - |OIDC| + + - metrics-server .. only:: partner @@ -130,22 +136,20 @@ The prestage image list must contain: If the available docker and docker-distribution storage is ample, prestage image list should also contain: -- (Optional) Images required for Kubernetes version upgrades post subcloud upgrade. +.. - (Optional) Images required for Kubernetes version upgrades post subcloud upgrade. -- (Optional) Images required for the update of end users' Helm applications - post subcloud upgrade. +(Optional) Images required for the update of end users' Helm applications +post major platform update. .. note:: - It is required to determine the total size of all images to be prestaged - in advance. Too many images can result in subcloud upgrade failure due to - docker-distribution (local registry) out of space error. - See the Prerequisites section above for more details. + It is recommended to determine the total size of all images to be prestaged + in advance and resize the docker-distribution (subcloud local registry) + filesystem using the :command:`system controllerfs-modify` command accordingly. .. rubric:: |proc| -#. To upload the prestage image list, use the following command after the - System Controller has been upgraded. +#. To upload the prestage image list, use the following command: .. code-block:: none @@ -197,74 +201,166 @@ Single Subcloud Prestage See :ref:`prestaging-prereqs` for preconditions prior to prestaging the subcloud. +Subcloud can be prestaged for install of current/previous software version or +for software deployment, and the previous release software must be uploaded +before prestaging. + .. code-block:: none - ~(keystone_admin)]$ dcmanager subcloud prestage subcloud2 - + [sysadmin@controller-0 ~(keystone_admin)]$ dcmanager subcloud prestage --for-install --release nn.nn subcloud7 Enter the sysadmin password for the subcloud: Re-enter sysadmin password to confirm: +-----------------------------+----------------------------+ | Field | Value | +-----------------------------+----------------------------+ - | id | 2 | - | name | subcloud2 | + | id | 7 | + | name | subcloud7 | | description | None | | location | None | - | software_version | nn.nn | + | software_version | 22.12 | | management | managed | | availability | online | - | deploy_status | prestage-prepare | - | management_subnet | 2620:10a:a001:ac01::20/123 | - | management_start_ip | 2620:10a:a001:ac01::22 | - | management_end_ip | 2620:10a:a001:ac01::3e | - | management_gateway_ip | 2620:10a:a001:ac01::21 | - | systemcontroller_gateway_ip | 2620:10a:a001:a113::1 | - | group_id | 3 | - | created_at | 2202-03-18 20:31:16.548903 | - | updated_at | 2202-03-22 18:55:56:251643 | + | deploy_status | complete | + | management_subnet | fdff:719a:bf60:4028::/64 | + | management_start_ip | fdff:719a:bf60:4028::2 | + | management_end_ip | fdff:719a:bf60:4028::ffff | + | management_gateway_ip | fdff:719a:bf60:4028::1 | + | systemcontroller_gateway_ip | fd00:8:16::1 | + | group_id | 1 | + | peer_group_id | None | + | created_at | 2024-10-25 06:06:39.527023 | + | updated_at | 2024-10-25 08:08:16.311164 | + | backup_status | None | + | backup_datetime | None | + | prestage_status | prestaging-packages | + | prestage_versions | | + | prestage_software_version | 22.12 | +-----------------------------+----------------------------+ +.. code-block:: + + [sysadmin@controller-0 ~(keystone_admin)]$ dcmanager subcloud prestage --for-install subcloud5 + Enter the sysadmin password for the subcloud: + Re-enter sysadmin password to confirm: + +-----------------------------+----------------------------+ + | Field | Value | + +-----------------------------+----------------------------+ + | id | 8 | + | name | subcloud5 | + | description | None | + | location | None | + | software_version | 24.09 | + | management | managed | + | availability | online | + | deploy_status | complete | + | management_subnet | fd00:8:28::/64 | + | management_start_ip | fd00:8:28::2 | + | management_end_ip | fd00:8:28::10 | + | management_gateway_ip | fd00:8:28::1 | + | systemcontroller_gateway_ip | fd00:8:16::1 | + | group_id | 1 | + | peer_group_id | None | + | created_at | 2024-10-25 06:28:57.197765 | + | updated_at | 2024-10-25 08:08:38.433790 | + | backup_status | None | + | backup_datetime | None | + | prestage_status | prestaging-packages | + | prestage_versions | | + | prestage_software_version | 24.09 | + +-----------------------------+----------------------------+ + +.. note:: + + - If the ``--release`` option is not specified, the subcloud will be + prestaged for install with the same software as the system controller. + Also, the release format must be MM.mm and any other format is not valid. + + - If the release is not in deployed/available state and if it tries to do prestaging + for the ``--for-sw-deploy`` option, prestaging will throw an error. + + - To verify the release, run the :command:`software list` command. + +.. code-block:: + + [sysadmin@controller-0 ~(keystone_admin)]$ dcmanager subcloud prestage --for-sw-deploy subcloud5 + Enter the sysadmin password for the subcloud: + Re-enter sysadmin password to confirm: + +-----------------------------+----------------------------+ + | Field | Value | + +-----------------------------+----------------------------+ + | id | 8 | + | name | subcloud5 | + | description | None | + | location | None | + | software_version | 24.09 | + | management | managed | + | availability | online | + | deploy_status | complete | + | management_subnet | fd00:8:28::/64 | + | management_start_ip | fd00:8:28::2 | + | management_end_ip | fd00:8:28::10 | + | management_gateway_ip | fd00:8:28::1 | + | systemcontroller_gateway_ip | fd00:8:16::1 | + | group_id | 3 | + | peer_group_id | None | + | created_at | 2024-09-21 15:21:20.418706 | + | updated_at | 2024-09-22 08:44:30.937440 | + | backup_status | None | + | backup_datetime | None | + | prestage_status | prestaging-packages | + | prestage_versions | | + | prestage_software_version | 24.09 | + +-----------------------------+----------------------------+ + +.. note:: + + To perform the software deployment prestaging, system controller should + have higher patch level than the subcloud. Post prestaging, the release + will be in available state in the subcloud. If the subcloud has higher patch + level than system controller, prestaging will be skipped. + ----------------------- Rerun Subcloud Prestage ----------------------- -A subcloud can be prestaged multiple times. However, only prestaging images -will be repeated. Once packages prestaging is successful, this step will be -skipped in subsequent prestage reruns for the same software version. +A subcloud can be prestaged multiple times. Software prestaging will be skipped +if the subcloud already has the specified software version. Prestaging of +container images will be repeated. ------------------------ Verify Subcloud Prestage ------------------------ -After a subcloud is successfully prestaged, the ``deploy_status`` will change to -``prestage-complete``. Use the :command:`dcmanager subcloud show` command to -verify the status. The packages directory, repodata directory, and container -image bundles, and md5 file can be found on the subcloud in -``/opt/platform-backup/``. - -Where, the `` number is the active load of the System Controller. +After a subcloud is successfully prestaged, ``prestage_status`` will change to +``complete`` and ``prestage_versions`` will show the software version(s) the +subcloud has prestaged data for. Use the :command:`dcmanager subcloud show` +command to verify the status and versions. ------------------------------ Troubleshoot Subcloud Prestage ------------------------------ -If the subcloud prestage fails, check ``/var/log/dcmanager/dcmanager.log`` -for the reason of failure. Once the issue has been resolved, prestage can be -retried using :command:`dcmanager subcloud prestage` command. +If the subcloud prestage fails, check ``/var/log/dcmanager/dcmanager.log`` and +``/var/log/dcmanager/ansible/_playbook_output.log`` for the reason +of failure. Once the issue has been resolved, prestage can be retried using +the :command:`dcmanager subcloud prestage` command. --------------------------------- Verifying Usage of Prestaged Data --------------------------------- -To verify that the prestaged data is used over subcloud upgrade, subcloud -redeploy, or subcloud remote restore: +To verify that the prestaged data is used over subcloud update with new +software or subcloud redeployment/restore: - Search for the the subcloud name in the log file, for example, subcloud1 from ``/www/var/log/lighttpd-access.log``. There should not be GET requests to download packages from ``/iso//nodes/subcloud1/Packages/``. -- Check subcloud ansible log in ``/var/log/dcmanager/ansible`` directory. - Images are imported from local archives and no images in the prestage image - list need to be downloaded from configured sources. +- Check subcloud ansible log in the ``/var/log/dcmanager/ansible`` directory. + Images are imported from local archives and no images in the prestage image + list need to be downloaded from the configured sources. + +- If we reinstall the subcloud, then we can check ``/root/install.log`` to + validate if the prestage data was used during installation. diff --git a/doc/source/dist_cloud/kubernetes/prestage-subcloud-orchestration-eb516473582f.rst b/doc/source/dist_cloud/kubernetes/prestage-subcloud-orchestration-eb516473582f.rst index 65fa7d3b9..849534ec3 100644 --- a/doc/source/dist_cloud/kubernetes/prestage-subcloud-orchestration-eb516473582f.rst +++ b/doc/source/dist_cloud/kubernetes/prestage-subcloud-orchestration-eb516473582f.rst @@ -12,7 +12,6 @@ subcloud group or a specific subcloud group. For more information on prerequisites for prestage upgrade and reinstall, see :ref:`prestage-a-subcloud-using-dcmanager-df756866163f`. - .. note:: Any existing strategy must be deleted first as only one type @@ -34,65 +33,59 @@ For more information on prerequisites for prestage upgrade and reinstall, see To create a prestage strategy for a specific subcloud, use the following command: - .. code-block:: none - - ~(keystone_admin)]$ dcmanager prestage-strategy create subcloud1 + .. parsed-literal:: + [sysadmin@controller-0 ~(keystone_admin)]$ dcmanager prestage-strategy create --for-install --release nn.nn subcloud7 Enter the sysadmin password for the subcloud: Re-enter sysadmin password to confirm: + +--------------------------+-----------------------------+ + | Field | Value | + +--------------------------+-----------------------------+ + | strategy type | prestage | + | subcloud apply type | None | + | max parallel subclouds | 2 | + | stop on failure | False | + | prestage software version| 22.12 | + | state | initial | + | created_at | 2024-10-25T05:37:54.607307 | + | updated_at | None | + +--------------------------+-----------------------------+ - +-----------------------------+----------------------------+ - | Field | Value | - +-----------------------------+----------------------------+ - | id | 1 | - | name | subcloud1 | - | description | None | - | location | False | - | software_version | nn.nn | - | management | managed | - | availability | online | - | deploy_status | prestage-prepare | - | management_subnet | 2620:10a:a001:ac01::20/123 | - | management_start_ip | 2620:10a:a001:ac01::22 | - | management_end_ip | 2620:10a:a001:ac01::3e | - | management_gateway_ip | 2620:10a:a001:ac01::21 | - | systemcontroller_gateway_ip | 2620:10a:a001:a113::1 | - | group_id | 3 | - | created_at | 2202-03-18 20:31:16.548903 | - | updated_at | 2202-03-22 18:55:56:251643 | - +-----------------------------+----------------------------+ To create a prestage strategy for the default subcloud group, use the following command: - .. code-block:: none + .. parsed-literal:: - ~(keystone_admin)]$ dcmanager prestage-strategy create + [sysadmin@controller-0 ~(keystone_admin)]$ dcmanager prestage-strategy create --for-sw-deploy Enter the sysadmin password for the subcloud: Re-enter sysadmin password to confirm: + +--------------------------+-----------------------------+ + | Field | Value | + +--------------------------+-----------------------------+ + | strategy type | prestage | + | subcloud apply type | None | + | max parallel subclouds | 50 | + | stop on failure | False | + | prestage software version| 24.09 | + | state | initial | + | created_at | 2024-10-25T05:37:54.607307 | + | updated_at | None | + +--------------------------+-----------------------------+ - +------------------------+-----------------------------+ - | Field | Value | - +------------------------+-----------------------------+ - | strategy type | prestage | - | subcloud apply type | parallel | - | max parallel subclouds | 50 | - | stop on failure | False | - | state | initial | - | created_at | 2202-03-22T18:54:45.037336 | - | updated_at | None | - +------------------------+-----------------------------+ + .. note:: + + If the ``--release`` option is not specified, the release version of the + system controller will be used. To create a prestage strategy for a specific subcloud group, use the following command: .. code-block:: none - ~(keystone_admin)]$ dcmanager prestage-strategy create –group First_10_Subclouds - + ~(keystone_admin)]$ dcmanager prestage-strategy create --group First_10_Subclouds Enter the sysadmin password for the subcloud: Re-enter sysadmin password to confirm: - +------------------------+-----------------------------+ | Field | Value | +------------------------+-----------------------------+ @@ -105,6 +98,11 @@ For more information on prerequisites for prestage upgrade and reinstall, see | updated_at | None | +------------------------+-----------------------------+ + .. note:: + + If the prestage type is not specified, the default prestage type + (``--for-install``) will be applied. + .. note:: Unlike other types of orchestration, prestage orchestration requires