starlingx/docs
Code Issues Proposed changes

Merge "Subcloud Remote Install, Upgrade and Prestaging Adaptation"

Browse Source
This commit is contained in:
Zuul 2024-11-22 18:06:52 +00:00 committed by Gerrit Code Review
commit d8ebebe083
2 changed files with 225 additions and 131 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

276
doc/source/dist_cloud/kubernetes/prestage-a-subcloud-using-dcmanager-df756866163f.rst
View File

@ -4,18 +4,11 @@
Prestage a Subcloud Prestage a Subcloud
=================== ===================
Before you start an |AIO-SX| subcloud upgrade or reinstall for the purpose of Before you deploy platform software updates on the subcloud or reinstall it to
restoring the subcloud; the subcloud can be prestaged with OSTree repo restore the subcloud, the subcloud can be prestaged with OSTree repo (software
(software updates) and container image archives outside the maintenance window updates) and container image archives outside the maintenance window. This can
using the dcmanager CLI. The prestaged data is stored in the subcloud be done using the dcmanager CLI. The prestaging reduces the duration of the
persistent file system ``/opt/platform-backup/<sw_version>``. This data will be subsequent software update or reinstall in the maintenance window.
used when the subcloud is reinstalled next.
Where, the `<sw_version>` number is the active load of the System Controller.
.. note::
Only |AIO-SX| subclouds can be prestaged using the dcmanager CLI.
For information on prestaging a batch of subclouds, see, For information on prestaging a batch of subclouds, see,
:ref:`prestage-subcloud-orchestration-eb516473582f`. :ref:`prestage-subcloud-orchestration-eb516473582f`.
@ -27,8 +20,9 @@ The main steps of this task are:
#. Ensure prestaging prerequisites are met, see :ref:`prestaging-prereqs`. #. Ensure prestaging prerequisites are met, see :ref:`prestaging-prereqs`.
#. Upload the list of container images to prestage. This step is relevant to #. Upload the list of container images to prestage. This step is relevant to
upgrade and must be performed after the System Controller major release software deployment and must be performed after the system
has been upgraded. See :ref:`Upload Prestage Image List <prestaging-image-list>`. controller has been updated with the new major release. See :ref:`Upload
Prestage Image List <prestaging-image-list>`.
#. Use dcmanager commands to prestage the subcloud(s). #. Use dcmanager commands to prestage the subcloud(s).
@ -48,19 +42,26 @@ orchestration. See :ref:`prestage-subcloud-orchestration-eb516473582f`.
There are two types of subcloud prestage: There are two types of subcloud prestage:
- **Prestage for upgrade**: when the subcloud is running a different (older) - **Prestage for software deployment** occurs when the subcloud is running an
load than the System Controller at the time of prestaging. older software than the system controller during prestaging. This prestage
transfers software update(s) and container images to the subcloud while
preparing for new software deployment. The prestaged data is stored in
various locations on the subcloud. For example, metadata is stored in
``/opt/software/<sw-version>`` and new images are added to the local docker
registry.
- **Prestage for reinstall**: When the subcloud is running the same load as the - **Prestage for install** occurs when the subcloud is running the same or older
System Controller at the time of prestaging. software than the system controller during prestaging. This prestage
transfers the entire platform software and container images of the
.. note:: specified release while preparing for the subcloud redeployment. This is
Only |AIO-SX| subclouds can be prestaged using the dcmanager CLI. used if the subcloud needs to be reinstalled and restored to a specific
release. The prestaged data is stored in the subcloud persistent
filesystem ``/opt/platform-backup/<sw-version>``.
**Pre-conditions common to both types of prestage**: **Pre-conditions common to both types of prestage**:
- Subclouds to be prestaged must be |AIO-SX|, online, managed and free - Subclouds to be prestaged must be online, managed, and free of any management
of any management affecting alarms. affecting alarms.
.. note:: .. note::
@ -75,31 +76,31 @@ There are two types of subcloud prestage:
image pulls and archive file generation. If the total size of prestage image pulls and archive file generation. If the total size of prestage
images is N GB, available Docker space should be N*2 GB. images is N GB, available Docker space should be N*2 GB.
**Pre-conditions specific to prestage for install**:
- The active controller must be controller-0.
- Subcloud ``/opt/platform-backup`` must have enough available disk space for
prestaged data.
.. note:: .. note::
For the prestaged remote subcloud installations, the ISO imported via In case of prestage for-install, the ISO to be uploaded using the
``load-import --inactive`` must be at the same patch level as the system :command:`system software upload` command must be at the same patch level
controller. If the system controller is patched after prestaging of as the system controller. If the system controller is patched after
subclouds, you need to repeat the prestaging of each subcloud. This prestaging of subclouds, you need to repeat the prestaging of each
ensures that the subcloud boot image aligns with the patch level of the subcloud. This ensures that the subcloud boot image aligns with the patch
load that needs to be installed on the subcloud. level of the load that needs to be installed on the subcloud.
.. warning:: **Pre-conditions specific to prestage for the deployment of major release**:
If the available docker space is inadequate, some application pods can get .. The total size of prestage images and custom images restored over upgrade
evicted due to temporary disk pressure during the prestaging process. The .. must not exceed docker-distribution capacity.
cert-manager application will fail subcloud upgrade if its evicted pods are
not cleaned up.
**Pre-conditions specific to prestage for upgrade**: Prestage images must already exist in the configured source(s) prior to
subcloud prestaging. For example, if the subcloud is configured to download
- The total size of prestage images and custom images restored over upgrade images from the central registry, the specified images must already exist in
must not exceed docker-distribution capacity. the registry on the system controller.
- Prestage images must already exist in the configured source(s) prior to
subcloud prestaging. For example, if the subcloud is configured to
download images from the central registry; the specified images must
already exist in the registry on the System Controller.
.. _prestaging-image-list: .. _prestaging-image-list:
@ -108,18 +109,23 @@ Upload Prestage Image List
-------------------------- --------------------------
The prestage image list specifies what container images are to be pulled from The prestage image list specifies what container images are to be pulled from
the configured sources and included in the image archive files during prestaging. the configured sources and included in the image archive files during
This list is only used if the prestage is intended for subcloud upgrade i.e. prestaging. This list is only used if the prestage is intended for deployment
the System Controller and subclouds are running different loads at the time of of a major release, that is, subclouds are running different loads at the time
prestaging. of prestaging.
The prestage image list must contain: The prestage image list must contain:
- Images required for subcloud platform upgrade. - Images required for the deployment of major platform update.
- Images required for the restore and update or |prod-long| applications, - Images required for the restore and update or |prod-long| applications,
currently applied on the subcloud, for example, cert-manager, |OIDC|, and currently applied on the subcloud. For example:
metrics-server.
- cert-manager
- |OIDC|
- metrics-server
.. only:: partner .. only:: partner
@ -130,22 +136,20 @@ The prestage image list must contain:
If the available docker and docker-distribution storage is ample, prestage If the available docker and docker-distribution storage is ample, prestage
image list should also contain: image list should also contain:
- (Optional) Images required for Kubernetes version upgrades post subcloud upgrade. .. - (Optional) Images required for Kubernetes version upgrades post subcloud upgrade.
- (Optional) Images required for the update of end users' Helm applications (Optional) Images required for the update of end users' Helm applications
post subcloud upgrade. post major platform update.
.. note:: .. note::
It is required to determine the total size of all images to be prestaged It is recommended to determine the total size of all images to be prestaged
in advance. Too many images can result in subcloud upgrade failure due to in advance and resize the docker-distribution (subcloud local registry)
docker-distribution (local registry) out of space error. filesystem using the :command:`system controllerfs-modify` command accordingly.
See the Prerequisites section above for more details.
.. rubric:: |proc| .. rubric:: |proc|
#. To upload the prestage image list, use the following command after the #. To upload the prestage image list, use the following command:
System Controller has been upgraded.
.. code-block:: none .. code-block:: none
@ -197,74 +201,166 @@ Single Subcloud Prestage
See :ref:`prestaging-prereqs` for preconditions prior to prestaging the subcloud. See :ref:`prestaging-prereqs` for preconditions prior to prestaging the subcloud.
Subcloud can be prestaged for install of current/previous software version or
for software deployment, and the previous release software must be uploaded
before prestaging.
.. code-block:: none .. code-block:: none
~(keystone_admin)]$ dcmanager subcloud prestage subcloud2 [sysadmin@controller-0 ~(keystone_admin)]$ dcmanager subcloud prestage --for-install --release nn.nn subcloud7
Enter the sysadmin password for the subcloud: Enter the sysadmin password for the subcloud:
Re-enter sysadmin password to confirm: Re-enter sysadmin password to confirm:
+-----------------------------+----------------------------+ +-----------------------------+----------------------------+
| Field | Value | | Field | Value |
+-----------------------------+----------------------------+ +-----------------------------+----------------------------+
| id | 2 | | id | 7 |
| name | subcloud2 | | name | subcloud7 |
| description | None | | description | None |
| location | None | | location | None |
| software_version | nn.nn | | software_version | 22.12 |
| management | managed | | management | managed |
| availability | online | | availability | online |
| deploy_status | prestage-prepare | | deploy_status | complete |
| management_subnet | 2620:10a:a001:ac01::20/123 | | management_subnet | fdff:719a:bf60:4028::/64 |
| management_start_ip | 2620:10a:a001:ac01::22 | | management_start_ip | fdff:719a:bf60:4028::2 |
| management_end_ip | 2620:10a:a001:ac01::3e | | management_end_ip | fdff:719a:bf60:4028::ffff |
| management_gateway_ip | 2620:10a:a001:ac01::21 | | management_gateway_ip | fdff:719a:bf60:4028::1 |
| systemcontroller_gateway_ip | 2620:10a:a001:a113::1 | | systemcontroller_gateway_ip | fd00:8:16::1 |
| group_id | 3 | | group_id | 1 |
| created_at | 2202-03-18 20:31:16.548903 | | peer_group_id | None |
| updated_at | 2202-03-22 18:55:56:251643 | | created_at | 2024-10-25 06:06:39.527023 |
| updated_at | 2024-10-25 08:08:16.311164 |
| backup_status | None |
| backup_datetime | None |
| prestage_status | prestaging-packages |
| prestage_versions | |
| prestage_software_version | 22.12 |
+-----------------------------+----------------------------+ +-----------------------------+----------------------------+
.. code-block::
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager subcloud prestage --for-install subcloud5
Enter the sysadmin password for the subcloud:
Re-enter sysadmin password to confirm:
+-----------------------------+----------------------------+
| Field | Value |
+-----------------------------+----------------------------+
| id | 8 |
| name | subcloud5 |
| description | None |
| location | None |
| software_version | 24.09 |
| management | managed |
| availability | online |
| deploy_status | complete |
| management_subnet | fd00:8:28::/64 |
| management_start_ip | fd00:8:28::2 |
| management_end_ip | fd00:8:28::10 |
| management_gateway_ip | fd00:8:28::1 |
| systemcontroller_gateway_ip | fd00:8:16::1 |
| group_id | 1 |
| peer_group_id | None |
| created_at | 2024-10-25 06:28:57.197765 |
| updated_at | 2024-10-25 08:08:38.433790 |
| backup_status | None |
| backup_datetime | None |
| prestage_status | prestaging-packages |
| prestage_versions | |
| prestage_software_version | 24.09 |
+-----------------------------+----------------------------+
.. note::
- If the ``--release`` option is not specified, the subcloud will be
prestaged for install with the same software as the system controller.
Also, the release format must be MM.mm and any other format is not valid.
- If the release is not in deployed/available state and if it tries to do prestaging
for the ``--for-sw-deploy`` option, prestaging will throw an error.
- To verify the release, run the :command:`software list` command.
.. code-block::
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager subcloud prestage --for-sw-deploy subcloud5
Enter the sysadmin password for the subcloud:
Re-enter sysadmin password to confirm:
+-----------------------------+----------------------------+
| Field | Value |
+-----------------------------+----------------------------+
| id | 8 |
| name | subcloud5 |
| description | None |
| location | None |
| software_version | 24.09 |
| management | managed |
| availability | online |
| deploy_status | complete |
| management_subnet | fd00:8:28::/64 |
| management_start_ip | fd00:8:28::2 |
| management_end_ip | fd00:8:28::10 |
| management_gateway_ip | fd00:8:28::1 |
| systemcontroller_gateway_ip | fd00:8:16::1 |
| group_id | 3 |
| peer_group_id | None |
| created_at | 2024-09-21 15:21:20.418706 |
| updated_at | 2024-09-22 08:44:30.937440 |
| backup_status | None |
| backup_datetime | None |
| prestage_status | prestaging-packages |
| prestage_versions | |
| prestage_software_version | 24.09 |
+-----------------------------+----------------------------+
.. note::
To perform the software deployment prestaging, system controller should
have higher patch level than the subcloud. Post prestaging, the release
will be in available state in the subcloud. If the subcloud has higher patch
level than system controller, prestaging will be skipped.
----------------------- -----------------------
Rerun Subcloud Prestage Rerun Subcloud Prestage
----------------------- -----------------------
A subcloud can be prestaged multiple times. However, only prestaging images A subcloud can be prestaged multiple times. Software prestaging will be skipped
will be repeated. Once packages prestaging is successful, this step will be if the subcloud already has the specified software version. Prestaging of
skipped in subsequent prestage reruns for the same software version. container images will be repeated.
------------------------ ------------------------
Verify Subcloud Prestage Verify Subcloud Prestage
------------------------ ------------------------
After a subcloud is successfully prestaged, the ``deploy_status`` will change to After a subcloud is successfully prestaged, ``prestage_status`` will change to
``prestage-complete``. Use the :command:`dcmanager subcloud show` command to ``complete`` and ``prestage_versions`` will show the software version(s) the
verify the status. The packages directory, repodata directory, and container subcloud has prestaged data for. Use the :command:`dcmanager subcloud show`
image bundles, and md5 file can be found on the subcloud in command to verify the status and versions.
``/opt/platform-backup/<sw_version>``.
Where, the `<sw_version>` number is the active load of the System Controller.
------------------------------ ------------------------------
Troubleshoot Subcloud Prestage Troubleshoot Subcloud Prestage
------------------------------ ------------------------------
If the subcloud prestage fails, check ``/var/log/dcmanager/dcmanager.log`` If the subcloud prestage fails, check ``/var/log/dcmanager/dcmanager.log`` and
for the reason of failure. Once the issue has been resolved, prestage can be ``/var/log/dcmanager/ansible/<subcloud-name>_playbook_output.log`` for the reason
retried using :command:`dcmanager subcloud prestage` command. of failure. Once the issue has been resolved, prestage can be retried using
the :command:`dcmanager subcloud prestage` command.
--------------------------------- ---------------------------------
Verifying Usage of Prestaged Data Verifying Usage of Prestaged Data
--------------------------------- ---------------------------------
To verify that the prestaged data is used over subcloud upgrade, subcloud To verify that the prestaged data is used over subcloud update with new
redeploy, or subcloud remote restore: software or subcloud redeployment/restore:
- Search for the the subcloud name in the log file, for example, - Search for the the subcloud name in the log file, for example,
subcloud1 from ``/www/var/log/lighttpd-access.log``. There should not be subcloud1 from ``/www/var/log/lighttpd-access.log``. There should not be
GET requests to download packages from ``/iso/<sw_version>/nodes/subcloud1/Packages/``. GET requests to download packages from ``/iso/<sw_version>/nodes/subcloud1/Packages/``.
- Check subcloud ansible log in ``/var/log/dcmanager/ansible`` directory. - Check subcloud ansible log in the ``/var/log/dcmanager/ansible`` directory.
Images are imported from local archives and no images in the prestage image Images are imported from local archives and no images in the prestage image
list need to be downloaded from configured sources. list need to be downloaded from the configured sources.
- If we reinstall the subcloud, then we can check ``/root/install.log`` to
validate if the prestage data was used during installation.

68
doc/source/dist_cloud/kubernetes/prestage-subcloud-orchestration-eb516473582f.rst
View File

@ -12,7 +12,6 @@ subcloud group or a specific subcloud group.
For more information on prerequisites for prestage upgrade and reinstall, see For more information on prerequisites for prestage upgrade and reinstall, see
:ref:`prestage-a-subcloud-using-dcmanager-df756866163f`. :ref:`prestage-a-subcloud-using-dcmanager-df756866163f`.
.. note:: .. note::
Any existing strategy must be deleted first as only one type Any existing strategy must be deleted first as only one type
@ -34,65 +33,59 @@ For more information on prerequisites for prestage upgrade and reinstall, see
To create a prestage strategy for a specific subcloud, use the following To create a prestage strategy for a specific subcloud, use the following
command: command:
.. code-block:: none .. parsed-literal::
~(keystone_admin)]$ dcmanager prestage-strategy create subcloud1
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager prestage-strategy create --for-install --release nn.nn subcloud7
Enter the sysadmin password for the subcloud: Enter the sysadmin password for the subcloud:
Re-enter sysadmin password to confirm: Re-enter sysadmin password to confirm:
+--------------------------+-----------------------------+
+-----------------------------+----------------------------+
| Field | Value | | Field | Value |
+-----------------------------+----------------------------+ +--------------------------+-----------------------------+
| id | 1 | | strategy type | prestage |
| name | subcloud1 | | subcloud apply type | None |
| description | None | | max parallel subclouds | 2 |
| location | False | | stop on failure | False |
| software_version | nn.nn | | prestage software version| 22.12 |
| management | managed | | state | initial |
| availability | online | | created_at | 2024-10-25T05:37:54.607307 |
| deploy_status | prestage-prepare | | updated_at | None |
| management_subnet | 2620:10a:a001:ac01::20/123 | +--------------------------+-----------------------------+
| management_start_ip | 2620:10a:a001:ac01::22 |
| management_end_ip | 2620:10a:a001:ac01::3e |
| management_gateway_ip | 2620:10a:a001:ac01::21 |
| systemcontroller_gateway_ip | 2620:10a:a001:a113::1 |
| group_id | 3 |
| created_at | 2202-03-18 20:31:16.548903 |
| updated_at | 2202-03-22 18:55:56:251643 |
+-----------------------------+----------------------------+
To create a prestage strategy for the default subcloud group, use the To create a prestage strategy for the default subcloud group, use the
following command: following command:
.. code-block:: none .. parsed-literal::
~(keystone_admin)]$ dcmanager prestage-strategy create [sysadmin@controller-0 ~(keystone_admin)]$ dcmanager prestage-strategy create --for-sw-deploy
Enter the sysadmin password for the subcloud: Enter the sysadmin password for the subcloud:
Re-enter sysadmin password to confirm: Re-enter sysadmin password to confirm:
+--------------------------+-----------------------------+
+------------------------+-----------------------------+
| Field | Value | | Field | Value |
+------------------------+-----------------------------+ +--------------------------+-----------------------------+
| strategy type | prestage | | strategy type | prestage |
| subcloud apply type | parallel | | subcloud apply type | None |
| max parallel subclouds | 50 | | max parallel subclouds | 50 |
| stop on failure | False | | stop on failure | False |
| prestage software version| 24.09 |
| state | initial | | state | initial |
| created_at | 2202-03-22T18:54:45.037336 | | created_at | 2024-10-25T05:37:54.607307 |
| updated_at | None | | updated_at | None |
+------------------------+-----------------------------+ +--------------------------+-----------------------------+
.. note::
If the ``--release`` option is not specified, the release version of the
system controller will be used.
To create a prestage strategy for a specific subcloud group, use the To create a prestage strategy for a specific subcloud group, use the
following command: following command:
.. code-block:: none .. code-block:: none
~(keystone_admin)]$ dcmanager prestage-strategy create –group First_10_Subclouds ~(keystone_admin)]$ dcmanager prestage-strategy create --group First_10_Subclouds
Enter the sysadmin password for the subcloud: Enter the sysadmin password for the subcloud:
Re-enter sysadmin password to confirm: Re-enter sysadmin password to confirm:
+------------------------+-----------------------------+ +------------------------+-----------------------------+
| Field | Value | | Field | Value |
+------------------------+-----------------------------+ +------------------------+-----------------------------+
@ -105,6 +98,11 @@ For more information on prerequisites for prestage upgrade and reinstall, see
| updated_at | None | | updated_at | None |
+------------------------+-----------------------------+ +------------------------+-----------------------------+
.. note::
If the prestage type is not specified, the default prestage type
(``--for-install``) will be applied.
.. note:: .. note::
Unlike other types of orchestration, prestage orchestration requires Unlike other types of orchestration, prestage orchestration requires