starlingx/docs
Code Issues Proposed changes

Merge "Subcloud Remote Install, Upgrade and Prestaging Adaptation"

Browse Source
This commit is contained in:
Zuul 2024-11-22 18:06:52 +00:00 committed by Gerrit Code Review
commit d8ebebe083
2 changed files with 225 additions and 131 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

276
doc/source/dist_cloud/kubernetes/prestage-a-subcloud-using-dcmanager-df756866163f.rst
View File

@ -4,18 +4,11 @@
Prestage a Subcloud
===================
Before you start an |AIO-SX| subcloud upgrade or reinstall for the purpose of
restoring the subcloud; the subcloud can be prestaged with OSTree repo
(software updates) and container image archives outside the maintenance window
using the dcmanager CLI. The prestaged data is stored in the subcloud
persistent file system ``/opt/platform-backup/<sw_version>``. This data will be
used when the subcloud is reinstalled next.
Where, the `<sw_version>` number is the active load of the System Controller.
.. note::
Only |AIO-SX| subclouds can be prestaged using the dcmanager CLI.
Before you deploy platform software updates on the subcloud or reinstall it to
restore the subcloud, the subcloud can be prestaged with OSTree repo (software
updates) and container image archives outside the maintenance window. This can
be done using the dcmanager CLI. The prestaging reduces the duration of the
subsequent software update or reinstall in the maintenance window.
For information on prestaging a batch of subclouds, see,
:ref:`prestage-subcloud-orchestration-eb516473582f`.
@ -27,8 +20,9 @@ The main steps of this task are:
#. Ensure prestaging prerequisites are met, see :ref:`prestaging-prereqs`.
#. Upload the list of container images to prestage. This step is relevant to
upgrade and must be performed after the System Controller
has been upgraded. See :ref:`Upload Prestage Image List <prestaging-image-list>`.
major release software deployment and must be performed after the system
controller has been updated with the new major release. See :ref:`Upload
Prestage Image List <prestaging-image-list>`.
#. Use dcmanager commands to prestage the subcloud(s).
@ -48,19 +42,26 @@ orchestration. See :ref:`prestage-subcloud-orchestration-eb516473582f`.
There are two types of subcloud prestage:
- **Prestage for upgrade**: when the subcloud is running a different (older)
load than the System Controller at the time of prestaging.
- **Prestage for software deployment** occurs when the subcloud is running an
older software than the system controller during prestaging. This prestage
transfers software update(s) and container images to the subcloud while
preparing for new software deployment. The prestaged data is stored in
various locations on the subcloud. For example, metadata is stored in
``/opt/software/<sw-version>`` and new images are added to the local docker
registry.
- **Prestage for reinstall**: When the subcloud is running the same load as the
System Controller at the time of prestaging.
.. note::
Only |AIO-SX| subclouds can be prestaged using the dcmanager CLI.
- **Prestage for install** occurs when the subcloud is running the same or older
software than the system controller during prestaging. This prestage
transfers the entire platform software and container images of the
specified release while preparing for the subcloud redeployment. This is
used if the subcloud needs to be reinstalled and restored to a specific
release. The prestaged data is stored in the subcloud persistent
filesystem ``/opt/platform-backup/<sw-version>``.
**Pre-conditions common to both types of prestage**:
- Subclouds to be prestaged must be |AIO-SX|, online, managed and free
of any management affecting alarms.
- Subclouds to be prestaged must be online, managed, and free of any management
affecting alarms.
.. note::
@ -75,31 +76,31 @@ There are two types of subcloud prestage:
image pulls and archive file generation. If the total size of prestage
images is N GB, available Docker space should be N*2 GB.
**Pre-conditions specific to prestage for install**:
- The active controller must be controller-0.
- Subcloud ``/opt/platform-backup`` must have enough available disk space for
prestaged data.
.. note::
For the prestaged remote subcloud installations, the ISO imported via
``load-import --inactive`` must be at the same patch level as the system
controller. If the system controller is patched after prestaging of
subclouds, you need to repeat the prestaging of each subcloud. This
ensures that the subcloud boot image aligns with the patch level of the
load that needs to be installed on the subcloud.
In case of prestage for-install, the ISO to be uploaded using the
:command:`system software upload` command must be at the same patch level
as the system controller. If the system controller is patched after
prestaging of subclouds, you need to repeat the prestaging of each
subcloud. This ensures that the subcloud boot image aligns with the patch
level of the load that needs to be installed on the subcloud.
.. warning::
**Pre-conditions specific to prestage for the deployment of major release**:
If the available docker space is inadequate, some application pods can get
evicted due to temporary disk pressure during the prestaging process. The
cert-manager application will fail subcloud upgrade if its evicted pods are
not cleaned up.
.. The total size of prestage images and custom images restored over upgrade
.. must not exceed docker-distribution capacity.
**Pre-conditions specific to prestage for upgrade**:
- The total size of prestage images and custom images restored over upgrade
must not exceed docker-distribution capacity.
- Prestage images must already exist in the configured source(s) prior to
subcloud prestaging. For example, if the subcloud is configured to
download images from the central registry; the specified images must
already exist in the registry on the System Controller.
Prestage images must already exist in the configured source(s) prior to
subcloud prestaging. For example, if the subcloud is configured to download
images from the central registry, the specified images must already exist in
the registry on the system controller.
.. _prestaging-image-list:
@ -108,18 +109,23 @@ Upload Prestage Image List
--------------------------
The prestage image list specifies what container images are to be pulled from
the configured sources and included in the image archive files during prestaging.
This list is only used if the prestage is intended for subcloud upgrade i.e.
the System Controller and subclouds are running different loads at the time of
prestaging.
the configured sources and included in the image archive files during
prestaging. This list is only used if the prestage is intended for deployment
of a major release, that is, subclouds are running different loads at the time
of prestaging.
The prestage image list must contain:
- Images required for subcloud platform upgrade.
- Images required for the deployment of major platform update.
- Images required for the restore and update or |prod-long| applications,
currently applied on the subcloud, for example, cert-manager, |OIDC|, and
metrics-server.
currently applied on the subcloud. For example:
- cert-manager
- |OIDC|
- metrics-server
.. only:: partner
@ -130,22 +136,20 @@ The prestage image list must contain:
If the available docker and docker-distribution storage is ample, prestage
image list should also contain:
- (Optional) Images required for Kubernetes version upgrades post subcloud upgrade.
.. - (Optional) Images required for Kubernetes version upgrades post subcloud upgrade.
- (Optional) Images required for the update of end users' Helm applications
post subcloud upgrade.
(Optional) Images required for the update of end users' Helm applications
post major platform update.
.. note::
It is required to determine the total size of all images to be prestaged
in advance. Too many images can result in subcloud upgrade failure due to
docker-distribution (local registry) out of space error.
See the Prerequisites section above for more details.
It is recommended to determine the total size of all images to be prestaged
in advance and resize the docker-distribution (subcloud local registry)
filesystem using the :command:`system controllerfs-modify` command accordingly.
.. rubric:: |proc|
#. To upload the prestage image list, use the following command after the
System Controller has been upgraded.
#. To upload the prestage image list, use the following command:
.. code-block:: none
@ -197,74 +201,166 @@ Single Subcloud Prestage
See :ref:`prestaging-prereqs` for preconditions prior to prestaging the subcloud.
Subcloud can be prestaged for install of current/previous software version or
for software deployment, and the previous release software must be uploaded
before prestaging.
.. code-block:: none
~(keystone_admin)]$ dcmanager subcloud prestage subcloud2
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager subcloud prestage --for-install --release nn.nn subcloud7
Enter the sysadmin password for the subcloud:
Re-enter sysadmin password to confirm:
+-----------------------------+----------------------------+
| Field | Value |
+-----------------------------+----------------------------+
| id | 2 |
| name | subcloud2 |
| id | 7 |
| name | subcloud7 |
| description | None |
| location | None |
| software_version | nn.nn |
| software_version | 22.12 |
| management | managed |
| availability | online |
| deploy_status | prestage-prepare |
| management_subnet | 2620:10a:a001:ac01::20/123 |
| management_start_ip | 2620:10a:a001:ac01::22 |
| management_end_ip | 2620:10a:a001:ac01::3e |
| management_gateway_ip | 2620:10a:a001:ac01::21 |
| systemcontroller_gateway_ip | 2620:10a:a001:a113::1 |
| group_id | 3 |
| created_at | 2202-03-18 20:31:16.548903 |
| updated_at | 2202-03-22 18:55:56:251643 |
| deploy_status | complete |
| management_subnet | fdff:719a:bf60:4028::/64 |
| management_start_ip | fdff:719a:bf60:4028::2 |
| management_end_ip | fdff:719a:bf60:4028::ffff |
| management_gateway_ip | fdff:719a:bf60:4028::1 |
| systemcontroller_gateway_ip | fd00:8:16::1 |
| group_id | 1 |
| peer_group_id | None |
| created_at | 2024-10-25 06:06:39.527023 |
| updated_at | 2024-10-25 08:08:16.311164 |
| backup_status | None |
| backup_datetime | None |
| prestage_status | prestaging-packages |
| prestage_versions | |
| prestage_software_version | 22.12 |
+-----------------------------+----------------------------+
.. code-block::
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager subcloud prestage --for-install subcloud5
Enter the sysadmin password for the subcloud:
Re-enter sysadmin password to confirm:
+-----------------------------+----------------------------+
| Field | Value |
+-----------------------------+----------------------------+
| id | 8 |
| name | subcloud5 |
| description | None |
| location | None |
| software_version | 24.09 |
| management | managed |
| availability | online |
| deploy_status | complete |
| management_subnet | fd00:8:28::/64 |
| management_start_ip | fd00:8:28::2 |
| management_end_ip | fd00:8:28::10 |
| management_gateway_ip | fd00:8:28::1 |
| systemcontroller_gateway_ip | fd00:8:16::1 |
| group_id | 1 |
| peer_group_id | None |
| created_at | 2024-10-25 06:28:57.197765 |
| updated_at | 2024-10-25 08:08:38.433790 |
| backup_status | None |
| backup_datetime | None |
| prestage_status | prestaging-packages |
| prestage_versions | |
| prestage_software_version | 24.09 |
+-----------------------------+----------------------------+
.. note::
- If the ``--release`` option is not specified, the subcloud will be
prestaged for install with the same software as the system controller.
Also, the release format must be MM.mm and any other format is not valid.
- If the release is not in deployed/available state and if it tries to do prestaging
for the ``--for-sw-deploy`` option, prestaging will throw an error.
- To verify the release, run the :command:`software list` command.
.. code-block::
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager subcloud prestage --for-sw-deploy subcloud5
Enter the sysadmin password for the subcloud:
Re-enter sysadmin password to confirm:
+-----------------------------+----------------------------+
| Field | Value |
+-----------------------------+----------------------------+
| id | 8 |
| name | subcloud5 |
| description | None |
| location | None |
| software_version | 24.09 |
| management | managed |
| availability | online |
| deploy_status | complete |
| management_subnet | fd00:8:28::/64 |
| management_start_ip | fd00:8:28::2 |
| management_end_ip | fd00:8:28::10 |
| management_gateway_ip | fd00:8:28::1 |
| systemcontroller_gateway_ip | fd00:8:16::1 |
| group_id | 3 |
| peer_group_id | None |
| created_at | 2024-09-21 15:21:20.418706 |
| updated_at | 2024-09-22 08:44:30.937440 |
| backup_status | None |
| backup_datetime | None |
| prestage_status | prestaging-packages |
| prestage_versions | |
| prestage_software_version | 24.09 |
+-----------------------------+----------------------------+
.. note::
To perform the software deployment prestaging, system controller should
have higher patch level than the subcloud. Post prestaging, the release
will be in available state in the subcloud. If the subcloud has higher patch
level than system controller, prestaging will be skipped.
-----------------------
Rerun Subcloud Prestage
-----------------------
A subcloud can be prestaged multiple times. However, only prestaging images
will be repeated. Once packages prestaging is successful, this step will be
skipped in subsequent prestage reruns for the same software version.
A subcloud can be prestaged multiple times. Software prestaging will be skipped
if the subcloud already has the specified software version. Prestaging of
container images will be repeated.
------------------------
Verify Subcloud Prestage
------------------------
After a subcloud is successfully prestaged, the ``deploy_status`` will change to
``prestage-complete``. Use the :command:`dcmanager subcloud show` command to
verify the status. The packages directory, repodata directory, and container
image bundles, and md5 file can be found on the subcloud in
``/opt/platform-backup/<sw_version>``.
Where, the `<sw_version>` number is the active load of the System Controller.
After a subcloud is successfully prestaged, ``prestage_status`` will change to
``complete`` and ``prestage_versions`` will show the software version(s) the
subcloud has prestaged data for. Use the :command:`dcmanager subcloud show`
command to verify the status and versions.
------------------------------
Troubleshoot Subcloud Prestage
------------------------------
If the subcloud prestage fails, check ``/var/log/dcmanager/dcmanager.log``
for the reason of failure. Once the issue has been resolved, prestage can be
retried using :command:`dcmanager subcloud prestage` command.
If the subcloud prestage fails, check ``/var/log/dcmanager/dcmanager.log`` and
``/var/log/dcmanager/ansible/<subcloud-name>_playbook_output.log`` for the reason
of failure. Once the issue has been resolved, prestage can be retried using
the :command:`dcmanager subcloud prestage` command.
---------------------------------
Verifying Usage of Prestaged Data
---------------------------------
To verify that the prestaged data is used over subcloud upgrade, subcloud
redeploy, or subcloud remote restore:
To verify that the prestaged data is used over subcloud update with new
software or subcloud redeployment/restore:
- Search for the the subcloud name in the log file, for example,
subcloud1 from ``/www/var/log/lighttpd-access.log``. There should not be
GET requests to download packages from ``/iso/<sw_version>/nodes/subcloud1/Packages/``.
- Check subcloud ansible log in ``/var/log/dcmanager/ansible`` directory.
- Check subcloud ansible log in the ``/var/log/dcmanager/ansible`` directory.
Images are imported from local archives and no images in the prestage image
list need to be downloaded from configured sources.
list need to be downloaded from the configured sources.
- If we reinstall the subcloud, then we can check ``/root/install.log`` to
validate if the prestage data was used during installation.

68
doc/source/dist_cloud/kubernetes/prestage-subcloud-orchestration-eb516473582f.rst
View File

@ -12,7 +12,6 @@ subcloud group or a specific subcloud group.
For more information on prerequisites for prestage upgrade and reinstall, see
:ref:`prestage-a-subcloud-using-dcmanager-df756866163f`.
.. note::
Any existing strategy must be deleted first as only one type
@ -34,65 +33,59 @@ For more information on prerequisites for prestage upgrade and reinstall, see
To create a prestage strategy for a specific subcloud, use the following
command:
.. code-block:: none
~(keystone_admin)]$ dcmanager prestage-strategy create subcloud1
.. parsed-literal::
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager prestage-strategy create --for-install --release nn.nn subcloud7
Enter the sysadmin password for the subcloud:
Re-enter sysadmin password to confirm:
+-----------------------------+----------------------------+
+--------------------------+-----------------------------+
| Field | Value |
+-----------------------------+----------------------------+
| id | 1 |
| name | subcloud1 |
| description | None |
| location | False |
| software_version | nn.nn |
| management | managed |
| availability | online |
| deploy_status | prestage-prepare |
| management_subnet | 2620:10a:a001:ac01::20/123 |
| management_start_ip | 2620:10a:a001:ac01::22 |
| management_end_ip | 2620:10a:a001:ac01::3e |
| management_gateway_ip | 2620:10a:a001:ac01::21 |
| systemcontroller_gateway_ip | 2620:10a:a001:a113::1 |
| group_id | 3 |
| created_at | 2202-03-18 20:31:16.548903 |
| updated_at | 2202-03-22 18:55:56:251643 |
+-----------------------------+----------------------------+
+--------------------------+-----------------------------+
| strategy type | prestage |
| subcloud apply type | None |
| max parallel subclouds | 2 |
| stop on failure | False |
| prestage software version| 22.12 |
| state | initial |
| created_at | 2024-10-25T05:37:54.607307 |
| updated_at | None |
+--------------------------+-----------------------------+
To create a prestage strategy for the default subcloud group, use the
following command:
.. code-block:: none
.. parsed-literal::
~(keystone_admin)]$ dcmanager prestage-strategy create
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager prestage-strategy create --for-sw-deploy
Enter the sysadmin password for the subcloud:
Re-enter sysadmin password to confirm:
+------------------------+-----------------------------+
+--------------------------+-----------------------------+
| Field | Value |
+------------------------+-----------------------------+
+--------------------------+-----------------------------+
| strategy type | prestage |
| subcloud apply type | parallel |
| subcloud apply type | None |
| max parallel subclouds | 50 |
| stop on failure | False |
| prestage software version| 24.09 |
| state | initial |
| created_at | 2202-03-22T18:54:45.037336 |
| created_at | 2024-10-25T05:37:54.607307 |
| updated_at | None |
+------------------------+-----------------------------+
+--------------------------+-----------------------------+
.. note::
If the ``--release`` option is not specified, the release version of the
system controller will be used.
To create a prestage strategy for a specific subcloud group, use the
following command:
.. code-block:: none
~(keystone_admin)]$ dcmanager prestage-strategy create –group First_10_Subclouds
~(keystone_admin)]$ dcmanager prestage-strategy create --group First_10_Subclouds
Enter the sysadmin password for the subcloud:
Re-enter sysadmin password to confirm:
+------------------------+-----------------------------+
| Field | Value |
+------------------------+-----------------------------+
@ -105,6 +98,11 @@ For more information on prerequisites for prestage upgrade and reinstall, see
| updated_at | None |
+------------------------+-----------------------------+
.. note::
If the prestage type is not specified, the default prestage type
(``--for-install``) will be applied.
.. note::
Unlike other types of orchestration, prestage orchestration requires