Platform Upgrades with Deployment Manager

Updated comments
Updated Patchset 1 comments
Resolved merge conflicts.
Fixed rst syntax on inline named literal.

Signed-off-by: Juanita-Balaraj <juanita.balaraj@windriver.com>
Change-Id: I20c339767de09e5f7fcbb88b9ae10366e6d790a2
Signed-off-by: Juanita-Balaraj <juanita.balaraj@windriver.com>
Signed-off-by: Ron Stone <ronald.stone@windriver.com>
This commit is contained in:
Juanita-Balaraj 2021-06-10 00:41:39 -04:00 committed by Ron Stone
parent ccb1d63cc2
commit e3c0887983
12 changed files with 302 additions and 90 deletions

View File

@ -0,0 +1,6 @@
.. syscontroller-begin
.. syscontroller-end
.. dcupgrade-begin
.. dcupgrade-end

View File

@ -0,0 +1,4 @@
.. Orchupgradecli-begin
.. Orchupgradecli-end

View File

@ -0,0 +1,3 @@
.. Orchupgrade-begin
.. Orchupgrade-end

View File

@ -0,0 +1,4 @@
.. upgradeDX-begin
.. upgradeDX-end

View File

@ -0,0 +1,4 @@
.. upgradeAIO-begin
.. upgradeAIO-end

View File

@ -0,0 +1,6 @@
.. license-begin
.. license-end
.. upgradeDM-begin
.. upgradeDM-end

View File

@ -43,7 +43,11 @@ following conditions:
- Redfish |BMC| is required for orchestrated subcloud upgrades. The install
values, and :command:`bmc\_password` for each |AIO-SX| subcloud controller
must be provided using the following |CLI| command on the System Controller:
must be provided using the following |CLI| command on the System Controller.
.. note::
This is only needed if the subcloud has not already been provisioned
with the Redfish BMC password.
.. code-block:: none
@ -115,9 +119,9 @@ following conditions:
.. code-block:: none
~(keystone_admin)]$ dcmanager subcloud show subcloud1
+-----------------------------+----------------------------+
+-----------------------------+------------------------------+
| Field | Value |
+-----------------------------+----------------------------+
+-----------------------------+------------------------------+
| id | 1 |
| name | subcloud1 |
| description | None |
@ -132,14 +136,16 @@ following conditions:
| management_gateway_ip | fd01:82::1 |
| systemcontroller_gateway_ip | fd01:81::1 |
| group_id | 1 |
| created_at | 2020-07-15 19:23:50.966984 |
| updated_at | 2020-07-17 12:36:28.815655 |
| created_at | 2021-06-07 21:05:16.224664 |
| updated_at | 2021-06-09 20:01:37.525012 |
| dc-cert_sync_status | in-sync |
| firmware_sync_status | in-sync |
| identity_sync_status | in-sync |
| load_sync_status | in-sync |
| kubernetes_sync_status | in-sync |
| load_sync_status | out-of-sync |
| patching_sync_status | in-sync |
| platform_sync_status | in-sync |
+-----------------------------+----------------------------+
+-----------------------------+------------------------------+
#. To create an upgrade strategy, use the :command:`dcmanager upgrade-strategy create`
command.
@ -223,9 +229,9 @@ following conditions:
+------------------------+----------------------------+
.. note::
A value of **None** for :command:`subcloud apply type`, and
:command:`max parallel subclouds` indicates that subcloud group values
are being used.
The values for `subcloud apply type` and `max parallel subclouds` will
be taken from the subcloud group if specified through the ``--group``
parameter.
#. Review the upgrade strategy for the subclouds.
@ -277,10 +283,10 @@ following conditions:
+------------------+-------+-------------+-----------------------------+----------------------------+----------------------------+
| cloud | stage | state | details | started_at | finished_at |
+------------------+-------+-------------+-----------------------------+----------------------------+----------------------------+
| subcloud-1 | 2 | applying... | apply phase is 66% complete | 2020-03-13 14:12:12.262001 | 2020-03-13 14:15:52.450908 |
| subcloud-4 | 2 | applying... | apply phase is 83% complete | 2020-03-13 14:16:02.457588 | None |
| subcloud-5 | 2 | finishing | | 2020-03-13 14:16:02.463213 | None |
| subcloud-6 | 2 | applying... | apply phase is 66% complete | 2020-03-13 14:16:02.473669 | None |
| subcloud-1 | 2 | applying... | apply phase is 66% complete | 2021-06-11 14:12:12.262001 | 2021-06-11 14:15:52.450908 |
| subcloud-4 | 2 | applying... | apply phase is 83% complete | 2021-06-11 14:16:02.457588 | None |
| subcloud-5 | 2 | finishing | | 2021-06-11 14:16:02.463213 | None |
| subcloud-6 | 2 | applying... | apply phase is 66% complete | 2021-06-11 14:16:02.473669 | None |
+------------------+-------+-------------+-----------------------------+----------------------------+----------------------------+
#. To show the step currently being performed on a subcloud, use the
@ -290,8 +296,9 @@ following conditions:
~(keystone_admin)]$ dcmanager strategy-step show <subcloud>
#. When the distributed upgrade orchestration complete, delete the upgrade
strategy, using the :command:`dcmanager upgrade-strategy delete` command.
#. When all the subclouds within the distributed upgrade orchestration indicate
they have entered the complete state, delete the upgrade strategy, using
the :command:`dcmanager upgrade-strategy delete` command.
.. code-block:: none
@ -319,3 +326,11 @@ the secret payload is, "username: admin password:<password>", see,
.. only:: partner
.. include:: ../_includes/distributed-upgrade-orchestration-process-using-the-cli.rest
:start-after: syscontroller-begin
:end-before: syscontroller-end
.. only:: partner
.. include:: ../_includes/distributed-upgrade-orchestration-process-using-the-cli.rest
:start-after: dcupgrade-begin
:end-before: dcupgrade-end

View File

@ -29,6 +29,8 @@ Follow the steps below to manually upgrade the System Controller:
.. only:: partner
.. include:: ../_includes/upgrading-the-systemcontroller-using-the-cli.rest
:start-after: license-begin
:end-before: license-end
#. Transfer iso and signature files to controller-0 \(active controller\) and import the load.
@ -122,10 +124,9 @@ Follow the steps below to manually upgrade the System Controller:
- started:
- State entered after :command:`system upgrade-start` completes.
- Release 20.04 system data \(for example, postgres databases\) has
- Release nn.nn system data \(for example, postgres databases\) has
been exported to be used in the upgrade.
As part of the upgrade, the upgrade process checks the health of the system
@ -472,3 +473,9 @@ Follow the steps below to manually upgrade the System Controller:
Run the :command:`system upgrade-show` command, and the status will display
"no upgrade in progress". The subclouds will be out-of-sync.
.. only:: partner
.. include:: /_includes/upgrading-the-systemcontroller-using-the-cli.rest
:start-after: upgradeDM-begin
:end-before: upgradeDM-end

View File

@ -61,9 +61,9 @@ upgrade orchestration to orchestrate the remaining nodes of the |prod|.
.. rubric:: |prereq|
See :ref:`Upgrading All-in-One Duplex / Standard
<upgrading-all-in-one-duplex-or-standard>` to manually upgrade the initial
controller node before doing the upgrade orchestration described below to
upgrade the remaining nodes of the |prod|.
<upgrading-all-in-one-duplex-or-standard>`, and perform Steps 1-8, to manually
upgrade the initial controller node before doing the upgrade orchestration
described below to upgrade the remaining nodes of the |prod|.
- The subclouds must use the Redfish platform management service if it is an All-in-one Simplex subcloud.
@ -73,19 +73,28 @@ upgrade the remaining nodes of the |prod|.
.. _performing-an-orchestrated-upgrade-using-the-cli-steps-e45-kh5-sy:
#. Create a update strategy using the :command:`sw-manager` upgrade-strategy
#. Create a update strategy using the :command:`sw-manager upgrade-strategy create`
command.
.. code-block:: none
~(keystone_admin)]$ sw-manager upgrade-strategy create
strategy-uuid: 5435e049-7002-4403-acfb-7886f6da14af
controller-apply-type: serial
storage-apply-type: serial
worker-apply-type: serial
default-instance-action: migrate
alarm-restrictions: strict
current-phase: build
current-phase-completion: 0%
state: building
inprogress: true
Create an upgrade strategy, specifying the following parameters:
- storage-apply-type:
- serial \(default\): storage hosts will be upgraded one at a time
- parallel: storage hosts will be upgraded in parallel, ensuring that
@ -167,16 +176,47 @@ upgrade the remaining nodes of the |prod|.
updated, but any additional pods on each worker host will be
relocated before it is upgraded.
#. Apply the upgrade-strategy. You can optionally apply a single stage at a time.
#. Run :command:`sw-manager upgrade-strategy show` command, to display the
current-phase-completion displaying the field goes from 0% to 100% in
various increments. Once at 100%, it returns:
.. code-block:: none
~(keystone_admin)]$ sw-manager upgrade-strategy show
strategy-uuid: 5435e049-7002-4403-acfb-7886f6da14af
controller-apply-type: serial
storage-apply-type: serial
worker-apply-type: serial
default-instance-action: migrate
alarm-restrictions: strict
current-phase: build
current-phase-completion: 100%
state: ready-to-apply
build-result: success
build-reason:
#. Apply the upgrade-strategy. You can optionally apply a single stage at a
time.
.. code-block:: none
~(keystone_admin)]$ sw-manager upgrade-strategy apply
strategy-uuid: 5435e049-7002-4403-acfb-7886f6da14af
controller-apply-type: serial
storage-apply-type: serial
worker-apply-type: serial
default-instance-action: migrate
alarm-restrictions: strict
current-phase: apply
current-phase-completion: 0%
state: applying
inprogress: true
While an upgrade-strategy is being applied, it can be aborted. This results
in:
- The current step will be allowed to complete.
- If necessary an abort phase will be created and applied, which will
@ -187,3 +227,106 @@ upgrade the remaining nodes of the |prod|.
upgrade-strategy application fails, you must address the issue that caused
the failure, then delete/re-create the strategy before attempting to apply
it again.
#. Run :command:`sw-manager upgrade-strategy show` command, to display the
current-phase-completion displaying the field goes from 0% to 100% in
various increments. Once at 100%, it returns:
.. code-block:: none
~(keystone_admin)]$ sw-manager upgrade-strategy show
strategy-uuid: b91d8332-9ece-4578-b4dd-e9cf87b73f18
controller-apply-type: serial
storage-apply-type: serial
worker-apply-type: serial
default-instance-action: migrate
alarm-restrictions: strict
current-phase: apply
current-phase-completion: 100%
state: applied
apply-result: success
apply-reason:
#. Activate the upgrade.
During the running of the :command:`upgrade-activate` command, new
configurations are applied to the controller. 250.001 \(**hostname
Configuration is out-of-date**\) alarms are raised and are cleared as the
configuration is applied. The upgrade state goes from **activating** to
**activation-complete** once this is done.
.. code-block:: none
~(keystone_admin)]$ system upgrade-activate
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state | activating |
| from_release | nn.nn |
| to_release | nn.nn |
+--------------+--------------------------------------+
The following states apply when this command is executed.
**activation-requested**
State entered when :command:`system upgrade-activate` is executed.
**activating**
State entered when we have started activating the upgrade by applying
new configurations to the controller and compute hosts.
**activation-complete**
State entered when new configurations have been applied to all
controller and compute hosts.
#. Check the status of the upgrade again to see it has reached
**activation-complete**
.. code-block:: none
~(keystone_admin)]$ system upgrade-show
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state | activation-complete |
| from_release | nn.nn |
| to_release | nn.nn |
+--------------+--------------------------------------+
#. Complete the upgrade.
.. code-block:: none
~(keystone_admin)]$ system upgrade-complete
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state | completing |
| from_release | nn.nn |
| to_release | nn.nn |
+--------------+--------------------------------------+
#. Delete the imported load.
.. code-block:: none
~(keystone_admin)]$ system load-list
+----+----------+------------------+
| id | state | software_version |
+----+----------+------------------+
| 1 | imported | nn.nn |
| 2 | active | nn.nn |
+----+----------+------------------+
~(keystone_admin)]$ system load-delete 1
Deleted load: load 1
.. only:: partner
.. include:: /_includes/performing-an-orchestrated-upgrade-using-the-cli.rest
:start-after: Orchupgradecli-begin
:end-before: Orchupgradecli-end

View File

@ -35,9 +35,9 @@ You can perform a partially-Orchestrated Upgrade of a |prod| system using the CL
.. rubric:: |prereq|
See :ref:`Upgrading All-in-One Duplex / Standard
<upgrading-all-in-one-duplex-or-standard>`, to manually upgrade the initial
controller node before doing the upgrade orchestration described below to
upgrade the remaining nodes of the |prod| system.
<upgrading-all-in-one-duplex-or-standard>`, and perform Steps 1-8, to manually
upgrade the initial controller node before doing the upgrade orchestration
described below to upgrade the remaining nodes of the |prod| system.
.. rubric:: |proc|
@ -167,3 +167,11 @@ upgrade the remaining nodes of the |prod| system.
upgrade-strategy application fails, you must address the issue that caused
the failure, then delete/re-create the strategy before attempting to apply
it again.
For more information, see: :ref:`Perform an Orchestrated Upgrade Using the CLI <performing-an-orchestrated-upgrade-using-the-cli>`
.. only:: partner
.. include:: /_includes/performing-an-orchestrated-upgrade.rest
:start-after: Orchupgrade-begin
:end-before: Orchupgrade-end

View File

@ -78,8 +78,8 @@ of |prod| software.
+--------------------+-----------+
| id | 2 |
| state | importing |
| software_version | 21.05 |
| compatible_version | 20.06 |
| software_version | nn.nn |
| compatible_version | nn.nn |
| required_patches | |
+--------------------+-----------+
@ -98,8 +98,8 @@ of |prod| software.
+----+----------+------------------+
| id | state | software_version |
+----+----------+------------------+
| 1 | active | 20.06 |
| 2 | imported | 21.05 |
| 1 | active | nn.nn |
| 2 | imported | nn.nn |
+----+----------+------------------+
@ -165,8 +165,8 @@ of |prod| software.
+--------------+--------------------------------------+
| uuid | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state | starting |
| from_release | 20.06 |
| to_release | 21.05 |
| from_release | nn.nn |
| to_release | nn.nn |
+--------------+--------------------------------------+
This will make a copy of the system data to be used in the upgrade.
@ -179,7 +179,7 @@ of |prod| software.
- State entered after :command:`system upgrade-start` completes.
- Release 20.06 system data \(for example, postgres databases\) has
- Release nn.nn system data \(for example, postgres databases\) has
been exported to be used in the upgrade.
- Configuration changes must not be made after this point, until the
@ -234,8 +234,8 @@ of |prod| software.
- State entered when controller-1 upgrade is complete.
- System data has been successfully migrated from release 20.06
to release 21.05.
- System data has been successfully migrated from release nn.nn
to release nn.nn.
- data-migration-failed:
@ -254,8 +254,8 @@ of |prod| software.
+--------------+--------------------------------------+
| uuid | e7c8f6bc-518c-46d4-ab81-7a59f8f8e64b |
| state | data-migration-complete |
| from_release | 20.06 |
| to_release | 21.05 |
| from_release | nn.nn |
| to_release | nn.nn |
+--------------+--------------------------------------+
If the :command:`upgrade-show` status indicates
@ -276,7 +276,7 @@ of |prod| software.
- upgrading-controllers:
- State entered when controller-1 has been unlocked and is
running release 21.05 software.
running release nn.nn software.
If it transitions to **unlocked-disabled-failed**, check the issue
before proceeding to the next step. The alarms may indicate a
@ -320,7 +320,7 @@ of |prod| software.
- upgrading-hosts:
- State entered when both controllers are running release 21.05
- State entered when both controllers are running release nn.nn
software.
#. Check the system health to ensure that there are no unexpected alarms.
@ -428,8 +428,8 @@ of |prod| software.
+--------------+--------------------------------------+
| uuid | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state | activating |
| from_release | 20.06 |
| to_release | 21.05 |
| from_release | nn.nn |
| to_release | nn.nn |
+--------------+--------------------------------------+
During the running of the :command:`upgrade-activate` command, new
@ -466,8 +466,8 @@ of |prod| software.
+--------------+--------------------------------------+
| uuid | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state | activation-complete |
| from_release | 20.06 |
| to_release | 21.05 |
| from_release | nn.nn |
| to_release | nn.nn |
+--------------+--------------------------------------+
.. note::
@ -483,8 +483,8 @@ of |prod| software.
+--------------+--------------------------------------+
| uuid | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state | completing |
| from_release | 20.06 |
| to_release | 21.05 |
| from_release | nn.nn |
| to_release | nn.nn |
+--------------+--------------------------------------+
#. Delete the imported load.
@ -495,11 +495,15 @@ of |prod| software.
+----+----------+------------------+
| id | state | software_version |
+----+----------+------------------+
| 1 | imported | 20.06 |
| 2 | active | 21.05 |
| 1 | imported | nn.nn |
| 2 | active | nn.nn |
+----+----------+------------------+
~(keystone_admin)]$ system load-delete 1
Deleted load: load 1
.. only:: partner
.. include:: /_includes/upgrading-all-in-one-duplex-or-standard.rest
:start-after: upgradeDX-begin
:end-before: upgradeDX-end

View File

@ -17,7 +17,7 @@ software.
- Perform a full backup to allow recovery.
.. note::
Back up files in the /home/sysadmin and /rootdirectories prior to doing
Back up files in the /home/sysadmin and /root directories prior to doing
an upgrade. Home directories are not preserved during backup or restore
operations, blade replacement, or upgrades.
@ -52,7 +52,7 @@ the system contains more than 5 GBytes of these images, the upgrade start will f
~(keystone_admin)]$
#. Install the license file for the release you are upgrading to, for example,
20.06.
21.05.
.. code-block:: none
@ -84,8 +84,8 @@ the system contains more than 5 GBytes of these images, the upgrade start will f
+--------------------+-----------+
| id | 2 |
| state | importing |
| software_version | 21.05 |
| compatible_version | 20.06 |
| software_version | nn.nn |
| compatible_version | nn.nn |
| required_patches | |
+--------------------+-----------+
@ -101,8 +101,8 @@ the system contains more than 5 GBytes of these images, the upgrade start will f
+----+----------+------------------+
| id | state | software_version |
+----+----------+------------------+
| 1 | active | 20.06 |
| 2 | imported | 21.05 |
| 1 | active | nn.nn |
| 2 | imported | nn.nn |
+----+----------+------------------+
.. note::
@ -166,8 +166,8 @@ the system contains more than 5 GBytes of these images, the upgrade start will f
+--------------+--------------------------------------+
| uuid | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state | starting |
| from_release | 20.06 |
| to_release | 21.05 |
| from_release | nn.nn |
| to_release | nn.nn |
+--------------+--------------------------------------+
This will back up the system data and images to /opt/platform-backup.
@ -187,7 +187,7 @@ the system contains more than 5 GBytes of these images, the upgrade start will f
- State entered after :command:`system upgrade-start` completes.
- Release 20.04 system data \(for example, postgres databases\) has
- Release nn.nn system data \(for example, postgres databases\) has
been exported to be used in the upgrade.
- Configuration changes must not be made after this point, until the
@ -215,8 +215,8 @@ the system contains more than 5 GBytes of these images, the upgrade start will f
+--------------+--------------------------------------+
| uuid | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state | started |
| from_release | 20.06 |
| to_release | 21.05 |
| from_release | nn.nn |
| to_release | nn.nn |
+--------------+--------------------------------------+
Ensure the upgrade state is **started**. It will take several minutes to
@ -304,8 +304,8 @@ the system contains more than 5 GBytes of these images, the upgrade start will f
+--------------+--------------------------------------+
| uuid | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state | upgrading-hosts |
| from_release | 20.06 |
| to_release | 21.05 |
| from_release | nn.nn |
| to_release | nn.nn |
+--------------+--------------------------------------+
#. Unlock controller-0.
@ -332,8 +332,8 @@ the system contains more than 5 GBytes of these images, the upgrade start will f
+--------------+--------------------------------------+
| uuid | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state | activating |
| from_release | 20.06 |
| to_release | 21.05 |
| from_release | nn.nn |
| to_release | nn.nn |
+--------------+--------------------------------------+
The following states apply when this command is executed.
@ -358,6 +358,10 @@ the system contains more than 5 GBytes of these images, the upgrade start will f
.. code-block:: none
.. note::
This can take more than half an hour to complete.
~(keystone_admin)]$ system upgrade-show
+--------------+--------------------------------------+
| Property | Value |
@ -368,8 +372,6 @@ the system contains more than 5 GBytes of these images, the upgrade start will f
| to_release | 21.05 |
+--------------+--------------------------------------+
.. note::
This can take more than half an hour to complete.
#. Complete the upgrade.
@ -381,8 +383,8 @@ the system contains more than 5 GBytes of these images, the upgrade start will f
+--------------+--------------------------------------+
| uuid | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state | completing |
| from_release | 20.06 |
| to_release | 21.05 |
| from_release | nn.nn |
| to_release | nn.nn |
+--------------+--------------------------------------+
#. Delete the imported load.
@ -393,9 +395,15 @@ the system contains more than 5 GBytes of these images, the upgrade start will f
+----+----------+------------------+
| id | state | software_version |
+----+----------+------------------+
| 1 | imported | 20.06 |
| 2 | active | 21.05 |
| 1 | imported | nn.nn |
| 2 | active | nn.nn |
+----+----------+------------------+
~(keystone_admin)]$ system load-delete 1
Deleted load: load 1
.. only:: partner
.. include:: /_includes/upgrading-all-in-one-simplex.rest
:start-after: upgradeAIO-begin
:end-before: upgradeAIO-end