Elaine Fonaro 952c0b53ae Fixed Errors in WRCP Distributed Cloud Upgrade management (r8, dsr8)

For System Controller Upgrade:
- Moved Duplex bullet to line 61.
- Removed a note and some comands from files.
- Removed information from postreq and added new text.
- Minor updates.
- Applied some fixes according the review.
- Added --local option to the command.
- Removed the example as it is identical to the command.
- Updated Step 3. In the second paragraph.
- Updated Step 4. second paragraph;
- Updated and moved Note to correct place. same for
Upgrade All-in-One Simplex and Upgrade All-in-One Duplex / Standard docs.
For Orchestrated Subcloud Upgrade section:
Prerquisites:
- Removed the 3rd bullet and updted the first.
- Reword the Note.
- Updated bullet 7;
Procedure:
- Updated step2: max-parallel-subclouds default is 2.


Signed-off-by: Elaine Fonaro <elaine.fonaro@windriver.com>
Change-Id: I095c6d14135d863e5117dc033028ef9e0dd79854

2023-05-17 19:58:19 +00:00

15 KiB

Raw Blame History

Distributed Upgrade Orchestration Process using the CLI

Distributed upgrade orchestration can be initiated after the System Controller has been successfully upgraded.

For more information on Prestaging Subcloud Orchestration see, prestage-subcloud-orchestration-eb516473582f.

The user first creates a distributed upgrade orchestration strategy, or plan, for the automated upgrade procedure. This customizes the upgrade orchestration, using parameters to specify:

whether to stop on failure of a subcloud upgrade or continue with the next subcloud
whether to upgrade hosts serially or in parallel

Based on these parameters, and the state of the subclouds, the upgrade orchestration creates a number of stages for the overall upgrade strategy. All the subclouds that are included in the same stage will be upgraded in parallel.

Distributed upgrade orchestration can only be done on a system that meets the following conditions:

The subclouds must use the Redfish platform management service if it is an subcloud. The install values as well as bmc_password for each subcloud must have already been saved in the system controller database before the start of orchestrated subcloud upgrade.

Note

The following command is only required if the subcloud install values and bmc_password have never been provided using dcmanager CLI command.
```
~(keystone_admin)]$ dcmanager subcloud update subcloud1 --install-values\
install-values.yaml --bmc-password <password>
```
For more information on install-values.yaml file, see Installing a Subcloud Using Redfish Platform Management Service <installing-a-subcloud-using-redfish-platform-management-service>.
Duplex (/Standard) upgrades are supported, and they do not require remote install using Redfish.
All subclouds are clear of management-affecting alarms (with the exception of the alarm upgrade in progress).
All hosts of all subclouds must be unlocked, enabled, and available.
No distributed upgrade orchestration strategy exists, to verify use the command dcmanager upgrade-strategy-show. An upgrade cannot be orchestrated while upgrade orchestration is in progress.
The size and format of the persistent filesystem, /opt/platform-backup, of each subcloud must be 30GiB (or larger) and ext4 respectively. From the shell on a subcloud, use the following command to view the details of its persistent file system:

df -Th /opt/platform-backup

The type must be ext4 and the size must be 30GiB. For example, on controller-0, run the following command:
```
~(keystone_admin)]$ df -Th /opt/platform-backup
Filesystem  Type    Size    Used     Avail  Use%  Mounted on
/dev/sda1   ext4    29G     6.9G     21G    26%   /opt/platform-backup
```
If a previous upgrade has been done on the subcloud, from the shell on each subcloud, use the following command to remove the previous upgrade data:

sudo rm /opt/platform-backup/upgrade_data\*

You can configure an upgrade Distributed Cloud orchestration strategy using the dcmanager CLI or the Horizon web interface. If you prefer to use Horizon, see create-an-upgrade-orchestration-using-horizon-9f8c6c2f3706.

Review the upgrade status for the subclouds.

After the System Controller upgrade is completed, wait for 10 minutes for the load_sync_status of all subclouds to be updated.

To identify which subclouds are upgrade-current (in-sync), use the subcloud list command. For example:

~(keystone_admin)]$ dcmanager subcloud list
+----+-----------+--------------+--------------------+-------------+
| id | name      | management   | availability       | sync        |
+----+-----------+--------------+--------------------+-------------+
|  1 | subcloud-1| managed      | online             | out-of-sync |
|  2 | subcloud-2| managed      | online             | out-of-sync |
|  3 | subcloud-3| managed      | online             | out-of-sync |
|  4 | subcloud-4| managed      | online             | out-of-sync |
+----+-----------+--------------+--------------------+-------------+

Note

The subclouds are out-of-sync because the load-sync-status is out-of-sync. All of the above subclouds are not upgrade-current and, therefore, need to be upgraded.

To see synchronization details for a subcloud, use the following command:

~(keystone_admin)]$ dcmanager subcloud show subcloud1
+-----------------------------+------------------------------+
| Field                       | Value                        |
+-----------------------------+------------------------------+
| id                          | 1                            |
| name                        | subcloud-1                   |
| description                 | None                         |
| location                    | None                         |
| software_version            | nn.nn                        |
| management                  | managed                      |
| availability                | online                       |
| deploy_status               | complete                     |
| management_subnet           | fd01:82::0/64                |
| management_start_ip         | fd01:82::2                   |
| management_end_ip           | fd01:82::11                  |
| management_gateway_ip       | fd01:82::1                   |
| systemcontroller_gateway_ip | fd01:81::1                   |
| group_id                    | 1                            |
| created_at                  | 2021-06-07 21:05:16.224664   |
| updated_at                  | 2021-06-09 20:01:37.525012   |
| dc-cert_sync_status         | in-sync                      |
| firmware_sync_status        | in-sync                      |
| identity_sync_status        | in-sync                      |
| kubernetes_sync_status      | in-sync                      |
| load_sync_status            | out-of-sync                  |
| patching_sync_status        | in-sync                      |
| platform_sync_status        | in-sync                      |
+-----------------------------+------------------------------+

To create an upgrade strategy, use the dcmanager upgrade-strategy create command.

The upgrade strategy for a system controls how upgrades are applied to subclouds.
```
~(keystone_admin)]$ dcmanager upgrade-strategy create \
[--subcloud-apply-type <type>] \
[–-max-parallel-subclouds <i>] \
[–-stop-on-failure <level>] \
[--group group] \
[--force] \
[<subcloud>]
```
where:

subcloud-apply-type

parallel or serial— determines whether the subclouds are upgraded in parallel, or serially.

If this is not specified using the CLI, the values for subcloud_update_type defined for each subcloud group will be used by default.

max-parallel-subclouds

Sets the maximum number of subclouds that can be upgraded in parallel (default 2).

If this is not specified using the CLI, the values for max_parallel_subclouds defined for each subcloud group will be used by default.

stop-on-failure

true(default) or false— determines whether upgrade orchestration failure for a subcloud prevents application to subsequent subclouds.

group

Optionally pass the name or ID of a subcloud group to the dcmanager upgrade-strategy create command. This results in a strategy that is only applied to all subclouds in the specified group. The subcloud group values are used for subcloud apply type and max parallel subclouds parameters.

force

Upgrade both online and offline subclouds. Can be used for a single subcloud, or a subcloud group.

For example:
```
~(keystone_admin)]$ dcmanager upgrade-strategy create
+------------------------+----------------------------+
| Field                  | Value                      |
+------------------------+----------------------------+
| strategy type          | upgrade                    |
| subcloud apply type    | parallel                   |
| max parallel subclouds | 2                          |
| stop on failure        | False                      |
| state                  | initial                    |
| created_at             | 2020-06-10T17:16:51.857207 |
| updated_at             | None                       |
+------------------------+----------------------------+
```

To show the settings for the upgrade strategy, use the dcmanager upgrade-strategy show command.

For example:

~(keystone_admin)]$ dcmanager upgrade-strategy show
+------------------------+----------------------------+
| Field                  | Value                      |
+------------------------+----------------------------+
| subcloud apply type    | parallel                   |
| max parallel subclouds | 2                          |
| stop on failure        | False                      |
| state                  | initial                    |
| created_at             | 2020-02-02T14:42:13.822499 |
| updated_at             | None                       |
+------------------------+----------------------------+

Note

The values for subcloud apply type and max parallel subclouds will be taken from the subcloud group if specified through the --group parameter.

Review the upgrade strategy for the subclouds.

To show the subclouds that will be upgraded when the upgrade strategy is applied, use the dcmanager strategy-step list command. For example:

~(keystone_admin)]$ dcmanager strategy-step list
+------------------+-------+---------+---------+------------+-------------+
| cloud            | stage | state   | details | started_at | finished_at |
+------------------+-------+---------+---------+------------+-------------+
| subcloud-1       |     1 | initial |         | None       | None        |
| subcloud-4       |     1 | initial |         | None       | None        |
| subcloud-5       |     2 | initial |         | None       | None        |
| subcloud-6       |     2 | initial |         | None       | None        |
+------------------+-------+---------+---------+------------+-------------+

Note

All the subclouds that are included in the same stage will be upgraded in parallel.

To apply the upgrade strategy, use the dcmanager upgrade-strategy apply command.

~(keystone_admin)]$ dcmanager upgrade-strategy apply
+------------------------+----------------------------+
| Field                  | Value                      |
+------------------------+----------------------------+
| subcloud apply type    | parallel                   |
| max parallel subclouds | 2                          |
| stop on failure        | False                      |
| state                  | applying                   |
| created_at             | 2020-02-02T14:42:13.822499 |
| updated_at             | 2020-02-02T14:42:19.376688 |
+------------------------+----------------------------+

To show the step currently being performed on each of the subclouds, use the dcmanager strategy-step list command.

For example:

~(keystone_admin)]$ dcmanager strategy-step list
+------------------+-------+-----------------------+-------------------+----------------------------+----------------------------+
| cloud            | stage | state                 | details           | started_at                 | finished_at                |
+------------------+-------+-----------------------+-------------------+----------------------------+----------------------------+
| subcloud-1       |     1 | complete              |                   | 2021-06-11 14:12:12.262001 | 2021-06-11 14:15:52.450908 |
| subcloud-4       |     1 | activating upgrade    |                   | 2021-06-11 14:16:02.457588 | None                       |
| subcloud-5       |     2 | initial               |                   | None                       | None                       |
| subcloud-6       |     2 | initial               |                   | None                       | None                       |
+------------------+-------+-------------+-----------------------------+----------------------------+----------------------------+

To show the step currently being performed on a subcloud, use the dcmanager strategy-step show <subcloud> command.
```
~(keystone_admin)]$ dcmanager strategy-step show <subcloud>
```

When all the subclouds within the distributed upgrade orchestration indicate they have entered the complete state, delete the upgrade strategy, using the dcmanager upgrade-strategy delete command.

~(keystone_admin)]$ dcmanager upgrade-strategy delete
+------------------------+----------------------------+
| Field                  | Value                      |
+------------------------+----------------------------+
| subcloud apply type    | parallel                   |
| max parallel subclouds | 2                          |
| stop on failure        | False                      |
| state                  | deleting                   |
| created_at             | 2020-03-23T20:04:50.992444 |
| updated_at             | 2020-03-23T20:05:14.157352 |
+------------------------+----------------------------+

partner

15 KiB Raw Blame History Unescape Escape

Distributed Upgrade Orchestration Process using the CLI

15 KiB

Raw Blame History