.. pek1594745988225
.. _distributed-software-deploy-orchestration-process-using-the-cli:

===============================================================
Distributed Software Deploy Orchestration Process using the CLI
===============================================================

Distributed software deploy orchestration can be initiated after the System
Controller has been successfully updated (patch) or deployed.

For more information on Prestaging Subcloud Orchestration see,
:ref:`prestage-subcloud-orchestration-eb516473582f`.

.. note::

    This section is applicable to users that use the dcmanager software deploy
    orchestration strategy to manage upgrades across subclouds.

    Refer to |updates-doc| for more details.

.. rubric:: |context|

The user first creates a distributed software deploy orchestration strategy, or
plan, for the automated software deploy procedure. This customizes the software
deploy orchestration, using parameters to specify:


.. _distributed-upgrade-orchestration-process-using-the-cli-ul-eyw-fyr-31b:

-   whether to stop on failure of a subcloud update/upgrade or continue with
    the next subcloud

-   whether to update/upgrade hosts serially or in parallel


Based on these parameters, and the state of the subclouds, the software deploy
orchestration creates a number of stages for the overall sw-deploy strategy.
All the subclouds that are included in the same stage will be updated/upgraded
in parallel.

.. rubric:: |prereq|

Distributed software deploy orchestration can only be done on a system that
meets the following conditions:

.. _distributed-upgrade-orchestration-process-using-the-cli-ul-blp-gcx-ry:

-   All subclouds are clear of management-affecting alarms (with the exception
    of the alarm upgrade in progress).

-   All hosts of all subclouds must be unlocked, enabled, and available.

-   All subclouds should have been prestaged (``--for-sw-deploy``).

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager prestage-strategy create --for-sw-deploy

-   All the subclouds do not have any VIM strategy different from ``sw-deploy``
    or in an invalid state.

-   No distributed software deploy orchestration strategy exists, to verify use
    the command :command:`dcmanager sw-deploy-strategy show`. An update/upgrade
    cannot be orchestrated while software deploy orchestration is in progress.

-   The size and format of the persistent filesystem, /opt/platform-backup, of
    each subcloud must be 30GiB (or larger) and ext4 respectively. From the shell
    on a subcloud, use the following command to view the details of its
    persistent file system:

    :command:`df -Th /opt/platform-backup`

    The type must be ext4 and the size must be 30GiB. For example, on
    controller-0, run the following command:

    .. code-block:: none

        ~(keystone_admin)]$ df -Th /opt/platform-backup
        Filesystem  Type    Size    Used     Avail  Use%  Mounted on
        /dev/sda1   ext4    29G     6.9G     21G    26%   /opt/platform-backup

-   **If a previous upgrade has been done on the subcloud**, from the shell on
    each subcloud, use the following command to remove the previous upgrade
    data:

    :command:`sudo rm /opt/platform-backup/upgrade_data\*`

You can configure a software deploy Distributed Cloud orchestration strategy
using the dcmanager CLI or the Horizon web interface. If you prefer to use
Horizon, see
:ref:`create-a-software-deploy-orchestration-using-horizon-9f8c6c2f3706`.


.. rubric:: |proc|

.. _distributed-upgrade-orchestration-process-using-the-cli-steps-vcm-pq4-3mb:

#.  Review the software status for the subclouds.

    After the System Controller upgrade/deploy is completed, wait for 40
    seconds for the **software_sync_status** of all subclouds to be updated.

    To identify which subclouds are deploy-current (in-sync), use the
    :command:`subcloud list` command. For example:

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager subcloud list
        +----+-----------+--------------+--------------------+-------------+
        | id | name      | management   | availability       | sync        |
        +----+-----------+--------------+--------------------+-------------+
        |  1 | subcloud-1| managed      | online             | out-of-sync |
        |  2 | subcloud-2| managed      | online             | out-of-sync |
        |  3 | subcloud-3| managed      | online             | out-of-sync |
        |  4 | subcloud-4| managed      | online             | out-of-sync |
        +----+-----------+--------------+--------------------+-------------+

    .. note::

        The subclouds are out-of-sync because the software-sync-status is
        out-of-sync. All of the above subclouds are not deploy-current and,
        therefore, need to be upgraded/updated.

    To see synchronization details for a subcloud, use the following command:

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager subcloud show subcloud1
        +-----------------------------+------------------------------+
        | Field                       | Value                        |
        +-----------------------------+------------------------------+
        | id                          | 1                            |
        | name                        | subcloud-1                   |
        | description                 | None                         |
        | location                    | None                         |
        | software_version            | nn.nn                        |
        | management                  | managed                      |
        | availability                | online                       |
        | deploy_status               | complete                     |
        | management_subnet           | fd01:82::0/64                |
        | management_start_ip         | fd01:82::2                   |
        | management_end_ip           | fd01:82::11                  |
        | management_gateway_ip       | fd01:82::1                   |
        | systemcontroller_gateway_ip | fd01:81::1                   |
        | group_id                    | 1                            |
        | created_at                  | 2021-06-07 21:05:16.224664   |
        | updated_at                  | 2021-06-09 20:01:37.525012   |
        | dc-cert_sync_status         | in-sync                      |
        | firmware_sync_status        | in-sync                      |
        | identity_sync_status        | in-sync                      |
        | kubernetes_sync_status      | in-sync                      |
        | load_sync_status            | not-available                |
        | patching_sync_status        | not-available                |
        | software_sync_status        | out-of-sync                  |
        | platform_sync_status        | in-sync                      |
        +-----------------------------+------------------------------+

#.  To create a sw-deploy strategy, use the :command:`dcmanager sw-deploy-strategy create <release_id>`
    command.

    The sw-deploy strategy for a |prod-dc| system controls how updates/upgrades
    are applied to subclouds.

    .. code-block:: none

        ~(keystone_admin)]$  dcmanager sw-deploy-strategy create \
        [--subcloud-apply-type <type>] \
        [–-max-parallel-subclouds <i>] \
        [–-stop-on-failure <level>] \
        [--group group] \
        [<subcloud>]

    where:

    **subcloud-apply-type**
        **parallel** or **serial**— determines whether the subclouds are
        upgraded in parallel, or serially.

        If this is not specified using the CLI, the values for
        :command:`subcloud_update_type` defined for each subcloud group will
        be used by default.

    **max-parallel-subclouds**
        Sets the maximum number of subclouds that can be upgraded in parallel
        (default 2).

        If this is not specified using the CLI, the values for
        :command:`max_parallel_subclouds` defined for each subcloud group
        will be used by default.

    **stop-on-failure**
        **true**\(default) or **false**— determines whether upgrade
        orchestration failure for a subcloud prevents application to subsequent
        subclouds.

    **group**
        Optionally pass the name or ID of a subcloud group to the
        :command:`dcmanager sw-deploy-strategy create` command. This results in a
        strategy that is only applied to all subclouds in the specified group.
        The subcloud group values are used for subcloud apply type and max
        parallel subclouds parameters.

    For example:

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager sw-deploy-strategy create
        +------------------------+----------------------------+
        | Field                  | Value                      |
        +------------------------+----------------------------+
        | strategy type          | sw-deploy                  |
        | subcloud apply type    | parallel                   |
        | max parallel subclouds | 2                          |
        | stop on failure        | False                      |
        | state                  | initial                    |
        | created_at             | 2024-11-06 12:56:17.111621 |
        | updated_at             | None                       |
        +------------------------+----------------------------+

#.  To show the settings for the ``sw-deploy-strategy``, use the
    :command:`dcmanager sw-deploy-strategy show` command.

    For example:

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager sw-deploy-strategy show
        +------------------------+----------------------------+
        | Field                  | Value                      |
        +------------------------+----------------------------+
        | strategy type          | sw-deploy                  |
        | subcloud apply type    | parallel                   |
        | max parallel subclouds | 2                          |
        | stop on failure        | False                      |
        | state                  | initial                    |
        | created_at             | 2020-02-02T14:42:13.822499 |
        | updated_at             | None                       |
        +------------------------+----------------------------+

    .. note::
        The values for `subcloud apply type` and `max parallel subclouds` will
        be taken from the subcloud group if specified through the ``--group``
        parameter.

#.  Review the sw-deploy strategy for the subclouds.

    To show the subclouds that will be upgraded when the sw-deploy strategy is
    applied, use the :command:`dcmanager strategy-step list` command. For
    example:

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager strategy-step list
        +------------------+--------+---------+---------+------------+-------------+
        | cloud            | stage  | state   | details | started_at | finished_at |
        +------------------+--------+---------+---------+------------+-------------+
        | subcloud-1       | 1      | initial |         | None       | None        |
        | subcloud-4       | 1      | initial |         | None       | None        |
        | subcloud-5       | 2      | initial |         | None       | None        |
        | subcloud-6       | 2      | initial |         | None       | None        |
        +------------------+--------+---------+---------+------------+-------------+

    .. note::

        All the subclouds that are included in the same stage will be deployed
        in parallel.

#.  To apply the software deploy strategy, use the :command:`dcmanager sw-deploy-strategy apply`
    command.

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager sw-deploy-strategy apply
        +------------------------+----------------------------+
        | Field                  | Value                      |
        +------------------------+----------------------------+
        | subcloud apply type    | parallel                   |
        | max parallel subclouds | 2                          |
        | stop on failure        | False                      |
        | state                  | applying                   |
        | created_at             | 2020-02-02T14:42:13.822499 |
        | updated_at             | 2020-02-02T14:42:19.376688 |
        +------------------------+----------------------------+

#.  To show the step currently being performed on each of the subclouds, use
    the :command:`dcmanager strategy-step list` command.

    For example:

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager strategy-step list
        +------------------+--------+-----------------------+-------------------+----------------------------+----------------------------+
        | cloud            | stage  | state                 | details           | started_at                 | finished_at                |
        +------------------+--------+-----------------------+-------------------+----------------------------+----------------------------+
        | subcloud-1       | 3      | complete              |                   | 2024-11-06 12:57:20.342429 | 2024-11-06 12:57:41.054664 |
        | subcloud-4       | 2      | apply VIM sw-deploy   |                   | 2024-11-06 12:57:20.342429 | None                       |
        | subcloud-5       | 1      | initial               |                   | None                       | None                       |
        | subcloud-6       | 1      | initial               |                   | None                       | None                       |
        +------------------+--------+-------------+-----------------------------+----------------------------+----------------------------+

#.  To show the step currently being performed on a subcloud, use the
    :command:`dcmanager strategy-step show` <subcloud> command.

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager strategy-step show <subcloud>

#.  When all the subclouds within the distributed software deploy orchestration indicate
    they have entered the complete state, delete the sw-deploy strategy, using
    the :command:`dcmanager sw-deploy-strategy delete` command.

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager sw-deploy-strategy delete
        +------------------------+----------------------------+
        | Field                  | Value                      |
        +------------------------+----------------------------+
        | subcloud apply type    | parallel                   |
        | max parallel subclouds | 2                          |
        | stop on failure        | False                      |
        | state                  | deleting                   |
        | created_at             | 2020-03-23T20:04:50.992444 |
        | updated_at             | 2020-03-23T20:05:14.157352 |
        +------------------------+----------------------------+

.. note::
    Before attempting to log in to the subclouds using the sysadmin account,
    verify that the subcloud ``platform_sync_status`` is synced. This would
    ensure that the sysadmin password is successfully resynced to the subclouds
    and that login attempts do not fail.

.. only:: partner

   .. include:: /_includes/distributed-upgrade-orchestration-process-using-the-cli.rest
      :start-after: DMupgrade-begin
      :end-before: DMupgrade-end