.. noc1590162360081 .. _configuring-kubernetes-update-orchestration: ============================================================== Create Kubernetes Version Upgrade Cloud Orchestration Strategy ============================================================== You can configure *Kubernetes Version Upgrade Orchestration Strategy* using the :command:`sw-manager` CLI. .. note:: You require administrator privileges to use :command:`sw-manager`. You must log in to the active controller as **user sysadmin** and source the script by using the command, source ``/etc/platform/openrc`` to obtain administrator privileges. Do not use :command:`sudo`. .. note:: Management-affecting alarms cannot be ignored using relaxed alarm rules during an orchestrated Kubernetes version upgrade operation. For a list of management-affecting alarms, see |fault-doc|: :ref:`Alarm Messages <100-series-alarm-messages-starlingx>`. To display management-affecting active alarms, use the following command: .. code-block:: none ~(keystone_admin)$ fm alarm-list --mgmt_affecting During an orchestrated Kubernetes version upgrade operation, the following alarms are ignored even when the default strict restrictions are selected: .. _noc1590162360081-ul-vhg-jxs-tlb: - 100.103: Memory threshold exceeded. - 200.001: Locked host. - 280.001: Subcloud resource off-line. - 280.002: Subcloud resource out-of-sync. - 700.004: |VM| stopped. - 750.006: Configuration change requires reapply of cert-manager. - 900.001: Patch in progress. - 900.007: Kube upgrade in progress. - 900.401: kube-upgrade-auto-apply-inprogress. You can use ``help`` for the overall commands and also for each sub-command. For example: .. code-block:: none ~(keystone_admin)$ sw-manager kube-upgrade-strategy –help usage: sw-manager kube-upgrade-strategy [-h] ... optional arguments: -h, --help show this help message and exit Kubernetes Update Commands: create Create a strategy delete Delete a strategy apply Apply a strategy abort Abort a strategy show Show a strategy .. rubric:: |prereq| .. _noc1590162360081-ul-ls2-pxs-tlb: - Hosts that need to be upgraded must be in the ``unlocked-enabled`` state. - If you are using NetApp Trident, ensure that your NetApp version is compatible with Trident 22.07 before upgrading Kubernetes to version |kube-ver| and after updating |prod| to version |prod-ver|. For more information, see :ref:`Upgrade the NetApp Trident Software `. .. only:: partner .. include:: /_includes/configuring-kubernetes-update-orchestration.rest .. rubric:: |proc| #. List available upgrades. .. code-block:: none ~(keystone_admin)$ system kube-version-list +-----------------+--------+-------------+ | version | target | state | +-----------------+--------+-------------+ | v1.18.1 | True | active | | v1.19.13 | False | available | | v1.20.9 | False | unavailable | | v1.21.8 | False | unavailable | +-----------------+--------+-------------+ #. Confirm that the system is healthy. Check the current system health status, resolve any alarms and other issues reported by the :command:`system health-query-kube-upgrade` command, then recheck the system health status to confirm that all **System Health** fields are set to **OK**. By default, the upgrade process cannot be run and is not recommended to be run with active alarms present. Use the :command:`system kube-upgrade-start --force` command to force the upgrade process to start and ignore non-management-affecting alarms. .. note:: It is strongly recommended that you clear your system of any and all alarms before doing an upgrade. While the :command:`--force` option is available to run the upgrade, it is a best practice to clear any alarms. .. code-block:: none ~(keystone_admin)]$ system health-query-kube-upgrade System Health: All hosts are provisioned: [OK] All hosts are unlocked/enabled: [OK] All hosts have current configurations: [OK] All hosts are patch current: [OK] Ceph Storage Healthy: [OK] No alarms: [OK] All kubernetes nodes are ready: [OK] All kubernetes control plane pods are ready: [OK] Required patches are applied: [OK] License valid for upgrade: [OK] No instances running on controller-1: [OK] All kubernetes applications are in a valid state: [OK] Active controller is controller-0: [OK] #. Create the strategy. The *Kubernetes Version Upgrade Orchestration Strategy* :command:`create` command creates a series of stages with steps that apply the Kubernetes version upgrade. Kubernetes Version upgrade requires a reboot. Therefore, the created strategy includes steps that automatically lock and unlock the host to bring the new image function into service. .. code-block:: none ~(keystone_admin)$ sw-manager kube-upgrade-strategy create --to-version v1.19.13 Strategy Kubernetes Upgrade Strategy: strategy-uuid: f7585178-cea6-4d2f-bda0-e0972145ebcf controller-apply-type: serial storage-apply-type: ignore worker-apply-type: serial default-instance-action: migrate alarm-restrictions: strict current-phase: build current-phase-completion: 0% state: building inprogress: true where: ``--to-version`` The version of Kubernetes to upgrade to. For example, ``v1.19.13``. This argument is required. ``--controller-apply-type`` and ``--storage-apply-type`` These options cannot be changed from ``serial`` because Kubernetes upgrade concurrency is only supported for worker hosts. .. note:: Setting the Kubernetes version upgrade apply type is only supported for hosts with only the worker function. Any attempt to modify the controller or storage apply type will be rejected. ``--worker-apply-type`` This option specifies the host concurrency of the Kubernetes version upgrade strategy: - serial (default): worker hosts will be patched one at a time - parallel: worker hosts will be upgraded in parallel - At most, ``parallel`` will be upgraded at the same time - At most, half of the hosts in a host aggregate will be upgraded at the same time - ignore: worker hosts will not be upgraded; strategy create will fail Worker hosts with no instances are upgraded before worker hosts with instances. ``--max-parallel-worker-hosts`` This option applies to the parallel worker apply type selection to specify the maximum worker hosts to upgrade in parallel (minimum: 2, maximum: 10). ``–instance-action`` This option only has significance when the |prefix|-openstack application is loaded and there are instances running on worker hosts. It specifies how the strategy deals with worker host instances over the strategy execution. ``stop-start`` (default) Instances will be stopped before the host lock operation following the upgrade and then started again following the host unlock. .. warning:: Using the ``stop-start`` option will result in an outage for each instance, as it is stopped while the worker host is locked/unlocked. In order to ensure this does not impact service, instances MUST be grouped into anti-affinity (or anti-affinity best effort) server groups, which will ensure that only a single instance in each server group is stopped at a time. ``migrate`` Instances will be migrated off a host before it is patched (this applies to reboot patching only). ``--alarm-restrictions`` This option sets how the how the Kubernetes version upgrade orchestration behaves when alarms are present. To display management-affecting active alarms, use the following command: .. code-block:: none ~(keystone_admin)$ fm alarm-list --mgmt_affecting ``strict`` (default) The default strict option will result in patch orchestration failing if there are any alarms present in the system (except for a small list of alarms). ``relaxed`` This option allows orchestration to proceed if alarms are present, as long as none of these alarms are management affecting. .. code-block:: none ~(keystone_admin)]$ sw-manager kube-upgrade-strategy create --help usage:sw-manager kube-upgrade-strategy [-h] --to-version [--controller-apply-type {ignore}] [--storage-apply-type {ignore}] [--worker-apply-type {serial,parallel,ignore}] [--max-parallel-worker-hosts {2,3,4,5,6,7,8,9,10}] [--instance-action {migrate,stop-start}] [--alarm-restrictions {strict,relaxed}] optional arguments: -h, --help show this help message and exit --controller-apply-type {serial,ignore} defaults to serial --storage-apply-type {serial,ignore} defaults to serial --worker-apply-type {serial,parallel,ignore} defaults to serial --max-parallel-worker-hosts {2,3,4,5,6,7,8,9,10} maximum worker hosts to update in parallel --instance-action {migrate,stop-start} defaults to stop-start --alarm-restrictions {strict,relaxed} defaults to strict #. |optional| Display the strategy in summary, if required. The Kubernetes upgrade strategy :command:`show` command displays the strategy in a summary. .. code-block:: none ~(keystone_admin)$ sw-manager kube-upgrade-strategy show Strategy Kubernetes Upgrade Strategy: strategy-uuid: f7585178-cea6-4d2f-bda0-e0972145ebcf controller-apply-type: serial storage-apply-type: ignore worker-apply-type: serial default-instance-action: migrate alarm-restrictions: strict current-phase: build current-phase-completion: 100% state: ready-to-apply build-result: success build-reason: The :command:`show` strategy subcommand displays a summary of the current state of the strategy. A complete view of the strategy can be shown using the ``--details`` option. The strategy steps and stages are displayed using the ``--details`` option. #. Apply the strategy. *Kubernetes Version Upgrade Orchestration Strategy* :command:`apply` command executes the strategy stages and steps consecutively until the Kubernetes upgrade on all the hosts in the strategy is complete. - Use the ``-stage-id`` option to specify a specific stage to apply; one at a time. .. note:: When applying a single stage, only the next stage will be applied; you cannot skip stages. .. code-block:: none ~(keystone_admin)$ sw-manager kube-upgrade-strategy apply Strategy Kubernetes upgrade Strategy: strategy-uuid: 3e43c018-9c75-4ba8-a276-472c3bcbb268 controller-apply-type: ignore storage-apply-type: ignore worker-apply-type: serial default-instance-action: stop-start alarm-restrictions: strict current-phase: apply current-phase-completion: 0% state: applying inprogress: true - Use the :command:`kube-upgrade-show` command to monitor Kubernetes upgrade state and percentage completion. .. code-block:: none ~(keystone_admin)$ system kube-upgrade-show +--------------+--------------------------------------+ | Property | Value | +--------------+--------------------------------------+ | uuid | 3d2da123-bff4-4b3a-a64a-b320c3b498cc | | from_version | v1.18.1 | | to_version | v1.19.13 | | state | downloading-images | | created_at | 2021-02-23T00:08:24.579257+00:00 | | updated_at | 2021-02-23T00:09:35.413307+00:00 | +--------------+--------------------------------------+ You will see the ``state`` property transition through values such as ``downloading-images``, ``downloaded-images``, ``upgrading-first-master``, ``upgraded-first-master``, etc. #. |optional| Abort the strategy, if required. This is only used to stop, and abort the entire strategy. The Kubernetes version upgrade strategy :command:`abort` command can be used to abort the Kubernetes version upgrade strategy after the current step of the currently applying stage is completed. #. Confirm that the upgrade has completed successfully. .. code-block:: none ~(keystone_admin)$ system kube-upgrade-show +--------------+--------------------------------------+ | Property | Value | +--------------+--------------------------------------+ | uuid | 426d7e11-2de2-40ba-b482-ed3691625383 | | from_version | v1.18.1 | | to_version | v1.19.13 | | state | upgrade-complete | | created_at | 2021-04-12T17:58:36.492523+00:00 | | updated_at | 2021-04-12T18:49:11.673259+00:00 | +--------------+--------------------------------------+ ~(keystone_admin)$ system kube-version-list +-----------------+--------+-------------+ | version | target | state | +-----------------+--------+-------------+ | v1.18.1 | True | unavailable | | v1.19.13 | False | active | | v1.20.9 | False | available | | v1.21.8 | False | unavailable | +-----------------+--------+-------------+ #. Delete the strategy. .. note:: After the *Kubernetes Version Upgrade Orchestration Strategy* has been applied (or aborted) it must be deleted before another Kubernetes version upgrade strategy can be created. If a Kubernetes version upgrade strategy application fails, you must address the issue that caused the failure, then delete and re-create the strategy before attempting to apply it again. .. code-block:: none ~(keystone_admin)$ sw-manager kube-upgrade-strategy delete Strategy deleted.