From c782df88929d1cfc3e090439c4380971dae41894 Mon Sep 17 00:00:00 2001 From: Ron Stone Date: Thu, 29 Apr 2021 12:41:33 -0400 Subject: [PATCH] Platform k8s upgrades Initial draft based on downstream content. Implemented patchset 1 review comments. Implemented patchset 2 rewiew comments. Implemented patchset 3 rewiew comments. Implemented patchset 4 rewiew comments. Implemented patchset 5 rewiew comments. Implemented patchset 6 rewiew comments. Story: 2008055 Task: 42401 Signed-off-by: Ron Stone Change-Id: I6262018778fae44726985853ec4e01f1abf5b890 Signed-off-by: Ron Stone --- ...uring-kubernetes-update-orchestration.rest | 0 ...ernetes-update-orchestration-failures.rest | 0 doc/source/shared/abbrevs.txt | 2 + doc/source/updates/index.rst | 25 +- ...about-kubernetes-orchestrated-upgrades.rst | 51 +++ ...guring-kubernetes-update-orchestration.rst | 354 ++++++++++++++++ ...bernetes-update-orchestration-failures.rst | 125 ++++++ doc/source/updates/kubernetes/index.rst | 29 ++ .../manual-kubernetes-components-upgrade.rst | 381 ++++++++++++++++++ ...ubernetes-update-orchestration-process.rst | 182 +++++++++ 10 files changed, 1144 insertions(+), 5 deletions(-) create mode 100644 doc/source/_includes/configuring-kubernetes-update-orchestration.rest create mode 100644 doc/source/_includes/handling-kubernetes-update-orchestration-failures.rest create mode 100644 doc/source/updates/kubernetes/about-kubernetes-orchestrated-upgrades.rst create mode 100644 doc/source/updates/kubernetes/configuring-kubernetes-update-orchestration.rst create mode 100644 doc/source/updates/kubernetes/handling-kubernetes-update-orchestration-failures.rst create mode 100644 doc/source/updates/kubernetes/index.rst create mode 100644 doc/source/updates/kubernetes/manual-kubernetes-components-upgrade.rst create mode 100644 doc/source/updates/kubernetes/the-kubernetes-update-orchestration-process.rst diff --git a/doc/source/_includes/configuring-kubernetes-update-orchestration.rest b/doc/source/_includes/configuring-kubernetes-update-orchestration.rest new file mode 100644 index 000000000..e69de29bb diff --git a/doc/source/_includes/handling-kubernetes-update-orchestration-failures.rest b/doc/source/_includes/handling-kubernetes-update-orchestration-failures.rest new file mode 100644 index 000000000..e69de29bb diff --git a/doc/source/shared/abbrevs.txt b/doc/source/shared/abbrevs.txt index 16064a867..17a1c04be 100755 --- a/doc/source/shared/abbrevs.txt +++ b/doc/source/shared/abbrevs.txt @@ -7,6 +7,8 @@ .. |ACL| replace:: :abbr:`ACL (Access Control List)` .. |AE| replace:: :abbr:`AE (Aggregated Ethernet)` .. |AIO| replace:: :abbr:`AIO (All-In-One)` +.. |AIO-DX| replace:: :abbr:`AIO-DX (All-In-One Duplex)` +.. |AIO-SX| replace:: :abbr:`AIO-SX (All-In-One Simplex)` .. |AVP| replace:: :abbr:`AVP (Accelerated Virtual Port)` .. |AVPs| replace:: :abbr:`AVPs (Accelerated Virtual Ports)` .. |AWS| replace:: :abbr:`AWS (Amazon Web Services)` diff --git a/doc/source/updates/index.rst b/doc/source/updates/index.rst index 1eb643ec9..9d3882d25 100644 --- a/doc/source/updates/index.rst +++ b/doc/source/updates/index.rst @@ -1,11 +1,26 @@ -======= -Updates -======= +==================== +Updates and Upgrades +==================== + ---------- -Openstack +Kubernetes ---------- -.. check what put here +Kubernetes version upgrade cloud orchestration allows the Kubernetes version on +all hosts of an entire |prod-long| cloud to be updated with a single operation. + +.. toctree:: + :maxdepth: 2 + + kubernetes/index + +--------- +Openstack +--------- + +The system application-update -n |prefix|-openstack -v +command is used for corrective content \(bug fixes\) -type updates to the +running containerized openstack application. .. toctree:: :maxdepth: 2 diff --git a/doc/source/updates/kubernetes/about-kubernetes-orchestrated-upgrades.rst b/doc/source/updates/kubernetes/about-kubernetes-orchestrated-upgrades.rst new file mode 100644 index 000000000..857c4539a --- /dev/null +++ b/doc/source/updates/kubernetes/about-kubernetes-orchestrated-upgrades.rst @@ -0,0 +1,51 @@ + +.. xkr1590157116928 +.. _about-kubernetes-orchestrated-upgrades: + +==================================================== +About Kubernetes Version Upgrade Cloud Orchestration +==================================================== + +Kubernetes version upgrade cloud orchestration allows the Kubernetes version on +all hosts of an entire |prod-long| cloud to be updated with a single operation. + +You can configure and run Kubernetes version upgrade cloud orchestration using +the CLI, or the stx-nfv VIM REST API. + + +.. _xkr1590157116928-section-phk-xls-tlb: + +----------------------------------------------------------- +Kubernetes Version Upgrade Cloud Orchestration Requirements +----------------------------------------------------------- + +Kubernetes version upgrade orchestration can only be done on a system that +meets the following conditions: + + +.. _xkr1590157116928-ul-frz-yls-tlb: + +- The system is clear of alarms \(with the exception of alarms for locked + hosts, stopped instances, and Kubernetes version upgrade cloud + orchestration in progress\). + + .. note:: + When configuring Kubernetes version upgrade cloud orchestration, you + have the option to ignore alarms that are not management-affecting + severity. For more information, see :ref:`Kubernetes Version Upgrade + Cloud Orchestration `. + +- There are unlocked-enabled worker function hosts in the system that + requires Kubernetes Version Upgrade Cloud Orchestration. The *Kubernetes + Upgrade Orchestration Strategy* creation step will fail if there are no + qualified hosts detected. + + A Kubernetes version upgrade is a reboot required operation. Therefore, in + systems that have the |prefix|-openstack application applied with running + instances, if the migrate option is selected there must be spare + openstack-compute \(worker\) capacity to move instances off the + openstack-compute \(worker\) host\(s\) being upgraded. + + .. note:: + Administrative controller ``swact`` operations should be avoided during + Kubernetes version upgrade orchestration. diff --git a/doc/source/updates/kubernetes/configuring-kubernetes-update-orchestration.rst b/doc/source/updates/kubernetes/configuring-kubernetes-update-orchestration.rst new file mode 100644 index 000000000..a6d666315 --- /dev/null +++ b/doc/source/updates/kubernetes/configuring-kubernetes-update-orchestration.rst @@ -0,0 +1,354 @@ + +.. noc1590162360081 +.. _configuring-kubernetes-update-orchestration: + +============================================== +Kubernetes Version Upgrade Cloud Orchestration +============================================== + +You can configure *Kubernetes Version Upgrade Orchestration Strategy* using the +:command:`sw-manager` CLI. + +.. note:: + You require administrator privileges to use :command:`sw-manager`. You must + log in to the active controller as **user sysadmin** and source the script + by using the command, source /etc/platform/openrc to obtain administrator + privileges. Do not use :command:`sudo`. + +.. note:: + Management-affecting alarms cannot be ignored using relaxed alarm rules + during an orchestrated Kubernetes version upgrade operation. For a list of + management-affecting alarms, see |fault-doc|: :ref:`Alarm Messages + <100-series-alarm-messages>`. To display management-affecting active + alarms, use the following command: + + .. code-block:: none + + ~(keystone_admin)$ fm alarm-list --mgmt_affecting + +During an orchestrated Kubernetes version upgrade operation, the following +alarms are ignored even when the default strict restrictions are selected: + + +.. _noc1590162360081-ul-vhg-jxs-tlb: + +- 100.103: Memory threshold exceeded. + +- 200.001: Locked host. + +- 280.001: Subcloud resource off-line. + +- 280.002: Subcloud resource out-of-sync. + +- 700.004: |VM| stopped. + +- 750.006: Configuration change requires reapply of cert-manager. + +- 900.001: Patch in progress. + +- 900.007: Kube upgrade in progress. + +- 900.401: kube-upgrade-auto-apply-inprogress. + + +You can use ``help`` for the overall commands and also for each sub-command. +For example: + +.. code-block:: none + + ~(keystone_admin)$ sw-manager kube-upgrade-strategy –help + usage: sw-manager kube-upgrade-strategy [-h] ... + optional arguments: + -h, --help show this help message and exit + Kubernetes Update Commands: + create Create a strategy + delete Delete a strategy + apply Apply a strategy + abort Abort a strategy + show Show a strategy + +.. rubric:: |prereq| + + +.. _noc1590162360081-ul-ls2-pxs-tlb: + +- Hosts that need to be upgraded must be in the ``unlocked-enabled`` state. + +.. only:: partner + + .. include:: ../../_includes/configuring-kubernetes-update-orchestration.rest + + +.. rubric:: |proc| + +#. List available upgrades. + + .. code-block: none + + ~(keystone_admin)$ system kube-version-list + +-----------------+--------+-----------+ + | version | target | state | + +-----------------+--------+-----------+ + | v1.18.1 | True | active | + | v1.18.1-upgrade | False | available | + +-----------------+--------+-----------+ + + +#. Create the strategy. + + The *Kubernetes Version Upgrade Orchestration Strategy* :command:`create` + command creates a series of stages with steps that apply the Kubernetes + version upgrade. + + Kubernetes Version upgrade requires a reboot. Therefore, the created strategy + includes steps that automatically lock and unlock the host to bring the new + image function into service. + + .. code-block:: none + + ~(keystone_admin)$ sw-manager kube-upgrade-strategy create --to-version v1.18.1-upgrade + Strategy Kubernetes Upgrade Strategy: + strategy-uuid: f7585178-cea6-4d2f-bda0-e0972145ebcf + controller-apply-type: serial + storage-apply-type: ignore + worker-apply-type: serial + default-instance-action: migrate + alarm-restrictions: strict + current-phase: build + current-phase-completion: 0% + state: building + inprogress: true + + where: + + ``--to-version`` + The version of Kubernetes to upgrade to. For example, + ``v1.18.1-upgrade``. This argument is required. + + ``--controller-apply-type`` and ``--storage-apply-type`` + These options cannot be changed from ``serial`` because Kubernetes + upgrade concurrency is only supported for worker hosts. + + .. note:: + Kubernetes version upgrade is currently only supported for hosts with + worker function. Any attempt to modify the controller or storage + apply type is rejected. + + ``--worker-apply-type`` + This option specifies the host concurrency of the Kubernetes version + upgrade strategy: + + - serial \(default\): worker hosts will be patched one at a time + + - parallel: worker hosts will be upgraded in parallel + + - At most, ``parallel`` will be upgraded at the same time + + - At most, half of the hosts in a host aggregate will be upgraded + at the same time + + - ignore: worker hosts will not be upgraded; strategy create will fail + + Worker hosts with no instances are upgraded before worker hosts with + instances. + + ``--max-parallel-worker-hosts`` + This option applies to the parallel worker apply type selection to + specify the maximum worker hosts to upgrade in parallel \(minimum: 2, + maximum: 10\). + + ``–instance-action`` + This option only has significance when the |prefix|-openstack + application is loaded and there are instances running on worker hosts. + It specifies how the strategy deals with worker host instances over the + strategy execution. + + ``stop-start`` \(default\) + Instances will be stopped before the host lock operation following the + upgrade and then started again following the host unlock. + + .. warning:: + Using the ``stop-start`` option will result in an outage for each + instance, as it is stopped while the worker host is locked/unlocked. + In order to ensure this does not impact service, instances MUST be + grouped into anti-affinity \(or anti-affinity best effort\) server + groups, which will ensure that only a single instance in each server + group is stopped at a time. + + ``migrate`` + Instances will be migrated off a host before it is patched \(this + applies to reboot patching only\). + + ``--alarm-restrictions`` + This option sets how the how the Kubernetes version upgrade + orchestration behaves when alarms are present. + + To display management-affecting active alarms, use the following + command: + + .. code-block:: none + + ~(keystone_admin)$ fm alarm-list --mgmt_affecting + + ``strict`` \(default\) + The default strict option will result in patch orchestration failing if + there are any alarms present in the system \(except for a small list of + alarms\). + + ``relaxed`` + This option allows orchestration to proceed if alarms are present, as + long as none of these alarms are management affecting. + + .. code-block:: none + + ~(keystone_admin)]$ sw-manager kube-upgrade-strategy create --help + usage:sw-manager kube-upgrade-strategy [-h] + --to-version + [--controller-apply-type {ignore}] + [--storage-apply-type {ignore}] + [--worker-apply-type + {serial,parallel,ignore}] + [--max-parallel-worker-hosts + {2,3,4,5,6,7,8,9,10}] + [--instance-action {migrate,stop-start}] + [--alarm-restrictions {strict,relaxed}] + + optional arguments: + -h, --help show this help message and exit + --controller-apply-type {serial,ignore} + defaults to serial + --storage-apply-type {serial,ignore} + defaults to serial + --worker-apply-type {serial,parallel,ignore} + defaults to serial + --max-parallel-worker-hosts {2,3,4,5,6,7,8,9,10} + maximum worker hosts to update in parallel + --instance-action {migrate,stop-start} + defaults to stop-start + --alarm-restrictions {strict,relaxed} + defaults to strict + + +#. Optional: Display the strategy in summary, if required. The Kubernetes + upgrade strategy :command:`show` command displays the strategy in a summary. + + .. code-block:: none + + ~(keystone_admin)$ sw-manager kube-upgrade-strategy show + Strategy Kubernetes Upgrade Strategy: + strategy-uuid: f7585178-cea6-4d2f-bda0-e0972145ebcf + controller-apply-type: serial + storage-apply-type: ignore + worker-apply-type: serial + default-instance-action: migrate + alarm-restrictions: strict + current-phase: build + current-phase-completion: 100% + state: ready-to-apply + build-result: success + build-reason: + + The :command:`show` strategy subcommand displays a summary of the current + state of the strategy. A complete view of the strategy can be shown using + the ``--details`` option. + + The strategy steps and stages are displayed using the ``--details`` option. + +#. Apply the strategy. + + *Kubernetes Version Upgrade Orchestration Strategy* :command:`apply` command + executes the strategy stages and steps consecutively until the Kubernetes + upgrade on all the hosts in the strategy is complete. + + + - Use the ``-stage-id`` option to specify a specific stage to apply; one + at a time. + + .. note:: + When applying a single stage, only the next stage will be applied; + you cannot skip stages. + + + .. code-block:: none + + ~(keystone_admin)$ sw-manager kube-upgrade-strategy apply + Strategy Kubernetes upgrade Strategy: + strategy-uuid: 3e43c018-9c75-4ba8-a276-472c3bcbb268 + controller-apply-type: ignore + storage-apply-type: ignore + worker-apply-type: serial + default-instance-action: stop-start + alarm-restrictions: strict + current-phase: apply + current-phase-completion: 0% + state: applying + inprogress: true + + + - Use the :command:`kube-upgrade-show` command to monitor Kubernetes + upgrade state and percentage completion. + + + .. code-block:: none + + ~(keystone_admin)$ system kube-upgrade-show + +--------------+--------------------------------------+ + | Property | Value | + +--------------+--------------------------------------+ + | uuid | 3d2da123-bff4-4b3a-a64a-b320c3b498cc | + | from_version | v1.18.1 | + | to_version | v1.18.1-upgrade | + | state | downloading-images | + | created_at | 2021-02-23T00:08:24.579257+00:00 | + | updated_at | 2021-02-23T00:09:35.413307+00:00 | + +--------------+--------------------------------------+ + + You will see the ``state`` property transition through values such as + ``downloading-images``, ``downloaded-images``, ``upgrading-first-master``, + ``upgraded-first-master``, etc. + +#. Optional: Abort the strategy, if required. This is only used to stop, and + abort the entire strategy. + + The Kubernetes version upgrade strategy :command:`abort` command can be + used to abort the Kubernetes version upgrade strategy after the current + step of the currently applying stage is completed. + +#. Confirm that the upgrade has completed successfully. + + .. code-block:: none + + ~(keystone_admin)$ system kube-upgrade-show + +--------------+--------------------------------------+ + | Property | Value | + +--------------+--------------------------------------+ + | uuid | 426d7e11-2de2-40ba-b482-ed3691625383 | + | from_version | v1.18.1 | + | to_version | v1.18.1-upgrade | + | state | upgrade-complete | + | created_at | 2021-04-12T17:58:36.492523+00:00 | + | updated_at | 2021-04-12T18:49:11.673259+00:00 | + +--------------+--------------------------------------+ + + ~(keystone_admin)$ system kube-version-list + +-----------------+--------+-----------+ + | version | target | state | + +-----------------+--------+-----------+ + | v1.18.1 | False | available | + | v1.18.1-upgrade | True | active | + +-----------------+--------+-----------+ + +#. Delete the strategy. + + .. note:: + After the *Kubernetes Version Upgrade Orchestration Strategy* has been + applied \(or aborted\) it must be deleted before another Kubernetes + version upgrade strategy can be created. If a Kubernetes version + upgrade strategy application fails, you must address the issue that + caused the failure, then delete and re-create the strategy before + attempting to apply it again. + + .. code-block:: none + + ~(keystone_admin)$ sw-manager kube-upgrade-strategy delete + Strategy deleted. diff --git a/doc/source/updates/kubernetes/handling-kubernetes-update-orchestration-failures.rst b/doc/source/updates/kubernetes/handling-kubernetes-update-orchestration-failures.rst new file mode 100644 index 000000000..05f28fdf5 --- /dev/null +++ b/doc/source/updates/kubernetes/handling-kubernetes-update-orchestration-failures.rst @@ -0,0 +1,125 @@ + +.. jkf1590184623714 +.. _handling-kubernetes-update-orchestration-failures: + +======================================================== +Handle Kubernetes Version Upgrade Orchestration Failures +======================================================== + +The creation or application of a strategy could fail for any of the listed +reasons described in this section. Follow the suggested actions in each case to +resolve the issue. + +.. _jkf1590184623714-section-fhk-nnq-5lb: + +------------------------- +Strategy creation failure +------------------------- + +.. _jkf1590184623714-ul-fvs-vnq-5lb: + +- **Reason**: build failed with no reason. + + - **Action**: + + - Verify that the ``--worker-apply-type`` was not set to ``ignore``. + + - Check recent logs added to /var/log/nfv-vim.log. + + +- **Reason**: alarms from platform are present. + + - **Action**: + + - Query for management affecting alarms and take actions to clear + them. + + .. code-block:: none + + ~(keystone_admin)$ fm alarm-list --mgmt_affecting + + - If there are no management affecting alarms present, take actions + to clear other reported alarms or try creating the strategy with + the ``relaxed`` alarms restrictions option ``--alarm-restrictions + relaxed``. + +- **Reason**: no Kubernetes version upgrade required. + + - **Action**: + + - Verify that the Kubernetes patches have been uploaded and applied. + Verify the version of Kubernetes on the hosts by executing "system + kube-host-upgrade-list. + + .. note:: + If the strategy create failed, first you must resolve it. You + must delete the failed strategy before you create another + strategy. + + +.. _jkf1590184623714-section-ppt-gpq-5lb: + +---------------------- +Strategy Apply Failure +---------------------- + +.. _jkf1590184623714-ul-rdf-4pq-5lb: + +- **Reason**: alarms from platform are present. + + - **Action**: suggests that an alarm has been raised since the creation + of the strategy. Address the cause of the new alarm, delete the + strategy and try creating and applying a new strategy. + + +- **Reason**: unable to migrate instances. + + - **Action**: See :ref:`Kubernetes Version Upgrade Operations Requiring + Manual Migration + ` for steps to + resolve migration issues. + +- **Reason**: Kubernetes version upgrade failed. Suggests that the Kubernetes + upgrade for the specified host has failed. + + .. only:: starlingx + + - **Action**: Consult the `StarlingX community + `_. + + .. only:: partner + + .. include:: ../../_includes/handling-kubernetes-update-orchestration-failures.rest + +- **Reason**: lock host failed. + + - **Action**: + + - Investigate the /var/log/sysinv.log, and /var/log/nfv-vim.log + files. + + - Address the underlying issue. + + - Manually lock and unlock the host. + + - Try recreating and re-applying the Kubernetes version upgrade + strategy to automatically finish the upgrade process. + + +- **Reason**: unlock host failed. + + - **Action**: + + - Investigate /var/log/mtcAgent.log file for cause logs files. + + - Address the underlying issue. + + - Manually lock and unlock the host to recover. + + - Try recreating and re-applying the Kubernetes version upgrade + strategy to automatically finish the upgrade process. + +.. note:: + If the strategy :command:`apply` fails, you must resolve the + strategy:command:`apply` failure, and delete the failed strategy before + trying to create and apply another strategy. diff --git a/doc/source/updates/kubernetes/index.rst b/doc/source/updates/kubernetes/index.rst new file mode 100644 index 000000000..94af1d888 --- /dev/null +++ b/doc/source/updates/kubernetes/index.rst @@ -0,0 +1,29 @@ + +========== +Kubernetes +========== + +--------------------------------- +Manual Kubernetes Version Upgrade +--------------------------------- + +.. toctree:: + :maxdepth: 1 + :caption: Contents: + + manual-kubernetes-components-upgrade + + +---------------------------------------------- +Kubernetes Version Upgrade Cloud Orchestration +---------------------------------------------- + +.. toctree:: + :maxdepth: 1 + :caption: Contents: + + about-kubernetes-orchestrated-upgrades + the-kubernetes-update-orchestration-process + configuring-kubernetes-update-orchestration + handling-kubernetes-update-orchestration-failures + diff --git a/doc/source/updates/kubernetes/manual-kubernetes-components-upgrade.rst b/doc/source/updates/kubernetes/manual-kubernetes-components-upgrade.rst new file mode 100644 index 000000000..ad21c5969 --- /dev/null +++ b/doc/source/updates/kubernetes/manual-kubernetes-components-upgrade.rst @@ -0,0 +1,381 @@ + +.. bfd1591638638205 +.. _manual-kubernetes-components-upgrade: + +=========================== +Kubernetes Version Upgrade +=========================== + +You can upgrade the Kubernetes version on a running system from one +supported version to another. + +.. rubric:: |context| + +To complete this task, you will apply the following three updates \(patches\) +and upgrade various systems. + +**Platform update** + The platform update contains metadata for the new Kubernetes version and the + Kubernetes networking pods templates for the new Kubernetes version. + +**Kubeadm update** + The kubeadm update upgrades the kubeadm RPM to the new Kubernetes version. + +**Kubelet and Kubectl update** + This Kubernetes update upgrades kubelet and kubectl RPMs to the new + Kubernetes version. + + +.. rubric:: |prereq| + + +.. _manual-kubernetes-components-upgrade-ul-jbr-vcn-ylb: + +- The system must be clear of alarms. + +- All hosts must be unlocked, enabled and available. + +- All Kubernetes pods must be ready. + +- The installed applications must be compatible with the new Kubernetes + version. + +- All updates required for the new Kubernetes version must be transferred to + the active controller. + + +.. rubric:: |proc| + +#. Upload, apply and install the platform update. + + Use the standard :command:`sw-patch`, :command:`upload`, :command:`apply` + and :command:`install` commands to perform these operations. + +#. List the available Kubernetes versions. + + .. code-block:: none + + ~(keystone_admin)]$ system kube-version-list + +---------+--------+-----------+ + | Version | Target | State | + +---------+--------+-----------+ + | v1.16.1 | True | active | + | v1.16.2 | False | available | + +---------+--------+-----------+ + + The following meanings apply to the output shown: + + **Target** + A value of True means that the target is currently selected for + installation. + + **State** + Can be one of: + + **active** + The version is running everywhere. + + **partial** + The version is running somewhere. + + **available** + The version can be upgraded to. + +#. Upload, apply and install the kubeadm update. + + Use the standard :command:`sw-patch`, :command:`upload`, :command:`apply` + and :command:`install` commands to perform these operations. + +#. Upload the kubelet update. + + .. note:: + Run the :command:`upload` command only: + + .. code-block:: none + + ~(keystone_admin)]$ sw-patch upload + + + The kubelet update cannot be applied before upgrading kubelet. + +#. Start the Kubernetes upgrade. + + For example: + + .. code-block:: none + + ~(keystone_admin)]$ system kube-upgrade-start v1.16.2 + +-------------------+-------------------+ + | Property | Value | + +-------------------+-------------------+ + | from_version | v1.16.1 | + | to_version | v1.16.2 | + | state | upgrade-started | + +-------------------+-------------------+ + + The upgrade process checks the applied/available updates, the upgrade path, + the health of the system, the installed applications compatibility and + validates the system is ready for an upgrade. + + .. warning:: + If you use the command :command:`system kube-upgrade-start --force` to + cause the upgrades process to ignore management affecting alarms and + start, first determine that these alarms will not compromise the + upgrade process. + +#. Download the Kubernetes images. + + For example: + + .. code-block:: none + + ~(keystone_admin)]$ system kube-upgrade-download-images + +--------------+--------------------------------------+ + | Property | Value | + +--------------+--------------------------------------+ + | uuid | b5f7dada-2537-4416-9d2c-f9ca9fcd0e22 | + | from_version | v1.16.1 | + | to_version | v1.16.2 | + | state | downloading-images | + | created_at | 2020-02-20T16:08:48.854869+00:00 | + | updated_at | None | + +--------------+--------------------------------------+ + +#. Confirm that the download has completed. + + .. code-block:: none + + ~(keystone_admin)]$ system-kube-upgrade-show + +--------------+--------------------------------------+ + | Property | Value | + +--------------+--------------------------------------+ + | uuid | b5f7dada-2537-4416-9d2c-f9ca9fcd0e22 | + | from_version | v1.16.1 | + | to_version | v1.16.2 | + | state | downloaded-images | + | created_at | 2020-02-20T16:08:48.854869+00:00 | + | updated_at | 2020-02-20T16:10:37.858661+00:00 | + +--------------+--------------------------------------+ + + +#. Upgrade the control plane on the first controller. + + .. code-block:: none + + ~(keystone_admin)]$ system kube-host-upgrade controller-1 control-plane + +-----------------------+-------------------------+ + | Property | Value | + +-----------------------+-------------------------+ + | control_plane_version | v1.16.1 | + | hostname | controller-1 | + | id | 2 | + | kubelet_version | v1.16.1 | + | personality | controller | + | status | upgrading-control-plane | + | target_version | v1.16.2 | + +-----------------------+-------------------------+ + + + You can upgrade either controller first. + + The state **upgraded-first-master** will be entered when the first control + plane upgrade has completed. + +#. Upgrade Kubernetes networking. + + This step must be completed after the first control plane has been upgraded + and before upgrading the second control plane. + + .. code-block:: none + + ~(keystone_admin)]$ system kube-upgrade-networking + +--------------+--------------------------------------+ + | Property | Value | + +--------------+--------------------------------------+ + | uuid | b5f7dada-2537-4416-9d2c-f9ca9fcd0e22 | + | from_version | v1.16.1 | + | to_version | v1.16.2 | + | state | upgrading-networking | + | created_at | 2020-02-20T16:08:48.854869+00:00 | + | updated_at | 2020-02-20T16:18:11.459736+00:00 | + +--------------+--------------------------------------+ + + The state **upgraded-networking** will be entered when the networking + upgrade has completed. + +#. Upgrade the control plane on the second controller. + + .. code-block:: none + + ~(keystone_admin)]$ system kube-host-upgrade controller-0 control-plane + +-----------------------+-------------------------+ + | Property | Value | + +-----------------------+-------------------------+ + | control_plane_version | v1.16.1 | + | hostname | controller-0 | + | id | 1 | + | kubelet_version | v1.16.1 | + | personality | controller | + | status | upgrading-control-plane | + | target_version | v1.16.2 | + +-----------------------+-------------------------+ + + The state **upgraded-second-master** will be entered when the upgrade has + completed. + +#. Show the Kubernetes upgrade status for all hosts. + + .. code-block:: none + + ~(keystone_admin)]$ system kube-host-upgrade-list + +----+--------------+-------------+----------------+-----------------------+-----------------+--------+ + | id | hostname | personality | target_version | control_plane_version | kubelet_version | status | + +----+--------------+-------------+----------------+-----------------------+-----------------+--------+ + | 1 | controller-0 | controller | v1.16.2 | v1.16.2 | v1.16.1 | None | + | 2 | controller-1 | controller | v1.16.2 | v1.16.2 | v1.16.1 | None | + | 3 | storage-0 | storage | v1.16.1 | N/A | N/A | None | + | 4 | storage-1 | storage | v1.16.1 | N/A | N/A | None | + | 5 | worker-0 | worker | v1.16.1 | N/A | v1.16.1 | None | + | 6 | worker- 1 | worker | v1.16.1 | N/A | v1.16.1 | None | + +----+--------------+-------------+----------------+-----------------------+-----------------+--------+ + + The control planes of both controllers are now upgraded to v1.16.2. + +#. Apply and install the kubectl update. + + Use the standard :command:`sw-patch`, :command:`apply` and + :command:`install` commands to perform these operations. + + This places the new version of kubelet binary on each host, but will not + restart kubelet. + +#. Upgrade kubelet on both controllers. + + Either controller can be upgraded first. + + The kubelets on all controller hosts must be upgraded before upgrading + kubelets on worker hosts. + + For each controller, do the following. + + + #. For non |AIO-SX| systems, lock the controller. + + For example: + + .. code-block:: none + + ~(keystone_admin)]$ system host-lock controller-1 + + .. note:: + For All-In-One Simplex systems, the controller must **not** be + locked. + + #. Apply the upgrade. + + For example: + + .. code-block:: none + + ~(keystone_admin)]$ system kube-host-upgrade controller-1 kubelet + +-----------------------+-------------------+ + | Property | Value | + +-----------------------+-------------------+ + | control_plane_version | v1.16.2 | + | hostname | controller-1 | + | id | 2 | + | kubelet_version | v1.16.1 | + | personality | controller | + | status | upgrading-kubelet | + | target_version | v1.16.2 | + +-----------------------+-------------------+ + + #. For non |AIO-SX| systems, unlock the controller. + + For example: + + .. code-block:: none + + ~(keystone_admin)]$ system host-unlock controller-1 + + +#. Show the Kubernetes upgrade status. + + .. code-block:: none + + ~(keystone_admin)]$ system kube-upgrade-show + +--------------+--------------------------------------+ + | Property | Value | + +--------------+--------------------------------------+ + | uuid | b5f7dada-2537-4416-9d2c-f9ca9fcd0e22 | + | from_version | v1.16.1 | + | to_version | v1.16.2 | + | state | upgrading-kubelets | + | created_at | 2020-02-20T16:08:48.854869+00:00 | + | updated_at | 2020-02-20T21:53:16.347406+00:00 | + +--------------+--------------------------------------+ + +#. Upgrade kubelet on all worker hosts. + + Multiple worker hosts can be upgraded simultaneously provided there is + sufficient capacity remaining on other worker hosts. + + For each worker host, do the following: + + + #. Lock the host. + + For example: + + .. code-block:: none + + ~(keystone_admin)]$ system host-lock worker-1 + + #. Perform the upgrade. + + For example: + + .. code-block:: none + + ~(keystone_admin)]$ system kube-host-upgrade worker-1 kubelet + +-----------------------+-------------------+ + | Property | Value | + +-----------------------+-------------------+ + | control_plane_version | v1.16.2 | + | hostname | worker-1 | + | id | 3 | + | kubelet_version | v1.16.1 | + | personality | worker | + | status | upgrading-kubelet | + | target_version | v1.16.2 | + +-----------------------+-------------------+ + + #. Unlock the host. + + For example: + + .. code-block:: none + + ~(keystone_admin)]$ system host-unlock worker-1 + + +#. Complete the Kubernetes upgrade. + + .. code-block:: none + + ~(keystone_admin)]$ system kube-upgrade-complete + +--------------+--------------------------------------+ + | Property | Value | + +--------------+--------------------------------------+ + | uuid | 4e942297-465e-47d4-9e1b-9fb1630be33c | + | from_version | v1.16.1 | + | to_version | v1.16.2 | + | state | upgrade-complete | + | created_at | 2020-02-19T20:59:51.079966+00:00 | + | updated_at | 2020-02-24T15:03:34.572199+00:00 | + +--------------+--------------------------------------+ + +.. from step 1 +.. For more + information, see the :ref:`Managing Software Updates + `. diff --git a/doc/source/updates/kubernetes/the-kubernetes-update-orchestration-process.rst b/doc/source/updates/kubernetes/the-kubernetes-update-orchestration-process.rst new file mode 100644 index 000000000..9b904842b --- /dev/null +++ b/doc/source/updates/kubernetes/the-kubernetes-update-orchestration-process.rst @@ -0,0 +1,182 @@ + +.. htb1590431033292 +.. _the-kubernetes-update-orchestration-process: + +======================================================= +Kubernetes Version Upgrade Cloud Orchestration Overview +======================================================= + +For an orchestrated Kubernetes version upgrade you need to first create a +*Kubernetes Upgrade Orchestration Strategy*, or plan for the automated +Kubernetes version upgrade procedure. + +You can customize the Kubernetes version upgrade orchestration by specifying +the following parameters: + + +.. _htb1590431033292-ul-pdh-5ms-tlb: + +- The host types to be upgraded; only **worker-apply-type** is supported. + +- The alarm restriction mode; **strict** or **relaxed** where the **relaxed** + mode allows the strategy to be created while there are alarms present that + are not management-affecting. + +- The concurrency of the upgrade process; whether to Kubernetes version + upgrade hosts **serially** or in **parallel**. + +- The maximum number of worker hosts that can be upgraded together when + **parallel** mode is selected. + + +For hosts that have the |prefix|-openstack application running with active +instances and since the Kubernetes version upgrade is a reboot required +operation for a host, the strategy offers **stop/start** or **migrate** options +for managing instances over the **lock/unlock** \(reboot\) steps in the upgrade +process. + +You must use the **sw-manager** CLI tool to **create**, and then **apply** the +upgrade strategy. A created strategy can be monitored with the **show** +command. + +Kubernetes version upgrade orchestration automatically iterates through all +**unlocked-enabled** hosts on the system looking for hosts with the worker +function that need Kubernetes version upgrade and then proceeds to upgrade them +on the strategy :command:`apply` action. + +.. note:: + Controllers (including |AIO| controllers) are upgraded before worker only + hosts. Storage hosts do not run Kubernetes so Kubernetes is not upgraded + on them, although they still may be patched. + +After creating the *Kubernetes Version Upgrade Orchestration Strategy*, you can +either apply the entire strategy automatically, or manually apply individual +stages to control and monitor the Kubernetes version upgrade progress one stage +at a time. + +When the Kubernetes version upgrade strategy is **applied**, if the system is +All-in-one, the controllers are upgraded first, one after the other with a +swact in between, followed by the remaining worker hosts according to the +selected worker apply concurrency \(**serial** or **parallel**\) method. + +The strategy creation default is to upgrade the worker hosts serially unless +the **parallel** worker apply type option is specified which configures the +Kubernetes version upgrade process for worker hosts to be in parallel \(up to a +maximum parallel number\) to reduce the overall Kubernetes version upgrade +installation time. + +The upgrade takes place in two phases. The first phase upgrades the patches +(controllers, storage and then workers), and the second phase upgrades +Kubernetes based on those patches (controllers, then hosts). + +.. _htb1590431033292-ol-a1b-v5s-tlb: + +#. Alarm query. This step is an upgrade pre-check. + +#. Initiate the Kubernetes version upgrade. + +#. Download Kubernetes Images. + +#. Upgrade the first Control Plane. + +#. Upgrade Kubernetes networking. + +#. Upgrade the second control plane (on duplex environments only). + +#. Patch the hosts. + + #. Patch controller nodes if they are not |AIO|. + + #. Patch storage nodes. + + #. Patch worker nodes (this includes |AIO| controllers). + +#. Upgrade Kubelets on the hosts (controllers then workers. If controllers + are |AIO| they are processed before regular workers). + + #. Swact if the host is the active controller. + + #. Lock the host. + + #. Upgrade kubelet. + + #. Unlock the host. + + #. Restore |VMs| (if applicable). + +#. Complete the Kubernetes version upgrade. + +Systems with |prefix|-openstack application enabled could include additional +instance management steps. For more information, see :ref:`Kubernetes Version +Upgrade Operations Requiring Manual Migration +`. + +On systems with |prefix|-openstack application, the *Kubernetes Version Upgrade +Orchestration Strategy* considers any configured server groups and host +aggregates when creating the stages to reduce the impact to running instances. +The *Kubernetes Version Upgrade Orchestration Strategy* automatically manages +the instances during the strategy application process. The instance management +options include **start-stop** or **migrate**. + + +.. _htb1590431033292-ul-vcp-dvs-tlb: + +- **start-stop**: where instances are stopped following the actual Kubernetes + upgrade but before the lock operation and then automatically started again + after the unlock completes. This is typically used for instances that do + not support migration or for cases where migration takes too long. To + ensure this does not impact the high-level service being provided by the + instance, the instance\(s\) should be protected and grouped into an + anti-affinity server group\(s\) with its standby instance. + +- **migrate**: where instances are moved off a host following the Kubernetes + upgrade but before the host is locked. Instances with **Live Migration** + support are **Live Migrated**. Otherwise, they are **Cold Migrated**. + + +.. _kubernetes-update-operations-requiring-manual-migration: + +---------------------------------------- +Manually Migrating Application Instances +---------------------------------------- + +On systems with the |prefix|-openstack application there may be some instances +that cannot be migrated automatically by orchestration. In these cases the +instance migration must be managed manually. + +Do the following to manage the instance re-location manually: + + +.. _rbp1590431075472-ul-mgr-kvs-tlb: + +- Manually perform Kubernetes version upgrade at least one openstack-compute worker host. This + assumes that at least one openstack-compute worker host does not have any + instances, or has instances that can be migrated. For more information on + manually updating a host, see :ref:`Kubernetes Version Upgrade + `. + +- If the migration is prevented by limitations in the VNF or virtual + application, perform the following: + + + - Create new instances on an already upgraded openstack-compute worker + host. + + - Manually migrate the data from the old instances to the new instances. + + .. note:: + This is specific to your environment and depends on the virtual + application running in the instance. + + - Terminate the old instances. + + +- If the migration is prevented by the size of the instances local disks: + + + - For each openstack-compute worker host that has instances that cannot + be migrated, manually move the instances using the CLI. + +Once all openstack-compute worker hosts containing instances that cannot be +migrated have been Kubernetes version upgraded, Kubernetes version upgrade +orchestration can then be used to upgrade the remaining worker hosts.