bbe8e4fc8d
Change-Id: I851c58ca9e9dd9ec3c7653a1c5646e851df74aee Signed-off-by: Keane Lim <keane.lim@windriver.com>
137 lines
4.8 KiB
ReStructuredText
137 lines
4.8 KiB
ReStructuredText
|
|
.. bla1593031188931
|
|
.. _orchestration-upgrade-overview:
|
|
|
|
==============================
|
|
Upgrade Orchestration Overview
|
|
==============================
|
|
|
|
Upgrade Orchestration automates much of the upgrade procedure, leaving a few
|
|
manual steps for operator oversight.
|
|
|
|
.. contents:: |minitoc|
|
|
:local:
|
|
:depth: 1
|
|
|
|
.. note::
|
|
Upgrading of |prod-dc| is distinct from upgrading other |prod|
|
|
configurations.
|
|
|
|
.. xbooklink For information on updating |prod-dc|, see |distcloud-doc|:
|
|
:ref:`Upgrade Management <upgrade-management-overview>`.
|
|
|
|
.. note::
|
|
The upgrade orchestration CLI is :command:`sw-manager`.To use upgrade
|
|
orchestration commands, you need administrator privileges. You must log in
|
|
to the active controller as user **sysadmin** and source the
|
|
/etc/platform/openrc script to obtain administrator privileges. Do not use
|
|
**sudo**.
|
|
|
|
.. code-block:: none
|
|
|
|
~(keystone_admin)]$ sw-manager upgrade-strategy --help
|
|
usage: sw-manager upgrade-strategy [-h] ...
|
|
|
|
optional arguments:
|
|
-h, --help show this help message and exit
|
|
|
|
Software Upgrade Commands:
|
|
|
|
create Create a strategy
|
|
delete Delete a strategy
|
|
apply Apply a strategy
|
|
abort Abort a strategy
|
|
show Show a strategy
|
|
|
|
.. _orchestration-upgrade-overview-section-N10029-N10026-N10001:
|
|
|
|
----------------------------------
|
|
Upgrade Orchestration Requirements
|
|
----------------------------------
|
|
|
|
Upgrade orchestration can only be done on a system that meets the following
|
|
conditions:
|
|
|
|
.. _orchestration-upgrade-overview-ul-blp-gcx-ry:
|
|
|
|
- The system is clear of alarms \(with the exception of the alarm upgrade in
|
|
progress\).
|
|
|
|
- All hosts must be unlocked, enabled, and available.
|
|
|
|
- The system is fully redundant \(two controller nodes available, at least
|
|
one complete storage replication group available for systems with Ceph
|
|
backend\).
|
|
|
|
- An upgrade has been started, and controller-1 has been upgraded and is
|
|
active.
|
|
|
|
- No orchestration strategy exists. Patch, upgrade, firmware, kubernetes,
|
|
and kube rootca are all types of orchestration. An upgrade cannot be
|
|
orchestrated while another orchestration is in progress.
|
|
|
|
- Sufficient free capacity or unused worker resources must be available
|
|
across the cluster. A rough calculation is: Required spare capacity \( %\)
|
|
= \(Number of hosts to upgrade in parallel divided by the total number of
|
|
hosts\) times 100.
|
|
|
|
.. _orchestration-upgrade-overview-section-N10081-N10026-N10001:
|
|
|
|
---------------------------------
|
|
The Upgrade Orchestration Process
|
|
---------------------------------
|
|
|
|
Upgrade orchestration can be initiated after the manual upgrade and stability
|
|
of the initial controller host. Upgrade orchestration automatically iterates
|
|
through the remaining hosts, installing the new software load on each one:
|
|
first the other controller host, then the storage hosts, and finally the worker
|
|
hosts. During worker host upgrades, pods are moved to alternate worker hosts
|
|
automatically.
|
|
|
|
The user first creates an upgrade orchestration strategy, or plan, for the
|
|
automated upgrade procedure. This customizes the upgrade orchestration, using
|
|
parameters to specify:
|
|
|
|
.. _orchestration-upgrade-overview-ul-eyw-fyr-31b:
|
|
|
|
- the host types to be upgraded
|
|
|
|
- whether to upgrade hosts serially or in parallel
|
|
|
|
Based on these parameters, and the state of the hosts, upgrade orchestration
|
|
creates a number of stages for the overall upgrade strategy. Each stage
|
|
generally consists of moving pods, locking hosts, installing upgrades, and
|
|
unlocking hosts for a subset of the hosts on the system.
|
|
|
|
After creating the upgrade orchestration strategy, the user can either apply
|
|
the entire strategy automatically, or apply individual stages to control and
|
|
monitor its progress manually.
|
|
|
|
Update and upgrade orchestration are mutually exclusive; they perform
|
|
conflicting operations. Only a single strategy \(sw-patch or sw-upgrade\) is
|
|
allowed to exist at a time. If you need to update during an upgrade, you can
|
|
abort/delete the sw-upgrade strategy, and then create and apply a sw-patch
|
|
strategy before going back to the upgrade.
|
|
|
|
Some stages of the upgrade could take a significant amount of time \(hours\).
|
|
For example, after upgrading a storage host, re-syncing the OSD data could take
|
|
30m per TB \(assuming 500MB/s sync rate, which is about half of a 10G
|
|
infrastructure link\).
|
|
|
|
.. _orchestration-upgrade-overview-section-N10101-N10026-N10001:
|
|
|
|
------------------------------
|
|
Upgrade Orchestration Workflow
|
|
------------------------------
|
|
|
|
The Upgrade Orchestration procedure has several major parts:
|
|
|
|
.. _orchestration-upgrade-overview-ul-r1k-wzj-wy:
|
|
|
|
- Manually upgrade controller-1.
|
|
|
|
- Orchestrate the automatic upgrade of the remaining controller, all the
|
|
storage nodes, and all the worker nodes.
|
|
|
|
- Manually complete the upgrade.
|