Upgrade documentation
This is a first pass at adding upgrades docs which we don't currently have. For now trying to cover version specific caveats with a single upgrades guide. We could split this out if it is preferable. For example, for Liberty-->Mitaka there is an extra step required after upgrade-init to deliver AODH at https://review.openstack.org/#/c/320150 Co-Authored-By: Marios Andreou <marios@redhat.com> Co-Authored-By: Michele Baldessari <michele@acksyn.com> Change-Id: I4af214d1dfc595ea61be40efbc0a04e44225e8b1
This commit is contained in:
Jiri Stransky
parent
3bca61a9db
commit
9b5421a14f
@ -71,6 +71,8 @@
|
||||
.source {background: #bbb;}
|
||||
.ssl {background: #ffe;}
|
||||
.selfsigned {background: #ffd;}
|
||||
.ltom {background: #dee;}
|
||||
.mton {background: #ded;}
|
||||
|
||||
/* admonition selector */
|
||||
#admonition_selector {
|
||||
|
||||
@ -52,6 +52,12 @@
|
||||
title="Step that should only be run when choosing to use some components directly from their git source code repositories instead of packages.">Install from source</label></li>
|
||||
</ul>
|
||||
|
||||
<span class="title">Upgrade Version</span>
|
||||
<ul>
|
||||
<li><input type="checkbox" id="ltom" checked="checked"><label for="ltom" title="Step that should only be run for upgrading from Liberty to Mitaka">Upgrading Liberty to Mitaka</label></li>
|
||||
<li><input type="checkbox" id="mton" checked="checked"><label for="mton" title="Step that should only be run for upgrading from Mitaka to Newton">Upgrading Mitaka to Newton</label></li>
|
||||
</ul>
|
||||
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@ -13,5 +13,6 @@ In this chapter you will find advanced management of various |project| areas.
|
||||
replace_controller
|
||||
vm_snapshot
|
||||
package_update
|
||||
upgrade
|
||||
build_single_image
|
||||
backup_restore_undercloud
|
||||
|
||||
265
doc/source/post_deployment/upgrade.rst
Normal file
265
doc/source/post_deployment/upgrade.rst
Normal file
@ -0,0 +1,265 @@
|
||||
Upgrading to a Next Major Release
|
||||
=================================
|
||||
|
||||
Upgrading a TripleO deployment to a next major release is done by
|
||||
first upgrading the undercloud, and then upgrading the overcloud.
|
||||
|
||||
Note that there are version specific caveats and notes which are pointed out as below:
|
||||
|
||||
.. admonition:: Liberty to Mitaka
|
||||
:class: ltom
|
||||
|
||||
liberty to mitaka specific note
|
||||
|
||||
.. admonition:: Mitaka to Newton
|
||||
:class: mton
|
||||
|
||||
mitaka to newton specific note
|
||||
|
||||
.. note::
|
||||
|
||||
You can use the "Limit Environment Specific Content" in the left hand nav
|
||||
bar to restrict content to the upgrade you are performing.
|
||||
|
||||
.. note::
|
||||
|
||||
Generic upgrade testing cannot cover all possible deployment
|
||||
configurations. Before performing the upgrade in production, test
|
||||
it in a matching staging environment, and create a backup of the
|
||||
production environment.
|
||||
|
||||
|
||||
Upgrading the Undercloud
|
||||
------------------------
|
||||
|
||||
1. Disable the old OpenStack release repositories and enable new
|
||||
release repositories on the undercloud:
|
||||
|
||||
.. admonition:: Liberty to Mitaka
|
||||
:class: ltom
|
||||
|
||||
::
|
||||
|
||||
export CURRENT_VERSION=liberty
|
||||
export NEW_VERSION=mitaka
|
||||
|
||||
.. admonition:: Mitaka to Newton
|
||||
:class: mton
|
||||
|
||||
::
|
||||
|
||||
export CURRENT_VERSION=mitaka
|
||||
export NEW_VERSION=newton
|
||||
|
||||
Backup and disable current repos. Note that the repository files might be
|
||||
named differently depending on your installation::
|
||||
|
||||
mkdir /home/stack/REPOBACKUP
|
||||
sudo mv /etc/yum.repos.d/delorean* /home/stack/REPOBACKUP/
|
||||
|
||||
Get and enable new repos for `NEW_VERSION`:
|
||||
|
||||
.. include:: ../repositories.txt
|
||||
|
||||
2. Run undercloud upgrade:
|
||||
|
||||
.. admonition:: Liberty to Mitaka
|
||||
:class: ltom
|
||||
|
||||
For liberty to mitaka upgrades we need to manually update
|
||||
mariadb including a database backup/restore::
|
||||
|
||||
mysqldump -u root --flush-privileges --single-transaction --all-databases > /home/stack/backup.sql
|
||||
sudo systemctl stop mariadb
|
||||
sudo mv /var/lib/mysql /home/stack/mysql-backup
|
||||
|
||||
sudo yum -y update mariadb
|
||||
|
||||
sudo systemctl start mariadb
|
||||
mysql -u root < /home/stack/backup.sql
|
||||
|
||||
|
||||
The following command will upgrade the undercloud::
|
||||
|
||||
# TODO: In the future this might be taken care of by tripleoclient automatically
|
||||
sudo systemctl stop openstack-*
|
||||
sudo systemctl stop neutron-*
|
||||
sudo yum -y update instack-undercloud openstack-puppet-modules openstack-tripleo-common python-tripleoclient
|
||||
openstack undercloud upgrade
|
||||
|
||||
Once the undercloud upgrade is fully completed you may
|
||||
remove the older mysql backup folder /home/stack/mysql-backup
|
||||
|
||||
.. note::
|
||||
|
||||
You may wish to use time and capture the output to a file for any debug::
|
||||
|
||||
time openstack undercloud upgrade 2>&1 | tee undercloud_upgrade.log
|
||||
|
||||
.. note::
|
||||
|
||||
If you added custom OVS ports to the undercloud (e.g. in a virtual
|
||||
testing environment) you may need to re-add them at this point.
|
||||
|
||||
|
||||
Upgrading the Overcloud
|
||||
-----------------------
|
||||
|
||||
.. note::
|
||||
|
||||
The `openstack overcloud deploy` calls in upgrade steps below are
|
||||
non-blocking. Make sure that the overcloud is `UPDATE_COMPLETE` in
|
||||
`openstack stack list` and `sudo pcs status` on a controller
|
||||
reports everything running fine before proceeding to the next step.
|
||||
|
||||
.. admonition:: Liberty to Mitaka
|
||||
:class: ltom
|
||||
|
||||
**Create the new CephClientKey**
|
||||
|
||||
If using a TripleO managed Ceph deployment, a new key for the
|
||||
"client.openstack" CephX user needs to be provided. A sample
|
||||
environment file would look like the following::
|
||||
|
||||
parameter_defaults:
|
||||
CephClientKey: 'my_cephx_key'
|
||||
|
||||
A proper value for the key parameter can be generated from any of
|
||||
the overcloud nodes with::
|
||||
|
||||
$ ceph-authtool --gen-print-key
|
||||
|
||||
.. admonition:: Liberty to Mitaka
|
||||
:class: ltom
|
||||
|
||||
|
||||
**Deliver the aodh migration.**
|
||||
|
||||
For Liberty to Mitaka we need to run an extra step in the upgrades workflow
|
||||
after the upgrade initialisation.
|
||||
|
||||
This is to deliver the migration from ceilometer-alarms to aodh. To execute
|
||||
this step run `overcloud deploy`, passing in the full set of environment
|
||||
files plus `major-upgrade-aodh.yaml`. Note that the `--force-postconfig`
|
||||
switch is needed in order to add the newly created aodh endpoint::
|
||||
|
||||
openstack overcloud deploy --templates \
|
||||
-e <full environment> \
|
||||
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-aodh.yaml \
|
||||
--force-postconfig
|
||||
|
||||
.. admonition:: Liberty to Mitaka
|
||||
:class: ltom
|
||||
|
||||
|
||||
**Deliver the migration for keystone to run under httpd.**
|
||||
|
||||
For Liberty to Mitaka we need to run an extra step in the upgrades workflow
|
||||
after the aodh migration.
|
||||
|
||||
This is to deliver the migration for keystone to be run under httpd (apache)
|
||||
rather than eventlet as was the case before. To execute this step run
|
||||
`overcloud deploy`, passing in the full set of environment files plus
|
||||
`major-upgrade-keystone-liberty-mitaka.yaml`::
|
||||
|
||||
openstack overcloud deploy --templates \
|
||||
-e <full environment> \
|
||||
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-keystone-liberty-mitaka.yaml
|
||||
|
||||
#. Upgrade initialization
|
||||
|
||||
The initialization step switches to new repositories on overcloud
|
||||
nodes, and it delivers upgrade scripts to nodes which are going to
|
||||
be upgraded one-by-one (this means non-controller nodes, except any
|
||||
stand-alone block storage nodes).
|
||||
|
||||
Create an environment file with commands to switch OpenStack
|
||||
repositories to a new release. This will likely be the same
|
||||
commands that were used to switch repositories on the undercloud::
|
||||
|
||||
cat > overcloud-repos.yaml <<EOF
|
||||
parameter_defaults:
|
||||
UpgradeInitCommand: |
|
||||
set -e
|
||||
# REPOSITORY SWITCH COMMANDS GO HERE
|
||||
EOF
|
||||
|
||||
And run `overcloud deploy`, passing in full set of environment
|
||||
files plus `major-upgrade-pacemaker-init.yaml` and
|
||||
`overcloud-repos.yaml`::
|
||||
|
||||
openstack overcloud deploy --templates \
|
||||
-e <full environment> \
|
||||
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml \
|
||||
-e overcloud-repos.yaml
|
||||
|
||||
|
||||
#. Object storage nodes upgrade
|
||||
|
||||
If the deployment has any standalone object storage nodes, upgrade
|
||||
them one-by-one using the `upgrade-non-controller.sh` script on the
|
||||
undercloud node::
|
||||
|
||||
upgrade-non-controller.sh --upgrade <nova-id of object storage node>
|
||||
|
||||
This is ran before controller node upgrade because swift storage
|
||||
services should be upgraded before swift proxy services.
|
||||
|
||||
#. Upgrade controller and block storage nodes
|
||||
|
||||
All controllers will be upgraded in sync in order to make services
|
||||
only talk to DB schema versions they expect. Services will be
|
||||
unavailable during this operation. Standalone block storage nodes
|
||||
are automatically upgraded in this step too, in sync with
|
||||
controllers, because block storage services don't have a version
|
||||
pinning mechanism.
|
||||
|
||||
Run the deploy command with `major-upgrade-pacemaker.yaml`::
|
||||
|
||||
openstack overcloud deploy --templates \
|
||||
-e <full environment> \
|
||||
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker.yaml
|
||||
|
||||
Services of the compute component on the controller nodes are now
|
||||
pinned to communicate like the older release, ensuring that they
|
||||
can talk to the compute nodes which haven't been upgraded yet.
|
||||
|
||||
.. note::
|
||||
|
||||
If this step fails, it may leave the pacemaker cluster stopped
|
||||
(together with all OpenStack services on the controller
|
||||
nodes). The root cause and restoration procedure may vary, but
|
||||
in simple cases the pacemaker cluster can be started by logging
|
||||
into one of the controllers and running `sudo pcs cluster start
|
||||
--all`.
|
||||
|
||||
#. Upgrade compute nodes
|
||||
|
||||
Upgrade compute nodes one-by-one using the
|
||||
`upgrade-non-controller.sh` script on the undercloud node::
|
||||
|
||||
upgrade-non-controller.sh --upgrade <nova-id of compute node>
|
||||
|
||||
#. Upgrade ceph storage nodes
|
||||
|
||||
If the deployment has any ceph storage nodes, upgrade them
|
||||
one-by-one using the `upgrade-non-controller.sh` script on the
|
||||
undercloud node::
|
||||
|
||||
upgrade-non-controller.sh --upgrade <nova-id of ceph storage node>
|
||||
|
||||
#. Apply configuration from upgraded tripleo-heat-templates
|
||||
|
||||
This step unpins compute services communication (upgrade level) on
|
||||
controller and compute nodes, and it triggers configuration
|
||||
management tooling to converge the overcloud configuration
|
||||
according to the new release of `tripleo-heat-templates`.
|
||||
|
||||
Make sure that all overcloud nodes have been upgraded to the new
|
||||
release, and then run the deploy command with
|
||||
`major-upgrade-pacemaker-converge.yaml`::
|
||||
|
||||
|
||||
openstack overcloud deploy --templates \
|
||||
-e <full environment> \
|
||||
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-converge.yaml
|
||||
Loading…
x
Reference in New Issue
Block a user