Upgrade documentation
This is a first pass at adding upgrades docs which we don't currently have. For now trying to cover version specific caveats with a single upgrades guide. We could split this out if it is preferable. For example, for Liberty-->Mitaka there is an extra step required after upgrade-init to deliver AODH at https://review.openstack.org/#/c/320150 Co-Authored-By: Marios Andreou <marios@redhat.com> Co-Authored-By: Michele Baldessari <michele@acksyn.com> Change-Id: I4af214d1dfc595ea61be40efbc0a04e44225e8b1
This commit is contained in:
Jiri Stransky
parent
3bca61a9db
commit
9b5421a14f
@ -71,6 +71,8 @@
|
|||||||
.source {background: #bbb;}
|
.source {background: #bbb;}
|
||||||
.ssl {background: #ffe;}
|
.ssl {background: #ffe;}
|
||||||
.selfsigned {background: #ffd;}
|
.selfsigned {background: #ffd;}
|
||||||
|
.ltom {background: #dee;}
|
||||||
|
.mton {background: #ded;}
|
||||||
|
|
||||||
/* admonition selector */
|
/* admonition selector */
|
||||||
#admonition_selector {
|
#admonition_selector {
|
||||||
|
|||||||
@ -52,6 +52,12 @@
|
|||||||
title="Step that should only be run when choosing to use some components directly from their git source code repositories instead of packages.">Install from source</label></li>
|
title="Step that should only be run when choosing to use some components directly from their git source code repositories instead of packages.">Install from source</label></li>
|
||||||
</ul>
|
</ul>
|
||||||
|
|
||||||
|
<span class="title">Upgrade Version</span>
|
||||||
|
<ul>
|
||||||
|
<li><input type="checkbox" id="ltom" checked="checked"><label for="ltom" title="Step that should only be run for upgrading from Liberty to Mitaka">Upgrading Liberty to Mitaka</label></li>
|
||||||
|
<li><input type="checkbox" id="mton" checked="checked"><label for="mton" title="Step that should only be run for upgrading from Mitaka to Newton">Upgrading Mitaka to Newton</label></li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
@ -13,5 +13,6 @@ In this chapter you will find advanced management of various |project| areas.
|
|||||||
replace_controller
|
replace_controller
|
||||||
vm_snapshot
|
vm_snapshot
|
||||||
package_update
|
package_update
|
||||||
|
upgrade
|
||||||
build_single_image
|
build_single_image
|
||||||
backup_restore_undercloud
|
backup_restore_undercloud
|
||||||
|
|||||||
265
doc/source/post_deployment/upgrade.rst
Normal file
265
doc/source/post_deployment/upgrade.rst
Normal file
@ -0,0 +1,265 @@
|
|||||||
|
Upgrading to a Next Major Release
|
||||||
|
=================================
|
||||||
|
|
||||||
|
Upgrading a TripleO deployment to a next major release is done by
|
||||||
|
first upgrading the undercloud, and then upgrading the overcloud.
|
||||||
|
|
||||||
|
Note that there are version specific caveats and notes which are pointed out as below:
|
||||||
|
|
||||||
|
.. admonition:: Liberty to Mitaka
|
||||||
|
:class: ltom
|
||||||
|
|
||||||
|
liberty to mitaka specific note
|
||||||
|
|
||||||
|
.. admonition:: Mitaka to Newton
|
||||||
|
:class: mton
|
||||||
|
|
||||||
|
mitaka to newton specific note
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
You can use the "Limit Environment Specific Content" in the left hand nav
|
||||||
|
bar to restrict content to the upgrade you are performing.
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
Generic upgrade testing cannot cover all possible deployment
|
||||||
|
configurations. Before performing the upgrade in production, test
|
||||||
|
it in a matching staging environment, and create a backup of the
|
||||||
|
production environment.
|
||||||
|
|
||||||
|
|
||||||
|
Upgrading the Undercloud
|
||||||
|
------------------------
|
||||||
|
|
||||||
|
1. Disable the old OpenStack release repositories and enable new
|
||||||
|
release repositories on the undercloud:
|
||||||
|
|
||||||
|
.. admonition:: Liberty to Mitaka
|
||||||
|
:class: ltom
|
||||||
|
|
||||||
|
::
|
||||||
|
|
||||||
|
export CURRENT_VERSION=liberty
|
||||||
|
export NEW_VERSION=mitaka
|
||||||
|
|
||||||
|
.. admonition:: Mitaka to Newton
|
||||||
|
:class: mton
|
||||||
|
|
||||||
|
::
|
||||||
|
|
||||||
|
export CURRENT_VERSION=mitaka
|
||||||
|
export NEW_VERSION=newton
|
||||||
|
|
||||||
|
Backup and disable current repos. Note that the repository files might be
|
||||||
|
named differently depending on your installation::
|
||||||
|
|
||||||
|
mkdir /home/stack/REPOBACKUP
|
||||||
|
sudo mv /etc/yum.repos.d/delorean* /home/stack/REPOBACKUP/
|
||||||
|
|
||||||
|
Get and enable new repos for `NEW_VERSION`:
|
||||||
|
|
||||||
|
.. include:: ../repositories.txt
|
||||||
|
|
||||||
|
2. Run undercloud upgrade:
|
||||||
|
|
||||||
|
.. admonition:: Liberty to Mitaka
|
||||||
|
:class: ltom
|
||||||
|
|
||||||
|
For liberty to mitaka upgrades we need to manually update
|
||||||
|
mariadb including a database backup/restore::
|
||||||
|
|
||||||
|
mysqldump -u root --flush-privileges --single-transaction --all-databases > /home/stack/backup.sql
|
||||||
|
sudo systemctl stop mariadb
|
||||||
|
sudo mv /var/lib/mysql /home/stack/mysql-backup
|
||||||
|
|
||||||
|
sudo yum -y update mariadb
|
||||||
|
|
||||||
|
sudo systemctl start mariadb
|
||||||
|
mysql -u root < /home/stack/backup.sql
|
||||||
|
|
||||||
|
|
||||||
|
The following command will upgrade the undercloud::
|
||||||
|
|
||||||
|
# TODO: In the future this might be taken care of by tripleoclient automatically
|
||||||
|
sudo systemctl stop openstack-*
|
||||||
|
sudo systemctl stop neutron-*
|
||||||
|
sudo yum -y update instack-undercloud openstack-puppet-modules openstack-tripleo-common python-tripleoclient
|
||||||
|
openstack undercloud upgrade
|
||||||
|
|
||||||
|
Once the undercloud upgrade is fully completed you may
|
||||||
|
remove the older mysql backup folder /home/stack/mysql-backup
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
You may wish to use time and capture the output to a file for any debug::
|
||||||
|
|
||||||
|
time openstack undercloud upgrade 2>&1 | tee undercloud_upgrade.log
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
If you added custom OVS ports to the undercloud (e.g. in a virtual
|
||||||
|
testing environment) you may need to re-add them at this point.
|
||||||
|
|
||||||
|
|
||||||
|
Upgrading the Overcloud
|
||||||
|
-----------------------
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
The `openstack overcloud deploy` calls in upgrade steps below are
|
||||||
|
non-blocking. Make sure that the overcloud is `UPDATE_COMPLETE` in
|
||||||
|
`openstack stack list` and `sudo pcs status` on a controller
|
||||||
|
reports everything running fine before proceeding to the next step.
|
||||||
|
|
||||||
|
.. admonition:: Liberty to Mitaka
|
||||||
|
:class: ltom
|
||||||
|
|
||||||
|
**Create the new CephClientKey**
|
||||||
|
|
||||||
|
If using a TripleO managed Ceph deployment, a new key for the
|
||||||
|
"client.openstack" CephX user needs to be provided. A sample
|
||||||
|
environment file would look like the following::
|
||||||
|
|
||||||
|
parameter_defaults:
|
||||||
|
CephClientKey: 'my_cephx_key'
|
||||||
|
|
||||||
|
A proper value for the key parameter can be generated from any of
|
||||||
|
the overcloud nodes with::
|
||||||
|
|
||||||
|
$ ceph-authtool --gen-print-key
|
||||||
|
|
||||||
|
.. admonition:: Liberty to Mitaka
|
||||||
|
:class: ltom
|
||||||
|
|
||||||
|
|
||||||
|
**Deliver the aodh migration.**
|
||||||
|
|
||||||
|
For Liberty to Mitaka we need to run an extra step in the upgrades workflow
|
||||||
|
after the upgrade initialisation.
|
||||||
|
|
||||||
|
This is to deliver the migration from ceilometer-alarms to aodh. To execute
|
||||||
|
this step run `overcloud deploy`, passing in the full set of environment
|
||||||
|
files plus `major-upgrade-aodh.yaml`. Note that the `--force-postconfig`
|
||||||
|
switch is needed in order to add the newly created aodh endpoint::
|
||||||
|
|
||||||
|
openstack overcloud deploy --templates \
|
||||||
|
-e <full environment> \
|
||||||
|
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-aodh.yaml \
|
||||||
|
--force-postconfig
|
||||||
|
|
||||||
|
.. admonition:: Liberty to Mitaka
|
||||||
|
:class: ltom
|
||||||
|
|
||||||
|
|
||||||
|
**Deliver the migration for keystone to run under httpd.**
|
||||||
|
|
||||||
|
For Liberty to Mitaka we need to run an extra step in the upgrades workflow
|
||||||
|
after the aodh migration.
|
||||||
|
|
||||||
|
This is to deliver the migration for keystone to be run under httpd (apache)
|
||||||
|
rather than eventlet as was the case before. To execute this step run
|
||||||
|
`overcloud deploy`, passing in the full set of environment files plus
|
||||||
|
`major-upgrade-keystone-liberty-mitaka.yaml`::
|
||||||
|
|
||||||
|
openstack overcloud deploy --templates \
|
||||||
|
-e <full environment> \
|
||||||
|
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-keystone-liberty-mitaka.yaml
|
||||||
|
|
||||||
|
#. Upgrade initialization
|
||||||
|
|
||||||
|
The initialization step switches to new repositories on overcloud
|
||||||
|
nodes, and it delivers upgrade scripts to nodes which are going to
|
||||||
|
be upgraded one-by-one (this means non-controller nodes, except any
|
||||||
|
stand-alone block storage nodes).
|
||||||
|
|
||||||
|
Create an environment file with commands to switch OpenStack
|
||||||
|
repositories to a new release. This will likely be the same
|
||||||
|
commands that were used to switch repositories on the undercloud::
|
||||||
|
|
||||||
|
cat > overcloud-repos.yaml <<EOF
|
||||||
|
parameter_defaults:
|
||||||
|
UpgradeInitCommand: |
|
||||||
|
set -e
|
||||||
|
# REPOSITORY SWITCH COMMANDS GO HERE
|
||||||
|
EOF
|
||||||
|
|
||||||
|
And run `overcloud deploy`, passing in full set of environment
|
||||||
|
files plus `major-upgrade-pacemaker-init.yaml` and
|
||||||
|
`overcloud-repos.yaml`::
|
||||||
|
|
||||||
|
openstack overcloud deploy --templates \
|
||||||
|
-e <full environment> \
|
||||||
|
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml \
|
||||||
|
-e overcloud-repos.yaml
|
||||||
|
|
||||||
|
|
||||||
|
#. Object storage nodes upgrade
|
||||||
|
|
||||||
|
If the deployment has any standalone object storage nodes, upgrade
|
||||||
|
them one-by-one using the `upgrade-non-controller.sh` script on the
|
||||||
|
undercloud node::
|
||||||
|
|
||||||
|
upgrade-non-controller.sh --upgrade <nova-id of object storage node>
|
||||||
|
|
||||||
|
This is ran before controller node upgrade because swift storage
|
||||||
|
services should be upgraded before swift proxy services.
|
||||||
|
|
||||||
|
#. Upgrade controller and block storage nodes
|
||||||
|
|
||||||
|
All controllers will be upgraded in sync in order to make services
|
||||||
|
only talk to DB schema versions they expect. Services will be
|
||||||
|
unavailable during this operation. Standalone block storage nodes
|
||||||
|
are automatically upgraded in this step too, in sync with
|
||||||
|
controllers, because block storage services don't have a version
|
||||||
|
pinning mechanism.
|
||||||
|
|
||||||
|
Run the deploy command with `major-upgrade-pacemaker.yaml`::
|
||||||
|
|
||||||
|
openstack overcloud deploy --templates \
|
||||||
|
-e <full environment> \
|
||||||
|
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker.yaml
|
||||||
|
|
||||||
|
Services of the compute component on the controller nodes are now
|
||||||
|
pinned to communicate like the older release, ensuring that they
|
||||||
|
can talk to the compute nodes which haven't been upgraded yet.
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
If this step fails, it may leave the pacemaker cluster stopped
|
||||||
|
(together with all OpenStack services on the controller
|
||||||
|
nodes). The root cause and restoration procedure may vary, but
|
||||||
|
in simple cases the pacemaker cluster can be started by logging
|
||||||
|
into one of the controllers and running `sudo pcs cluster start
|
||||||
|
--all`.
|
||||||
|
|
||||||
|
#. Upgrade compute nodes
|
||||||
|
|
||||||
|
Upgrade compute nodes one-by-one using the
|
||||||
|
`upgrade-non-controller.sh` script on the undercloud node::
|
||||||
|
|
||||||
|
upgrade-non-controller.sh --upgrade <nova-id of compute node>
|
||||||
|
|
||||||
|
#. Upgrade ceph storage nodes
|
||||||
|
|
||||||
|
If the deployment has any ceph storage nodes, upgrade them
|
||||||
|
one-by-one using the `upgrade-non-controller.sh` script on the
|
||||||
|
undercloud node::
|
||||||
|
|
||||||
|
upgrade-non-controller.sh --upgrade <nova-id of ceph storage node>
|
||||||
|
|
||||||
|
#. Apply configuration from upgraded tripleo-heat-templates
|
||||||
|
|
||||||
|
This step unpins compute services communication (upgrade level) on
|
||||||
|
controller and compute nodes, and it triggers configuration
|
||||||
|
management tooling to converge the overcloud configuration
|
||||||
|
according to the new release of `tripleo-heat-templates`.
|
||||||
|
|
||||||
|
Make sure that all overcloud nodes have been upgraded to the new
|
||||||
|
release, and then run the deploy command with
|
||||||
|
`major-upgrade-pacemaker-converge.yaml`::
|
||||||
|
|
||||||
|
|
||||||
|
openstack overcloud deploy --templates \
|
||||||
|
-e <full environment> \
|
||||||
|
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-converge.yaml
|
||||||
Loading…
x
Reference in New Issue
Block a user