diff --git a/doc/source/_custom/custom.css b/doc/source/_custom/custom.css index 7f454691..1bf7ddc8 100644 --- a/doc/source/_custom/custom.css +++ b/doc/source/_custom/custom.css @@ -71,6 +71,8 @@ .source {background: #bbb;} .ssl {background: #ffe;} .selfsigned {background: #ffd;} +.ltom {background: #dee;} +.mton {background: #ded;} /* admonition selector */ #admonition_selector { diff --git a/doc/source/_templates/layout.html b/doc/source/_templates/layout.html index d61f008a..31d3ce52 100644 --- a/doc/source/_templates/layout.html +++ b/doc/source/_templates/layout.html @@ -52,6 +52,12 @@ title="Step that should only be run when choosing to use some components directly from their git source code repositories instead of packages.">Install from source + Upgrade Version + + diff --git a/doc/source/post_deployment/post_deployment.rst b/doc/source/post_deployment/post_deployment.rst index 7ecff57c..fb6de83d 100644 --- a/doc/source/post_deployment/post_deployment.rst +++ b/doc/source/post_deployment/post_deployment.rst @@ -13,5 +13,6 @@ In this chapter you will find advanced management of various |project| areas. replace_controller vm_snapshot package_update + upgrade build_single_image backup_restore_undercloud diff --git a/doc/source/post_deployment/upgrade.rst b/doc/source/post_deployment/upgrade.rst new file mode 100644 index 00000000..4379ce19 --- /dev/null +++ b/doc/source/post_deployment/upgrade.rst @@ -0,0 +1,265 @@ +Upgrading to a Next Major Release +================================= + +Upgrading a TripleO deployment to a next major release is done by +first upgrading the undercloud, and then upgrading the overcloud. + +Note that there are version specific caveats and notes which are pointed out as below: + +.. admonition:: Liberty to Mitaka + :class: ltom + + liberty to mitaka specific note + +.. admonition:: Mitaka to Newton + :class: mton + + mitaka to newton specific note + +.. note:: + + You can use the "Limit Environment Specific Content" in the left hand nav + bar to restrict content to the upgrade you are performing. + +.. note:: + + Generic upgrade testing cannot cover all possible deployment + configurations. Before performing the upgrade in production, test + it in a matching staging environment, and create a backup of the + production environment. + + +Upgrading the Undercloud +------------------------ + +1. Disable the old OpenStack release repositories and enable new + release repositories on the undercloud: + + .. admonition:: Liberty to Mitaka + :class: ltom + + :: + + export CURRENT_VERSION=liberty + export NEW_VERSION=mitaka + + .. admonition:: Mitaka to Newton + :class: mton + + :: + + export CURRENT_VERSION=mitaka + export NEW_VERSION=newton + + Backup and disable current repos. Note that the repository files might be + named differently depending on your installation:: + + mkdir /home/stack/REPOBACKUP + sudo mv /etc/yum.repos.d/delorean* /home/stack/REPOBACKUP/ + + Get and enable new repos for `NEW_VERSION`: + +.. include:: ../repositories.txt + +2. Run undercloud upgrade: + + .. admonition:: Liberty to Mitaka + :class: ltom + + For liberty to mitaka upgrades we need to manually update + mariadb including a database backup/restore:: + + mysqldump -u root --flush-privileges --single-transaction --all-databases > /home/stack/backup.sql + sudo systemctl stop mariadb + sudo mv /var/lib/mysql /home/stack/mysql-backup + + sudo yum -y update mariadb + + sudo systemctl start mariadb + mysql -u root < /home/stack/backup.sql + + + The following command will upgrade the undercloud:: + + # TODO: In the future this might be taken care of by tripleoclient automatically + sudo systemctl stop openstack-* + sudo systemctl stop neutron-* + sudo yum -y update instack-undercloud openstack-puppet-modules openstack-tripleo-common python-tripleoclient + openstack undercloud upgrade + + Once the undercloud upgrade is fully completed you may + remove the older mysql backup folder /home/stack/mysql-backup + +.. note:: + + You may wish to use time and capture the output to a file for any debug:: + + time openstack undercloud upgrade 2>&1 | tee undercloud_upgrade.log + +.. note:: + + If you added custom OVS ports to the undercloud (e.g. in a virtual + testing environment) you may need to re-add them at this point. + + +Upgrading the Overcloud +----------------------- + +.. note:: + + The `openstack overcloud deploy` calls in upgrade steps below are + non-blocking. Make sure that the overcloud is `UPDATE_COMPLETE` in + `openstack stack list` and `sudo pcs status` on a controller + reports everything running fine before proceeding to the next step. + +.. admonition:: Liberty to Mitaka + :class: ltom + + **Create the new CephClientKey** + + If using a TripleO managed Ceph deployment, a new key for the + "client.openstack" CephX user needs to be provided. A sample + environment file would look like the following:: + + parameter_defaults: + CephClientKey: 'my_cephx_key' + + A proper value for the key parameter can be generated from any of + the overcloud nodes with:: + + $ ceph-authtool --gen-print-key + +.. admonition:: Liberty to Mitaka + :class: ltom + + + **Deliver the aodh migration.** + + For Liberty to Mitaka we need to run an extra step in the upgrades workflow + after the upgrade initialisation. + + This is to deliver the migration from ceilometer-alarms to aodh. To execute + this step run `overcloud deploy`, passing in the full set of environment + files plus `major-upgrade-aodh.yaml`. Note that the `--force-postconfig` + switch is needed in order to add the newly created aodh endpoint:: + + openstack overcloud deploy --templates \ + -e \ + -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-aodh.yaml \ + --force-postconfig + +.. admonition:: Liberty to Mitaka + :class: ltom + + + **Deliver the migration for keystone to run under httpd.** + + For Liberty to Mitaka we need to run an extra step in the upgrades workflow + after the aodh migration. + + This is to deliver the migration for keystone to be run under httpd (apache) + rather than eventlet as was the case before. To execute this step run + `overcloud deploy`, passing in the full set of environment files plus + `major-upgrade-keystone-liberty-mitaka.yaml`:: + + openstack overcloud deploy --templates \ + -e \ + -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-keystone-liberty-mitaka.yaml + +#. Upgrade initialization + + The initialization step switches to new repositories on overcloud + nodes, and it delivers upgrade scripts to nodes which are going to + be upgraded one-by-one (this means non-controller nodes, except any + stand-alone block storage nodes). + + Create an environment file with commands to switch OpenStack + repositories to a new release. This will likely be the same + commands that were used to switch repositories on the undercloud:: + + cat > overcloud-repos.yaml < \ + -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml \ + -e overcloud-repos.yaml + + +#. Object storage nodes upgrade + + If the deployment has any standalone object storage nodes, upgrade + them one-by-one using the `upgrade-non-controller.sh` script on the + undercloud node:: + + upgrade-non-controller.sh --upgrade + + This is ran before controller node upgrade because swift storage + services should be upgraded before swift proxy services. + +#. Upgrade controller and block storage nodes + + All controllers will be upgraded in sync in order to make services + only talk to DB schema versions they expect. Services will be + unavailable during this operation. Standalone block storage nodes + are automatically upgraded in this step too, in sync with + controllers, because block storage services don't have a version + pinning mechanism. + + Run the deploy command with `major-upgrade-pacemaker.yaml`:: + + openstack overcloud deploy --templates \ + -e \ + -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker.yaml + + Services of the compute component on the controller nodes are now + pinned to communicate like the older release, ensuring that they + can talk to the compute nodes which haven't been upgraded yet. + + .. note:: + + If this step fails, it may leave the pacemaker cluster stopped + (together with all OpenStack services on the controller + nodes). The root cause and restoration procedure may vary, but + in simple cases the pacemaker cluster can be started by logging + into one of the controllers and running `sudo pcs cluster start + --all`. + +#. Upgrade compute nodes + + Upgrade compute nodes one-by-one using the + `upgrade-non-controller.sh` script on the undercloud node:: + + upgrade-non-controller.sh --upgrade + +#. Upgrade ceph storage nodes + + If the deployment has any ceph storage nodes, upgrade them + one-by-one using the `upgrade-non-controller.sh` script on the + undercloud node:: + + upgrade-non-controller.sh --upgrade + +#. Apply configuration from upgraded tripleo-heat-templates + + This step unpins compute services communication (upgrade level) on + controller and compute nodes, and it triggers configuration + management tooling to converge the overcloud configuration + according to the new release of `tripleo-heat-templates`. + + Make sure that all overcloud nodes have been upgraded to the new + release, and then run the deploy command with + `major-upgrade-pacemaker-converge.yaml`:: + + + openstack overcloud deploy --templates \ + -e \ + -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-converge.yaml