openstack/tripleo-docs
Code Issues Proposed changes

Merge "Adds operator docs for Pike to Queens upgrade"

Browse Source
This commit is contained in:
Zuul 2018-05-22 09:04:49 +00:00 committed by Gerrit Code Review
commit 9d1763cd28
1 changed files with 485 additions and 236 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

539
doc/source/install/post_deployment/upgrade.rst
View File

@ -1,15 +1,11 @@
Upgrading to a Next Major Release
=================================
Upgrading a TripleO deployment to a next major release is done by
first upgrading the undercloud, and then upgrading the overcloud.
Upgrading a TripleO deployment to the next major release is done by first
upgrading the undercloud and using it to upgrade the overcloud.
Note that there are version specific caveats and notes which are pointed out as below:
.. admonition:: Mitaka to Newton
:class: mton
mitaka to newton specific note
Note that there are version specific caveats and notes which are pointed out
as below:
.. note::
@ -26,9 +22,240 @@ Note that there are version specific caveats and notes which are pointed out as
.. Undercloud upgrade section
.. include:: ../../install/installation/updating.rst
Upgrading the Overcloud to Ocata or Pike
Upgrading the Overcloud from Pike to Queens
-------------------------------------------
The Queens upgrade workflow for TripleO is significantly different to previous
cycles. At a high level, the workflow no longer relies on Heat to deliver the
upgrade configuration, but instead uses ansible playbooks. Those
playbooks *are* still generated using a Heat stack update of the overcloud
stack, as the first step in the workflow. However there is no configuration
applied during that stack update and so comparatively it takes *much* less
time to complete than a 'normal' stack update. More information and pointers
can be found in the relevant queens-upgrade-spec_ and the
queens-upgrade-dev-docs_.
The tripleo cli has been updated accordingly to accommodate this new
workflow. In Queens a new `openstack overcloud upgrade` command is introduced
and it expects one of three subcommands: **prepare**, **run** and
**converge**. Each subcommand has its own set of options which you can explore
with --help:
.. code-block:: bash
source /home/stack/stackrc
openstack overcloud upgrade run --help
The Queens upgrade workflow essentially consists of the following steps:
#. **Prepare you environment, backup, get container images**.
Generate any environment files you need for the upgrade such as the
references to the latest container images or commands used to switch repos.
#. **openstack overcloud upgrade prepare $OPTS**.
Run a heat stack update to generate the upgrade playbooks.
#. **openstack overcloud upgrade run $OPTS**.
Run the upgrade on specific nodes or groups of nodes. Repeat until all nodes
are successfully upgraded.
#. **openstack overcloud upgrade converge $OPTS**.
Finally run a heat stack update, unsetting any upgrade specific variables
and leaving the heat stack in a healthy state for future updates.
.. _queens-upgrade-dev-docs: https://docs.openstack.org/tripleo-docs/latest/install/developer/upgrades/major_upgrade.html # WIP @ https://review.openstack.org/#/c/569443/
Prepare your environment - get latest container images
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
As a part of the upgrade the container images for the target release should be
downloaded to the Undercloud. Please see the `openstack overcloud container image prepare`
:doc:`../containers_deployment/overcloud` for more information.
The output of this step will be a Heat environment file that contains
references to the latest container images. You will need to pass this file
into the **upgrade prepare** command using the --container-registry-file
option.
You will also need to create an environment file to override the
UpgradeInitCommand_ tripleo-heat-templates parameter, that can be used to
switch the yum repos in use by the nodes during the upgrade. This will likely
be the same commands that were used to switch repositories on the undercloud.
.. code-block:: bash
cat <<EOF > init-repo.yaml
parameter_defaults:
UpgradeInitCommand: |
set -e
# -- REPLACE LINES WITH YOUR REPO SWITCH COMMANDS --
curl -L -o /etc/yum.repos.d/delorean.repo https://trunk.rdoproject.org/centos7-queens/current/delorean.repo
curl -L -o /etc/yum.repos.d/delorean-deps.repo https://trunk.rdoproject.org/centos7-queens/delorean-deps.repo
yum clean all
EOF
The resulting init-repo.yaml will then be passed into the upgrade prepare using
the -e option.
.. _Upgradeinitcommand: https://github.com/openstack/tripleo-heat-templates/blob/1d9629ec0b3320bcbc5a4150c8be19c6eb4096eb/puppet/role.role.j2.yaml#L468-L493
openstack overcloud upgrade prepare
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
prepare
queens-upgrade-dev-docs.
.. note::
Before running the overcloud upgrade prepare ensure you have a valid backup
of the current state, including the **undercloud** since there will be a
Heat stack update performed here.
.. note::
After running the upgrade prepare and until successful completion
of the upgrade converge operation, stack updates to the deployment Heat
stack are expected to fail. That is, operations such as scaling to add
a new node or to apply any new TripleO configuration via Heat stack
update **must not** be performed on a Heat stack that has been prepared
for upgrade with the 'prepare' command and only consider doing so after
running the converge step. See the queens-upgrade-dev-docs_ for more.
Run **overcloud upgrade prepare**. NOTE: this command expects the full set
of environment files that were passed into the deploy command, as well as the
roles_data.yaml file used to deploy the overcloud you are about to upgrade. The
--container-registry-file should point to the file that was output by the image
prepare command you ran to get the latest container image references.
.. note::
It is especially important to remember that you **must** include all
environment files that were used to deploy the overcloud that you are about
to upgrade.
.. code-block:: bash
openstack overcloud upgrade prepare --templates \
--container-registry-file /home/stack/containers-default-parameters.yaml \
-e <ALL Templates from overcloud-deploy.sh> \
-e init-repo.yaml
-r /path/to/roles_data.yaml
This will begin an update on the overcloud Heat stack but without
applying any of the TripleO configuration, as explained above. Once this
`upgrade prepare` operation has successfully completed the heat stack will be
in the UPDATE_COMPLETE state. At that point you can use `config download` to
download and inspect the configuration ansible playbooks that will be used
to deliver the upgrade in the next step:
.. code-block:: bash
openstack overcloud config download --config-dir SOMEDIR
# playbooks will be downloaded to SOMEDIR directory
openstack overcloud upgrade run
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This will run the ansible playbooks to deliver the upgrade configuration.
By default, 3 playbooks are executed: the upgrade_steps_playbook, then the
deploy_steps_playbook and finally the post_upgrade_steps_playbook. These
playbooks are invoked on those overcloud nodes specified by the --roles or
--nodes parameters, which are mutually exclusive. You are expected to use
--roles for controlplane nodes, since these need to be upgraded in the same
step. For non controlplane nodes, such as Compute or Storage, you can use
--nodes to specify a single node or list of nodes to upgrade.
.. code-block:: bash
openstack overcloud upgrade run --roles Controller
Optionally specify `--playbook` to manually step through the upgrade
playbooks: You need to run all three in this order and as specified below
(no path) for the full upgrade run
.. code-block:: bash
openstack overcloud upgrade run --roles Controller --playbook upgrade_steps_playbook.yaml
openstack overcloud upgrade run --roles Controller --playbook deploy_steps_playbook.yaml
openstack overcloud upgrade run --roles Controller --playbook post_upgrade_steps_playbook.yaml
After all three playbooks have been executed without error on all nodes of
the controller role, all controllers will have been fully upgraded to Queens.
At a minimum an operator mayb check the health of the pacemaker cluster and see
if there are any errors
.. code-block:: bash
[root@overcloud-controller-0 ~]# pcs status | grep -C 10 -i "error\|fail"
The operator may also want to confirm that openstack and
related service containers are all in a good state and using the image
references passed during upgrade prepare with the --container-registry-file
parameter.
.. code-block:: bash
[root@overcloud-controller-0 ~]# docker ps -a
For non controlplane, you use `--nodes overcloud-compute-0` to upgrade particular
nodes, or even "compute0,compute1,compute3" for multiple nodes. Note these
are again upgraded in parallel
.. code-block:: bash
openstack overcloud upgrade run --nodes overcloud-compute-0
This allows the operator to upgrade some subset, perhaps just one, compute
or other non controlplane node and verify that the upgrade has successfully
completed. You might even try to migrate workloads onto the newly upgraded
compute and confirm there are no problems, before deciding to proceed with
upgrading the remaining nodes in the role.
Again you can optionally step through the upgrade playbooks if you prefer. Be
sure to run upgrade_steps_playbook.yaml then deploy_steps_playbook.yaml and
finally post_upgrade_steps_playbook.yaml in that order.
.. code-block:: bash
openstack overcloud upgrade run --nodes overcloud-compute-1 \
--playbook upgrade_steps_playbook.yaml
# etc for the other 2 as above example for controller
For re-run, you can specify --skip-tags validation to skip those step 0
ansible tasks that check if services are running, in case you can't or
don't want to start them all.
.. code-block:: bash
openstack overcloud upgrade run --roles Controller --skip-tags validation
openstack overcloud upgrade converge
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Finally, run the converge heat stack update. This will re-apply all Queens
configuration across all nodes and unset all variables that were used during
the upgrade. Until you have successfully completed this step, heat stack
updates against the overcloud stack are expected to fail.
.. note::
It is especially important to remember that you **must** include all
environment files that were used to deploy the overcloud that you are about
to upgrade converge, including the list of Queens container image references
and the roles_data.yaml roles and services definition. You should omit
any repo switch commands, or override of the UpgradeInitCommand.
.. code-block:: bash
openstack overcloud upgrade converge --templates
-e /home/stack/containers-default-parameters.yaml \
-e <ALL Templates from overcloud-deploy.sh> \
-r /path/to/roles_data.yaml
Upgrading the Overcloud to Ocata or Pike
----------------------------------------
As of the Ocata release, the upgrades workflow in tripleo has changed
significantly to accommodate the operators' new ability to deploy custom roles
with the Newton release (see the Composable Service Upgrade spec_ for more
@ -68,14 +295,14 @@ For those roles with the ``disable_upgrade_deployment`` flag set True, the
operator will upgrade the corresponding nodes with the
upgrade-non-controller.sh_. The operator uses that script to invoke the
tripleo_upgrade_node.sh_ which is delivered during the
major-upgrade-composable-steps that comes first, as described above.
major-upgrade-composable-steps that come first, as described above.
1. Run the major upgrade composable ansible steps
#. Run the major upgrade composable ansible steps
This step will upgrade the nodes of all roles that do not explicitly set the
``disable_upgrade_deployment`` flag to ``True`` in the roles_data.yaml_
(this is an operator decision, and the current default is for the 'Compute'
and' ObjectStorage' roles to have this set).
(this is an operator decision, and the current default is for the **Compute**
and **ObjectStorage** roles to have this set).
The ansible upgrades tasks are collected from all service manifests_ and
executed in a series of steps as described in the introduction above.
@ -85,11 +312,11 @@ major-upgrade-composable-steps that comes first, as described above.
completed.
From Ocata to Pike, the Overcloud will be upgraded to a containerized
environment. All the services will run under Docker.
environment. All OpenStack related services will run in containers.
If you deploy TripleO with custom roles, you want to synchronize
them with roles_data.yaml visible in default roles and make sure
parameters and new services are present in your roles.
If you deploy TripleO with custom roles, you want to synchronize them with
`roles_data.yaml` visible in default roles and make sure parameters and new
services are present in your roles.
.. admonition:: Newton
:class: newton
@ -109,9 +336,18 @@ major-upgrade-composable-steps that comes first, as described above.
Pike roles_data.yaml is available here:
https://github.com/openstack/tripleo-heat-templates/blob/stable/pike/roles_data.yaml
.. admonition:: Queens
:class: queens
Queens roles_data.yaml is available here:
https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/roles_data.yaml
Create an environment file with commands to switch OpenStack repositories to
a new release. This will likely be the same commands that were used to switch
repositories on the undercloud::
repositories on the undercloud
.. code-block:: bash
cat > overcloud-repos.yaml <<EOF
parameter_defaults:
@ -123,9 +359,10 @@ major-upgrade-composable-steps that comes first, as described above.
.. admonition:: Newton to Ocata
:class: ntoo
Run `overcloud deploy`, passing in full set of environment
files plus `major-upgrade-composable-steps.yaml` and
`overcloud-repos.yaml`::
Run ``overcloud deploy``, passing in full set of environment files plus
`major-upgrade-composable-steps.yaml` and ``overcloud-repos.yaml``
.. code-block:: bash
openstack overcloud deploy --templates \
-e <full environment> \
@ -135,10 +372,12 @@ major-upgrade-composable-steps that comes first, as described above.
.. admonition:: Ceph
:class: ceph
If Ceph has been deployed in the Overcloud, then use the ceph-ansible.yaml
environment file ''instead of'' storage-environment.yaml and make sure
to move any customization into ceph-ansible.yaml (or a copy of
ceph-ansible.yaml)::
If Ceph has been deployed in the Overcloud, then use the
`ceph-ansible.yaml` environment file **instead of**
`storage-environment.yaml`. Make sure to move any customization into
`ceph-ansible.yaml` (or a copy of ceph-ansible.yaml)
.. code-block:: bash
openstack overcloud deploy --templates \
-e <full environment> \
@ -149,13 +388,13 @@ major-upgrade-composable-steps that comes first, as described above.
Customizations for the Ceph deployment previously passed as hieradata
via \*ExtraConfig should be removed as they are ignored, specifically
the deployment will stop if ``ceph::profile::params::osds`` is found
to ensure the devices list has been migrated to the format expected
by ceph-ansible. It is possible to use the ``CephAnsibleExtraConfig``
and ``CephAnsibleDisksConfig`` parameters to pass arbitrary variables
to ceph-ansible, like ``devices`` and ``dedicated_devices``.
See the `ceph-ansible scenarios`_ or
the :doc:`TripleO Ceph config guide <../advanced_deployment/ceph_config>`
the deployment will stop if ``ceph::profile::params::osds`` is found to
ensure the devices list has been migrated to the format expected by
ceph-ansible. It is possible to use the ``CephAnsibleExtraConfig`` and
``CephAnsibleDisksConfig`` parameters to pass arbitrary variables to
ceph-ansible, like ``devices`` and ``dedicated_devices``. See the
`ceph-ansible scenarios`_ or the :doc:`TripleO Ceph config guide
<../advanced_deployment/ceph_config>`
The other parameters (for example ``CinderRbdPoolName``,
``CephClientUserName``, ...) will behave as they used to with puppet-ceph
@ -188,7 +427,9 @@ major-upgrade-composable-steps that comes first, as described above.
Run `overcloud deploy`, passing in full set of environment
files plus `major-upgrade-composable-steps-docker.yaml` and
`overcloud-repos.yaml` (and docker registry if upgrading to containers)::
`overcloud-repos.yaml` (and docker registry if upgrading to containers)
.. code-block:: bash
openstack overcloud deploy --templates \
-e <full environment> \
@ -204,8 +445,10 @@ major-upgrade-composable-steps that comes first, as described above.
.. note::
If the Overcloud has been deployed with Pacemaker, then add the docker-ha.yaml
environment file to the upgrade command::
If the Overcloud has been deployed with Pacemaker, then add the
`docker-ha.yaml` environment file to the upgrade command
.. code-block:: bash
openstack overcloud deploy --templates \
-e <full environment> \
@ -230,7 +473,7 @@ major-upgrade-composable-steps that comes first, as described above.
SkipUpgradeConfigTags: [validation]
2. Upgrade remaining nodes for roles with ``disable_upgrade_deployment: True``
#. Upgrade remaining nodes for roles with ``disable_upgrade_deployment: True``
It is expected that the operator will want to upgrade the roles that have the
``openstack-nova-compute`` and ``openstack-swift-object`` services deployed
@ -250,13 +493,15 @@ major-upgrade-composable-steps that comes first, as described above.
the nodes with ``disable_upgrade_deployment`` set ``True`` during the initial
major upgrade composable steps in step 1 above.
For Ocata to Pike, the tripleo_upgrade_node.sh is still delivered to the
For Ocata to Pike, the tripleo_upgrade_node.sh_ is still delivered to the
``disable_upgrade_deployment`` nodes but is now empty. Instead, the
upgrade_non_controller.sh downloads ansible playbooks and those are executed
to deliver the upgrade. See the Queens-upgrade-spec_ for more information
on this mechanism.
`upgrade_non_controller.sh` downloads ansible playbooks and those are
executed to deliver the upgrade. See the Queens-upgrade-spec_ for more
information on this mechanism.
To upgrade remaining roles (at your convenience)::
To upgrade remaining roles (at your convenience)
.. code-block:: bash
upgrade-non-controller.sh --upgrade overcloud-compute-0
@ -264,19 +509,20 @@ major-upgrade-composable-steps that comes first, as described above.
upgrade-non-controller.sh --upgrade overcloud-objectstorage-$i &
done
3. Converge to unpin Nova RPC
#. Converge to unpin Nova RPC
The final step is required to unpin Nova RPC version. Unlike in previous
releases, for Ocata the puppet configuration has already been applied to nodes
as part of each upgrades step, i.e. after the ansible tasks or when invoking
the tripleo_upgrade_node.sh_ script to upgrade compute nodes. Thus the
significance of this step is somewhat diminished compared to previously.
releases, for Ocata the puppet configuration has already been applied to
nodes as part of each upgrades step, i.e. after the ansible tasks or when
invoking the tripleo_upgrade_node.sh_ script to upgrade compute nodes. Thus
the significance of this step is somewhat diminished compared to previously.
However a re-application of puppet configuration across all nodes here will
also serve as a sanity check and hopefully show any issues that an operator
may have missed during any of the previous upgrade steps.
To converge, run the deploy command with
`major-upgrade-converge-docker.yaml`::
To converge, run the deploy command with `major-upgrade-converge-docker.yaml`
.. code-block:: bash
openstack overcloud deploy --templates \
-e <full environment> \
@ -287,7 +533,9 @@ major-upgrade-composable-steps that comes first, as described above.
:class: ntoo
For Newton to Ocata, run the deploy command with
`major-upgrade-pacemaker-converge.yaml`::
`major-upgrade-pacemaker-converge.yaml`
.. code-block:: bash
openstack overcloud deploy --templates \
-e <full environment> \
@ -295,8 +543,10 @@ major-upgrade-composable-steps that comes first, as described above.
.. note::
If the Overcloud has been deployed with Pacemaker, then add the docker-ha.yaml
environment file to the upgrade command::
If the Overcloud has been deployed with Pacemaker, then add the
`docker-ha.yaml` environment file to the upgrade command
.. code-block:: bash
openstack overcloud deploy --templates \
-e <full environment> \
@ -330,19 +580,20 @@ Upgrading the Overcloud to Newton and earlier
The `openstack overcloud deploy` calls in upgrade steps below are
non-blocking. Make sure that the overcloud is `UPDATE_COMPLETE` in
`openstack stack list` and `sudo pcs status` on a controller
reports everything running fine before proceeding to the next step.
`openstack stack list` and `sudo pcs status` on a controller reports
everything running fine before proceeding to the next step.
.. admonition:: Mitaka to Newton
:class: mton
**Deliver the migration for ceilometer to run under httpd.**
This is to deliver the migration for ceilometer to be run under httpd (apache)
rather than eventlet as was the case before. To execute this step run
`overcloud deploy`, passing in the full set of environment files plus
`major-upgrade-ceilometer-wsgi-mitaka-newton.yaml`::
`major-upgrade-ceilometer-wsgi-mitaka-newton.yaml`
.. code-block:: bash
openstack overcloud deploy --templates \
-e <full environment> \
@ -350,14 +601,16 @@ Upgrading the Overcloud to Newton and earlier
#. Upgrade initialization
The initialization step switches to new repositories on overcloud
nodes, and it delivers upgrade scripts to nodes which are going to
be upgraded one-by-one (this means non-controller nodes, except any
stand-alone block storage nodes).
The initialization step switches to new repositories on overcloud nodes, and
it delivers upgrade scripts to nodes which are going to be upgraded
one-by-one (this means non-controller nodes, except any stand-alone block
storage nodes).
Create an environment file with commands to switch OpenStack
repositories to a new release. This will likely be the same
commands that were used to switch repositories on the undercloud::
Create an environment file with commands to switch OpenStack repositories to
a new release. This will likely be the same commands that were used to
switch repositories on the undercloud
.. code-block:: bash
cat > overcloud-repos.yaml <<EOF
parameter_defaults:
@ -366,104 +619,112 @@ Upgrading the Overcloud to Newton and earlier
# REPOSITORY SWITCH COMMANDS GO HERE
EOF
And run `overcloud deploy`, passing in full set of environment
files plus `major-upgrade-pacemaker-init.yaml` and
`overcloud-repos.yaml`::
And run `overcloud deploy`, passing in full set of environment files plus
`major-upgrade-pacemaker-init.yaml` and `overcloud-repos.yaml`
.. code-block:: bash
openstack overcloud deploy --templates \
-e <full environment> \
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml \
-e overcloud-repos.yaml
#. Object storage nodes upgrade
If the deployment has any standalone object storage nodes, upgrade
them one-by-one using the `upgrade-non-controller.sh` script on the
undercloud node::
If the deployment has any standalone object storage nodes, upgrade them
one-by-one using the `upgrade-non-controller.sh` script on the undercloud
node
.. code-block:: bash
upgrade-non-controller.sh --upgrade <nova-id of object storage node>
This is ran before controller node upgrade because swift storage
services should be upgraded before swift proxy services.
This is ran before controller node upgrade because swift storage services
should be upgraded before swift proxy services.
#. Upgrade controller and block storage nodes
.. admonition:: Mitaka to Newton
:class: mton
**Explicitly disable sahara services if so desired:**
As discussed at bug1630247_ sahara services are disabled by default
in the Newton overcloud deployment. This special case is handled for
the duration of the upgrade by defaulting to 'keep sahara-\*'.
As discussed at bug1630247_ sahara services are disabled by default in
the Newton overcloud deployment. This special case is handled for the
duration of the upgrade by defaulting to 'keep sahara-\*'.
That is by default sahara services are restarted after the mitaka to
newton upgrade of controller nodes and sahara config is re-applied
during the final upgrade converge step.
newton upgrade of controller nodes and sahara config is re-applied during
the final upgrade converge step.
If an operator wishes to **disable** sahara services as part of the mitaka
to newton upgrade they need to include the major-upgrade-remove-sahara.yaml_
environment file during the controller upgrade step as well as during
the converge step later::
If an operator wishes to **disable** sahara services as part of the
mitaka to newton upgrade they need to include the
major-upgrade-remove-sahara.yaml_ environment file during the controller
upgrade step as well as during the converge step later
.. code-block:: bash
openstack overcloud deploy --templates \
-e <full environment> \
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker.yaml
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-remove-sahara.yaml
All controllers will be upgraded in sync in order to make services
only talk to DB schema versions they expect. Services will be
unavailable during this operation. Standalone block storage nodes
are automatically upgraded in this step too, in sync with
controllers, because block storage services don't have a version
pinning mechanism.
All controllers will be upgraded in sync in order to make services only talk
to DB schema versions they expect. Services will be unavailable during this
operation. Standalone block storage nodes are automatically upgraded in this
step too, in sync with controllers, because block storage services don't
have a version pinning mechanism.
Run the deploy command with `major-upgrade-pacemaker.yaml`::
Run the deploy command with `major-upgrade-pacemaker.yaml`
.. code-block:: bash
openstack overcloud deploy --templates \
-e <full environment> \
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker.yaml
Services of the compute component on the controller nodes are now
pinned to communicate like the older release, ensuring that they
can talk to the compute nodes which haven't been upgraded yet.
Services of the compute component on the controller nodes are now pinned to
communicate like the older release, ensuring that they can talk to the
compute nodes which haven't been upgraded yet.
.. note::
If this step fails, it may leave the pacemaker cluster stopped
(together with all OpenStack services on the controller
nodes). The root cause and restoration procedure may vary, but
in simple cases the pacemaker cluster can be started by logging
into one of the controllers and running `sudo pcs cluster start
--all`.
If this step fails, it may leave the pacemaker cluster stopped (together
with all OpenStack services on the controller nodes). The root cause and
restoration procedure may vary, but in simple cases the pacemaker cluster
can be started by logging into one of the controllers and running `sudo
pcs cluster start --all`.
.. note::
After this step, or if this step failed with the error: `ERROR:
upgrade cannot start with some cluster nodes being
offlineAfter`, it's possible that some pacemaker resources needs
to be clean. Check the failed actions and clean them by running
on *only one* controller node as root::
After this step, or if this step failed with the error: `ERROR: upgrade
cannot start with some cluster nodes being offlineAfter`, it's possible
that some pacemaker resources needs to be clean. Check the failed
actions and clean them by running on `only one` controller node as root
.. code-block:: bash
pcs status
pcs resource cleanup
It can take few minutes for the cluster to go back to a “normal”
state as displayed by `crm_mon`. This is expected.
It can take few minutes for the cluster to go back to a “normal” state as
displayed by `crm_mon`. This is expected.
#. Upgrade ceph storage nodes
If the deployment has any ceph storage nodes, upgrade them
one-by-one using the `upgrade-non-controller.sh` script on the
undercloud node::
If the deployment has any ceph storage nodes, upgrade them one-by-one using
the `upgrade-non-controller.sh` script on the undercloud node
.. code-block:: bash
upgrade-non-controller.sh --upgrade <nova-id of ceph storage node>
#. Upgrade compute nodes
Upgrade compute nodes one-by-one using the
`upgrade-non-controller.sh` script on the undercloud node::
Upgrade compute nodes one-by-one using the `upgrade-non-controller.sh`
script on the undercloud node
.. code-block:: bash
upgrade-non-controller.sh --upgrade <nova-id of compute node>
@ -473,18 +734,20 @@ Upgrading the Overcloud to Newton and earlier
:class: mton
**Explicitly disable sahara services if so desired:**
As discussed at bug1630247_ sahara services are disabled by default
in the Newton overcloud deployment. This special case is handled for
the duration of the upgrade by defaulting to 'keep sahara-\*'.
As discussed at bug1630247_ sahara services are disabled by default in
the Newton overcloud deployment. This special case is handled for the
duration of the upgrade by defaulting to 'keep sahara-\*'.
That is by default sahara services are restarted after the mitaka to
newton upgrade of controller nodes and sahara config is re-applied
during the final upgrade converge step.
newton upgrade of controller nodes and sahara config is re-applied during
the final upgrade converge step.
If an operator wishes to **disable** sahara services as part of the mitaka
to newton upgrade they need to include the major-upgrade-remove-sahara.yaml_
environment file during the controller upgrade earlier and converge
step here::
If an operator wishes to **disable** sahara services as part of the
mitaka to newton upgrade they need to include the
major-upgrade-remove-sahara.yaml_ environment file during the controller
upgrade earlier and converge step here
.. code-block:: bash
openstack overcloud deploy --templates \
-e <full environment> \
@ -495,16 +758,15 @@ Upgrading the Overcloud to Newton and earlier
.. _major-upgrade-remove-sahara.yaml: https://github.com/openstack/tripleo-heat-templates/blob/2e6cc07c1a74c2dd7be70568f49834bace499937/environments/major-upgrade-remove-sahara.yaml
This step unpins compute services communication (upgrade level) on
controller and compute nodes, and it triggers configuration
management tooling to converge the overcloud configuration
according to the new release of `tripleo-heat-templates`.
controller and compute nodes, and it triggers configuration management
tooling to converge the overcloud configuration according to the new release
of `tripleo-heat-templates`.
Make sure that all overcloud nodes have been upgraded to the new
release, and then run the deploy command with
`major-upgrade-pacemaker-converge.yaml`::
Make sure that all overcloud nodes have been upgraded to the new release,
and then run the deploy command with `major-upgrade-pacemaker-converge.yaml`
.. code-block:: bash
openstack overcloud deploy --templates \
-e <full environment> \
@ -513,29 +775,16 @@ Upgrading the Overcloud to Newton and earlier
.. note::
After the converge step, it's possible that some pacemaker
resources needs to be cleaned. Check the failed actions and clean
them by running on *only one* controller as root::
After the converge step, it's possible that some pacemaker resources
needs to be cleaned. Check the failed actions and clean them by running
on **only one** controller as root
.. code-block:: bash
pcs status
pcs resource cleanup
It can take few minutes for the cluster to go back to a “normal”
state as displayed by `crm_mon`. This is expected.
It can take few minutes for the cluster to go back to a “normal” state as
displayed by ``crm_mon``. This is expected.
.. admonition:: Mitaka to Newton
:class: mton
**Deliver the data migration for aodh.**
This is to deliver the data migration for aodh. In Newton, aodh uses its
own mysql backend. This step migrates all the existing alarm data from
mongodb to the new mysql backend. To execute this step run
`overcloud deploy`, passing in the full set of environment files plus
`major-upgrade-aodh-migration.yaml`::
openstack overcloud deploy --templates \
-e <full environment> \
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-aodh-migration.yaml