docs: add info on adding and removing hosts

Forward-ported from Train commit
I19c7f05b538a7abc9253194bf041c037b1998378.

Change-Id: If07b84e0bbdcb7da8dbef87cc8826987f1d11cf8

doc/source/conf.py

@@ -101,4 +101,5 @@ openstack_projects = [
     'oslo.messaging',
     'oslotest',
     'swift',
+    'watcher',
 ]

doc/source/user/adding-and-removing-hosts.rst

@@ -0,0 +1,224 @@
+=========================
+Adding and removing hosts
+=========================
+
+This page discusses how to add and remove nodes from an existing cluster. The
+procedure differs depending on the type of nodes being added or removed, which
+services are running, and how they are configured. Here we will consider two
+types of nodes - controllers and compute nodes. Other types of nodes will need
+consideration.
+
+Any procedure being used should be tested before being applied in a production
+environment.
+
+Adding new hosts
+================
+
+.. _adding-new-controllers:
+
+Adding new controllers
+----------------------
+
+The :doc:`bootstrap-servers command
+</reference/deployment-and-bootstrapping/bootstrap-servers>` can be used to
+prepare the new hosts that are being added to the system.  It adds an entry to
+``/etc/hosts`` for the new hosts, and some services, such as RabbitMQ, require
+entries to exist for all controllers on every controller. If using a
+``--limit`` argument, ensure that all controllers are included, e.g. via
+``--limit control``. Be aware of the :ref:`potential issues <rebootstrapping>`
+with running ``bootstrap-servers`` on an existing system.
+
+.. code-block:: console
+
+   kolla-ansible -i <inventory> bootstrap-servers [ --limit <limit> ]
+
+Pull down container images to the new hosts. The ``--limit`` argument may be
+used and only needs to include the new hosts.
+
+.. code-block:: console
+
+   kolla-ansible -i <inventory> pull [ --limit <limit> ]
+
+Deploy containers to the new hosts. If using a ``--limit`` argument, ensure
+that all controllers are included, e.g. via ``--limit control``.
+
+.. code-block:: console
+
+   kolla-ansible -i <inventory> deploy [ --limit <limit> ]
+
+The new controllers are now deployed. It is recommended to perform testing
+of the control plane at this point to verify that the new controllers are
+functioning correctly.
+
+Some resources may not be automatically balanced onto the new controllers. It
+may be helpful to manually rebalance these resources onto the new controllers.
+Examples include networks hosted by Neutron DHCP agent, and routers hosted by
+Neutron L3 agent. The `removing-existing-controllers`_ section provides an
+example of how to do this.
+
+.. _adding-new-compute-nodes:
+
+Adding new compute nodes
+------------------------
+
+The :doc:`bootstrap-servers command
+</reference/deployment-and-bootstrapping/bootstrap-servers>`, can be used to
+prepare the new hosts that are being added to the system.  Be aware of the
+:ref:`potential issues <rebootstrapping>` with running ``bootstrap-servers`` on
+an existing system.
+
+.. code-block:: console
+
+   kolla-ansible -i <inventory> bootstrap-servers [ --limit <limit> ]
+
+Pull down container images to the new hosts. The ``--limit`` argument may be
+used and only needs to include the new hosts.
+
+.. code-block:: console
+
+   kolla-ansible -i <inventory> pull [ --limit <limit> ]
+
+Deploy containers on the new hosts. The ``--limit`` argument may be used and
+only needs to include the new hosts.
+
+.. code-block:: console
+
+   kolla-ansible -i <inventory> deploy [ --limit <limit> ]
+
+The new compute nodes are now deployed. It is recommended to perform
+testing of the compute nodes at this point to verify that they are functioning
+correctly.
+
+Server instances are not automatically balanced onto the new compute nodes. It
+may be helpful to live migrate some server instances onto the new hosts.
+
+.. code-block:: console
+
+   openstack server migrate <server> --live-migration --host <target host> --os-compute-api-version 2.30
+
+Alternatively, a service such as :watcher-doc:`Watcher </>` may be used to do
+this automatically.
+
+Removing existing hosts
+=======================
+
+.. _removing-existing-controllers:
+
+Removing existing controllers
+-----------------------------
+
+When removing controllers or other hosts running clustered services, consider
+whether enough hosts remain in the cluster to form a quorum. For example, in a
+system with 3 controllers, only one should be removed at a time. Consider also
+the effect this will have on redundancy.
+
+Before removing existing controllers from a cluster, it is recommended to move
+resources they are hosting. Here we will cover networks hosted by Neutron DHCP
+agent and routers hosted by Neutron L3 agent. Other actions may be necessary,
+depending on your environment and configuration.
+
+For each host being removed, find Neutron routers on that host and move them.
+Disable the L3 agent. For example:
+
+.. code-block:: console
+
+   l3_id=$(openstack network agent list --host <host> --agent-type l3 -f value -c ID)
+   target_l3_id=$(openstack network agent list --host <target host> --agent-type l3 -f value -c ID)
+   openstack router list --agent $l3_id -f value -c ID | while read router; do
+     openstack network agent remove router $l3_id $router --l3
+     openstack network agent add router $target_l3_id $router --l3
+   done
+   openstack network agent set $l3_id --disable
+
+Repeat for DHCP agents:
+
+.. code-block:: console
+
+   dhcp_id=$(openstack network agent list --host <host> --agent-type dhcp -f value -c ID)
+   target_dhcp_id=$(openstack network agent list --host <target host> --agent-type dhcp -f value -c ID)
+   openstack network list --agent $dhcp_id -f value -c ID | while read network; do
+     openstack network agent remove network $dhcp_id $network --dhcp
+     openstack network agent add network $target_dhcp_id $network --dhcp
+   done
+
+Stop all services running on the hosts being removed:
+
+.. code-block:: console
+
+   kolla-ansible -i <inventory> stop --yes-i-really-really-mean-it [ --limit <limit> ]
+
+Remove the hosts from the Ansible inventory.
+
+Reconfigure the remaining controllers to update the membership of clusters such
+as MariaDB and RabbitMQ. Use a suitable limit, such as ``--limit control``.
+
+.. code-block:: console
+
+   kolla-ansible -i <inventory> deploy [ --limit <limit> ]
+
+Perform testing to verify that the remaining cluster hosts are operating
+correctly.
+
+For each host, clean up its services:
+
+.. code-block:: console
+
+   openstack network agent list --host <host> -f value -c ID | while read id; do
+     openstack network agent delete $id
+   done
+
+   openstack compute service list --os-compute-api-version 2.53 --host <host> -f value -c ID | while read id; do
+     openstack compute service delete --os-compute-api-version 2.53 $id
+   done
+
+.. _removing-existing-compute-nodes:
+
+Removing existing compute nodes
+-------------------------------
+
+When removing compute nodes from a system, consider whether there is capacity
+to host the running workload on the remaining compute nodes. Include overhead
+for failures that may occur.
+
+Before removing compute nodes from a system, it is recommended to migrate or
+destroy any instances that they are hosting.
+
+For each host, disable the compute service to ensure that no new instances are
+scheduled to it.
+
+.. code-block:: console
+
+   openstack compute service set <host> nova-compute --disable
+
+If possible, live migrate instances to another host.
+
+.. code-block:: console
+
+   openstack server list --host <host> -f value -c ID | while read server; do
+     openstack server migrate --live-migration $server
+   done
+
+Verify that the migrations were successful.
+
+Stop all services running on the hosts being removed:
+
+.. code-block:: console
+
+   kolla-ansible -i <inventory> stop --yes-i-really-really-mean-it [ --limit <limit> ]
+
+Remove the hosts from the Ansible inventory.
+
+Perform testing to verify that the remaining cluster hosts are operating
+correctly.
+
+For each host, clean up its services:
+
+.. code-block:: console
+
+   openstack network agent list --host <host> -f value -c ID | while read id; do
+     openstack network agent delete $id
+   done
+
+   openstack compute service list --os-compute-api-version 2.53 --host <host> -f value -c ID | while read id; do
+     openstack compute service delete --os-compute-api-version 2.53 $id
+   done

doc/source/user/index.rst

@@ -11,5 +11,6 @@ User Guides
    multinode
    multi-regions
    operating-kolla
+   adding-and-removing-hosts
    security
    troubleshooting