========================= Adding and removing hosts ========================= This page discusses how to add and remove nodes from an existing cluster. The procedure differs depending on the type of nodes being added or removed, which services are running, and how they are configured. Here we will consider two types of nodes - controllers and compute nodes. Other types of nodes will need consideration. Any procedure being used should be tested before being applied in a production environment. Adding new hosts ================ .. _adding-new-controllers: Adding new controllers ---------------------- The :doc:`bootstrap-servers command ` can be used to prepare the new hosts that are being added to the system. It adds an entry to ``/etc/hosts`` for the new hosts, and some services, such as RabbitMQ, require entries to exist for all controllers on every controller. If using a ``--limit`` argument, ensure that all controllers are included, e.g. via ``--limit control``. Be aware of the :ref:`potential issues ` with running ``bootstrap-servers`` on an existing system. .. code-block:: console kolla-ansible -i bootstrap-servers [ --limit ] Pull down container images to the new hosts. The ``--limit`` argument may be used and only needs to include the new hosts. .. code-block:: console kolla-ansible -i pull [ --limit ] Deploy containers to the new hosts. If using a ``--limit`` argument, ensure that all controllers are included, e.g. via ``--limit control``. .. code-block:: console kolla-ansible -i deploy [ --limit ] The new controllers are now deployed. It is recommended to perform testing of the control plane at this point to verify that the new controllers are functioning correctly. Some resources may not be automatically balanced onto the new controllers. It may be helpful to manually rebalance these resources onto the new controllers. Examples include networks hosted by Neutron DHCP agent, and routers hosted by Neutron L3 agent. The `removing-existing-controllers`_ section provides an example of how to do this. .. _adding-new-compute-nodes: Adding new compute nodes ------------------------ The :doc:`bootstrap-servers command `, can be used to prepare the new hosts that are being added to the system. Be aware of the :ref:`potential issues ` with running ``bootstrap-servers`` on an existing system. .. code-block:: console kolla-ansible -i bootstrap-servers [ --limit ] Pull down container images to the new hosts. The ``--limit`` argument may be used and only needs to include the new hosts. .. code-block:: console kolla-ansible -i pull [ --limit ] Deploy containers on the new hosts. The ``--limit`` argument may be used and only needs to include the new hosts. .. code-block:: console kolla-ansible -i deploy [ --limit ] The new compute nodes are now deployed. It is recommended to perform testing of the compute nodes at this point to verify that they are functioning correctly. Server instances are not automatically balanced onto the new compute nodes. It may be helpful to live migrate some server instances onto the new hosts. .. code-block:: console openstack server migrate --live-migration --host --os-compute-api-version 2.30 Alternatively, a service such as :watcher-doc:`Watcher ` may be used to do this automatically. Removing existing hosts ======================= .. _removing-existing-controllers: Removing existing controllers ----------------------------- When removing controllers or other hosts running clustered services, consider whether enough hosts remain in the cluster to form a quorum. For example, in a system with 3 controllers, only one should be removed at a time. Consider also the effect this will have on redundancy. Before removing existing controllers from a cluster, it is recommended to move resources they are hosting. Here we will cover networks hosted by Neutron DHCP agent and routers hosted by Neutron L3 agent. Other actions may be necessary, depending on your environment and configuration. For each host being removed, find Neutron routers on that host and move them. Disable the L3 agent. For example: .. code-block:: console l3_id=$(openstack network agent list --host --agent-type l3 -f value -c ID) target_l3_id=$(openstack network agent list --host --agent-type l3 -f value -c ID) openstack router list --agent $l3_id -f value -c ID | while read router; do openstack network agent remove router $l3_id $router --l3 openstack network agent add router $target_l3_id $router --l3 done openstack network agent set $l3_id --disable Repeat for DHCP agents: .. code-block:: console dhcp_id=$(openstack network agent list --host --agent-type dhcp -f value -c ID) target_dhcp_id=$(openstack network agent list --host --agent-type dhcp -f value -c ID) openstack network list --agent $dhcp_id -f value -c ID | while read network; do openstack network agent remove network $dhcp_id $network --dhcp openstack network agent add network $target_dhcp_id $network --dhcp done Stop all services running on the hosts being removed: .. code-block:: console kolla-ansible -i stop --yes-i-really-really-mean-it [ --limit ] Remove the hosts from the Ansible inventory. Reconfigure the remaining controllers to update the membership of clusters such as MariaDB and RabbitMQ. Use a suitable limit, such as ``--limit control``. .. code-block:: console kolla-ansible -i deploy [ --limit ] Perform testing to verify that the remaining cluster hosts are operating correctly. For each host, clean up its services: .. code-block:: console openstack network agent list --host -f value -c ID | while read id; do openstack network agent delete $id done openstack compute service list --os-compute-api-version 2.53 --host -f value -c ID | while read id; do openstack compute service delete --os-compute-api-version 2.53 $id done .. _removing-existing-compute-nodes: Removing existing compute nodes ------------------------------- When removing compute nodes from a system, consider whether there is capacity to host the running workload on the remaining compute nodes. Include overhead for failures that may occur. Before removing compute nodes from a system, it is recommended to migrate or destroy any instances that they are hosting. For each host, disable the compute service to ensure that no new instances are scheduled to it. .. code-block:: console openstack compute service set nova-compute --disable If possible, live migrate instances to another host. .. code-block:: console openstack server list --all-projects --host -f value -c ID | while read server; do openstack server migrate --live-migration $server done Verify that the migrations were successful. Stop all services running on the hosts being removed: .. code-block:: console kolla-ansible -i stop --yes-i-really-really-mean-it [ --limit ] Remove the hosts from the Ansible inventory. Perform testing to verify that the remaining cluster hosts are operating correctly. For each host, clean up its services: .. code-block:: console openstack network agent list --host -f value -c ID | while read id; do openstack network agent delete $id done openstack compute service list --os-compute-api-version 2.53 --host -f value -c ID | while read id; do openstack compute service delete --os-compute-api-version 2.53 $id done