7a12374003
This patch does the following: - Move the CONTRIBUTING doc content into the developer documentation - Move the development-stack doc content into the developer documentation - Rename all instances of 'OpenStack Ansible' to 'OpenStack-Ansible' - Implements a simpler navigation index for the bottom of the page - Implements a navigation index at the bottom of all developer docs - Updates the README to reflect where to get documentation and further information - Adds more information into the 'scripts' developer doc - Updates the 'Additional Resources' Appendix with Ansible information Change-Id: Ic08ba072f3b3d1e4e446d1e81c20c0cdf01833de
3.6 KiB
Home OpenStack-Ansible Installation Guide
Rebuilding a container
Sometimes recovering from a failure requires rebuilding one or more containers.
Disable the failed node on the load balancer.
Do not rely on the load balancer health checks to disable the node. If the node is not disabled, the load balancer will send SQL requests to it before it rejoins the cluster and cause data inconsistencies.
Use the following commands to destroy the container and remove MariaDB data stored outside of the container. In this example, node 3 failed.
$ lxc-stop -n node3_galera_container-3ea2cbd3 $ lxc-destroy -n node3_galera_container-3ea2cbd3 $ rm -rf /openstack/node3_galera_container-3ea2cbd3/*Run the host setup playbook to rebuild the container specifically on node 3:
$ openstack-ansible setup-hosts.yml -l node3 \ -l node3_galera_container-3ea2cbd3The playbook will also restart all other containers on the node.
Run the infrastructure playbook to configure the container specifically on node 3:
$ openstack-ansible infrastructure-setup.yml \ -l node3_galera_container-3ea2cbd3The new container runs a single-node Galera cluster, a dangerous state because the environment contains more than one active database with potentially different data.
$ ansible galera_container -m shell -a "mysql \ -h localhost -e 'show status like \"%wsrep_cluster_%\";'" node3_galera_container-3ea2cbd3 | success | rc=0 >> Variable_name Value wsrep_cluster_conf_id 1 wsrep_cluster_size 1 wsrep_cluster_state_uuid da078d01-29e5-11e4-a051-03d896dbdb2d wsrep_cluster_status Primary node2_galera_container-49a47d25 | success | rc=0 >> Variable_name Value wsrep_cluster_conf_id 4 wsrep_cluster_size 2 wsrep_cluster_state_uuid 338b06b0-2948-11e4-9d06-bef42f6c52f1 wsrep_cluster_status Primary node4_galera_container-76275635 | success | rc=0 >> Variable_name Value wsrep_cluster_conf_id 4 wsrep_cluster_size 2 wsrep_cluster_state_uuid 338b06b0-2948-11e4-9d06-bef42f6c52f1 wsrep_cluster_status PrimaryRestart MariaDB in the new container and verify that it rejoins the cluster.
$ ansible galera_container -m shell -a "mysql \ -h localhost -e 'show status like \"%wsrep_cluster_%\";'" node2_galera_container-49a47d25 | success | rc=0 >> Variable_name Value wsrep_cluster_conf_id 5 wsrep_cluster_size 3 wsrep_cluster_state_uuid 338b06b0-2948-11e4-9d06-bef42f6c52f1 wsrep_cluster_status Primary node3_galera_container-3ea2cbd3 | success | rc=0 >> Variable_name Value wsrep_cluster_conf_id 5 wsrep_cluster_size 3 wsrep_cluster_state_uuid 338b06b0-2948-11e4-9d06-bef42f6c52f1 wsrep_cluster_status Primary node4_galera_container-76275635 | success | rc=0 >> Variable_name Value wsrep_cluster_conf_id 5 wsrep_cluster_size 3 wsrep_cluster_state_uuid 338b06b0-2948-11e4-9d06-bef42f6c52f1 wsrep_cluster_status PrimaryEnable the failed node on the load balancer.