openstack/openstack-ansible
Code Issues Proposed changes

Docs: Add note about slow galera recovery

Browse Source 
Rebuilding a galera container is stressful enough and deployers
may find that restarting the MariaDB daemon on a re-created
container takes a lot of time.

This patch adds a note to advise deployers about the time required to
sync data from the older nodes to the new, empty node. This will hopefully
restore some sanity and alleviate some stress.

Change-Id: Ife5ffed913e7ca21f7d003b7f09b740e1f678a21
This commit is contained in:
Major Hayden 2016-05-19 07:54:49 -05:00
parent 0984490358
commit c52755ef4b
No known key found for this signature in database
GPG Key ID: 737051E0C1011FB1
1 changed files with 16 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
doc/source/install-guide/ops-galera-recovery.rst
View File

@ -194,20 +194,20 @@ Recovering from certain failures require rebuilding one or more containers.
#. Disable the failed node on the load balancer.
.. note::
Do not rely on the load balancer health checks to disable the node.
If the node is not disabled, the load balancer sends SQL requests
to it before it rejoins the cluster and cause data inconsistencies.
#. Destroy the container and remove MariaDB data stored outside
of the container:
of the container:
.. code-block:: shell-session
# lxc-stop -n node3_galera_container-3ea2cbd3
# lxc-destroy -n node3_galera_container-3ea2cbd3
# rm -rf /openstack/node3_galera_container-3ea2cbd3/*
In this example, node 3 failed.
#. Run the host setup playbook to rebuild the container on node 3:
@ -230,7 +230,7 @@ Recovering from certain failures require rebuilding one or more containers.
.. warning::
The new container runs a single-node Galera cluster, which is a dangerous
state because the environment contains more than one active database
with potentially different data.
@ -263,6 +263,18 @@ Recovering from certain failures require rebuilding one or more containers.
#. Restart MariaDB in the new container and verify that it rejoins the
cluster.
.. note::
In larger deployments, it may take some time for the MariaDB daemon to
start in the new container. It will be synchronizing data from the other
MariaDB servers during this time. You can monitor the status during this
process by tailing the ``/var/log/mysql_logs/galera_server_error.log``
log file.
Lines starting with ``WSREP_SST`` will appear during the sync process
and you should see a line with ``WSREP: SST complete, seqno: <NUMBER>``
if the sync was successful.
.. code-block:: shell-session
# ansible galera_container -m shell -a "mysql \