openstack/openstack-helm
Code Issues Proposed changes
d78f8e0901
openstack-helm/mariadb
History
Marcus d78f8e0901 Improve fault tolerance of MariaDB startup 
* Changed podManagementPolicy to parallel in order to allow recovery
  from secondary or third master

  After rebooting the whole cluster on purpose or after a power failure
  a primary node the cluster can synchronize with is required. This is
  usually done automatically by selecting the node with the highest
  transaction id. The current implementation of the stateful set starts
  the nodes in sequence, preventing the start of further nodes if the
  process terminates with error state. Because of this, the cluster may
  not come up if the first or second node are not in primary state.

* Elects first node started in primary state as bootstrap source.
* Display warnings and runs mysqld with wsrep-recover on crashed nodes
* Introduces FORCE_RECOVERY argument for crash recovery

  In case the primary selection failed, the cluster bootstrap process
  must be manually initiated from the most advanced node (highest
  committed transaction id). This information is available from the
  grastate.dat file in case of a clean shutdown. On crashed nodes an
  InnoDB recovery is required to get the last committed transaction id.

  start.sh will handle both cases and gives instructions on how to
  recover the cluster on a hard failure. If FORCE_RECOVERY was set to
  the name of a POD (mariadb-0, mariadb-1, ...), the bootstrap process
  will be initiated from the specified node.

DocImpact
Closes-Bug: #1716461

Change-Id: I96a8cb52124f64920a7d9cf21a8924ede78ebf7b
2018-02-13 11:30:26 +00:00
..
templates Improve fault tolerance of MariaDB startup 2018-02-13 11:30:26 +00:00
.helmignore Adding MariaDB 2016-11-18 16:34:36 +01:00
Chart.yaml Mariadb: Update to use endpoint values layout and lookups 2017-07-03 13:17:48 +00:00
README.rst Update remaining markdown docs to RST 2017-05-22 10:56:51 +02:00
requirements.yaml Rename common chart, and update all references to functions within it. (#167) 2017-02-14 16:52:38 -08:00
values.yaml Add prometheus exporters to osh charts 2018-02-07 15:55:49 -06:00

README.rst

openstack-helm/mariadb

By default, this chart creates a 3-member mariadb galera cluster.

This chart leverages StatefulSets, with persistent storage.

It creates a job that acts as a temporary standalone galera cluster. This host is bootstrapped with authentication and then the WSREP bindings are exposed publicly. The cluster members being StatefulSets are provisioned one at a time. The first host must be marked as Ready before the next host will be provisioned. This is determined by the readinessProbes which actually validate that MySQL is up and responsive.

The configuration leverages xtrabackup-v2 for synchronization. This may later be augmented to leverage rsync which has some benefits.

Once the seed job completes, which completes only when galera reports that it is Synced and all cluster members are reporting in thus matching the cluster count according to the job to the replica count in the helm values configuration, the job is terminated. When the job is no longer active, future StatefulSets provisioned will leverage the existing cluster members as gcomm endpoints. It is only when the job is running that the cluster members leverage the seed job as their gcomm endpoint. This ensures you can restart members and scale the cluster.

The StatefulSets all leverage PVCs to provide stateful storage to /var/lib/mysql.

You must ensure that your control nodes that should receive mariadb instances are labeled with openstack-control-plane=enabled, or whatever you have configured in values.yaml for the label configuration:

kubectl label nodes openstack-control-plane=enabled --all