MariaDB: avoid synchronization in state configmap updates

Each MariaDB instance updates the grastate configmap on a periodic
basis, every 10s by default. Collisions can occur when multiple
instances try to write their state at the same time (within a few
milliseconds). One instance will write successfully, and the other will
get a 409 error. There is nothing to break the synchronization, so the
failures tend to be persistent.

This change adds a small sleep after a collision is encountered,
creating an offset between the cycles.

Change-Id: Ib8a64f8f7ee15a6579e901d80ae759c38e0e901e
This commit is contained in:
Phil Sphicas 2020-03-03 13:56:41 -08:00
parent 444a5ae67f
commit 2a4b369f25

View File

@ -327,6 +327,10 @@ def safe_update_configmap(configmap_dict, configmap_patch):
# This status code indicates a collision trying to write to the
# config map while another instance is also trying the same.
logger.warning("Collision writing configmap: {0}".format(error))
# This often happens when the replicas were started at the same
# time, and tends to be persistent. Sleep briefly to break the
# synchronization.
time.sleep(1)
return True
else:
logger.error("Failed to set configmap: {0}".format(error))