openstack/openstack-helm
Code Issues Proposed changes

Merge branch 'master' into horizon_merge

Browse Source
This commit is contained in:
Alan Meadows 2016-12-20 07:37:56 -08:00
commit 6ec92626e8
54 changed files with 805 additions and 769 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
Makefile
View File

@ -3,10 +3,10 @@
B64_DIRS := common/secrets
B64_EXCLUDE := $(wildcard common/secrets/*.b64)
CHARTS := ceph mariadb rabbitmq memcached keystone openstack
CHARTS := ceph mariadb rabbitmq GLANCE memcached keystone glance horizon openstack
COMMON_TPL := common/templates/_globals.tpl
all: common bootstrap mariadb rabbitmq memcached keystone horizon openstack
all: common bootstrap mariadb rabbitmq memcached keystone glance horizon openstack
common: build-common
@ -22,6 +22,8 @@ horizon: build-horizon
rabbitmq: build-rabbitmq
glance: build-glance
memcached: build-memcached
openstack: build-openstack

101
README.md
View File

@ -1,102 +1,7 @@
# aic-helm
This is a fully self-contained OpenStack deployment on Kubernetes. This collection is a work in progress so components will continue to be added over time.
This is a fully self-contained helm-based OpenStack deployment on Kubernetes, including baremetal provisioning, persistent storage, and resiliency considerations.
## Requirements
This collection is a work in progress so components will continue to be added over time.
The aic-helm project is fairly opinionated. We will work to generalize the configuration but since we are targeting a fully functional proof of concept end-to-end, we will have to limit the plugin like functionality within this project.
### helm
The entire aic-helm project is obviously helm driven. All components should work with 2.0.0-rc2 or later.
### baremetal provisioning
The aic-helm project assumes Canonical's MaaS as the foundational bootstrap provider. We create the MaaS service inside Kubernetes for ease of deployment and upgrades. This has a few requirements for external network connectivity to provide bootstrapping noted in the maas chart README.
### dynamic volume provisioning
At the moment, this is not optional. We will strive to make a non-1.5.0 requirement path in all charts using an alternative persistent storage approach but that currently, all charts make the assumption that dynamic volume provisioning is supported.
To support dynamic volume provisioning, the aic-helm project requires Kubernetes 1.5.0-beta1 in order to obtain rbd dynamic volume support. Although rbd volumes are supported in the stable v1.4 version, dynamic rbd volumes allowing PVCs are only supported in 1.5.0-beta.1 and beyond. Note that you can still use helm-2.0.0 with 1.5.0-beta.1, but you will not be able to use PetSets until the following helm [issue](https://github.com/kubernetes/helm/issues/1581) is resolved.
This can be accomplished with a [kubeadm](http://kubernetes.io/docs/getting-started-guides/kubeadm/) based cluster install:
```
kubeadm init --use-kubernetes-version v1.5.0-beta.1
```
Note that in addition to Kubernetes 1.5.0-beta.1, you will need to replace the kube-controller-manager container with one that supports the rbd utilities. We have made a convenient container that you can drop in as a replacement. It is an ubuntu based container with the ceph tools and the kube-controller-manager binary from the 1.5.0-beta.1 release available as a [Dockerfile](https://github.com/att-comdev/dockerfiles/tree/master/kube-controller-manager) or a quay.io image you can update in your kubeadm manifest ```/etc/kubernetes/manifests/kube-controller-manager.json``` directly with ```image: quay.io/attcomdev/kube-controller-manager```
The kubelet should pick up the change and restart the container.
For the kube-controller-manager to be able to talk to the ceph-mon instances, ensure it can resolve ceph-mon.ceph (assuming you install the ceph chart into the ceph namespace). This is done by ensuring that both the baremetal host running the kubelet process and the kube-controller-manager container have the SkyDNS address and the appropriate search string in /etc/resolv.conf. This is covered in more detail in the [ceph](ceph/README.md) but a typical resolv.conf would look like this:
```
nameserver 10.32.0.2 ### skydns instance ip
nameserver 8.8.8.8
nameserver 8.8.4.4
search svc.cluster.local
```
Finally, you need to install Sigil to help in the generation of Ceph Secrets. You can do this by running the following command as root:
```
curl -L https://github.com/gliderlabs/sigil/releases/download/v0.4.0/sigil_0.4.0_Linux_x86_64.tgz | tar -zxC /usr/local/bin
```
## QuickStart
You can start aic-helm fairly quickly. Assuming the above requirements are met, you can install the charts in a layered approach. Today, the openstack chart is only tied to the mariadb sub-chart. We will continue to add other OpenStack components into the openstack parent chart as they are validated.
Note that the openstack parent chart should always be used as it does some prepatory work for the openstack namespace for subcharts, such as ensuring ceph secrets are available to all subcharts.
```
# label all known nodes as candidates for pods
kubectl label nodes node-type=storage --all
kubectl label nodes openstack-control-plane=enabled --all
# move into the aic-helm directory
cd aic-helm
# generate secrets (ceph, etc.)
export osd_cluster_network=10.32.0.0/12
export osd_public_network=10.32.0.0/12
cd common/utils/secret-generator
./generate_secrets.sh all `./generate_secrets.sh fsid`
cd ../../..
# now you are ready to build aic-helm
helm serve . &
helm repo add local http://localhost:8879/charts
make
# install
helm install local/chef --namespace=ceph
helm install local/openstack --namespace=openstack
```
## Control Plane Charts
The following charts form the foundation to help establish an OpenStack control plane, including shared storage and bare metal provisioning:
- [ceph](ceph/README.md)
- maas (in progress)
- aic-kube (in progress)
These charts, unlike the OpenStack charts below, are designed to run directly. They form the foundational layers necessary to bootstrap an environment in may run in separate namespaces. The intention is to layer them. Please see the direct links above as they become available for README instructions crafted for each chart. Please walk through each of these as some of them require build steps that should be done before running make.
## Infrastructure Charts
- [mariadb](mariadb/README.md)
- rabbitmq (in progress)
- memcached (in progress)
## OpenStack Charts
- keystone (in progress)
The OpenStack charts under development will focus on container images leveraging the entrypoint model. This differs somewhat from the existing [openstack-helm](https://github.com/sapcc/openstack-helm) repository maintained by SAP right now although we have shamelessly "borrowed" many oncepts from them. For these charts, we will be following the same region approach as openstack-helm, namely that these charts will not install and run directly. They are included in the "openstack" chart as requirements, the openstack chart is effectively an abstract region and is intended to be required by a concrete region chart. We will provide an example region chart as well as sample region specific settings and certificate generation instructions.
Similar to openstack-helm, much of the 'make' complexity in this repository surrounds the fact that helm does not support directory based config maps or secrets. This will continue to be the case until (https://github.com/kubernetes/helm/issues/950) receives more attention.
Please see the [wiki](https://github.com/att-comdev/aic-helm/wiki) for more details.

154
common/utils/values/values.py Executable file
View File

@ -0,0 +1,154 @@
#! /usr/bin/env python
# vim: tabstop=4 shiftwidth=4 softtabstop=4
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""
Usage
$ cat /tmp/globals.yaml
global:
neener: teener
ceph:
foo: bar
coo: zoo
foo:
hoo: woo
keystone:
goo: baz
---
$ ./values.py -y /tmp/globals.yaml
2016-12-14 13:02:31,950 values INFO Processing YAML file /tmp/globals.yaml
2016-12-14 13:02:31,951 values INFO Writing out /tmp/helmVccaOA/ceph.yaml
2016-12-14 13:02:31,952 values INFO Writing out /tmp/helmVccaOA/keystone.yaml
$ cat /tmp/helmVccaOA/ceph.yaml
global:
neener: teener
coo: zoo
foo:
hoo: woo
$ cat /tmp/helmVccaOA/keystone.yaml
global:
neener: teener
goo: baz
---
$ helm install --values=/tmp/helmVccaOA/ceph.yaml ./ceph-0.1.0.tgz --namespace=ceph
"""
import yaml
import tempfile
import sys
import os
import logging
import argparse
from logging.handlers import SysLogHandler
LOG = logging.getLogger('values')
LOG_FORMAT='%(asctime)s %(name)-12s %(levelname)-8s %(message)s'
LOG_DATE = '%m-%d %H:%M'
DESCRIPTION = "Values Generator"
# special top level parameters
# we will copy to all configs
COPY_LIST=['global']
def parse_args():
ap = argparse.ArgumentParser(description=DESCRIPTION)
ap.add_argument('-d', '--debug', action='store_true',
default=False, help='Enable debugging')
ap.add_argument('-y', '--yaml', action='store',
required=True, help='Path to Master YAML File')
return ap.parse_args()
def setup_logging(args):
level = logging.DEBUG if args.debug else logging.INFO
logging.basicConfig(level=level, format=LOG_FORMAT, date_fmt=LOG_DATE)
handler = SysLogHandler(address='/dev/log')
syslog_formatter = logging.Formatter('%(name)s: %(levelname)s %(message)s')
handler.setFormatter(syslog_formatter)
LOG.addHandler(handler)
def is_path_creatable(pathname):
'''
`True` if the current user has sufficient permissions to create the passed
pathname; `False` otherwise.
'''
# Parent directory of the passed path. If empty, we substitute the current
# working directory (CWD) instead.
dirname = os.path.dirname(pathname) or os.getcwd()
return os.access(dirname, os.W_OK)
def process_yaml(yaml_path):
'''
Processes YAML file and generates breakdown of top level
keys
'''
# perform some basic validation
if not os.path.isfile(yaml_path):
LOG.exception("The path %s does not point to a valid file", yaml_path)
sys.exit(1)
tmp_dir = tempfile.mkdtemp(prefix='helm', dir='/tmp')
yamldata = yaml.load(open(yaml_path).read())
for top_level in yamldata.keys():
if top_level in COPY_LIST:
continue
top_level_path = os.path.join(tmp_dir, top_level + '.yaml')
if is_path_creatable(top_level_path):
with open(top_level_path, 'w') as f:
LOG.info("Writing out %s", top_level_path)
for item in COPY_LIST:
if item in yamldata.keys():
f.write(yaml.dump({item: yamldata[item]}, default_flow_style=False))
f.write(yaml.dump(yamldata[top_level], default_flow_style=False))
else:
LOG.exception("Unable to create path %s based on top level key in %s" % (top_level_path, yaml_path))
def run(args):
LOG.info("Processing YAML file %s", args.yaml)
process_yaml(args.yaml)
if __name__ == '__main__':
args = parse_args()
setup_logging(args)
try:
run(args)
sys.exit(0)
except Exception as err:
LOG.exception(err)
sys.exit(1)

15
glance/requirements.yaml
View File

@ -2,18 +2,3 @@ dependencies:
- name: common
repository: http://localhost:8879/charts
version: 0.1.0
- name: mariadb
repository: http://localhost:8879/charts
version: 0.1.0
- name: rabbitmq
repository: http://localhost:8879/charts
version: 0.1.0
- name: memcached
repository: http://localhost:8879/charts
version: 0.1.0
- name: keystone
repository: http://localhost:8879/charts
version: 0.1.0
- name: keystone
repository: http://localhost:8879/charts
version: 0.1.0

2
glance/templates/_helpers.tpl
View File

@ -2,4 +2,4 @@
{{ range $k, $v := . }}{{ if $k }},{{ end }}{{ $v }}{{ end }}
{{- end -}}
{{ define "keystone_auth" }}auth: "{'auth_url':'{{ .Values.keystone.auth_url }}', 'username':'{{ .Values.keystone.admin_user }}','password':'{{ .Values.keystone.admin_password }}','project_name':'{{ .Values.keystone.admin_project_name }}','domain_name':'default'}"{{end}}
{{ define "keystone_auth" }}{'auth_url':'{{ .Values.keystone.auth_url }}', 'username':'{{ .Values.keystone.admin_user }}','password':'{{ .Values.keystone.admin_password }}','project_name':'{{ .Values.keystone.admin_project_name }}','domain_name':'default'}{{end}}

3
glance/templates/api.yaml
View File

@ -13,6 +13,7 @@ spec:
{
"name": "init",
"image": "quay.io/stackanetes/kubernetes-entrypoint:v0.1.0",
"imagePullPolicy": "{{ .Values.images.pull_policy }}",
"env": [
{
"name": "NAMESPACE",
@ -39,7 +40,7 @@ spec:
containers:
- name: glance-api
image: {{ .Values.images.api }}
imagePullPolicy: Always
imagePullPolicy: {{ .Values.images.pull_policy }}
command:
- bash
- /tmp/start.sh

3
glance/templates/db-sync.yaml
View File

@ -10,6 +10,7 @@ spec:
{
"name": "init",
"image": "quay.io/stackanetes/kubernetes-entrypoint:v0.1.0",
"imagePullPolicy": "{{ .Values.images.pull_policy }}",
"env": [
{
"name": "NAMESPACE",
@ -35,7 +36,7 @@ spec:
containers:
- name: glance-db-sync
image: {{ .Values.images.db_sync }}
imagePullPolicy: Always
imagePullPolicy: {{ .Values.images.pull_policy }}
command:
- bash
- /tmp/db-sync.sh

3
glance/templates/init.yaml
View File

@ -10,6 +10,7 @@ spec:
{
"name": "init",
"image": "quay.io/stackanetes/kubernetes-entrypoint:v0.1.0",
"imagePullPolicy": "{{ .Values.images.pull_policy }}",
"env": [
{
"name": "NAMESPACE",
@ -35,7 +36,7 @@ spec:
containers:
- name: glance-init
image: {{ .Values.images.init }}
imagePullPolicy: Always
imagePullPolicy: {{ .Values.images.pull_policy }}
env:
- name: ANSIBLE_LIBRARY
value: /usr/share/ansible/

42
glance/templates/post.sh.yaml
View File

@ -6,9 +6,41 @@ data:
post.sh: |+
#!/bin/bash
set -ex
export HOME=/tmp
ansible localhost -vvv -m kolla_keystone_service -a "service_name=glance service_type=image description='Openstack Image' endpoint_region={{ .Values.keystone.glance_region_name }} url='http://glance-api:{{ .Values.network.port.api }}' interface=admin region_name={{ .Values.keystone.admin_region_name }} auth='{{ include "keystone_auth" . }}'" -e "{ 'openstack_glance_auth': {{ include "keystone_auth" . }} }"
ansible localhost -vvv -m kolla_keystone_service -a "service_name=glance service_type=image description='Openstack Image' endpoint_region={{ .Values.keystone.glance_region_name }} url='http://glance-api:{{ .Values.network.port.api }}' interface=internal region_name={{ .Values.keystone.admin_region_name }} auth='{{ include "keystone_auth" . }}'" -e "{ 'openstack_glance_auth': {{ include "keystone_auth" . }} }"
ansible localhost -vvv -m kolla_keystone_service -a "service_name=glance service_type=image description='Openstack Image' endpoint_region={{ .Values.keystone.glance_region_name }} url='http://glance-api:{{ .Values.network.port.api }}' interface=public region_name={{ .Values.keystone.admin_region_name }} auth='{{ include "keystone_auth" . }}' " -e "{ 'openstack_glance_auth': {{ include "keystone_auth" . }} }"
ansible localhost -vvv -m kolla_keystone_user -a "project=service user={{ .Values.keystone.glance_user }} password={{ .Values.keystone.glance_password }} role=admin region_name={{ .Values.keystone.admin_region_name }} auth='{{ include "keystone_auth" . }}'" -e "{'openstack_glance_auth': {{ include "keystone_auth" .}}}"
ansible localhost -vvv -m kolla_keystone_service -a 'service_name=glance \
service_type=image \
description="Openstack Image" \
endpoint_region="{{ .Values.keystone.glance_region_name }}" \
url="http://glance-api:{{ .Values.network.port.api }}" \
interface=admin \
region_name="{{ .Values.keystone.admin_region_name }}" \
auth="{# openstack_glance_auth #}"' \
-e "{ 'openstack_glance_auth': {{ include "keystone_auth" . }} }"
ansible localhost -vvv -m kolla_keystone_service -a 'service_name=glance \
service_type=image \
description="Openstack Image" \
endpoint_region="{{ .Values.keystone.glance_region_name }}" \
url="http://glance-api:{{ .Values.network.port.api }}" \
interface=internal \
region_name="{{ .Values.keystone.admin_region_name }}" \
auth="{# openstack_glance_auth #}"' \
-e "{ 'openstack_glance_auth': {{ include "keystone_auth" . }} }"
ansible localhost -vvv -m kolla_keystone_service -a 'service_name=glance \
service_type=image \
description="Openstack Image" \
endpoint_region="{{ .Values.keystone.glance_region_name }}" \
url="http://glance-api:{{ .Values.network.port.api }}" \
interface=public \
region_name="{{ .Values.keystone.admin_region_name }}" \
auth="{# openstack_glance_auth #}"' \
-e "{ 'openstack_glance_auth': {{ include "keystone_auth" . }} }"
ansible localhost -vvv -m kolla_keystone_user -a "project=service \
user={{ .Values.keystone.glance_user }} \
password={{ .Values.keystone.glance_password }} \
role=admin \
region_name={{ .Values.keystone.admin_region_name }} \
auth="{# openstack_glance_auth #}"' \
-e "{ 'openstack_glance_auth': {{ include "keystone_auth" . }} }"

3
glance/templates/post.yaml
View File

@ -10,6 +10,7 @@ spec:
{
"name": "init",
"image": "quay.io/stackanetes/kubernetes-entrypoint:v0.1.0",
"imagePullPolicy": "{{ .Values.images.pull_policy }}",
"env": [
{
"name": "NAMESPACE",
@ -35,7 +36,7 @@ spec:
containers:
- name: glance-post
image: {{ .Values.images.post }}
imagePullPolicy: Always
imagePullPolicy: {{ .Values.images.pull_policy }}
command:
- bash
- /tmp/post.sh

5
glance/templates/registry.yaml
View File

@ -13,6 +13,7 @@ spec:
{
"name": "init",
"image": "quay.io/stackanetes/kubernetes-entrypoint:v0.1.0",
"imagePullPolicy": "{{ .Values.images.pull_policy }}",
"env": [
{
"name": "NAMESPACE",
@ -39,9 +40,9 @@ spec:
containers:
- name: glance-registry
image: {{ .Values.images.registry }}
imagePullPolicy: Always
imagePullPolicy: {{ .Values.images.pull_policy }}
command:
- glance-registr
- glance-registry
ports:
- containerPort: {{ .Values.network.port.registry }}
readinessProbe:

3
glance/values.yaml
View File

@ -15,6 +15,7 @@ images:
init: quay.io/stackanetes/stackanetes-kolla-toolbox:newton
registry: quay.io/stackanetes/stackanetes-glance-registry:newton
post: quay.io/stackanetes/stackanetes-kolla-toolbox:newton
pull_policy: "IfNotPresent"
keystone:
auth_uri: "http://keystone-api:5000"
@ -91,7 +92,7 @@ dependencies:
- glance-db-sync
- keystone-db-sync
- keystone-init
- mariadb-init
- mariadb-seed
service:
- mariadb
- keystone-api

8
keystone/templates/bin/_start.sh.tpl Normal file
View File

@ -0,0 +1,8 @@
#!/bin/bash
set -ex
# Loading Apache2 ENV variables
source /etc/apache2/envvars
# start apache with any container arguments
apache2 -DFOREGROUND $*

2
keystone/templates/bin-configmap.yaml → keystone/templates/configmap-bin.yaml
View File

@ -7,3 +7,5 @@ data:
{{ tuple "bin/_db-sync.sh.tpl" . | include "template" | indent 4 }}
init.sh: |
{{ tuple "bin/_init.sh.tpl" . | include "template" | indent 4 }}
start.sh: |
{{ tuple "bin/_start.sh.tpl" . | include "template" | indent 4 }}

2
keystone/templates/etc-configmap.yaml → keystone/templates/configmap-etc.yaml
View File

@ -5,7 +5,7 @@ metadata:
data:
keystone.conf: |+
{{ tuple "etc/_keystone.conf.tpl" . | include "template" | indent 4 }}
mpm-event.conf: |+
mpm_event.conf: |+
{{ tuple "etc/_mpm_event.conf.tpl" . | include "template" | indent 4 }}
wsgi-keystone.conf: |+
{{ tuple "etc/_wsgi-keystone.conf.tpl" . | include "template" | indent 4 }}

24
keystone/templates/deployment.yaml
View File

@ -13,6 +13,7 @@ spec:
{
"name": "init",
"image": "{{ .Values.images.entrypoint }}",
"imagePullPolicy": "{{ .Values.images.pull_policy }}",
"env": [
{
"name": "NAMESPACE",
@ -43,9 +44,10 @@ spec:
containers:
- name: keystone-api
image: {{ .Values.images.api }}
imagePullPolicy: {{ .Values.images.pull_policy }}
command:
- apache2
- -DFOREGROUND
- bash
- /tmp/start.sh
ports:
- containerPort: {{ .Values.network.port.public }}
- containerPort: {{ .Values.network.port.admin }}
@ -58,26 +60,24 @@ spec:
subPath: keystone.conf
- name: wsgikeystone
mountPath: /etc/apache2/conf-enabled/wsgi-keystone.conf
subPath: wsgi_keystone.conf
subPath: wsgi-keystone.conf
- name: mpmeventconf
mountPath: /etc/apache2/mods-available/mpm_event.conf
subPath: mpm_event.conf
- name: startsh
mountPath: /tmp/start.sh
subPath: start.sh
volumes:
- name: keystoneconf
configMap:
name: keystone-etc
items:
- key: keystone.conf
path: keystone.conf
- name: wsgikeystone
configMap:
name: keystone-etc
items:
- key: wsgi-keystone.conf
path: wsgi_keystone.conf
- name: mpmeventconf
configMap:
name: keystone-etc
items:
- key: mpm-event.conf
path: mpm_event.conf
- name: startsh
configMap:
name: keystone-bin

3
keystone/templates/job-db-sync.yaml
View File

@ -10,6 +10,7 @@ spec:
{
"name": "init",
"image": "{{ .Values.images.entrypoint }}",
"imagePullPolicy": "{{ .Values.images.pull_policy }}",
"env": [
{
"name": "NAMESPACE",
@ -35,7 +36,7 @@ spec:
containers:
- name: keystone-db-sync
image: {{ .Values.images.db_sync }}
imagePullPolicy: Always
imagePullPolicy: {{ .Values.images.pull_policy }}
command:
- bash
- /tmp/db-sync.sh

3
keystone/templates/job-init.yaml
View File

@ -10,6 +10,7 @@ spec:
{
"name": "init",
"image": "{{ .Values.images.entrypoint }}",
"imagePullPolicy": "{{ .Values.images.pull_policy }}",
"env": [
{
"name": "NAMESPACE",
@ -35,7 +36,7 @@ spec:
containers:
- name: keystone-init
image: {{ .Values.images.init }}
imagePullPolicy: Always
imagePullPolicy: {{ .Values.images.pull_policy }}
command:
- bash
- /tmp/init.sh

1
keystone/values.yaml
View File

@ -14,6 +14,7 @@ images:
api: quay.io/stackanetes/stackanetes-keystone-api:newton
init: quay.io/stackanetes/stackanetes-kolla-toolbox:newton
entrypoint: quay.io/stackanetes/kubernetes-entrypoint:v0.1.0
pull_policy: "IfNotPresent"
keystone:
version: v2.0

14
mariadb/README.md
View File

@ -2,12 +2,18 @@
By default, this chart creates a 3-member mariadb galera cluster.
PetSets would be ideal to use for this purpose, but as they are going through a transition in 1.5.0-beta.1 and not supported by Helm 2.0.0 under their new name, StatefulSets, we have opted to leverage helms template generation ability to create Values.replicas * POD+PVC+PV resources. Essentially, we create a mariadb-0, mariadb-1, and mariadb-2 Pod and associated unique PersistentVolumeClaims for each. This removes the previous daemonset limitations in other mariadb approaches.
This chart leverages StatefulSets, with persistent storage.
You must ensure that your control nodes that should receive mariadb instances are labeled with openstack-control-plane=enabled:
It creates a job that acts as a temporary standalone galera cluster. This host is bootstrapped with authentication and then the WSREP bindings are exposed publicly. The cluster members being StatefulSets are provisioned one at a time. The first host must be marked as ```Ready``` before the next host will be provisioned. This is determined by the readinessProbes which actually validate that MySQL is up and responsive.
The configuration leverages xtrabackup-v2 for synchronization. This may later be augmented to leverage rsync which has some benefits.
Once the seed job completes, which completes only when galera reports that it is Synced and all cluster members are reporting in thus matching the cluster count according to the job to the replica count in the helm values configuration, the job is terminated. When the job is no longer active, future StatefulSets provisioned will leverage the existing cluster members as gcomm endpoints. It is only when the job is running that the cluster members leverage the seed job as their gcomm endpoint. This ensures you can restart members and scale the cluster.
The StatefulSets all leverage PVCs to provide stateful storage to /var/lib/mysql.
You must ensure that your control nodes that should receive mariadb instances are labeled with openstack-control-plane=enabled, or whatever you have configured in values.yaml for the label configuration:
```
kubectl label nodes openstack-control-plane=enabled --all
```
We will continue to refine our labeling so that it is consistent throughout the project.

53
mariadb/templates/bin/_bootstrap-db.sh.tpl Normal file
View File

@ -0,0 +1,53 @@
#!/bin/sh
set -ex
SLEEP_TIMEOUT=5
# Initialize system .Values.database.
mysql_install_db --datadir=/var/lib/mysql
# Start mariadb and wait for it to be ready.
#
# note that we bind to 127.0.0.1 here because we want
# to interact with the database but we dont want to expose it
# yet for other cluster members to accidently connect yet
mysqld_safe --defaults-file=/etc/my.cnf \
--console \
--wsrep-new-cluster \
--wsrep_cluster_address='gcomm://' \
--bind-address='127.0.0.1' \
--wsrep_node_address="127.0.0.1:{{ .Values.network.port.wsrep }}" \
--wsrep_provider_options="gcache.size=512M; gmcast.listen_addr=tcp://127.0.0.1:{{ .Values.network.port.wsrep }}" &
TIMEOUT=120
while [[ ! -f /var/lib/mysql/mariadb.pid ]]; do
if [[ ${TIMEOUT} -gt 0 ]]; then
let TIMEOUT-=1
sleep 1
else
exit 1
fi
done
# Reset permissions.
# kolla_security_reset requires to be run from home directory
cd /var/lib/mysql ; DB_ROOT_PASSWORD="{{ .Values.database.root_password }}" kolla_security_reset
mysql -u root --password="{{ .Values.database.root_password }}" --port="{{ .Values.network.port.mariadb }}" -e "GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' IDENTIFIED BY '{{ .Values.database.root_password }}' WITH GRANT OPTION;"
mysql -u root --password="{{ .Values.database.root_password }}" --port="{{ .Values.network.port.mariadb }}" -e "GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '{{ .Values.database.root_password }}' WITH GRANT OPTION;"
# Restart .Values.database.
mysqladmin -uroot -p"{{ .Values.database.root_password }}" --port="{{ .Values.network.port.mariadb }}" shutdown
# Wait for the mariadb server to shut down
SHUTDOWN_TIMEOUT=60
while [[ -f /var/lib/mysql/mariadb.pid ]]; do
if [[ ${SHUTDOWN_TIMEOUT} -gt 0 ]]; then
let SHUTDOWN_TIMEOUT-=1
sleep 1
else
echo "MariaDB instance couldn't be properly shut down"
exit 1
fi
done

91
mariadb/templates/bin/_peer-finder.py.tpl Normal file
View File

@ -0,0 +1,91 @@
#!/usr/bin/env python
import json
import os
import urllib2
import ssl
import socket
import sys
import time
URL = ('https://kubernetes.default.svc.cluster.local/api/v1/namespaces/{namespace}'
'/endpoints/{service_name}')
TOKEN_FILE = '/var/run/secrets/kubernetes.io/serviceaccount/token'
def get_service_endpoints(service_name):
url = URL.format(namespace=os.environ['NAMESPACE'], service_name=service_name)
try:
token = file (TOKEN_FILE, 'r').read()
except KeyError:
exit("Unable to open a file with token.")
header = {'Authorization': " Bearer {}".format(token)}
req = urllib2.Request(url=url, headers=header)
ctx = create_ctx()
connection = urllib2.urlopen(req, context=ctx)
data = connection.read()
# parse to dict
json_acceptable_string = data.replace("'", "\"")
output = json.loads(json_acceptable_string)
return output
def get_ip_addresses(output, force_only_members=False):
subsets = output['subsets'][0]
if not 'addresses' in subsets:
return []
# where we are seeding, the only cluster member is the seed job
if not force_only_members:
for subset in subsets['addresses']:
if subset.has_key('name'):
if 'seed' in subset['name']:
return [subset['ip']]
# otherwise use the other cluster members
ip_addresses = [x['ip'] for x in subsets['addresses']]
my_ip = get_my_ip_address()
if my_ip in ip_addresses:
ip_addresses.remove(my_ip)
return ip_addresses
def get_my_ip_address():
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(('kubernetes.default.svc.cluster.local', 0))
return s.getsockname()[0]
def create_ctx():
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
return ctx
def print_galera_cluster_address(service_name, force_only_members):
while True:
output = get_service_endpoints(service_name)
ips = get_ip_addresses(output, force_only_members)
#print "=== OUTPUT: %s" % output
#print "=== IPS: %s" % ips
if len(ips):
wsrep_cluster_address = '--wsrep_cluster_address=gcomm://{}'.format(
','.join(get_ip_addresses(output)))
print wsrep_cluster_address
break
time.sleep(5)
def main():
if len(sys.argv) != 3:
exit('peer-finder: You need to pass argument <service name> <1|0 for force cluster members>')
service_name = sys.argv[1]
force_only_members = int(sys.argv[2])
print_galera_cluster_address(service_name, force_only_members)
if __name__ == '__main__':
main()

27
mariadb/templates/bin/_readiness.py.tpl Normal file
View File

@ -0,0 +1,27 @@
#!/usr/bin/env python
import os
import sys
import time
import pymysql
DB_HOST = "127.0.0.1"
DB_PORT = int(os.environ.get('MARIADB_SERVICE_PORT', '3306'))
while True:
try:
pymysql.connections.Connection(host=DB_HOST, port=DB_PORT,
connect_timeout=1)
sys.exit(0)
except pymysql.err.OperationalError as e:
code, message = e.args
if code == 2003 and 'time out' in message:
print('Connection timeout, sleeping')
time.sleep(1)
continue
if code == 1045:
print('Mysql ready to use. Exiting')
sys.exit(0)
# other error
raise

79
mariadb/templates/bin/_seed.sh.tpl Normal file
View File

@ -0,0 +1,79 @@
#!/bin/sh
set -ex
SLEEP_TIMEOUT=5
function wait_for_cluster {
# Wait for the mariadb server to be "Ready" before starting the security reset with a max timeout
TIMEOUT=600
while [[ ! -f /var/lib/mysql/mariadb.pid ]]; do
if [[ ${TIMEOUT} -gt 0 ]]; then
let TIMEOUT-=1
sleep 1
else
exit 1
fi
done
REPLICAS={{ .Values.replicas }}
# We need to count seed instance here.
MINIMUM_CLUSTER_SIZE=$(( $REPLICAS + 1 ))
# wait until we have at least two more members in a cluster.
while true ; do
CLUSTER_SIZE=`mysql -uroot -h ${POD_IP} -p"{{ .Values.database.root_password }}" --port="{{ .Values.network.port.mariadb }}" -e'show status' | grep wsrep_cluster_size | awk ' { if($2 ~ /[0-9]/){ print $2 } else { print 0 } } '`
if [ "${CLUSTER_SIZE}" -lt ${MINIMUM_CLUSTER_SIZE} ] ; then
echo "Cluster seed not finished, waiting."
sleep ${SLEEP_TIMEOUT}
continue
fi
CLUSTER_STATUS=`mysql -uroot -h ${POD_IP} -p"{{ .Values.database.root_password }}" --port="{{ .Values.network.port.mariadb }}" -e'show status' | grep wsrep_local_state_comment | awk ' { print $2 } '`
if [ "${CLUSTER_STATUS}" != "Synced" ] ; then
echo "Cluster not synced, waiting."
sleep ${SLEEP_TIMEOUT}
continue
fi
# Count number of endpoint separators.
ENDPOINTS_CNT=`python /tmp/peer-finder.py mariadb 1 | grep -o ',' | wc -l`
# TODO(tomasz.paszkowski): Fix a corner case when only one endpoint is on the list.
# Add +1 for seed node and +1 as first item does not have a separator
ENDPOINTS_CNT=$(($ENDPOINTS_CNT+2))
if [ "${ENDPOINTS_CNT}" != "${CLUSTER_SIZE}" ] ; then
echo "Cluster not synced, waiting."
sleep ${SLEEP_TIMEOUT}
continue
fi
echo "Cluster ready, exiting seed."
kill -- -$$
break
done
}
# With the DaemonSet implementation, there may be a difference
# in the number of replicas and actual number of nodes matching
# mariadb node selector label. Problem will be solved when
# the implementation will be switched to Deployment
# (using anti-affinity feature).
REPLICAS={{ .Values.replicas }}
if [ "$REPLICAS" -eq 1 ] ; then
echo "Requested to build one-instance MariaDB cluster. There is no need to run seed. Exiting."
exit 0
elif [ "$REPLICAS" -eq 2 ] ; then
echo "2-instance cluster is not a valid MariaDB configuration."
exit 1
fi
. /tmp/bootstrap-db.sh
mysqld_safe --defaults-file=/etc/my.cnf \
--console \
--wsrep-new-cluster \
--wsrep_cluster_address='gcomm://' \
--bind-address="0.0.0.0" \
--wsrep_node_address="${POD_IP}:{{ .Values.network.port.wsrep }}" \
--wsrep_provider_options="gcache.size=512M; gmcast.listen_addr=tcp://${POD_IP}:{{ .Values.network.port.wsrep }}" &
wait_for_cluster
exit 0

48
mariadb/templates/bin/_start.sh.tpl Normal file
View File

@ -0,0 +1,48 @@
#!/bin/bash
set -ex
trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT
sudo chown mysql: /var/lib/mysql
rm -rf /var/lib/mysql/lost+found
REPLICAS={{ .Values.replicas }}
PETSET_NAME={{ printf "%s" .Values.service_name }}
INIT_MARKER="/var/lib/mysql/init_done"
function join_by { local IFS="$1"; shift; echo "$*"; }
# Remove mariadb.pid if exists
if [[ -f /var/lib/mysql/mariadb.pid ]]; then
if [[ `pgrep -c $(cat /var/lib/mysql/mariadb.pid)` -eq 0 ]]; then
rm -vf /var/lib/mysql/mariadb.pid
fi
fi
if [ "$REPLICAS" -eq 1 ] ; then
if [[ ! -f ${INIT_MARKER} ]]; then
cd /var/lib/mysql
echo "Creating one-instance MariaDB."
bash /tmp/bootstrap-db.sh
touch ${INIT_MARKER}
fi
exec mysqld_safe --defaults-file=/etc/my.cnf \
--console \
--wsrep-new-cluster \
--wsrep_cluster_address='gcomm://'
else
# give the seed more of a chance to be ready by the time
# we start the first pet so we succeed on the first pass
# a little hacky, but prevents restarts as we aren't waiting
# for job completion here so I'm not sure what else
# to look for
sleep 30
export WSREP_OPTIONS=`python /tmp/peer-finder.py mariadb 0`
exec mysqld --defaults-file=/etc/my.cnf \
--console \
--bind-address="0.0.0.0" \
--wsrep_node_address="${POD_IP}:{{ .Values.network.port.wsrep }}" \
--wsrep_provider_options="gcache.size=512M; gmcast.listen_addr=tcp://${POD_IP}:{{ .Values.network.port.wsrep }}" \
$WSREP_OPTIONS
fi

56
mariadb/templates/bootstrap-db.sh.yaml
View File

@ -1,56 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: bootstrap-db
data:
bootstrap-db.sh: |
#!/bin/sh
set -ex
SLEEP_TIMEOUT=5
# Initialize system .Values.database.
mysql_install_db --datadir=/var/lib/mysql
# Start mariadb and wait for it to be ready.
mysqld_safe --defaults-file=/etc/my.cnf \
--console \
--wsrep-new-cluster \
--wsrep_cluster_address='gcomm://' \
--bind-address='127.0.0.1' \
--wsrep_node_address='127.0.0.1' \
--wsrep_provider_options='gcache.size=512M; gmcast.listen_addr=tcp://127.0.0.1:{{ .Values.network.port.wsrep }}' &
TIMEOUT=120
while [[ ! -f /var/lib/mysql/mariadb.pid ]]; do
if [[ ${TIMEOUT} -gt 0 ]]; then
let TIMEOUT-=1
sleep 1
else
exit 1
fi
done
# Reset permissions.
# kolla_security_reset requires to be run from home directory
cd /var/lib/mysql ; DB_ROOT_PASSWORD="{{ .Values.database.root_password }}" kolla_security_reset
mysql -u root --password="{{ .Values.database.root_password }}" --port="{{ .Values.network.port.mariadb }}" -e "GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' IDENTIFIED BY '{{ .Values.database.root_password }}' WITH GRANT OPTION;"
mysql -u root --password="{{ .Values.database.root_password }}" --port="{{ .Values.network.port.mariadb }}" -e "GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '{{ .Values.database.root_password }}' WITH GRANT OPTION;"
# Restart .Values.database.
mysqladmin -uroot -p"{{ .Values.database.root_password }}" --port="{{ .Values.network.port.mariadb }}" shutdown
# Wait for the mariadb server to shut down
SHUTDOWN_TIMEOUT=60
while [[ -f /var/lib/mysql/mariadb.pid ]]; do
if [[ ${SHUTDOWN_TIMEOUT} -gt 0 ]]; then
let SHUTDOWN_TIMEOUT-=1
sleep 1
else
echo "MariaDB instance couldn't be properly shut down"
exit 1
fi
done

13
mariadb/templates/charsets.cnf.yaml
View File

@ -1,13 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: mariadb-charsets
data:
charsets.cnf: |+
[mysqld]
character_set_server=utf8
collation_server=utf8_unicode_ci
skip-character-set-client-handshake
[client]
default_character_set=utf8

15
mariadb/templates/configmap-bin.yaml Normal file
View File

@ -0,0 +1,15 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: mariadb-bin
data:
start.sh: |
{{ tuple "bin/_start.sh.tpl" . | include "template" | indent 4 }}
peer-finder.py: |
{{ tuple "bin/_peer-finder.py.tpl" . | include "template" | indent 4 }}
readiness.py: |
{{ tuple "bin/_readiness.py.tpl" . | include "template" | indent 4 }}
bootstrap-db.sh: |
{{ tuple "bin/_bootstrap-db.sh.tpl" . | include "template" | indent 4 }}
seed.sh: |
{{ tuple "bin/_seed.sh.tpl" . | include "template" | indent 4 }}

21
mariadb/templates/configmap-etc.yaml Normal file
View File

@ -0,0 +1,21 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: mariadb-etc
data:
charsets.cnf: |
{{ tuple "etc/_charsets.cnf.tpl" . | include "template" | indent 4 }}
engine.cnf: |
{{ tuple "etc/_engine.cnf.tpl" . | include "template" | indent 4 }}
my.cnf: |
{{ tuple "etc/_galera-my.cnf.tpl" . | include "template" | indent 4 }}
log.cnf: |
{{ tuple "etc/_log.cnf.tpl" . | include "template" | indent 4 }}
pid.cnf: |
{{ tuple "etc/_pid.cnf.tpl" . | include "template" | indent 4 }}
tuning.cnf: |
{{ tuple "etc/_tuning.cnf.tpl" . | include "template" | indent 4 }}
networking.cnf: |
{{ tuple "etc/_networking.cnf.tpl" . | include "template" | indent 4 }}
wsrep.cnf: |
{{ tuple "etc/_wsrep.cnf.tpl" . | include "template" | indent 4 }}

169
mariadb/templates/mariadb-deployment.yaml → mariadb/templates/deployment.yaml
View File

@ -1,57 +1,17 @@
{{- $root := . -}}
{{ range $k, $v := until (atoi .Values.replicas) }}
---
apiVersion: v1
kind: Service
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: mariadb-{{$v}}
labels:
release: {{ $root.Release.Name | quote }}
chart: "{{ $root.Chart.Name }}-{{ $root.Chart.Version }}"
name: {{ .Values.service_name }}
spec:
ports:
- name: db
port: {{ $root.Values.network.port.mariadb }}
- name: wsrep
port: {{ $root.Values.network.port.wsrep }}
selector:
app: mariadb
server-id: "{{$v}}"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: mariadb-{{$v}}
annotations:
{{ $root.Values.volume.class_path }}: {{ $root.Values.volume.class_name }}
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: {{ $root.Values.volume.size }}
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
labels:
app: mariadb
galera: enabled
server-id: "{{$v}}"
name: mariadb-{{$v}}
spec:
replicas: 1
serviceName: "{{ .Values.service_name }}"
replicas: 3
template:
securityContext:
runAsUser: 0
metadata:
name: mariadb-{{$v}}
labels:
app: mariadb
app: {{ .Values.service_name }}
galera: enabled
server-id: "{{$v}}"
annotations:
pod.beta.kubernetes.io/hostname: mariadb-{{$v}}
helm.sh/created: {{ $root.Release.Time.Seconds | quote }}
# alanmeadows: this soft requirement allows single
# host deployments to spawn several mariadb containers
# but in a larger environment, would attempt to spread
@ -71,13 +31,13 @@ spec:
"weight": 10
}]
}
}
}
spec:
nodeSelector:
{{ $root.Values.labels.node_selector_key }}: {{ $root.Values.labels.node_selector_value }}
{{ .Values.labels.node_selector_key }}: {{ .Values.labels.node_selector_value }}
containers:
- name: mariadb-{{$v}}
image: {{ $root.Values.images.mariadb }}
- name: {{ .Values.service_name }}
image: {{ .Values.images.mariadb }}
imagePullPolicy: Always
env:
- name: INTERFACE_NAME
@ -86,22 +46,30 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: COMMAND
value: "bash /tmp/start.sh"
- name: DEPENDENCY_CONFIG
value: "/etc/my.cnf.d/wsrep.cnf"
ports:
- containerPort: {{ $root.Values.network.port.mariadb }}
- containerPort: {{ $root.Values.network.port.wsrep }}
- containerPort: {{ .Values.network.port.mariadb }}
- containerPort: {{ .Values.network.port.wsrep }}
- containerPort: {{ .Values.network.port.ist }}
# a readinessprobe is a much more complex affair with
# statefulsets, as the container must be "live"
# before the next stateful member is created
# and with galera this is problematic
readinessProbe:
exec:
command:
- python
- /mariadb-readiness.py
initialDelaySeconds: 60
volumeMounts:
- name: mycnfd
mountPath: /etc/my.cnf.d
@ -111,9 +79,12 @@ spec:
- name: bootstrapdb
mountPath: /tmp/bootstrap-db.sh
subPath: bootstrap-db.sh
- name: peer-finder
- name: peerfinder
mountPath: /tmp/peer-finder.py
subPath: peer-finder.py
- name: readiness
mountPath: /mariadb-readiness.py
subPath: readiness.py
- name: charsets
mountPath: /etc/my.cnf.d/charsets.cnf
subPath: charsets.cnf
@ -136,60 +107,56 @@ spec:
mountPath: /etc/my.cnf.d/tuning.cnf
subPath: tuning.cnf
- name: wsrep
mountPath: /configmaps/wsrep.cnf
- name: replicas
mountPath: /tmp/replicas.py
subPath: replicas.py
- name: readiness
mountPath: /mariadb-readiness.py
subPath: mariadb-readiness.py
mountPath: /etc/my.cnf.d/wsrep.cnf
subPath: wsrep.cnf
- name: mysql-data
mountPath: /var/lib/mysql
mountPath: /var/lib/mysql
volumes:
- name: mycnfd
emptyDir: {}
- name: startsh
configMap:
name: mariadb-startsh
name: mariadb-bin
- name: bootstrapdb
configMap:
name: bootstrap-db
- name: peer-finder
name: mariadb-bin
- name: peerfinder
configMap:
name: mariadb-peer-finder
- name: charsets
configMap:
name: mariadb-charsets
- name: engine
configMap:
name: mariadb-engine
- name: log
configMap:
name: mariadb-log
- name: mycnf
configMap:
name: mariadb-mycnf
- name: networking
configMap:
name: mariadb-networking
- name: pid
configMap:
name: mariadb-pid
- name: tuning
configMap:
name: mariadb-tuning
- name: wsrep
configMap:
name: mariadb-wsrep
- name: replicas
configMap:
name: mariadb-replicas
name: mariadb-bin
- name: readiness
configMap:
name: mariadb-readiness
- name: mysql-data
persistentVolumeClaim:
matchLabels:
server-id: "{{$v}}"
claimName: mariadb-{{$v}}
{{ end }}
name: mariadb-bin
- name: charsets
configMap:
name: mariadb-etc
- name: engine
configMap:
name: mariadb-etc
- name: log
configMap:
name: mariadb-etc
- name: mycnf
configMap:
name: mariadb-etc
- name: networking
configMap:
name: mariadb-etc
- name: pid
configMap:
name: mariadb-etc
- name: tuning
configMap:
name: mariadb-etc
- name: wsrep
configMap:
name: mariadb-etc
volumeClaimTemplates:
- metadata:
name: mysql-data
annotations:
{{ .Values.volume.class_path }}: {{ .Values.volume.class_name }}
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: {{ .Values.volume.size }}

10
mariadb/templates/engine.cnf.yaml
View File

@ -1,10 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: mariadb-engine
data:
engine.cnf: |+
[mysqld]
default-storage-engine=InnoDB
innodb=FORCE
binlog_format=ROW

7
mariadb/templates/etc/_charsets.cnf.tpl Normal file
View File

@ -0,0 +1,7 @@
[mysqld]
character_set_server=utf8
collation_server=utf8_unicode_ci
skip-character-set-client-handshake
[client]
default_character_set=utf8

4
mariadb/templates/etc/_engine.cnf.tpl Normal file
View File

@ -0,0 +1,4 @@
[mysqld]
default-storage-engine=InnoDB
innodb=FORCE
binlog_format=ROW

6
mariadb/templates/etc/_galera-my.cnf.tpl Normal file
View File

@ -0,0 +1,6 @@
[mysqld]
datadir=/var/lib/mysql
basedir=/usr
[client-server]
!includedir /etc/my.cnf.d/

11
mariadb/templates/etc/_log.cnf.tpl Normal file
View File

@ -0,0 +1,11 @@
[mysqld]
slow_query_log=off
slow_query_log_file=/var/log/mysql/mariadb-slow.log
log_warnings=2
# General logging has huge performance penalty therefore is disabled by default
general_log=off
general_log_file=/var/log/mysql/mariadb-error.log
long_query_time=3
log_queries_not_using_indexes=on

14
mariadb/templates/etc/_networking.cnf.tpl Normal file
View File

@ -0,0 +1,14 @@
[mysqld]
bind_address=0.0.0.0
port={{ .Values.network.port.mariadb }}
# When a client connects, the server will perform hostname resolution,
# and when DNS is slow, establishing the connection will become slow as well.
# It is therefore recommended to start the server with skip-name-resolve to
# disable all DNS lookups. The only limitation is that the GRANT statements
# must then use IP addresses only.
skip_name_resolve
[client]
protocol=tcp
port={{ .Values.network.port.mariadb }}

2
mariadb/templates/etc/_pid.cnf.tpl Normal file
View File

@ -0,0 +1,2 @@
[mysqld]
pid_file=/var/lib/mysql/mariadb.pid

47
mariadb/templates/etc/_tuning.cnf.tpl Normal file
View File

@ -0,0 +1,47 @@
[mysqld]
user=mysql
max_allowed_packet=256M
open_files_limit=10240
max_connections=8192
max-connect-errors=1000000
## Generally, it is unwise to set the query cache to be larger than 64-128M
## as the costs associated with maintaining the cache outweigh the performance
## gains.
## The query cache is a well known bottleneck that can be seen even when
## concurrency is moderate. The best option is to disable it from day 1
## by setting query_cache_size=0 (now the default on MySQL 5.6)
## and to use other ways to speed up read queries: good indexing, adding
## replicas to spread the read load or using an external cache.
query_cache_size =0
query_cache_type=0
sync_binlog=0
thread_cache_size=16
table_open_cache=2048
table_definition_cache=1024
#
# InnoDB
#
# The buffer pool is where data and indexes are cached: having it as large as possible
# will ensure you use memory and not disks for most read operations.
# Typical values are 50..75% of available RAM.
# TODO(tomasz.paszkowski): This needs to by dynamic based on avaliable RAM.
innodb_buffer_pool_size=4096M
innodb_log_file_size=2000M
innodb_flush_method=O_DIRECT
innodb_flush_log_at_trx_commit=2
innodb_old_blocks_time=1000
innodb_autoinc_lock_mode=2
innodb_doublewrite=0
innodb_file_format=Barracuda
innodb_file_per_table=1
innodb_io_capacity=500
innodb_locks_unsafe_for_binlog=1
innodb_read_io_threads=8
innodb_write_io_threads=8
[mysqldump]
max-allowed-packet=16M

15
mariadb/templates/etc/_wsrep.cnf.tpl Normal file
View File

@ -0,0 +1,15 @@
[mysqld]
wsrep_cluster_name="{{ .Values.database.cluster_name }}"
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_provider_options="gcache.size=512M"
wsrep_slave_threads=12
wsrep_sst_auth=root:{{ .Values.database.root_password }}
# xtrabackup-v2 would be more desirable here, but its
# not in the upstream stackanetes images
# ()[mysql@mariadb-seed-gdqr8 /]$ xtrabackup --version
# xtrabackup version 2.2.13 based on MySQL server 5.6.24 Linux (x86_64) (revision id: 70f4be3)
wsrep_sst_method=xtrabackup-v2
wsrep_node_name={{ .Values.database.node_name }}
datadir=/var/lib/mysql
tmpdir=/tmp
user=mysql

12
mariadb/templates/galera-my.cnf.yaml
View File

@ -1,12 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: mariadb-mycnf
data:
my.cnf: |+
[mysqld]
datadir=/var/lib/mysql
basedir=/usr
[client-server]
!includedir /etc/my.cnf.d/

45
mariadb/templates/mariadb-seed-job.yaml → mariadb/templates/job-seed.yaml
View File

@ -1,3 +1,4 @@
---
apiVersion: batch/v1
kind: Job
metadata:
@ -9,7 +10,6 @@ spec:
app: mariadb
spec:
restartPolicy: Never
terminationGracePeriodSeconds: 30
containers:
- name: mariadb-init
image: {{ .Values.images.mariadb }}
@ -21,16 +21,20 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: COMMAND
value: "bash /tmp/seed.sh"
- name: DEPENDENCY_CONFIG
value: "/etc/my.cnf.d/wsrep.cnf"
ports:
- containerPort: {{ .Values.network.port.mariadb }}
- containerPort: {{ .Values.network.port.wsrep }}
- containerPort: {{ .Values.network.port.ist }}
volumeMounts:
- name: mycnfd
mountPath: /etc/my.cnf.d
@ -40,7 +44,7 @@ spec:
- name: bootstrapdb
mountPath: /tmp/bootstrap-db.sh
subPath: bootstrap-db.sh
- name: peer-finder
- name: peerfinder
mountPath: /tmp/peer-finder.py
subPath: peer-finder.py
- name: charsets
@ -65,46 +69,41 @@ spec:
mountPath: /etc/my.cnf.d/tuning.cnf
subPath: tuning.cnf
- name: wsrep
mountPath: /configmaps/wsrep.cnf
- name: replicas
mountPath: /tmp/replicas.py
subPath: replicas.py
mountPath: /etc/my.cnf.d/wsrep.cnf
subPath: wsrep.cnf
volumes:
- name: mycnfd
emptyDir: {}
- name: seedsh
configMap:
name: mariadb-seedsh
name: mariadb-bin
- name: bootstrapdb
configMap:
name: bootstrap-db
- name: peer-finder
name: mariadb-bin
- name: peerfinder
configMap:
name: mariadb-peer-finder
name: mariadb-bin
- name: charsets
configMap:
name: mariadb-charsets
name: mariadb-etc
- name: engine
configMap:
name: mariadb-engine
name: mariadb-etc
- name: log
configMap:
name: mariadb-log
name: mariadb-etc
- name: mycnf
configMap:
name: mariadb-mycnf
name: mariadb-etc
- name: networking
configMap:
name: mariadb-networking
name: mariadb-etc
- name: pid
configMap:
name: mariadb-pid
name: mariadb-etc
- name: tuning
configMap:
name: mariadb-tuning
name: mariadb-etc
- name: wsrep
configMap:
name: mariadb-wsrep
- name: replicas
configMap:
name: mariadb-replicas
name: mariadb-etc

17
mariadb/templates/log.cnf.yaml
View File

@ -1,17 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: mariadb-log
data:
log.cnf: |+
[mysqld]
slow_query_log=off
slow_query_log_file=/var/log/mysql/mariadb-slow.log
log_warnings=2
# General logging has huge performance penalty therefore is disabled by default
general_log=off
general_log_file=/var/log/mysql/mariadb-error.log
long_query_time=3
log_queries_not_using_indexes=on

33
mariadb/templates/mariadb-readiness.py.yaml
View File

@ -1,33 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: mariadb-readiness
data:
mariadb-readiness.py: |+
#!/usr/bin/env python
import os
import sys
import time
import pymysql
DB_HOST = "127.0.0.1"
DB_PORT = int(os.environ.get('MARIADB_SERVICE_PORT', '3306'))
while True:
try:
pymysql.connections.Connection(host=DB_HOST, port=DB_PORT,
connect_timeout=1)
sys.exit(0)
except pymysql.err.OperationalError as e:
code, message = e.args
if code == 2003 and 'time out' in message:
print('Connection timeout, sleeping')
time.sleep(1)
continue
if code == 1045:
print('Mysql ready to use. Exiting')
sys.exit(0)
# other error
raise

10
mariadb/templates/mariadb-service.yaml
View File

@ -1,10 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: mariadb
spec:
ports:
- name: db
port: {{ .Values.network.port.mariadb }}
selector:
app: mariadb

20
mariadb/templates/networking.cnf.yaml
View File

@ -1,20 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: mariadb-networking
data:
networking.cnf: |+
[mysqld]
bind_address=0.0.0.0
port={{ .Values.network.port.mariadb }}
# When a client connects, the server will perform hostname resolution,
# and when DNS is slow, establishing the connection will become slow as well.
# It is therefore recommended to start the server with skip-name-resolve to
# disable all DNS lookups. The only limitation is that the GRANT statements
# must then use IP addresses only.
skip_name_resolve
[client]
protocol=tcp
port={{ .Values.network.port.mariadb }}

84
mariadb/templates/peer-finder.py.yaml
View File

@ -1,84 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: mariadb-peer-finder
data:
peer-finder.py: |+
import json
import os
import urllib2
import ssl
import socket
import sys
import time
URL = ('https://kubernetes.default.svc.cluster.local/api/v1/namespaces/{namespace}'
'/endpoints/{service_name}')
TOKEN_FILE = '/var/run/secrets/kubernetes.io/serviceaccount/token'
def get_service_endpoints(service_name):
url = URL.format(namespace=os.environ['NAMESPACE'], service_name=service_name)
try:
token = file (TOKEN_FILE, 'r').read()
except KeyError:
exit("Unable to open a file with token.")
header = {'Authorization': " Bearer {}".format(token)}
req = urllib2.Request(url=url, headers=header)
ctx = create_ctx()
connection = urllib2.urlopen(req, context=ctx)
data = connection.read()
# parse to dict
json_acceptable_string = data.replace("'", "\"")
output = json.loads(json_acceptable_string)
return output
def get_ip_addresses(output):
subsets = output['subsets'][0]
if not 'addresses' in subsets:
return []
ip_addresses = [x['ip'] for x in subsets['addresses']]
my_ip = get_my_ip_address()
if my_ip in ip_addresses:
ip_addresses.remove(my_ip)
return ip_addresses
def get_my_ip_address():
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(('kubernetes.default.svc.cluster.local', 0))
return s.getsockname()[0]
def create_ctx():
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
return ctx
def print_galera_cluster_address(service_name):
while True:
output = get_service_endpoints(service_name)
if len(get_ip_addresses(output)):
wsrep_cluster_address = '--wsrep_cluster_address=gcomm://{}'.format(
','.join(get_ip_addresses(output)))
print wsrep_cluster_address
break
time.sleep(5)
def main():
if len(sys.argv) != 2:
exit('peer-finder: You need to pass argument')
service_name = sys.argv[1]
print_galera_cluster_address(service_name)
if __name__ == '__main__':
main()

8
mariadb/templates/pid.cnf.yaml
View File

@ -1,8 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: mariadb-pid
data:
pid.cnf: |+
[mysqld]
pid_file=/var/lib/mysql/mariadb.pid

46
mariadb/templates/replicas.py.yaml
View File

@ -1,46 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: mariadb-replicas
data:
replicas.py: |
#!/usr/bin/env python
import json
import os
import ssl
import sys
import urllib2
URL = ('https://kubernetes.default.svc.{{ .Values.network.dns.kubernetes_domain }}/apis/extensions/v1beta1/deployments')
TOKEN_FILE = '/var/run/secrets/kubernetes.io/serviceaccount/token'
def create_ctx():
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
return ctx
def get_deployments():
url = URL.format()
try:
token = file(TOKEN_FILE, 'r').read()
except KeyError:
exit("Unable to open a file with token.")
header = {'Authorization': " Bearer {}".format(token)}
req = urllib2.Request(url=url, headers=header)
ctx = create_ctx()
response = urllib2.urlopen(req, context=ctx)
output = json.load(response)
return output
def main():
reply = get_deployments()
name = "mariadb"
namespace = "default" if not os.environ["NAMESPACE"] else os.environ["NAMESPACE"]
mariadb = filter(lambda d: d["metadata"]["namespace"] == namespace and d["metadata"]["name"].startswith(name), reply["items"])
print len(mariadb)
if __name__ == "__main__":
main()

82
mariadb/templates/seed.sh.yaml
View File

@ -1,82 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: mariadb-seedsh
data:
seed.sh: |+
#!/bin/sh
set -ex
SLEEP_TIMEOUT=5
function wait_for_cluster {
# Wait for the mariadb server to be "Ready" before starting the security reset with a max timeout
TIMEOUT=120
while [[ ! -f /var/lib/mysql/mariadb.pid ]]; do
if [[ ${TIMEOUT} -gt 0 ]]; then
let TIMEOUT-=1
sleep 1
else
exit 1
fi
done
REPLICAS=$(python /tmp/replicas.py)
# We need to count seed instance here.
MINIMUM_CLUSTER_SIZE=$(( $REPLICAS + 1 ))
# wait until we have at least two more members in a cluster.
while true ; do
CLUSTER_SIZE=`mysql -uroot -p"{{ .Values.database.root_password }}" --port="{{ .Values.network.port.mariadb }}" -e'show status' | grep wsrep_cluster_size | awk ' { if($2 ~ /[0-9]/){ print $2 } else { print 0 } } '`
if [ "${CLUSTER_SIZE}" -lt ${MINIMUM_CLUSTER_SIZE} ] ; then
echo "Cluster seed not finished, waiting."
sleep ${SLEEP_TIMEOUT}
continue
fi
CLUSTER_STATUS=`mysql -uroot -p"{{ .Values.database.root_password }}" --port="{{ .Values.network.port.mariadb }}" -e'show status' | grep wsrep_local_state_comment | awk ' { print $2 } '`
if [ "${CLUSTER_STATUS}" != "Synced" ] ; then
echo "Cluster not synced, waiting."
sleep ${SLEEP_TIMEOUT}
continue
fi
# Count number of endpoint separators.
ENDPOINTS_CNT=`python /tmp/peer-finder.py mariadb | grep -o ',' | wc -l`
# TODO(tomasz.paszkowski): Fix a corner case when only one endpoint is on the list.
# Add +1 for seed node and +1 as first item does not have a separator
ENDPOINTS_CNT=$(($ENDPOINTS_CNT+2))
if [ "${ENDPOINTS_CNT}" != "${CLUSTER_SIZE}" ] ; then
echo "Cluster not synced, waiting."
sleep ${SLEEP_TIMEOUT}
continue
fi
echo "Cluster ready, exiting seed."
kill -- -$$
break
done
}
# With the DaemonSet implementation, there may be a difference
# in the number of replicas and actual number of nodes matching
# mariadb node selector label. Problem will be solved when
# the implementation will be switched to Deployment
# (using anti-affinity feature).
REPLICAS=$(python /tmp/replicas.py)
if [ "$REPLICAS" -eq 1 ] ; then
echo "Requested to build one-instance MariaDB cluster. There is no need to run seed. Exiting."
exit 0
elif [ "$REPLICAS" -eq 2 ] ; then
echo "2-instance cluster is not a valid MariaDB configuration."
exit 1
fi
bash /tmp/bootstrap-db.sh
mysqld_safe --defaults-file=/etc/my.cnf \
--console \
--wsrep-new-cluster \
--wsrep_cluster_address='gcomm://' &
wait_for_cluster
exit 0

18
mariadb/templates/service.yaml Normal file
View File

@ -0,0 +1,18 @@
apiVersion: v1
kind: Service
metadata:
name: {{ .Values.service_name }}
annotations:
# This is needed to make the peer-finder work properly and to help avoid
# edge cases where instance 0 comes up after losing its data and needs to
# decide whether it should create a new cluster or try to join an existing
# one. If it creates a new cluster when it should have joined an existing
# one, we'd end up with two separate clusters listening at the same service
# endpoint, which would be very bad.
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
spec:
ports:
- name: db
port: {{ .Values.network.port.mariadb }}
selector:
app: {{ .Values.service_name }}

37
mariadb/templates/start.sh.yaml
View File

@ -1,37 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: mariadb-startsh
data:
start.sh: |+
#!/bin/bash
set -ex
trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT
sudo chown mysql: /var/lib/mysql
REPLICAS=$(python /tmp/replicas.py)
INIT_MARKER="/var/lib/mysql/init_done"
# Remove mariadb.pid if exists
if [[ -f /var/lib/mysql/mariadb.pid ]]; then
if [[ `pgrep -c $(cat /var/lib/mysql/mariadb.pid)` -eq 0 ]]; then
rm -vf /var/lib/mysql/mariadb.pid
fi
fi
if [ "$REPLICAS" -eq 1 ] ; then
if [[ ! -f ${INIT_MARKER} ]]; then
cd /var/lib/mysql
echo "Creating one-instance MariaDB."
bash /tmp/bootstrap-db.sh
touch ${INIT_MARKER}
fi
exec mysqld_safe --defaults-file=/etc/my.cnf \
--console \
--wsrep-new-cluster \
--wsrep_cluster_address='gcomm://'
else
export WSREP_OPTIONS=`python /tmp/peer-finder.py mariadb`
exec mysqld --defaults-file=/etc/my.cnf --console $WSREP_OPTIONS
fi

53
mariadb/templates/tuning.cnf.yaml
View File

@ -1,53 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: mariadb-tuning
data:
tuning.cnf: |+
[mysqld]
user=mysql
max_allowed_packet=256M
open_files_limit=10240
max_connections=8192
max-connect-errors=1000000
## Generally, it is unwise to set the query cache to be larger than 64-128M
## as the costs associated with maintaining the cache outweigh the performance
## gains.
## The query cache is a well known bottleneck that can be seen even when
## concurrency is moderate. The best option is to disable it from day 1
## by setting query_cache_size=0 (now the default on MySQL 5.6)
## and to use other ways to speed up read queries: good indexing, adding
## replicas to spread the read load or using an external cache.
query_cache_size =0
query_cache_type=0
sync_binlog=0
thread_cache_size=16
table_open_cache=2048
table_definition_cache=1024
#
# InnoDB
#
# The buffer pool is where data and indexes are cached: having it as large as possible
# will ensure you use memory and not disks for most read operations.
# Typical values are 50..75% of available RAM.
# TODO(tomasz.paszkowski): This needs to by dynamic based on avaliable RAM.
innodb_buffer_pool_size=4096M
innodb_log_file_size=2000M
innodb_flush_method=O_DIRECT
innodb_flush_log_at_trx_commit=2
innodb_old_blocks_time=1000
innodb_autoinc_lock_mode=2
innodb_doublewrite=0
innodb_file_format=Barracuda
innodb_file_per_table=1
innodb_io_capacity=500
innodb_locks_unsafe_for_binlog=1
innodb_read_io_threads=8
innodb_write_io_threads=8
[mysqldump]
max-allowed-packet=16M

15
mariadb/templates/wsrep.cnf.yaml
View File

@ -1,15 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: mariadb-wsrep
data:
wsrep.cnf: |+
[mysqld]
wsrep_cluster_name="{{ .Values.database.cluster_name }}"
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_provider_options="gcache.size=512M"
wsrep_slave_threads=12
wsrep_sst_auth=root:{{ .Values.database.root_password }}
wsrep_sst_method=xtrabackup-v2
wsrep_node_name={{ .Values.database.node_name }}
wsrep_node_address={{ .Values.network.ip_address }}:{{ .Values.network.port.wsrep }}

9
mariadb/values.yaml
View File

@ -3,7 +3,12 @@
# Declare name/value pairs to be passed into your templates.
# name: value
replicas: "3" # this must be quoted to deal with atoi
# note that you need to update the gcomm member list
# below when changing this value
replicas: 3
# this drives the service name, and statefulset name
service_name: mariadb
images:
mariadb: quay.io/stackanetes/stackanetes-mariadb:newton
@ -21,9 +26,9 @@ network:
port:
wsrep: 4567
mariadb: 3306
ist: 4444
dns:
kubernetes_domain: cluster.local
ip_address: "{{ .IP }}"
database:
root_password: password