openstack/openstack-helm
Code Issues Proposed changes

Remove hacked OSLO genconfig tool

Browse Source 
This PS removes the pregenerated config templates producing using
the hacked oslo-genconfig tool. This results in both a much smaller
codebase and also more readable configuration by removing the
requirement to specify settings via oslo namespaced references.

This initial PS applies only to Keystone, A follow up will extend to
all remaining services.

Partially implements: blueprint remove-pregenerated-config-templates
See: https://blueprints.launchpad.net/openstack-helm/+spec/remove-pregenerated-config-templates

Change-Id: I3ced7ad02c703c767925a17b1a18f6158a878e83
This commit is contained in:
Pete Birley 2017-09-14 09:50:07 -06:00
parent ae4da1e7ef
commit 5e8be5c339
21 changed files with 234 additions and 4336 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
doc/source/devref/index.rst
View File

@ -9,4 +9,5 @@ Contents:
pod-disruption-budgets
images
endpoints
oslo-config
upgrades

99
doc/source/devref/oslo-config.rst Normal file
View File

@ -0,0 +1,99 @@
OSLO-Config Values
------------------
OpenStack-Helm generates oslo-config compatible formatted configuration files for
services dynamically from values specified in a yaml tree. This allows operators to
control any and all aspects of an OpenStack services configuration. An example
snippet for an imaginary Keystone configuration is described here:
::
conf:
keystone:
DEFAULT: # Keys at this level are used for section headings
max_token_size: 255
token:
provider: fernet
fernet_tokens:
key_repository: /etc/keystone/fernet-keys/
credential:
key_repository: /etc/keystone/credential-keys/
database:
max_retries: -1
cache:
enabled: true
backend: dogpile.cache.memcached
oslo_messaging_notifications:
driver: # An example of a multistring option's syntax
type: multistring
values:
- messagingv2
- log
security_compliance:
password_expires_ignore_user_ids:
# Values in a list will be converted to a comma seperated key
- "123"
- "456"
This will be consumed by the templated ``configmap-etc.yaml`` manifest to
produce the following config file:
::
---
# Source: keystone/templates/configmap-etc.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: keystone-etc
data:
keystone.conf: |+
[DEFAULT]
max_token_size = 255
transport_url = rabbit://keystone:password@rabbitmq.default.svc.cluster.local:5672/openstack
[cache]
backend = dogpile.cache.memcached
enabled = true
memcache_servers = memcached.default.svc.cluster.local:11211
[credential]
key_repository = /etc/keystone/credential-keys/
[database]
connection = mysql+pymysql://keystone:password@mariadb.default.svc.cluster.local:3306/keystone
max_retries = -1
[fernet_tokens]
key_repository = /etc/keystone/fernet-keys/
[oslo_messaging_notifications]
driver = messagingv2
driver = log
[security_compliance]
password_expires_ignore_user_ids = 123,456
[token]
provider = fernet
Note that some additional values have been injected into the config file, this is
performed via statements in the configmap template, which also calls the
``helm-toolkit.utils.to_oslo_conf`` to convert the yaml to the required layout:
::
{{- if empty .Values.conf.keystone.database.connection -}}
{{- tuple "oslo_db" "internal" "user" "mysql" . | include "helm-toolkit.endpoints.authenticated_endpoint_uri_lookup"| set .Values.conf.keystone.database "connection" | quote | trunc 0 -}}
{{- end -}}
{{- if empty .Values.conf.keystone.DEFAULT.transport_url -}}
{{- tuple "oslo_messaging" "internal" "user" "amqp" . | include "helm-toolkit.endpoints.authenticated_endpoint_uri_lookup" | set .Values.conf.keystone.DEFAULT "transport_url" | quote | trunc 0 -}}
{{- end -}}
{{- if empty .Values.conf.keystone.cache.memcache_servers -}}
{{- tuple "oslo_cache" "internal" "memcache" . | include "helm-toolkit.endpoints.host_and_port_endpoint_uri_lookup" | set .Values.conf.keystone.cache "memcache_servers" | quote | trunc 0 -}}
{{- end -}}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: keystone-etc
data:
keystone.conf: |+
{{ include "helm-toolkit.utils.to_oslo_conf" .Values.conf.keystone | indent 4 }}
{{- end }}

2
helm-toolkit/templates/utils/_to_ini.tpl
View File

@ -20,7 +20,7 @@ limitations under the License.
[{{ $section }}]
{{range $key, $value := $values -}}
{{- if kindIs "slice" $value -}}
{{ $key }} = {{ include "helm-toolkit.joinListWithComma" $value }}
{{ $key }} = {{ include "helm-toolkit.utils.joinListWithComma" $value }}
{{else -}}
{{ $key }} = {{ $value }}
{{end -}}

36
helm-toolkit/templates/utils/_to_oslo_conf.tpl Normal file
View File

@ -0,0 +1,36 @@
{{/*
Copyright 2017 The Openstack-Helm Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/}}
{{- define "helm-toolkit.utils.to_oslo_conf" -}}
{{- range $section, $values := . -}}
{{- if kindIs "map" $values -}}
[{{ $section }}]
{{ range $key, $value := $values -}}
{{- if kindIs "slice" $value -}}
{{ $key }} = {{ include "helm-toolkit.utils.joinListWithComma" $value }}
{{ else if kindIs "map" $value -}}
{{- if eq $value.type "multistring" }}
{{- range $k, $multistringValue := $value.values -}}
{{ $key }} = {{ $multistringValue }}
{{ end -}}
{{- end -}}
{{- else -}}
{{ $key }} = {{ $value }}
{{ end -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{- end -}}

18
keystone/templates/configmap-etc.yaml
View File

@ -17,18 +17,16 @@ limitations under the License.
{{- if .Values.manifests.configmap_etc }}
{{- $envAll := . }}
{{- include "keystone.conf.keystone_values_skeleton" .Values.conf.keystone | trunc 0 -}}
{{- if empty .Values.conf.keystone.database.oslo.db.connection -}}
{{- tuple "oslo_db" "internal" "user" "mysql" . | include "helm-toolkit.endpoints.authenticated_endpoint_uri_lookup"| set .Values.conf.keystone.database.oslo.db "connection" | quote | trunc 0 -}}
{{- if empty .Values.conf.keystone.database.connection -}}
{{- tuple "oslo_db" "internal" "user" "mysql" . | include "helm-toolkit.endpoints.authenticated_endpoint_uri_lookup"| set .Values.conf.keystone.database "connection" | quote | trunc 0 -}}
{{- end -}}
{{- if empty .Values.conf.keystone.default.oslo.messaging.transport_url -}}
{{- tuple "oslo_messaging" "internal" "user" "amqp" . | include "helm-toolkit.endpoints.authenticated_endpoint_uri_lookup" | set .Values.conf.keystone.default.oslo.messaging "transport_url" | quote | trunc 0 -}}
{{- if empty .Values.conf.keystone.DEFAULT.transport_url -}}
{{- tuple "oslo_messaging" "internal" "user" "amqp" . | include "helm-toolkit.endpoints.authenticated_endpoint_uri_lookup" | set .Values.conf.keystone.DEFAULT "transport_url" | quote | trunc 0 -}}
{{- end -}}
{{- if empty .Values.conf.keystone.cache.oslo.cache.memcache_servers -}}
{{- tuple "oslo_cache" "internal" "memcache" . | include "helm-toolkit.endpoints.host_and_port_endpoint_uri_lookup" | set .Values.conf.keystone.cache.oslo.cache "memcache_servers" | quote | trunc 0 -}}
{{- if empty .Values.conf.keystone.cache.memcache_servers -}}
{{- tuple "oslo_cache" "internal" "memcache" . | include "helm-toolkit.endpoints.host_and_port_endpoint_uri_lookup" | set .Values.conf.keystone.cache "memcache_servers" | quote | trunc 0 -}}
{{- end -}}
---
@ -40,9 +38,9 @@ data:
rally_tests.yaml: |+
{{- tuple .Values.conf.rally_tests "etc/_rally_tests.yaml.tpl" . | include "helm-toolkit.utils.configmap_templater" }}
keystone.conf: |+
{{- tuple .Values.conf.keystone "etc/_keystone.conf.tpl" . | include "helm-toolkit.utils.configmap_templater" }}
{{ include "helm-toolkit.utils.to_oslo_conf" .Values.conf.keystone | indent 4 }}
keystone-paste.ini: |+
{{- tuple .Values.conf.paste "etc/_keystone-paste.ini.tpl" . | include "helm-toolkit.utils.configmap_templater" }}
{{ include "helm-toolkit.utils.to_ini" .Values.conf.paste | indent 4 }}
policy.json: |+
{{ toJson .Values.conf.policy | indent 4 }}
mpm_event.conf: |+

2
keystone/templates/cron-job-credential-rotate.yaml
View File

@ -52,7 +52,7 @@ spec:
- name: KUBERNETES_NAMESPACE
value: {{ .Release.Namespace | quote }}
- name: KEYSTONE_KEYS_REPOSITORY
value: {{ .Values.conf.keystone.credential.keystone.key_repository | quote }}
value: {{ .Values.conf.keystone.credential.key_repository | quote }}
- name: KEYSTONE_CREDENTIAL_MIGRATE_WAIT
value: {{ .Values.jobs.credential_rotate.migrate_wait | quote }}
command:

4
keystone/templates/cron-job-fernet-rotate.yaml
View File

@ -15,7 +15,7 @@ limitations under the License.
*/}}
{{- if .Values.manifests.cron_fernet_rotate }}
{{- if and (eq .Values.conf.keystone.token.keystone.provider "fernet") (.Capabilities.APIVersions.Has "batch/v2alpha1") }}
{{- if and (eq .Values.conf.keystone.token.provider "fernet") (.Capabilities.APIVersions.Has "batch/v2alpha1") }}
{{- $envAll := . }}
{{- $dependencies := .Values.dependencies.fernet_rotate }}
{{- $mounts_keystone_fernet_rotate := .Values.pod.mounts.keystone_fernet_rotate.keystone_fernet_rotate }}
@ -52,7 +52,7 @@ spec:
- name: KUBERNETES_NAMESPACE
value: {{ .Release.Namespace | quote }}
- name: KEYSTONE_KEYS_REPOSITORY
value: {{ .Values.conf.keystone.fernet_tokens.keystone.key_repository | quote }}
value: {{ .Values.conf.keystone.fernet_tokens.key_repository | quote }}
command:
- python
- /tmp/fernet-manage.py

8
keystone/templates/deployment-api.yaml
View File

@ -97,12 +97,12 @@ spec:
mountPath: /tmp/keystone-api.sh
subPath: keystone-api.sh
readOnly: true
{{- if eq .Values.conf.keystone.token.keystone.provider "fernet" }}
{{- if eq .Values.conf.keystone.token.provider "fernet" }}
- name: keystone-fernet-keys
mountPath: {{ .Values.conf.keystone.fernet_tokens.keystone.key_repository }}
mountPath: {{ .Values.conf.keystone.fernet_tokens.key_repository }}
{{- end }}
- name: keystone-credential-keys
mountPath: {{ .Values.conf.keystone.credential.keystone.key_repository }}
mountPath: {{ .Values.conf.keystone.credential.key_repository }}
{{ if $mounts_keystone_api.volumeMounts }}{{ toYaml $mounts_keystone_api.volumeMounts | indent 10 }}{{ end }}
volumes:
- name: etckeystone
@ -117,7 +117,7 @@ spec:
configMap:
name: keystone-bin
defaultMode: 0555
{{- if eq .Values.conf.keystone.token.keystone.provider "fernet" }}
{{- if eq .Values.conf.keystone.token.provider "fernet" }}
- name: keystone-fernet-keys
secret:
secretName: keystone-fernet-keys

105
keystone/templates/etc/_keystone-paste.ini.tpl
View File

@ -1,105 +0,0 @@
{{/*
Copyright 2017 The Openstack-Helm Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/}}
# Keystone PasteDeploy configuration file.
[filter:debug]
use = egg:oslo.middleware#debug
[filter:request_id]
use = egg:oslo.middleware#request_id
[filter:build_auth_context]
use = egg:keystone#build_auth_context
[filter:token_auth]
use = egg:keystone#token_auth
[filter:json_body]
use = egg:keystone#json_body
[filter:cors]
use = egg:oslo.middleware#cors
oslo_config_project = keystone
[filter:http_proxy_to_wsgi]
use = egg:oslo.middleware#http_proxy_to_wsgi
[filter:ec2_extension]
use = egg:keystone#ec2_extension
[filter:ec2_extension_v3]
use = egg:keystone#ec2_extension_v3
[filter:s3_extension]
use = egg:keystone#s3_extension
[filter:url_normalize]
use = egg:keystone#url_normalize
[filter:sizelimit]
use = egg:oslo.middleware#sizelimit
[filter:osprofiler]
use = egg:osprofiler#osprofiler
[app:public_service]
use = egg:keystone#public_service
[app:service_v3]
use = egg:keystone#service_v3
[app:admin_service]
use = egg:keystone#admin_service
[pipeline:public_api]
# The last item in this pipeline must be public_service or an equivalent
# application. It cannot be a filter.
pipeline = cors sizelimit http_proxy_to_wsgi osprofiler url_normalize request_id build_auth_context token_auth json_body ec2_extension public_service
[pipeline:admin_api]
# The last item in this pipeline must be admin_service or an equivalent
# application. It cannot be a filter.
pipeline = cors sizelimit http_proxy_to_wsgi osprofiler url_normalize request_id build_auth_context token_auth json_body ec2_extension s3_extension admin_service
[pipeline:api_v3]
# The last item in this pipeline must be service_v3 or an equivalent
# application. It cannot be a filter.
pipeline = cors sizelimit http_proxy_to_wsgi osprofiler url_normalize request_id build_auth_context token_auth json_body ec2_extension_v3 s3_extension service_v3
[app:public_version_service]
use = egg:keystone#public_version_service
[app:admin_version_service]
use = egg:keystone#admin_version_service
[pipeline:public_version_api]
pipeline = cors sizelimit osprofiler url_normalize public_version_service
[pipeline:admin_version_api]
pipeline = cors sizelimit osprofiler url_normalize admin_version_service
[composite:main]
use = egg:Paste#urlmap
/v2.0 = public_api
/v3 = api_v3
/ = public_version_api
[composite:admin]
use = egg:Paste#urlmap
/v2.0 = admin_api
/v3 = api_v3
/ = admin_version_api

3341
keystone/templates/etc/_keystone.conf.tpl
View File

File diff suppressed because it is too large Load Diff

2
keystone/templates/job-credential-setup.yaml
View File

@ -47,7 +47,7 @@ spec:
- name: KUBERNETES_NAMESPACE
value: {{ .Release.Namespace | quote }}
- name: KEYSTONE_KEYS_REPOSITORY
value: {{ .Values.conf.keystone.credential.keystone.key_repository | quote }}
value: {{ .Values.conf.keystone.credential.key_repository | quote }}
command:
- python
- /tmp/fernet-manage.py

6
keystone/templates/job-db-sync.yaml
View File

@ -63,9 +63,9 @@ spec:
mountPath: /tmp/db-sync.sh
subPath: db-sync.sh
readOnly: true
{{- if eq .Values.conf.keystone.token.keystone.provider "fernet" }}
{{- if eq .Values.conf.keystone.token.provider "fernet" }}
- name: keystone-fernet-keys
mountPath: {{ .Values.conf.keystone.fernet_tokens.keystone.key_repository }}
mountPath: {{ .Values.conf.keystone.fernet_tokens.key_repository }}
{{- end }}
{{ if $mounts_keystone_db_sync.volumeMounts }}{{ toYaml $mounts_keystone_db_sync.volumeMounts | indent 10 }}{{ end }}
volumes:
@ -79,7 +79,7 @@ spec:
configMap:
name: keystone-bin
defaultMode: 0555
{{- if eq .Values.conf.keystone.token.keystone.provider "fernet" }}
{{- if eq .Values.conf.keystone.token.provider "fernet" }}
- name: keystone-fernet-keys
secret:
secretName: keystone-fernet-keys

4
keystone/templates/job-fernet-setup.yaml
View File

@ -15,7 +15,7 @@ limitations under the License.
*/}}
{{- if .Values.manifests.job_fernet_setup }}
{{- if eq .Values.conf.keystone.token.keystone.provider "fernet" }}
{{- if eq .Values.conf.keystone.token.provider "fernet" }}
{{- $envAll := . }}
{{- $dependencies := .Values.dependencies.fernet_setup }}
{{- $mounts_keystone_fernet_setup := .Values.pod.mounts.keystone_fernet_setup.keystone_fernet_setup }}
@ -48,7 +48,7 @@ spec:
- name: KUBERNETES_NAMESPACE
value: {{ .Release.Namespace | quote }}
- name: KEYSTONE_KEYS_REPOSITORY
value: {{ .Values.conf.keystone.fernet_tokens.keystone.key_repository | quote }}
value: {{ .Values.conf.keystone.fernet_tokens.key_repository | quote }}
command:
- python
- /tmp/fernet-manage.py

2
keystone/templates/secret-fernet-keys.yaml
View File

@ -16,7 +16,7 @@ limitations under the License.
{{- if .Values.manifests.secret_fernet_keys }}
{{- $envAll := . }}
{{- if eq .Values.conf.keystone.token.keystone.provider "fernet" }}
{{- if eq .Values.conf.keystone.token.provider "fernet" }}
---
apiVersion: v1
kind: Secret

105
keystone/values.yaml
View File

@ -240,13 +240,78 @@ jobs:
group: keystone
conf:
rally_tests:
run_tempest: false
override:
append:
keystone:
DEFAULT:
max_token_size: 255
token:
provider: fernet
fernet_tokens:
key_repository: /etc/keystone/fernet-keys/
credential:
key_repository: /etc/keystone/credential-keys/
database:
max_retries: -1
cache:
enabled: true
backend: dogpile.cache.memcached
paste:
override:
append:
filter:debug:
use: egg:oslo.middleware#debug
filter:request_id:
use: egg:oslo.middleware#request_id
filter:build_auth_context:
use: egg:keystone#build_auth_context
filter:token_auth:
use: egg:keystone#token_auth
filter:json_body:
use: egg:keystone#json_body
filter:cors:
use: egg:oslo.middleware#cors
oslo_config_project: keystone
filter:http_proxy_to_wsgi:
use: egg:oslo.middleware#http_proxy_to_wsgi
filter:ec2_extension:
use: egg:keystone#ec2_extension
filter:ec2_extension_v3:
use: egg:keystone#ec2_extension_v3
filter:s3_extension:
use: egg:keystone#s3_extension
filter:url_normalize:
use: egg:keystone#url_normalize
filter:sizelimit:
use: egg:oslo.middleware#sizelimit
filter:osprofiler:
use: egg:osprofiler#osprofiler
app:public_service:
use: egg:keystone#public_service
app:service_v3:
use: egg:keystone#service_v3
app:admin_service:
use: egg:keystone#admin_service
pipeline:public_api:
pipeline: cors sizelimit http_proxy_to_wsgi osprofiler url_normalize request_id build_auth_context token_auth json_body ec2_extension public_service
pipeline:admin_api:
pipeline: cors sizelimit http_proxy_to_wsgi osprofiler url_normalize request_id build_auth_context token_auth json_body ec2_extension s3_extension admin_service
pipeline:api_v3:
pipeline: cors sizelimit http_proxy_to_wsgi osprofiler url_normalize request_id build_auth_context token_auth json_body ec2_extension_v3 s3_extension service_v3
app:public_version_service:
use: egg:keystone#public_version_service
app:admin_version_service:
use: egg:keystone#admin_version_service
pipeline:public_version_api:
pipeline: cors sizelimit osprofiler url_normalize public_version_service
pipeline:admin_version_api:
pipeline: cors sizelimit osprofiler url_normalize admin_version_service
composite:main:
use: egg:Paste#urlmap
/v2.0: public_api
/v3: api_v3
/: public_version_api
composite:admin:
use: egg:Paste#urlmap
/v2.0: admin_api
/v3: api_v3
/: admin_version_api
policy:
admin_required: role:admin or is_admin:1
service_role: role:service
@ -414,6 +479,10 @@ conf:
identity:update_domain_config: rule:admin_required
identity:delete_domain_config: rule:admin_required
identity:get_domain_config_default: rule:admin_required
rally_tests:
run_tempest: false
override:
append:
mpm_event:
override:
append:
@ -423,30 +492,6 @@ conf:
sso_callback_template:
override:
append:
keystone:
override:
append:
default:
keystone:
max_token_size: 255
token:
keystone:
provider: fernet
fernet_tokens:
keystone:
key_repository: /etc/keystone/fernet-keys/
credential:
keystone:
key_repository: /etc/keystone/credential-keys/
database:
oslo:
db:
max_retries: -1
cache:
oslo:
cache:
enabled: true
backend: dogpile.cache.memcached
# Names of secrets used by bootstrap and environmental checks
secrets:

33
tools/gen-oslo-openstack-helm/Dockerfile
View File

@ -1,33 +0,0 @@
FROM debian:jessie-slim
MAINTAINER Pete Birley <pete.birley@att.com>
ENV PROJECT="keystone" \
PROJECT_BRANCH="stable/newton" \
PROJECT_REPO="https://git.openstack.org/openstack/keystone.git"
RUN set -x \
&& apt-get update \
&& apt-get install -y \
build-essential \
curl \
git \
libffi-dev \
libldap2-dev \
libpq-dev \
libsasl2-dev \
libssl-dev \
python \
python-dev \
&& curl -sSL https://bootstrap.pypa.io/get-pip.py -o get-pip.py \
&& python get-pip.py \
&& rm get-pip.py \
&& pip install \
oslo.config \
tox \
crudini
COPY generate.py /opt/gen-oslo-openstack-helm/generate.py
COPY entrypoint.sh /entrypoint.sh
COPY oslo-config-generator /opt/gen-oslo-openstack-helm/oslo-config-generator
ENTRYPOINT ["/entrypoint.sh"]

24
tools/gen-oslo-openstack-helm/Makefile
View File

@ -1,24 +0,0 @@
# Copyright 2017 The Openstack-Helm Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
.PHONY: build push
PREFIX=quay.io/att-comdev
TAG = 0.4.0
build:
docker build --pull -t $(PREFIX)/gen-oslo-openstack-helm:$(TAG) .
push:
docker -- push $(PREFIX)/gen-oslo-openstack-helm:$(TAG)

27
tools/gen-oslo-openstack-helm/README.rst
View File

@ -1,27 +0,0 @@
gen-oslo-openstack-helm
=======================
Oslo Config Generator Hack to Generate Helm Configs
Usage
-----
From the tools/gen-oslo-openstack-helm directory
run the following commands, adjusting for the
OpenStack project and/or branch desired as necessary.
.. code:: bash
docker build . -t gen-oslo-openstack-helm
PROJECT=heat
sudo rm -rf /tmp/${PROJECT} || true
docker run -it --rm \
-e PROJECT="${PROJECT}" \
-e PROJECT_BRANCH="stable/newton" \
-e PROJECT_REPO=https://git.openstack.org/openstack/${PROJECT}.git \
-v /tmp:/tmp:rw \
gen-oslo-openstack-helm
This container will then drop you into a shell, at the project root with
OpenStack-Helm formatted configuration files in the standard locations
produced by genconfig for the project.

47
tools/gen-oslo-openstack-helm/entrypoint.sh
View File

@ -1,47 +0,0 @@
#!/bin/bash
# Copyright 2017 The Openstack-Helm Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
set -xe
# Shallow clone the Project Repo and desired branch
git clone --depth 1 --branch ${PROJECT_BRANCH} ${PROJECT_REPO} /tmp/${PROJECT}
# Move into the project directory
cd /tmp/${PROJECT}
# Hack the tox.ini to load our oslo-config-generator script
TOX_VENV_DIR=$(crudini --get tox.ini testenv:genconfig envdir | sed 's|^{toxworkdir}|.tox|')
if [ -z "$TOX_VENV_DIR" ]; then
TOX_VENV_DIR=".tox/genconfig"
fi
echo "mv -f /opt/gen-oslo-openstack-helm/oslo-config-generator $(pwd)/${TOX_VENV_DIR}/bin/oslo-config-generator" > /tmp/oslo-config-generator-cmds
crudini --get tox.ini testenv:genconfig commands >> /tmp/oslo-config-generator-cmds
crudini --set tox.ini testenv:genconfig commands "$(cat /tmp/oslo-config-generator-cmds)"
# Setting up the whitelisted commands to include mv, letting us drop in the
# hacked oslo-config-generator.
(crudini --get tox.ini testenv:genconfig whitelist_externals > /tmp/whitelist_externals) || true
echo "mv" >> /tmp/whitelist_externals
crudini --set tox.ini testenv:genconfig whitelist_externals "$(cat /tmp/whitelist_externals)"
# Setting environment variables for OpenStack-Helm's oslo-config-generator args
echo "HELM_CHART=${PROJECT}" > /opt/gen-oslo-openstack-helm-env
echo "HELM_NAMESPACE=${PROJECT}" >> /opt/gen-oslo-openstack-helm-env
# Run tox to generate the standard config files, and generate the venv
tox -egenconfig
# Drop into a shell in the container.
bash

681
tools/gen-oslo-openstack-helm/generate.py
View File

@ -1,681 +0,0 @@
#!/usr/bin/env python
# Copyright 2012 SINA Corporation
# Copyright 2014 Cisco Systems, Inc.
# All Rights Reserved.
# Copyright 2014 Red Hat, Inc.
# Copyright 2017 The Openstack-Helm Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import logging
import operator
import sys
import textwrap
import re
import traceback
import pkg_resources
import six
from oslo_config._i18n import _
from oslo_config import cfg
from oslo_config.generator import _get_groups, _list_opts, _format_defaults, \
_TYPE_NAMES
import stevedore.named # noqa
LOG = logging.getLogger(__name__)
UPPER_CASE_GROUP_NAMES = ['DEFAULT']
def _format_type_name(opt_type):
"""Format the type name to use in describing an option"""
try:
return opt_type.type_name
except AttributeError: # nosec
pass
try:
return _TYPE_NAMES[opt_type]
except KeyError: # nosec
pass
return 'unknown value'
_generator_opts = [
cfg.StrOpt(
'output-file',
help='Path of the file to write to. Defaults to stdout.'),
cfg.IntOpt(
'wrap-width',
default=70,
help='The maximum length of help lines.'),
cfg.MultiStrOpt(
'namespace',
required=True,
help='Option namespace under "oslo.config.opts" in which to query '
'for options.'),
cfg.StrOpt(
'helm_namespace',
required=True,
help="Helm Namespace, e.g. 'keystone'"),
cfg.StrOpt(
'helm_chart',
required=True,
help="Helm Chart Name, e.g. 'keystone'"),
cfg.BoolOpt(
'minimal',
default=False,
help='Generate a minimal required configuration.'),
cfg.BoolOpt(
'summarize',
default=False,
help='Only output summaries of help text to config files. Retain '
'longer help text for Sphinx documents.'),
]
def register_cli_opts(conf):
"""Register the formatter's CLI options with a ConfigOpts instance.
Note, this must be done before the ConfigOpts instance is called to parse
the configuration.
:param conf: a ConfigOpts instance
:raises: DuplicateOptError, ArgsAlreadyParsedError
"""
conf.register_cli_opts(_generator_opts)
def _output_opts_null(f, group, group_data, minimal=False, summarize=False):
pass
f.format_group(group_data['object'] or group)
for (namespace, opts) in sorted(group_data['namespaces'],
key=operator.itemgetter(0)):
for opt in sorted(opts, key=operator.attrgetter('advanced')):
try:
if minimal and not opt.required:
pass
else:
f.format(opt, group, namespace, minimal, summarize)
except Exception as err:
pass
def _output_opts(f, group, group_data, minimal=False, summarize=False):
f.format_group(group_data['object'] or group)
for (namespace, opts) in sorted(group_data['namespaces'],
key=operator.itemgetter(0)):
f.write('\n#\n# From %s\n#\n' % namespace)
for opt in sorted(opts, key=operator.attrgetter('advanced')):
try:
if minimal and not opt.required:
pass
else:
f.write('\n')
f.format(opt, group, namespace, minimal, summarize)
except Exception as err:
f.write('# Warning: Failed to format sample for %s\n' %
(opt.dest,))
f.write('# %s\n' % (traceback.format_exc(),))
class _ValuesSkeletonFormatter(object):
"""Format configuration option descriptions to a file."""
def __init__(self, output_file=None, wrap_width=70):
"""Construct an OptFormatter object.
:param output_file: a writeable file object
:param wrap_width: The maximum length of help lines, 0 to not wrap
"""
self.output_file = output_file or sys.stdout
self.wrap_width = wrap_width
self.done = []
def _format_help(self, help_text):
pass
def _get_choice_text(self, choice):
pass
def format_group(self, group_or_groupname):
pass
def format(self, opt, group_name, namespace, minimal=False,
summarize=False):
"""Format a description of an option to the output file.
:param opt: a cfg.Opt instance
:param group_name: name of the group to which the opt is assigned
:param minimal: enable option by default, marking it as required
:param summarize: output a summarized description of the opt
:returns: a formatted opt description string
"""
if hasattr(opt.type, 'format_defaults'):
defaults = opt.type.format_defaults(opt.default,
opt.sample_default)
else:
LOG.debug(
"The type for option %(name)s which is %(type)s is not a "
"subclass of types.ConfigType and doesn't provide a "
"'format_defaults' method. A default formatter is not "
"available so the best-effort formatter will be used.",
{'type': opt.type, 'name': opt.name})
defaults = _format_defaults(opt)
lines = []
for default_str in defaults:
if len(group_name.split('.')) > 1:
line = '{{- if not .%s -}}\
{{- set . "%s" dict -}}\
{{- end -}}\n' % (
group_name.lower().split('.')[0],
group_name.lower().split('.')[0])
if line not in self.done:
self.done.append(line)
lines.append(line)
line = '{{- if not .%s.%s -}}\
{{- set .%s "%s" dict -}}\
{{- end -}}\n' % (
group_name.lower().split('.')[0],
group_name.lower().split('.')[1],
group_name.lower().split('.')[0],
group_name.lower().split('.')[1])
if line not in self.done:
self.done.append(line)
lines.append(line)
else:
line = '{{- if not .%s -}}\
{{- set . "%s" dict -}}\
{{- end -}}\n' % (
group_name.lower(),
group_name.lower())
if line not in self.done:
self.done.append(line)
lines.append(line)
if len(namespace.split('.')) == 1:
line = '{{- if not .%s.%s -}}\
{{- set .%s "%s" dict -}}\
{{- end -}}\n' % (
group_name.lower(),
namespace,
group_name.lower(),
namespace)
if line not in self.done:
self.done.append(line)
lines.append(line)
if len(namespace.split('.')) > 1:
line = '{{- if not .%s.%s -}}\
{{- set .%s "%s" dict -}}\
{{- end -}}\n' % (
group_name.lower(),
namespace.split('.')[0],
group_name.lower(),
namespace.split('.')[0])
if line not in self.done:
self.done.append(line)
lines.append(line)
line = '{{- if not .%s.%s.%s -}}\
{{- set .%s.%s "%s" dict -}}\
{{- end -}}\n' % (
group_name.lower(),
namespace.split('.')[0],
namespace.split('.')[1],
group_name.lower(),
namespace.split('.')[0],
namespace.split('.')[1])
if line not in self.done:
self.done.append(line)
lines.append(line)
if len(namespace.split('.')) > 2:
line = '{{- if not .%s.%s.%s.%s -}}\
{{- set .%s.%s.%s "%s" dict -}}\
{{- end -}}\n' % (
group_name.lower(),
namespace.split('.')[0],
namespace.split('.')[1],
namespace.split('.')[2],
group_name.lower(),
namespace.split('.')[0],
namespace.split('.')[1],
namespace.split('.')[2])
if line not in self.done:
self.done.append(line)
lines.append(line)
if len(namespace.split('.')) > 3:
line = '{{- if not .%s.%s.%s.%s.%s -}}\
{{- set .%s.%s.%s.%s "%s" dict -}}\
{{- end -}}\n' % (
group_name.lower(),
namespace.split('.')[0],
namespace.split('.')[1],
namespace.split('.')[2],
namespace.split('.')[3],
group_name.lower(),
namespace.split('.')[0],
namespace.split('.')[1],
namespace.split('.')[2],
namespace.split('.')[3])
if line not in self.done:
self.done.append(line)
lines.append(line)
if len(opt.dest.split('.')) > 1:
line = '{{- if not .%s.%s.%s -}}\
{{- set .%s.%s "%s" dict -}}\
{{- end -}}\n' % (
group_name.lower(),
namespace,
opt.dest.split('.')[0],
group_name.lower(),
namespace,
opt.dest.split('.')[0])
if line not in self.done:
self.done.append(line)
lines.append(line)
if len(opt.dest.split('.')) > 2:
line = '{{- if not .%s.%s.%s.%s -}}\
{{- set .%s.%s.%s "%s" dict -}}\
{{- end -}}\n' % (
group_name.lower(),
namespace,
opt.dest.split('.')[0],
opt.dest.split('.')[1],
group_name.lower(),
namespace,
opt.dest.split('.')[0],
opt.dest.split('.')[1])
if line not in self.done:
self.done.append(line)
lines.append(line)
if len(opt.dest.split('.')) > 3:
line = '{{- if not .%s.%s.%s.%s.%s -}}\
{{- set .%s.%s.%s.%s "%s" dict -}}\
{{- end -}}\n' % (
group_name.lower(),
namespace,
opt.dest.split('.')[0],
opt.dest.split('.')[1],
opt.dest.split('.')[2],
group_name.lower(),
namespace,
opt.dest.split('.')[0],
opt.dest.split('.')[1],
opt.dest.split('.')[2])
if line not in self.done:
self.done.append(line)
lines.append(line)
if lines:
self.writelines(lines)
def write(self, s):
"""Write an arbitrary string to the output file.
:param s: an arbitrary string
"""
self.output_file.write(s)
def writelines(self, l):
"""Write an arbitrary sequence of strings to the output file.
:param l: a list of arbitrary strings
"""
self.output_file.writelines(l)
class _HelmOptFormatter(object):
"""Format configuration option descriptions to a file."""
def __init__(self, output_file=None, wrap_width=70):
"""Construct an OptFormatter object.
:param output_file: a writeable file object
:param wrap_width: The maximum length of help lines, 0 to not wrap
"""
self.output_file = output_file or sys.stdout
self.wrap_width = wrap_width
def _format_help(self, help_text):
"""Format the help for a group or option to the output file.
:param help_text: The text of the help string
"""
if self.wrap_width is not None and self.wrap_width > 0:
wrapped = ""
for line in help_text.splitlines():
text = "\n".join(textwrap.wrap(line, self.wrap_width,
initial_indent='# ',
subsequent_indent='# ',
break_long_words=False,
replace_whitespace=False))
wrapped += "#" if text == "" else text
wrapped += "\n"
lines = [wrapped]
else:
lines = ['# ' + help_text + '\n']
return lines
def _get_choice_text(self, choice):
if choice is None:
return '<None>'
elif choice == '':
return "''"
return six.text_type(choice)
def format_group(self, group_or_groupname):
"""Format the description of a group header to the output file
:param group_or_groupname: a cfg.OptGroup instance or a name of group
:returns: a formatted group description string
"""
if isinstance(group_or_groupname, cfg.OptGroup):
group = group_or_groupname
lines = ['[%s]\n' % group.name]
if group.help:
lines += self._format_help(group.help)
else:
groupname = group_or_groupname
lines = ['[%s]\n' % groupname]
self.writelines(lines)
def format(self, opt, group_name, namespace, minimal=False,
summarize=False):
"""Format a description of an option to the output file.
:param opt: a cfg.Opt instance
:param group_name: name of the group to which the opt is assigned
:param minimal: enable option by default, marking it as required
:param summarize: output a summarized description of the opt
:returns: a formatted opt description string
"""
if not opt.help:
LOG.warning(_('"%s" is missing a help string'), opt.dest)
opt_type = _format_type_name(opt.type)
opt_prefix = ''
if (opt.deprecated_for_removal and
not opt.help.startswith('DEPRECATED')):
opt_prefix = 'DEPRECATED: '
if opt.help:
# an empty line signifies a new paragraph. We only want the
# summary line
if summarize:
_split = opt.help.split('\n\n')
opt_help = _split[0].rstrip(':').rstrip('.')
if len(_split) > 1:
opt_help += '. For more information, refer to the '
opt_help += 'documentation.'
else:
opt_help = opt.help
help_text = u'%s%s (%s)' % (opt_prefix,
opt_help,
opt_type)
else:
help_text = u'(%s)' % opt_type
lines = self._format_help(help_text)
if getattr(opt.type, 'min', None) is not None:
lines.append('# Minimum value: %d\n' % opt.type.min)
if getattr(opt.type, 'max', None) is not None:
lines.append('# Maximum value: %d\n' % opt.type.max)
if getattr(opt.type, 'choices', None):
choices_text = ', '.join([self._get_choice_text(choice)
for choice in opt.type.choices])
lines.append('# Allowed values: %s\n' % choices_text)
try:
if opt.mutable:
lines.append(
'# Note: This option can be changed without restarting.\n'
)
except AttributeError as err:
# NOTE(dhellmann): keystoneauth defines its own Opt class,
# and neutron (at least) returns instances of those
# classes instead of oslo_config Opt instances. The new
# mutable attribute is the first property where the API
# isn't supported in the external class, so we can use
# this failure to emit a warning. See
# https://bugs.launchpad.net/keystoneauth/+bug/1548433 for
# more details.
import warnings
if not isinstance(opt, cfg.Opt):
warnings.warn(
'Incompatible option class for %s (%r): %s' %
(opt.dest, opt.__class__, err),
)
else:
warnings.warn('Failed to fully format sample for %s: %s' %
(opt.dest, err))
for d in opt.deprecated_opts:
lines.append('# Deprecated group/name - [%s]/%s\n' %
(d.group or group_name, d.name or opt.dest))
if opt.deprecated_for_removal:
if opt.deprecated_since:
lines.append(
'# This option is deprecated for removal since %s.\n' % (
opt.deprecated_since))
else:
lines.append(
'# This option is deprecated for removal.\n')
lines.append(
'# Its value may be silently ignored in the future.\n')
if opt.deprecated_reason:
lines.extend(
self._format_help('Reason: ' + opt.deprecated_reason))
if opt.advanced:
lines.append(
'# Advanced Option: intended for advanced users and not used\n'
'# by the majority of users, and might have a significant\n'
'# effect on stability and/or performance.\n'
)
if hasattr(opt.type, 'format_defaults'):
defaults = opt.type.format_defaults(opt.default,
opt.sample_default)
else:
LOG.debug(
"The type for option %(name)s which is %(type)s is not a "
"subclass of types.ConfigType and doesn't provide a "
"'format_defaults' method. A default formatter is not "
"available so the best-effort formatter will be used.",
{'type': opt.type, 'name': opt.name})
defaults = _format_defaults(opt)
for default_str in defaults:
if type(opt) in [cfg.MultiOpt, cfg.MultiStrOpt]:
lines.append('# from .%s.%s.%s (multiopt)\n' % (
group_name.lower(),
namespace, opt.dest))
lines.append('{{ if not .%s.%s.%s }}#%s = '
'{{ .%s.%s.%s | default "%s" }}{{ else }}'
'{{ range .%s.%s.%s }}%s = {{ . }}\n{{ end }}'
'{{ end }}\n' % (
group_name.lower(),
namespace,
opt.dest,
opt.dest,
group_name.lower(),
namespace,
opt.dest,
default_str.replace('"', r'\"'),
group_name.lower(),
namespace,
opt.dest,
opt.dest))
else:
lines.append('# from .%s.%s.%s\n' % (
group_name.lower(),
namespace,
opt.dest))
if minimal:
lines.append('%s = {{ .%s.%s.%s | default "%s" }}\n' % (
opt.dest,
group_name.lower(),
namespace,
opt.dest,
default_str.replace('"', r'\"')))
else:
lines.append('{{ if not .%s.%s.%s }}#{{ end }}%s = '
'{{ .%s.%s.%s | default "%s" }}\n' % (
group_name.lower(),
namespace,
opt.dest,
opt.dest,
group_name.lower(),
namespace,
opt.dest,
default_str.replace('"', r'\"')))
self.writelines(lines)
def write(self, s):
"""Write an arbitrary string to the output file.
:param s: an arbitrary string
"""
self.output_file.write(s)
def writelines(self, l):
"""Write an arbitrary sequence of strings to the output file.
:param l: a list of arbitrary strings
"""
self.output_file.writelines(l)
def generate(conf):
"""Generate a sample config file.
List all of the options available via the namespaces specified in the given
configuration and write a description of them to the specified output file.
:param conf: a ConfigOpts instance containing the generator's configuration
"""
conf.register_opts(_generator_opts)
output_file = (open(conf.output_file, 'w')
if conf.output_file else sys.stdout)
output_file.write('''# Copyright 2017 The Openstack-Helm Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
{{ include "%s.conf.%s_values_skeleton" .Values.conf.%s | trunc 0 }}
{{ include "%s.conf.%s" .Values.conf.%s }}
\n''' % (conf.helm_chart,
conf.helm_namespace,
conf.helm_namespace,
conf.helm_chart,
conf.helm_namespace,
conf.helm_namespace))
output_file.write('''
{{- define "%s.conf.%s_values_skeleton" -}}
\n''' % (conf.helm_chart, conf.helm_namespace))
# values skeleton
formatter = _ValuesSkeletonFormatter(output_file=output_file,
wrap_width=conf.wrap_width)
groups = _get_groups(_list_opts(conf.namespace))
# Output the "DEFAULT" section as the very first section
_output_opts_null(formatter, 'DEFAULT', groups.pop('DEFAULT'),
conf.minimal, conf.summarize)
# output all other config sections with groups in alphabetical order
for group, group_data in sorted(groups.items()):
_output_opts_null(formatter, group, group_data, conf.minimal,
conf.summarize)
output_file.write('''
{{- end -}}
\n''')
output_file.write('''
{{- define "%s.conf.%s" -}}
\n''' % (conf.helm_chart, conf.helm_namespace))
# helm options
formatter = _HelmOptFormatter(output_file=output_file,
wrap_width=conf.wrap_width)
groups = _get_groups(_list_opts(conf.namespace))
# Output the "DEFAULT" section as the very first section
_output_opts(formatter, 'DEFAULT', groups.pop('DEFAULT'), conf.minimal,
conf.summarize)
# output all other config sections with groups in alphabetical order
for group, group_data in sorted(groups.items()):
formatter.write('\n\n')
_output_opts(formatter, group, group_data, conf.minimal,
conf.summarize)
output_file.write('''
{{- end -}}
\n''')
# generate helm defaults
def main(args=None):
"""The main function of oslo-config-generator."""
version = pkg_resources.get_distribution('oslo.config').version
logging.basicConfig(level=logging.WARN)
conf = cfg.ConfigOpts()
register_cli_opts(conf)
conf(args, version=version)
generate(conf)
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
sys.exit(main())

23
tools/gen-oslo-openstack-helm/oslo-config-generator
View File

@ -1,23 +0,0 @@
#!/bin/bash
# Copyright 2017 The Openstack-Helm Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
set -xe
source /opt/gen-oslo-openstack-helm-env
python /opt/gen-oslo-openstack-helm/generate.py \
--helm_chart ${HELM_CHART} \
--helm_namespace ${HELM_NAMESPACE} \
"${@}"