openstack/openstack-helm-infra
Code Issues Proposed changes

Add ceph-adapter-rook chart

Browse Source 
When using Rook for managing Ceph clusters we have
to provision a minimal set of assets (keys, endpoints, etc.)
to make Openstack-Helm charts work with these Ceph clusters.

Rook provides CRDs that can be used for managing Ceph assets
like pools/keyrings/buckets etc. but Openstack-Helm can not
utilize these CRDs. To support these CRDs in OSH would
require having lots of conditionals in OSH templates since
we still want OSH to work with OSH ceph-* charts.

Change-Id: If7fe29052640e48c37b653e13a74d95e360a6d16
This commit is contained in:
Vladimir Kozhukalov 2023-12-05 13:28:26 -06:00
parent 4a95f75b6b
commit 978507351f
18 changed files with 985 additions and 80 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
ceph-adapter-rook/Chart.yaml Normal file
View File

@ -0,0 +1,20 @@
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
---
apiVersion: v1
appVersion: v1.0.0
description: OpenStack-Helm Ceph Adapter Rook
name: ceph-adapter-rook
version: 0.1.0
home: https://github.com/ceph/ceph
...

51
ceph-adapter-rook/README.md Normal file
View File

@ -0,0 +1,51 @@
# Summary
This is the minimal set of templates necessary to make the rest
of Openstack-Helm charts work with Ceph clusters managed by the
Rook operator. Rook operator not only deploys Ceph clusters but
also provides convenience when interfacing with those clusters
via CRDs which can be used for managing pools/keys/users etc.
However Openstack-Helm charts do not utilize Rook CRDs but instead
manage Ceph assets like pools/keyrings/users/buckets etc. by means
of running bootstrap scripts. Before using Openstack-Helm charts we
have to provision a minimal set of assets like Ceph admin keys and
endpoints and this chart provides exactly this minimal set of templates.
# Usage
Deploy Ceph admin key and Ceph mon endpoint in the namespace where Ceph cluster is deployed.
```
tee > /tmp/ceph-adapter-rook-ceph.yaml <<EOF
manifests:
configmap_bin: true
configmap_templates: true
configmap_etc: false
job_storage_admin_keys: true
job_namespace_client_key: false
job_namespace_client_ceph_config: false
service_mon_discovery: true
EOF
helm upgrade --install ceph-adapter-rook ./ceph-adapter-rook \
--namespace=ceph \
--values=/tmp/ceph-adapter-ceph.yaml
```
Now wait until all jobs are finished and deploy client key and client
configuration in the namespace where Openstack charts are going to be deployed.
tee > /tmp/ceph-adapter-rook-openstack.yaml <<EOF
manifests:
configmap_bin: true
configmap_templates: false
configmap_etc: true
job_storage_admin_keys: false
job_namespace_client_key: true
job_namespace_client_ceph_config: true
service_mon_discovery: false
EOF
helm upgrade --install ceph-adapter-rook ./ceph-adapter-rook \
--namespace=openstack \
--values=/tmp/ceph-adapter-rook-openstack.yaml
```
Again wait until all jobs are finished and then you can deploy other Openstack-Helm charts.

18
ceph-adapter-rook/requirements.yaml Normal file
View File

@ -0,0 +1,18 @@
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
---
dependencies:
- name: helm-toolkit
repository: file://../helm-toolkit
version: ">= 0.1.0"
...

36
ceph-adapter-rook/templates/bin/_namespace-client-ceph-config-manager.sh.tpl Normal file
View File

@ -0,0 +1,36 @@
#!/bin/bash
{{/*
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/}}
set -ex
{{- $envAll := . }}
{{ include "helm-toolkit.snippets.mon_host_from_k8s_ep" . }}
# TODO: Get endpoint from rook-ceph-mon-endpoints configmap
ENDPOINT=$(mon_host_from_k8s_ep ${PVC_CEPH_RBD_STORAGECLASS_DEPLOYED_NAMESPACE} ceph-mon-discovery)
if [ -z "$ENDPOINT" ]; then
echo "Ceph Mon endpoint is empty"
exit 1
else
echo $ENDPOINT
fi
kubectl get cm ${CEPH_CONF_ETC} -n ${DEPLOYMENT_NAMESPACE} -o yaml | \
sed "s#mon_host.*#mon_host = ${ENDPOINT}#g" | \
kubectl apply -f -
kubectl get cm ${CEPH_CONF_ETC} -n ${DEPLOYMENT_NAMESPACE} -o yaml

51
ceph-adapter-rook/templates/bin/_namespace-client-key-manager.sh.tpl Normal file
View File

@ -0,0 +1,51 @@
#!/bin/bash
{{/*
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/}}
set -ex
{{- $envAll := . }}
CEPH_RBD_KEY=$(kubectl get secret ${PVC_CEPH_RBD_STORAGECLASS_ADMIN_SECRET_NAME} \
--namespace=${PVC_CEPH_RBD_STORAGECLASS_DEPLOYED_NAMESPACE} \
-o json )
# CONNECT_TO_ROOK_CEPH_CLUSTER is unset by default
if [[ ${CONNECT_TO_ROOK_CEPH_CLUSTER} == "true" ]] ; then
CEPH_CLUSTER_KEY=$(echo "${CEPH_RBD_KEY}" | jq -r '.data["ceph-secret"]')
else
CEPH_CLUSTER_KEY=$(echo "${CEPH_RBD_KEY}" | jq -r '.data.key')
fi
ceph_activate_namespace() {
kube_namespace=$1
secret_type=$2
secret_name=$3
ceph_key=$4
{
cat <<EOF
apiVersion: v1
kind: Secret
metadata:
name: "${secret_name}"
labels:
{{ tuple $envAll "ceph" "rbd" | include "helm-toolkit.snippets.kubernetes_metadata_labels" | indent 4 }}
type: "${secret_type}"
data:
key: $( echo ${ceph_key} )
EOF
} | kubectl apply --namespace ${kube_namespace} -f -
}
ceph_activate_namespace ${DEPLOYMENT_NAMESPACE} "kubernetes.io/rbd" ${PVC_CEPH_RBD_STORAGECLASS_USER_SECRET_NAME} "${CEPH_CLUSTER_KEY}"

91
ceph-adapter-rook/templates/bin/_storage-keyring-manager.sh.tpl Normal file
View File

@ -0,0 +1,91 @@
#!/bin/bash
{{/*
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/}}
set -ex
{{ if .Release.IsInstall }}
{{- $envAll := . }}
function kube_ceph_keyring_gen () {
CEPH_KEY=$1
CEPH_KEY_TEMPLATE=$2
sed "s|{{"{{"}} key {{"}}"}}|${CEPH_KEY}|" ${CEPH_TEMPLATES_DIR}/${CEPH_KEY_TEMPLATE} | base64 -w0 | tr -d '\n'
}
CEPH_CLIENT_KEY=""
ROOK_CEPH_TOOLS_POD=$(kubectl -n ${DEPLOYMENT_NAMESPACE} get pods --no-headers | awk '/rook-ceph-tools/{print $1}')
if [[ -n "${ROOK_CEPH_TOOLS_POD}" ]]; then
CEPH_AUTH_KEY_NAME=$(echo "${CEPH_KEYRING_NAME}" | awk -F. '{print $2 "." $3}')
CEPH_CLIENT_KEY=$(kubectl -n ${DEPLOYMENT_NAMESPACE} exec ${ROOK_CEPH_TOOLS_POD} -- ceph auth ls | grep -A1 "${CEPH_AUTH_KEY_NAME}" | awk '/key:/{print $2}')
fi
function create_kube_key () {
CEPH_KEYRING=$1
CEPH_KEYRING_NAME=$2
CEPH_KEYRING_TEMPLATE=$3
KUBE_SECRET_NAME=$4
if ! kubectl get --namespace ${DEPLOYMENT_NAMESPACE} secrets ${KUBE_SECRET_NAME}; then
{
cat <<EOF
---
apiVersion: v1
kind: Secret
metadata:
name: ${KUBE_SECRET_NAME}
labels:
{{ tuple $envAll "ceph" "admin" | include "helm-toolkit.snippets.kubernetes_metadata_labels" | indent 4 }}
type: Opaque
data:
${CEPH_KEYRING_NAME}: $( kube_ceph_keyring_gen ${CEPH_KEYRING} ${CEPH_KEYRING_TEMPLATE} )
EOF
} | kubectl apply --namespace ${DEPLOYMENT_NAMESPACE} -f -
fi
}
#create_kube_key <ceph_key> <ceph_keyring_name> <ceph_keyring_template> <kube_secret_name>
create_kube_key ${CEPH_CLIENT_KEY} ${CEPH_KEYRING_NAME} ${CEPH_KEYRING_TEMPLATE} ${CEPH_KEYRING_ADMIN_NAME}
function create_kube_storage_key () {
CEPH_KEYRING=$1
KUBE_SECRET_NAME=$2
if ! kubectl get --namespace ${DEPLOYMENT_NAMESPACE} secrets ${KUBE_SECRET_NAME}; then
{
cat <<EOF
---
apiVersion: v1
kind: Secret
metadata:
name: ${KUBE_SECRET_NAME}
labels:
{{ tuple $envAll "ceph" "admin" | include "helm-toolkit.snippets.kubernetes_metadata_labels" | indent 4 }}
type: kubernetes.io/rbd
data:
key: $( echo ${CEPH_KEYRING} | base64 | tr -d '\n' )
userID: $( echo -n "admin" | base64 | tr -d '\n' )
userKey: $( echo -n ${CEPH_KEYRING} | base64 | tr -d '\n' )
EOF
} | kubectl apply --namespace ${DEPLOYMENT_NAMESPACE} -f -
fi
}
#create_kube_storage_key <ceph_key> <kube_secret_name>
create_kube_storage_key ${CEPH_CLIENT_KEY} ${CEPH_STORAGECLASS_ADMIN_SECRET_NAME}
{{ else }}
echo "Not touching ${KUBE_SECRET_NAME} as this is not the initial deployment"
{{ end }}

30
ceph-adapter-rook/templates/configmap-bin.yaml Normal file
View File

@ -0,0 +1,30 @@
{{/*
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/}}
{{- if .Values.manifests.configmap_bin }}
{{- $envAll := . }}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ printf "%s-%s" $envAll.Release.Name "bin" | quote }}
data:
keys-storage-keyring-manager.sh: |
{{ tuple "bin/_storage-keyring-manager.sh.tpl" . | include "helm-toolkit.utils.template" | indent 4 }}
provisioner-rbd-namespace-client-key-manager.sh: |
{{ tuple "bin/_namespace-client-key-manager.sh.tpl" . | include "helm-toolkit.utils.template" | indent 4 }}
provisioner-rbd-namespace-client-ceph-config-manager.sh: |
{{ tuple "bin/_namespace-client-ceph-config-manager.sh.tpl" . | include "helm-toolkit.utils.template" | indent 4 }}
{{- end }}

49
ceph-adapter-rook/templates/configmap-etc-client.yaml Normal file
View File

@ -0,0 +1,49 @@
{{/*
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/}}
{{- define "ceph.configmap.etc" }}
{{- $configMapName := index . 0 }}
{{- $envAll := index . 1 }}
{{- with $envAll }}
{{/*
{{- if empty .Values.conf.ceph.global.mon_host -}}
{{- $monHost := tuple "ceph_mon" "internal" "mon_msgr2" . | include "helm-toolkit.endpoints.host_and_port_endpoint_uri_lookup" }}
{{- $_ := $monHost | set .Values.conf.ceph.global "mon_host" -}}
{{- end -}}
{{- if empty .Values.conf.ceph.osd.cluster_network -}}
{{- $_ := .Values.network.cluster | set .Values.conf.ceph.osd "cluster_network" -}}
{{- end -}}
{{- if empty .Values.conf.ceph.osd.public_network -}}
{{- $_ := .Values.network.public | set .Values.conf.ceph.osd "public_network" -}}
{{- end -}}
*/}}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ $configMapName }}
data:
ceph.conf: |
{{ include "helm-toolkit.utils.to_ini" .Values.conf.ceph | indent 4 }}
{{- end }}
{{- end }}
{{- if .Values.manifests.configmap_etc }}
{{- list .Values.ceph_configmap_name . | include "ceph.configmap.etc" }}
{{- end }}

25
ceph-adapter-rook/templates/configmap-templates.yaml Normal file
View File

@ -0,0 +1,25 @@
{{/*
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/}}
{{- if .Values.manifests.configmap_templates }}
{{- $envAll := . }}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ printf "%s-%s" $envAll.Release.Name "templates" | quote }}
data:
admin.keyring: |
{{ .Values.conf.templates.keyring.admin | indent 4 }}
{{- end }}

134
ceph-adapter-rook/templates/job-namespace-client-ceph-config.yaml Normal file
View File

@ -0,0 +1,134 @@
{{/*
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/}}
{{- if and .Values.manifests.job_namespace_client_ceph_config }}
{{- $envAll := . }}
{{- $randStringSuffix := randAlphaNum 5 | lower }}
{{- $serviceAccountName := print $envAll.Release.Name "-ceph-ns-ceph-config-generator" }}
{{ tuple $envAll "namespace_client_ceph_config_generator" $serviceAccountName | include "helm-toolkit.snippets.kubernetes_pod_rbac_serviceaccount" }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ $serviceAccountName }}
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- create
- update
- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: {{ $serviceAccountName }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: {{ $serviceAccountName }}
subjects:
- kind: ServiceAccount
name: {{ $serviceAccountName }}
namespace: {{ $envAll.Release.Namespace }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ printf "%s-%s" $serviceAccountName $randStringSuffix }}
namespace: {{ .Values.admin_secret_namespace }}
rules:
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: {{ printf "%s-%s" $serviceAccountName $randStringSuffix }}
namespace: {{ .Values.admin_secret_namespace }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: {{ printf "%s-%s" $serviceAccountName $randStringSuffix }}
subjects:
- kind: ServiceAccount
name: {{ $serviceAccountName }}
namespace: {{ $envAll.Release.Namespace }}
---
apiVersion: batch/v1
kind: Job
metadata:
name: {{ $serviceAccountName }}
labels:
{{ tuple $envAll "ceph" "client-ceph-config-generator" | include "helm-toolkit.snippets.kubernetes_metadata_labels" | indent 4 }}
spec:
template:
metadata:
labels:
{{ tuple $envAll "ceph" "client-ceph-config-generator" | include "helm-toolkit.snippets.kubernetes_metadata_labels" | indent 8 }}
annotations:
{{ dict "envAll" $envAll "podName" $serviceAccountName "containerNames" (list "ceph-storage-keys-generator" "init") | include "helm-toolkit.snippets.kubernetes_mandatory_access_control_annotation" | indent 8 }}
spec:
{{ dict "envAll" $envAll "application" "client_ceph_config_generator" | include "helm-toolkit.snippets.kubernetes_pod_security_context" | indent 6 }}
serviceAccountName: {{ $serviceAccountName }}
restartPolicy: OnFailure
nodeSelector:
{{ $envAll.Values.labels.job.node_selector_key }}: {{ $envAll.Values.labels.job.node_selector_value }}
initContainers:
{{ tuple $envAll "namespace_client_ceph_config_generator" list | include "helm-toolkit.snippets.kubernetes_entrypoint_init_container" | indent 8 }}
containers:
- name: ceph-storage-keys-generator
{{ tuple $envAll "ceph_config_helper" | include "helm-toolkit.snippets.image" | indent 10 }}
{{ tuple $envAll $envAll.Values.pod.resources.jobs.secret_provisioning | include "helm-toolkit.snippets.kubernetes_resources" | indent 10 }}
{{ dict "envAll" $envAll "application" "client_ceph_config_generator" "container" "ceph_storage_keys_generator" | include "helm-toolkit.snippets.kubernetes_container_security_context" | indent 10 }}
env:
- name: CEPH_CONF_ETC
value: {{ .Values.ceph_configmap_name }}
- name: DEPLOYMENT_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: PVC_CEPH_RBD_STORAGECLASS_DEPLOYED_NAMESPACE
value: {{ .Values.admin_secret_namespace }}
command:
- /tmp/provisioner-rbd-namespace-client-ceph-config-manager.sh
volumeMounts:
- name: pod-tmp
mountPath: /tmp
- name: pod-etc-ceph
mountPath: /etc/ceph
- name: ceph-provisioners-bin-clients
mountPath: /tmp/provisioner-rbd-namespace-client-ceph-config-manager.sh
subPath: provisioner-rbd-namespace-client-ceph-config-manager.sh
readOnly: true
volumes:
- name: pod-tmp
emptyDir: {}
- name: pod-etc-ceph
emptyDir: {}
- name: ceph-provisioners-bin-clients
configMap:
name: {{ printf "%s-%s" $envAll.Release.Name "bin" | quote }}
defaultMode: 0555
{{- end }}

136
ceph-adapter-rook/templates/job-namespace-client-key.yaml Normal file
View File

@ -0,0 +1,136 @@
{{/*
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/}}
{{- if .Values.manifests.job_namespace_client_key }}
{{- $envAll := . }}
{{- $randStringSuffix := randAlphaNum 5 | lower }}
{{- $serviceAccountName := print $envAll.Release.Name "-ceph-ns-key-generator" }}
{{ tuple $envAll "namespace_client_key_generator" $serviceAccountName | include "helm-toolkit.snippets.kubernetes_pod_rbac_serviceaccount" }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ $serviceAccountName }}
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- create
- update
- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: {{ $serviceAccountName }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: {{ $serviceAccountName }}
subjects:
- kind: ServiceAccount
name: {{ $serviceAccountName }}
namespace: {{ $envAll.Release.Namespace }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ printf "%s-%s" $serviceAccountName $randStringSuffix }}
namespace: {{ .Values.admin_secret_namespace }}
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: {{ printf "%s-%s" $serviceAccountName $randStringSuffix }}
namespace: {{ .Values.admin_secret_namespace }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: {{ printf "%s-%s" $serviceAccountName $randStringSuffix }}
subjects:
- kind: ServiceAccount
name: {{ $serviceAccountName }}
namespace: {{ $envAll.Release.Namespace }}
---
apiVersion: batch/v1
kind: Job
metadata:
name: {{ $serviceAccountName }}
labels:
{{ tuple $envAll "ceph" "client-key-generator" | include "helm-toolkit.snippets.kubernetes_metadata_labels" | indent 4 }}
spec:
template:
metadata:
labels:
{{ tuple $envAll "ceph" "client-key-generator" | include "helm-toolkit.snippets.kubernetes_metadata_labels" | indent 8 }}
annotations:
{{ dict "envAll" $envAll "podName" $serviceAccountName "containerNames" (list "ceph-storage-keys-generator" "init") | include "helm-toolkit.snippets.kubernetes_mandatory_access_control_annotation" | indent 8 }}
spec:
{{ dict "envAll" $envAll "application" "client_key_generator" | include "helm-toolkit.snippets.kubernetes_pod_security_context" | indent 6 }}
serviceAccountName: {{ $serviceAccountName }}
restartPolicy: OnFailure
nodeSelector:
{{ $envAll.Values.labels.job.node_selector_key }}: {{ $envAll.Values.labels.job.node_selector_value }}
initContainers:
{{ tuple $envAll "namespace_client_key_generator" list | include "helm-toolkit.snippets.kubernetes_entrypoint_init_container" | indent 8 }}
containers:
- name: ceph-storage-keys-generator
{{ tuple $envAll "ceph_config_helper" | include "helm-toolkit.snippets.image" | indent 10 }}
{{ tuple $envAll $envAll.Values.pod.resources.jobs.secret_provisioning | include "helm-toolkit.snippets.kubernetes_resources" | indent 10 }}
{{ dict "envAll" $envAll "application" "client_key_generator" "container" "ceph_storage_keys_generator" | include "helm-toolkit.snippets.kubernetes_container_security_context" | indent 10 }}
env:
- name: DEPLOYMENT_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: PVC_CEPH_RBD_STORAGECLASS_USER_SECRET_NAME
value: {{ .Values.secrets.keys.user }}
- name: PVC_CEPH_RBD_STORAGECLASS_ADMIN_SECRET_NAME
value: {{ .Values.secrets.keys.admin }}
- name: PVC_CEPH_RBD_STORAGECLASS_DEPLOYED_NAMESPACE
value: {{ .Values.admin_secret_namespace }}
command:
- /tmp/provisioner-rbd-namespace-client-key-manager.sh
volumeMounts:
- name: pod-tmp
mountPath: /tmp
- name: pod-etc-ceph
mountPath: /etc/ceph
- name: ceph-provisioners-bin-clients
mountPath: /tmp/provisioner-rbd-namespace-client-key-manager.sh
subPath: provisioner-rbd-namespace-client-key-manager.sh
readOnly: true
volumes:
- name: pod-tmp
emptyDir: {}
- name: pod-etc-ceph
emptyDir: {}
- name: ceph-provisioners-bin-clients
configMap:
name: {{ printf "%s-%s" $envAll.Release.Name "bin" | quote }}
defaultMode: 0555
{{- end }}

128
ceph-adapter-rook/templates/job-storage-admin-keys.yaml Normal file
View File

@ -0,0 +1,128 @@
{{/*
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/}}
{{- if .Values.manifests.job_storage_admin_keys }}
{{- $envAll := . }}
{{- $serviceAccountName := "ceph-storage-keys-generator" }}
{{ tuple $envAll "storage_keys_generator" $serviceAccountName | include "helm-toolkit.snippets.kubernetes_pod_rbac_serviceaccount" }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ $serviceAccountName }}
namespace: {{ .Values.admin_secret_namespace }}
rules:
- apiGroups:
- ""
resources:
- pods
- pods/exec
- secrets
verbs:
- get
- create
- patch
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: {{ $serviceAccountName }}
namespace: {{ .Values.admin_secret_namespace }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: {{ $serviceAccountName }}
subjects:
- kind: ServiceAccount
name: {{ $serviceAccountName }}
namespace: {{ $envAll.Release.Namespace }}
---
apiVersion: batch/v1
kind: Job
metadata:
name: ceph-storage-keys-generator
namespace: {{ .Values.admin_secret_namespace }}
labels:
{{ tuple $envAll "ceph" "storage-keys-generator" | include "helm-toolkit.snippets.kubernetes_metadata_labels" | indent 4 }}
annotations:
{{ tuple $envAll | include "helm-toolkit.snippets.release_uuid" }}
spec:
template:
metadata:
labels:
{{ tuple $envAll "ceph" "storage-keys-generator" | include "helm-toolkit.snippets.kubernetes_metadata_labels" | indent 8 }}
annotations:
{{ tuple $envAll | include "helm-toolkit.snippets.release_uuid" | indent 8 }}
configmap-bin-hash: {{ tuple "configmap-bin.yaml" . | include "helm-toolkit.utils.hash" }}
{{ dict "envAll" $envAll "podName" "ceph-storage-keys-generator" "containerNames" (list "ceph-storage-keys-generator" "init") | include "helm-toolkit.snippets.kubernetes_mandatory_access_control_annotation" | indent 8 }}
spec:
{{ dict "envAll" $envAll "application" "storage_keys_generator" | include "helm-toolkit.snippets.kubernetes_pod_security_context" | indent 6 }}
serviceAccountName: {{ $serviceAccountName }}
restartPolicy: OnFailure
nodeSelector:
{{ $envAll.Values.labels.job.node_selector_key }}: {{ $envAll.Values.labels.job.node_selector_value }}
initContainers:
{{ tuple $envAll "storage_keys_generator" list | include "helm-toolkit.snippets.kubernetes_entrypoint_init_container" | indent 8 }}
containers:
- name: ceph-storage-keys-generator
{{ tuple $envAll "ceph_config_helper" | include "helm-toolkit.snippets.image" | indent 10 }}
{{ tuple $envAll $envAll.Values.pod.resources.jobs.secret_provisioning | include "helm-toolkit.snippets.kubernetes_resources" | indent 10 }}
{{ dict "envAll" $envAll "application" "storage_keys_generator" "container" "ceph_storage_keys_generator" | include "helm-toolkit.snippets.kubernetes_container_security_context" | indent 10 }}
env:
- name: DEPLOYMENT_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: CEPH_GEN_DIR
value: /tmp
- name: CEPH_TEMPLATES_DIR
value: /tmp/templates
- name: CEPH_KEYRING_NAME
value: ceph.client.admin.keyring
- name: CEPH_KEYRING_TEMPLATE
value: admin.keyring
- name: CEPH_KEYRING_ADMIN_NAME
value: {{ .Values.secrets.keyrings.admin }}
- name: CEPH_STORAGECLASS_ADMIN_SECRET_NAME
value: {{ .Values.secrets.keys.admin }}
command:
- /tmp/keys-storage-keyring-manager.sh
volumeMounts:
- name: pod-tmp
mountPath: /tmp
- name: pod-etc-ceph
mountPath: /etc/ceph
- name: ceph-mon-bin
mountPath: /tmp/keys-storage-keyring-manager.sh
subPath: keys-storage-keyring-manager.sh
readOnly: true
- name: ceph-templates
mountPath: /tmp/templates
readOnly: true
volumes:
- name: pod-tmp
emptyDir: {}
- name: pod-etc-ceph
emptyDir: {}
- name: ceph-mon-bin
configMap:
name: {{ printf "%s-%s" $envAll.Release.Name "bin" | quote }}
defaultMode: 0555
- name: ceph-templates
configMap:
name: {{ printf "%s-%s" $envAll.Release.Name "templates" | quote }}
defaultMode: 0444
{{- end }}

37
ceph-adapter-rook/templates/service-mon-discovery.yaml Normal file
View File

@ -0,0 +1,37 @@
{{/*
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/}}
{{- if and .Values.manifests.service_mon_discovery }}
{{- $envAll := . }}
---
kind: Service
apiVersion: v1
metadata:
name: {{ tuple "ceph_mon" "discovery" . | include "helm-toolkit.endpoints.hostname_short_endpoint_lookup" }}
spec:
ports:
- name: mon
port: {{ tuple "ceph_mon" "discovery" "mon" $envAll | include "helm-toolkit.endpoints.endpoint_port_lookup" }}
protocol: TCP
targetPort: {{ tuple "ceph_mon" "discovery" "mon" $envAll | include "helm-toolkit.endpoints.endpoint_port_lookup" }}
- name: mon-msgr2
port: {{ tuple "ceph_mon" "discovery" "mon_msgr2" $envAll | include "helm-toolkit.endpoints.endpoint_port_lookup" }}
protocol: TCP
targetPort: {{ tuple "ceph_mon" "discovery" "mon_msgr2" $envAll | include "helm-toolkit.endpoints.endpoint_port_lookup" }}
selector:
app: rook-ceph-mon
ceph_daemon_type: mon
clusterIP: None
publishNotReadyAddresses: true
{{- end }}

119
ceph-adapter-rook/values.yaml Normal file
View File

@ -0,0 +1,119 @@
---
images:
pull_policy: IfNotPresent
tags:
ceph_config_helper: 'docker.io/openstackhelm/ceph-config-helper:ubuntu_focal_18.2.0-1-20231013'
dep_check: 'quay.io/airshipit/kubernetes-entrypoint:v1.0.0'
image_repo_sync: 'docker.io/library/docker:17.07.0'
local_registry:
active: false
exclude:
- dep_check
- image_repo_sync
labels:
job:
node_selector_key: openstack-control-plane
node_selector_value: enabled
pod:
security_context:
storage_keys_generator:
pod:
runAsUser: 65534
container:
ceph_storage_keys_generator:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
client_key_generator:
pod:
runAsUser: 99
container:
ceph_storage_keys_generator:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
dns_policy: "ClusterFirstWithHostNet"
affinity:
anti:
type:
default: preferredDuringSchedulingIgnoredDuringExecution
topologyKey:
default: kubernetes.io/hostname
weight:
default: 10
resources:
enabled: false
jobs:
secret_provisioning:
limits:
memory: "1024Mi"
cpu: "2000m"
requests:
memory: "128Mi"
cpu: "500m"
secrets:
keyrings:
admin: ceph-client-admin-keyring
keys:
admin: pvc-ceph-conf-combined-storageclass
user: pvc-ceph-client-key
admin_secret_namespace: ceph
ceph_configmap_name: ceph-etc
conf:
templates:
keyring:
admin: |
[client.admin]
key = {{ key }}
auid = 0
caps mds = "allow"
caps mon = "allow *"
caps osd = "allow *"
caps mgr = "allow *"
ceph:
global:
# auth
cephx: true
cephx_require_signatures: false
cephx_cluster_require_signatures: true
cephx_service_require_signatures: false
objecter_inflight_op_bytes: "1073741824"
objecter_inflight_ops: 10240
debug_ms: "0/0"
log_file: /dev/stdout
mon_cluster_log_file: /dev/stdout
# TODO: Get mon host from rook-ceph-mon-endpoints configmap
mon_host: "will be discovered"
endpoints:
cluster_domain_suffix: cluster.local
ceph_mon:
namespace: ceph
hosts:
default: ceph-mon
discovery: ceph-mon-discovery
host_fqdn_override:
default: null
port:
mon:
default: 6789
mon_msgr2:
default: 3300
dependencies:
static:
storage_keys_generator:
jobs: null
manifests:
configmap_bin: true
configmap_templates: true
configmap_etc: true
job_storage_admin_keys: true
job_namespace_client_key: true
job_namespace_client_ceph_config: true
service_mon_discovery: true
...

4
releasenotes/notes/ceph-adapter-rook.yaml Normal file
View File

@ -0,0 +1,4 @@
---
ceph-adapter-rook:
- 0.1.0 Initial Chart
...

54
tools/deployment/ceph/ceph-adapter-rook.sh Executable file
View File

@ -0,0 +1,54 @@
#!/bin/bash
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
set -xe
make ceph-adapter
tee > /tmp/ceph-adapter-rook-ceph.yaml <<EOF
manifests:
configmap_bin: true
configmap_templates: true
configmap_etc: false
job_storage_admin_keys: true
job_namespace_client_key: false
job_namespace_client_ceph_config: false
service_mon_discovery: true
EOF
helm upgrade --install ceph-adapter-rook ./ceph-adapter-rook \
--namespace=ceph \
--values=/tmp/ceph-adapter-rook-ceph.yaml
#NOTE: Wait for deploy
./tools/deployment/common/wait-for-pods.sh ceph
tee > /tmp/ceph-adapter-rook-openstack.yaml <<EOF
manifests:
configmap_bin: true
configmap_templates: false
configmap_etc: true
job_storage_admin_keys: false
job_namespace_client_key: true
job_namespace_client_ceph_config: true
service_mon_discovery: false
EOF
helm upgrade --install ceph-adapter-rook ./ceph-adapter-rook \
--namespace=openstack \
--values=/tmp/ceph-adapter-rook-openstack.yaml
#NOTE: Wait for deploy
./tools/deployment/common/wait-for-pods.sh openstack

78
tools/deployment/ceph/ceph-rook.sh
View File

@ -632,20 +632,6 @@ cephObjectStores:
volumeBindingMode: "Immediate"
parameters:
region: us-east-1
storageclass:
rbd:
parameters:
adminSecretName: pvc-ceph-conf-combined-storageclass
cephfs:
provision_storage_class: true
provisioner: ceph.com/cephfs
metadata:
name: cephfs
parameters:
adminId: admin
userSecretName: pvc-ceph-cephfs-client-key
adminSecretName: pvc-ceph-conf-combined-storageclass
adminSecretNamespace: ceph
EOF
helm upgrade --install --create-namespace --namespace ceph rook-ceph-cluster --set operatorNamespace=rook-ceph rook-release/rook-ceph-cluster --version ${ROOK_RELEASE} -f /tmp/ceph.yaml
@ -671,67 +657,3 @@ TOOLS_POD=$(kubectl get pods \
--selector="app=rook-ceph-tools" \
--no-headers | awk '{ print $1; exit }')
kubectl exec -n ceph ${TOOLS_POD} -- ceph -s
tee /tmp/ceph-supplemental.yaml <<EOF
endpoints:
ceph_mon:
namespace: null
hosts:
default: rook-ceph-mon-a
discovery: ceph-mon-discovery
port:
mon:
default: 6789
mon_msgr2:
default: 3300
deployment:
storage_secrets: true
ceph: true
csi_rbd_provisioner: false
client_secrets: false
rgw_keystone_user_and_endpoints: false
bootstrap:
enabled: false
manifests:
daemonset_mon: false
daemonset_osd: false
deployment_mds: false
deployment_mgr: false
deployment_mgr_sa: false
deployment_moncheck: false
helm_tests: false
job_bootstrap: false
service_mgr: false
service_mon: false
service_mon_discovery: true
job_storage_admin_keys: true
job_keyring: true
EOF
helm upgrade --install ceph-mon ./ceph-mon --namespace=ceph --values=/tmp/ceph-supplemental.yaml
./tools/deployment/common/wait-for-pods.sh ceph
# credentials for this object store user will be placed
# to the rook-ceph-object-user-default-s3-admin secret
# AccessKey is the secret field where the access key is stored
# SecretKey is the secret field where the secret key is stored
# cat > /tmp/s3_admin.yaml <<EOF
# apiVersion: ceph.rook.io/v1
# kind: CephObjectStoreUser
# metadata:
# name: s3-admin
# namespace: osh-infra
# spec:
# store: default
# clusterNamespace: ceph
# # this is what is passed to radosgw-admin as uid argument
# displayName: s3_admin
# capabilities:
# user: "*"
# bucket: "*"
# EOF
# kubectl apply -f /tmp/s3_admin.yaml

4
zuul.d/jobs.yaml
View File

@ -123,7 +123,7 @@
- ./tools/deployment/osh-infra-logging/000-prepare-k8s.sh
- ./tools/deployment/osh-infra-logging/010-ingress.sh
- ./tools/deployment/ceph/ceph-rook.sh
- ./tools/deployment/osh-infra-logging/025-ceph-ns-activate.sh
- ./tools/deployment/ceph/ceph-adapter-rook.sh
- ./tools/deployment/osh-infra-logging/040-ldap.sh
- ./tools/deployment/osh-infra-logging/050-elasticsearch.sh
- ./tools/deployment/osh-infra-logging/060-fluentd.sh
@ -226,7 +226,7 @@
- ./tools/deployment/openstack-support-rook/007-namespace-config.sh
- ./tools/deployment/openstack-support-rook/010-ingress.sh
- ./tools/deployment/ceph/ceph-rook.sh
- ./tools/deployment/openstack-support-rook/025-ceph-ns-activate.sh
- ./tools/deployment/ceph/ceph-adapter-rook.sh
- ./tools/deployment/openstack-support-rook/030-rabbitmq.sh
- ./tools/deployment/openstack-support-rook/070-mariadb.sh
- ./tools/deployment/openstack-support-rook/040-memcached.sh