openstack/openstack-helm
Code Issues Proposed changes
3e62ce320b
openstack-helm/doc/source/specs/tenant-ceph.rst
Parsons, Cliff (cp769u) 48738f5426 Update gate scripts for Ceph CSI RBD Provisioner 
This patch updates the gate scripts so that the Ceph CSI RBD Provisioner
will be tested appropriately and is documented properly.

Change-Id: I0ad86c9d53db3533b65a41387bbd426c9023d6ee
2021-06-11 14:27:53 +00:00

49 KiB
Raw Blame History

Deploying multuple Ceph clusters

This guide shows how to setup multiple Ceph clusters. One Ceph cluster will be used for k8s RBD storage and while other Ceph cluster will be for tenant facing storage backend for Cinder and Glance.

Ceph Clusters:

Ceph for RBD:

This Ceph cluster will be used for k8s RBD storage (pvc). This can be used by entire Kubernetes cluster.

  • k8s namespace: ceph
  • mon endpoint port: 6789
  • mgr endpoint port: 7000
  • metric port: 9283
  • storage classes: general (rbd based for pvc)
  • no ceph-mds and ceph-rgw

Ceph for Tenant:

This Ceph cluster will be used by Cinder and Glance as storage backend.

  • k8s namespace: tenant-ceph
  • mon endpoint port: 6790
  • mgr endpoint port: 7001
  • metric port: 9284
  • no storage classes
  • no ceph-mds

Env Setup:

6 VM based hosts (node1, node2, node3, node4, node5, node6)

k8s node labels:

Ceph for RBD related labels:

Labels assigned to nodes: node1, node2, node3:

openstack-control-plane=enabled, ceph-mon=enabled, ceph-mgr=enabled, ceph-rgw=enabled, ceph-mds=enabled, ceph-osd=enabled

Ceph for Tenant related labels:

Labels assigned to nodes: node1, node2, node3:

tenant-ceph-control-plane=enabled, ceph-mon-tenant=enabled, ceph-mgr-tenant=enabled, ceph-rgw-tenant=enabled

Labels assigned to nodes: node4, node5, node6:

openstack-data-plane=enabled, openstack-compute-node=enabled, ceph-osd-tenant=enabled, openstack-data-plane=enabled

k8s node list with labels After applying above labels, node labels should look like following.

ubuntu@node1:~$ kubectl get nodes --show-labels=true
NAME      STATUS    ROLES     AGE       VERSION   LABELS
node1     Ready     <none>    9m        v1.10.6   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ceph-mds=enabled,ceph-mgr-tenant=enabled,ceph-mgr=enabled,ceph-mon-tenant=enabled,ceph-mon=enabled,ceph-osd=enabled,ceph-rgw-tenant=enabled,ceph-rgw=enabled,kubernetes.io/hostname=node1,linuxbridge=enabled,openstack-control-plane=enabled,openstack-helm-node-class=primary,openvswitch=enabled,tenant-ceph-control-plane=enabled
node2     Ready     <none>    6m        v1.10.6   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ceph-mds=enabled,ceph-mgr-tenant=enabled,ceph-mgr=enabled,ceph-mon-tenant=enabled,ceph-mon=enabled,ceph-osd=enabled,ceph-rgw-tenant=enabled,ceph-rgw=enabled,kubernetes.io/hostname=node2,linuxbridge=enabled,openstack-control-plane=enabled,openstack-helm-node-class=general,openvswitch=enabled,tenant-ceph-control-plane=enabled
node3     Ready     <none>    6m        v1.10.6   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ceph-mds=enabled,ceph-mgr-tenant=enabled,ceph-mgr=enabled,ceph-mon-tenant=enabled,ceph-mon=enabled,ceph-osd=enabled,ceph-rgw-tenant=enabled,ceph-rgw=enabled,kubernetes.io/hostname=node3,linuxbridge=enabled,openstack-control-plane=enabled,openstack-helm-node-class=general,openvswitch=enabled,tenant-ceph-control-plane=enabled
node4     Ready     <none>    7m        v1.10.6   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ceph-osd-tenant=enabled,kubernetes.io/hostname=node4,linuxbridge=enabled,openstack-compute-node=enabled,openstack-data-plane=enabled,openstack-helm-node-class=general,openvswitch=enabled
node5     Ready     <none>    6m        v1.10.6   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ceph-osd-tenant=enabled,kubernetes.io/hostname=node5,linuxbridge=enabled,openstack-compute-node=enabled,openstack-data-plane=enabled,openstack-helm-node-class=general,openvswitch=enabled
node6     Ready     <none>    6m        v1.10.6   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ceph-osd-tenant=enabled,kubernetes.io/hostname=node6,linuxbridge=enabled,openstack-compute-node=enabled,openstack-data-plane=enabled,openstack-helm-node-class=general,openvswitch=enabled

Test Steps:

1) Prepare scripts:

OpenStack-Helm multinode guide includes scripts which are used to specify overrides and deploy charts.

Duplicate scripts as shows below for later use.

cd tools/deployment/multinode/
cp 030-ceph.sh 030-tenant-ceph.sh
cp 040-ceph-ns-activate.sh 040-tenant-ceph-ns-activate.sh
cp 090-ceph-radosgateway.sh 090-tenant-ceph-radosgateway.sh

2) Deploy ingress chart:

Script to update and execute: 020-ingress.sh

Update script to include namespace tenant-ceph as shown below.

for NAMESPACE in openstack ceph tenant-ceph; do

Execute script.

3) Deploy Ceph for RBD:

Script to update and execute: 030-ceph.sh

Update script with following overrides. Note: The original RBD provisioner is now deprecated. The CSI RBD provisioner is selected here. If you prefer the original non-CSI RBD provisioner, then set rbd_provisioner to true instead.

deployment:
  storage_secrets: true
  ceph: true
  rbd_provisioner: false
  csi_rbd_provisioner: true
  cephfs_provisioner: false
  client_secrets: false
endpoints:
  ceph_mon:
    namespace: ceph
    port:
      mon:
        default: 6789
  ceph_mgr:
    namespace: ceph
    port:
      mgr:
        default: 7000
      metrics:
        default: 9283
manifests:
  deployment_mds: false
bootstrap:
  enabled: true
conf:
  pool:
    target:
      osd: 3
storageclass:
  rbd:
    ceph_configmap_name: ceph-etc
  cephfs:
    provision_storage_class: false
ceph_mgr_modules_config:
  prometheus:
    server_port: 9283
monitoring:
  prometheus:
    enabled: true
    ceph_mgr:
      port: 9283

Note

cephfs_provisioner: false and provision_storage_class: false are set to false to disable cephfs. deployment_mds: false is set to disable ceph-mds

Execute script.

4) Deploy MariaDB, RabbitMQ, Memcached and Keystone:

Use default overrides and execute following scripts as per OSH guide steps:

  • 040-ceph-ns-activate.sh
  • 050-mariadb.sh
  • 060-rabbitmq.sh
  • 070-memcached.sh
  • 080-keystone.sh

Result from Steps 2, 3, 4:

Ceph Pods

ubuntu@node1:~$  kubectl get pods -n ceph -o wide
NAME                                    READY     STATUS      RESTARTS   AGE       IP              NODE
ceph-bootstrap-g45qc                    0/1       Completed   0          28m       192.168.5.16    node3
ceph-mds-keyring-generator-gsw4m        0/1       Completed   0          28m       192.168.2.11    node2
ceph-mgr-5746dd89db-mmrg4               1/1       Running     0          23m       10.0.0.12       node2
ceph-mgr-5746dd89db-q25lt               1/1       Running     0          23m       10.0.0.9        node3
ceph-mgr-keyring-generator-t4s8l        0/1       Completed   0          28m       192.168.2.9     node2
ceph-mon-6n4hk                          1/1       Running     0          28m       10.0.0.9        node3
ceph-mon-b2d9w                          1/1       Running     0          28m       10.0.0.12       node2
ceph-mon-check-d85994946-2dcpg          1/1       Running     0          28m       192.168.5.17    node3
ceph-mon-keyring-generator-rmvfz        0/1       Completed   0          28m       192.168.2.10    node2
ceph-mon-svkdl                          1/1       Running     0          28m       10.0.0.16       node1
ceph-osd-default-83945928-2mhrj         1/1       Running     0          25m       10.0.0.9        node3
ceph-osd-default-83945928-gqbd9         1/1       Running     0          25m       10.0.0.16       node1
ceph-osd-default-83945928-krrl8         1/1       Running     0          25m       10.0.0.12       node2
ceph-osd-keyring-generator-zg8s5        0/1       Completed   0          28m       192.168.0.195   node1
ceph-rbd-pool-92nbv                     0/1       Completed   0          23m       192.168.5.18    node3
ceph-rbd-provisioner-599895579c-jl6qk   1/1       Running     0          21m       192.168.2.15    node2
ceph-rbd-provisioner-599895579c-n4hbk   1/1       Running     0          21m       192.168.5.19    node3
ceph-rgw-keyring-generator-2wv4j        0/1       Completed   0          28m       192.168.5.15    node3
ceph-storage-keys-generator-8vzrx       0/1       Completed   0          28m       192.168.2.12    node2
ingress-796d8cf8d6-9khkm                1/1       Running     0          28m       192.168.2.6     node2
ingress-796d8cf8d6-nznvc                1/1       Running     0          28m       192.168.5.12    node3
ingress-error-pages-54454dc79b-bgc5m    1/1       Running     0          28m       192.168.2.5     node2
ingress-error-pages-54454dc79b-hwnv4    1/1       Running     0          28m       192.168.5.7     node3

Openstack Pods:

ubuntu@node1:~$ kubectl get pods -n openstack -o wide
NAME                                                READY     STATUS      RESTARTS   AGE       IP              NODE
ceph-openstack-config-ceph-ns-key-generator-mcxrs   0/1       Completed   0          11m       192.168.2.16    node2
ingress-7b4bc84cdd-7wslz                            1/1       Running     0          30m       192.168.5.5     node3
ingress-7b4bc84cdd-z6t2z                            1/1       Running     0          30m       192.168.2.4     node2
ingress-error-pages-586c7f86d6-7m58l                1/1       Running     0          30m       192.168.5.6     node3
ingress-error-pages-586c7f86d6-n9tzv                1/1       Running     0          30m       192.168.2.3     node2
keystone-api-7974676d5d-5k27d                       1/1       Running     0          6m        192.168.5.24    node3
keystone-api-7974676d5d-cd9kv                       1/1       Running     0          6m        192.168.2.21    node2
keystone-bootstrap-twfrj                            0/1       Completed   0          6m        192.168.0.197   node1
keystone-credential-setup-txf5p                     0/1       Completed   0          6m        192.168.5.25    node3
keystone-db-init-tjxgm                              0/1       Completed   0          6m        192.168.2.20    node2
keystone-db-sync-zl9t4                              0/1       Completed   0          6m        192.168.2.22    node2
keystone-domain-manage-thwdm                        0/1       Completed   0          6m        192.168.0.198   node1
keystone-fernet-setup-qm424                         0/1       Completed   0          6m        192.168.5.26    node3
keystone-rabbit-init-6699r                          0/1       Completed   0          6m        192.168.2.23    node2
keystone-test                                       0/1       Completed   0          4m        192.168.3.3     node4
mariadb-ingress-84894687fd-wfc9b                    1/1       Running     0          11m       192.168.2.17    node2
mariadb-ingress-error-pages-78fb865f84-bg8sg        1/1       Running     0          11m       192.168.5.20    node3
mariadb-server-0                                    1/1       Running     0          11m       192.168.5.22    node3
memcached-memcached-5db74ddfd5-m5gw2                1/1       Running     0          7m        192.168.2.19    node2
rabbitmq-rabbitmq-0                                 1/1       Running     0          8m        192.168.2.18    node2
rabbitmq-rabbitmq-1                                 1/1       Running     0          8m        192.168.5.23    node3
rabbitmq-rabbitmq-2                                 1/1       Running     0          8m        192.168.0.196   node1

Ceph Status

ubuntu@node1:~$ kubectl exec -n ceph ceph-mon-b2d9w -- ceph -s
  cluster:
    id:     3e53e3b7-e5d9-4bab-9701-134687f4954e
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum node3,node2,node1
    mgr: node3(active), standbys: node2
    osd: 3 osds: 3 up, 3 in

  data:
    pools:   18 pools, 93 pgs
    objects: 127 objects, 218 MB
    usage:   46820 MB used, 186 GB / 232 GB avail
    pgs:     93 active+clean

Ceph ConfigMaps

ubuntu@node1:~$ kubectl get cm -n ceph
NAME                                      DATA      AGE
ceph-client-bin                           7         25m
ceph-client-etc                           1         25m
ceph-etc                                  1         23m
ceph-mon-bin                              10        29m
ceph-mon-etc                              1         29m
ceph-osd-bin                              7         27m
ceph-osd-default                          1         27m
ceph-osd-etc                              1         27m
ceph-provisioners-ceph-provisioners-bin   4         23m
ceph-templates                            6         29m
ingress-bin                               2         30m
ingress-ceph-nginx                        0         30m
ingress-conf                              3         30m
ingress-services-tcp                      0         30m
ingress-services-udp                      0         30m

ceph-mon-etc (ceph.conf)

ubuntu@node1:~$ kubectl get cm -n ceph ceph-mon-etc -o yaml
apiVersion: v1
data:
  ceph.conf: |
    [global]
    cephx = true
    cephx_cluster_require_signatures = true
    cephx_require_signatures = false
    cephx_service_require_signatures = false
    fsid = 3e53e3b7-e5d9-4bab-9701-134687f4954e
    mon_addr = :6789
    mon_host = ceph-mon-discovery.ceph.svc.cluster.local:6789
    [osd]
    cluster_network = 10.0.0.0/24
    ms_bind_port_max = 7100
    ms_bind_port_min = 6800
    osd_max_object_name_len = 256
    osd_mkfs_options_xfs = -f -i size=2048
    osd_mkfs_type = xfs
    public_network = 10.0.0.0/24
kind: ConfigMap
metadata:
  creationTimestamp: 2018-08-27T04:55:32Z
  name: ceph-mon-etc
  namespace: ceph
  resourceVersion: "3218"
  selfLink: /api/v1/namespaces/ceph/configmaps/ceph-mon-etc
  uid: 6d9fdcba-a9b5-11e8-bb1d-fa163ec12213

Note

Note that mon_addr and mon_host have default mon port 6789.

k8s storageclass

ubuntu@node1:~$ kubectl get storageclasses
NAME      PROVISIONER    AGE
general   ceph.com/rbd   14m

Ceph services

ubuntu@node1:~$ kubectl get svc -n ceph
NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
ceph-mgr              ClusterIP   10.111.185.73    <none>        7000/TCP,9283/TCP   27m
ceph-mon              ClusterIP   None             <none>        6789/TCP            31m
ceph-mon-discovery    ClusterIP   None             <none>        6789/TCP            31m
ingress               ClusterIP   10.100.23.32     <none>        80/TCP,443/TCP      32m
ingress-error-pages   ClusterIP   None             <none>        80/TCP              32m
ingress-exporter      ClusterIP   10.109.196.155   <none>        10254/TCP           32m

Ceph endpoints

ubuntu@node1:~$ kubectl get endpoints -n ceph
NAME                  ENDPOINTS                                                    AGE
ceph-mgr              10.0.0.12:9283,10.0.0.9:9283,10.0.0.12:7000 + 1 more...      27m
ceph-mon              10.0.0.12:6789,10.0.0.16:6789,10.0.0.9:6789                  31m
ceph-mon-discovery    10.0.0.12:6789,10.0.0.16:6789,10.0.0.9:6789                  31m
ingress               192.168.2.6:80,192.168.5.12:80,192.168.2.6:443 + 1 more...   32m
ingress-error-pages   192.168.2.5:8080,192.168.5.7:8080                            32m
ingress-exporter      192.168.2.6:10254,192.168.5.12:10254                         32m

netstat ceph mon port

ubuntu@node1: netstat -ntlp | grep 6789
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 10.0.0.16:6789          0.0.0.0:*               LISTEN      -

ubuntu@node1: netstat -ntlp | grep 6790
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)

Ceph secrets

ubuntu@node1:~$ kubectl get secrets -n ceph
NAME                                                 TYPE                                  DATA      AGE
ceph-bootstrap-mds-keyring                           Opaque                                1         34m
ceph-bootstrap-mgr-keyring                           Opaque                                1         34m
ceph-bootstrap-osd-keyring                           Opaque                                1         34m
ceph-bootstrap-rgw-keyring                           Opaque                                1         34m
ceph-bootstrap-token-w2sqp                           kubernetes.io/service-account-token   3         34m
ceph-client-admin-keyring                            Opaque                                1         34m
ceph-mds-keyring-generator-token-s9kst               kubernetes.io/service-account-token   3         34m
ceph-mgr-keyring-generator-token-h5sw6               kubernetes.io/service-account-token   3         34m
ceph-mgr-token-hr88m                                 kubernetes.io/service-account-token   3         30m
ceph-mon-check-token-bfvgk                           kubernetes.io/service-account-token   3         34m
ceph-mon-keyring                                     Opaque                                1         34m
ceph-mon-keyring-generator-token-5gs5q               kubernetes.io/service-account-token   3         34m
ceph-mon-token-zsd6w                                 kubernetes.io/service-account-token   3         34m
ceph-osd-keyring-generator-token-h97wb               kubernetes.io/service-account-token   3         34m
ceph-osd-token-4wfm5                                 kubernetes.io/service-account-token   3         32m
ceph-provisioners-ceph-rbd-provisioner-token-f92tw   kubernetes.io/service-account-token   3         28m
ceph-rbd-pool-token-p2nxt                            kubernetes.io/service-account-token   3         30m
ceph-rgw-keyring-generator-token-wmfx6               kubernetes.io/service-account-token   3         34m
ceph-storage-keys-generator-token-dq5ts              kubernetes.io/service-account-token   3         34m
default-token-j8h48                                  kubernetes.io/service-account-token   3         35m
ingress-ceph-ingress-token-68rws                     kubernetes.io/service-account-token   3         35m
ingress-error-pages-token-mpvhm                      kubernetes.io/service-account-token   3         35m
pvc-ceph-conf-combined-storageclass                  kubernetes.io/rbd                     1         34m

Openstack secrets

ubuntu@node1:~$ kubectl get secrets -n openstack
NAME                                                      TYPE                                  DATA      AGE
ceph-openstack-config-ceph-ns-key-cleaner-token-jj7n6     kubernetes.io/service-account-token   3         17m
ceph-openstack-config-ceph-ns-key-generator-token-5sqfw   kubernetes.io/service-account-token   3         17m
default-token-r5knr                                       kubernetes.io/service-account-token   3         35m
ingress-error-pages-token-xxjxt                           kubernetes.io/service-account-token   3         35m
ingress-openstack-ingress-token-hrvv8                     kubernetes.io/service-account-token   3         35m
keystone-api-token-xwczg                                  kubernetes.io/service-account-token   3         12m
keystone-bootstrap-token-dhnb6                            kubernetes.io/service-account-token   3         12m
keystone-credential-keys                                  Opaque                                2         12m
keystone-credential-rotate-token-68lnk                    kubernetes.io/service-account-token   3         12m
keystone-credential-setup-token-b2smc                     kubernetes.io/service-account-token   3         12m
keystone-db-admin                                         Opaque                                1         12m
keystone-db-init-token-brzkj                              kubernetes.io/service-account-token   3         12m
keystone-db-sync-token-xzqj9                              kubernetes.io/service-account-token   3         12m
keystone-db-user                                          Opaque                                1         12m
keystone-domain-manage-token-48gn5                        kubernetes.io/service-account-token   3         12m
keystone-etc                                              Opaque                                9         12m
keystone-fernet-keys                                      Opaque                                2         12m
keystone-fernet-rotate-token-djtzb                        kubernetes.io/service-account-token   3         12m
keystone-fernet-setup-token-n9st2                         kubernetes.io/service-account-token   3         12m
keystone-keystone-admin                                   Opaque                                8         12m
keystone-keystone-test                                    Opaque                                8         12m
keystone-rabbit-init-token-pt5b2                          kubernetes.io/service-account-token   3         12m
keystone-rabbitmq-admin                                   Opaque                                1         12m
keystone-rabbitmq-user                                    Opaque                                1         12m
keystone-test-token-z8mb6                                 kubernetes.io/service-account-token   3         12m
mariadb-db-root-password                                  Opaque                                1         17m
mariadb-ingress-error-pages-token-cnrqp                   kubernetes.io/service-account-token   3         17m
mariadb-ingress-token-gfrg4                               kubernetes.io/service-account-token   3         17m
mariadb-secrets                                           Opaque                                1         17m
mariadb-token-pr5lp                                       kubernetes.io/service-account-token   3         17m
memcached-memcached-token-gq96p                           kubernetes.io/service-account-token   3         13m
pvc-ceph-client-key                                       kubernetes.io/rbd                     1         17m
rabbitmq-rabbitmq-token-5bj85                             kubernetes.io/service-account-token   3         14m
rabbitmq-test-token-w4clj                                 kubernetes.io/service-account-token   3         14m

Openstack PV list

ubuntu@node1:~$ kubectl get pv -n openstack
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                                         STORAGECLASS   REASON    AGE
pvc-348f4c52-a9b8-11e8-bb1d-fa163ec12213   256Mi      RWO            Delete           Bound     openstack/rabbitmq-data-rabbitmq-rabbitmq-0   general                  15m
pvc-4418c745-a9b8-11e8-bb1d-fa163ec12213   256Mi      RWO            Delete           Bound     openstack/rabbitmq-data-rabbitmq-rabbitmq-1   general                  14m
pvc-524d4213-a9b8-11e8-bb1d-fa163ec12213   256Mi      RWO            Delete           Bound     openstack/rabbitmq-data-rabbitmq-rabbitmq-2   general                  14m
pvc-da9c9dd2-a9b7-11e8-bb1d-fa163ec12213   5Gi        RWO            Delete           Bound     openstack/mysql-data-mariadb-server-0         general                  17m

Openstack endpoints

ubuntu@node1:~$ openstack endpoint list
+----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------------------+
| ID                               | Region    | Service Name | Service Type | Enabled | Interface | URL                                                     |
+----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------------------+
| 480cc7360752498e822cbbc7211d213a | RegionOne | keystone     | identity     | True    | internal  | http://keystone-api.openstack.svc.cluster.local:5000/v3 |
| 8dfe4e4725b84e51a5eda564dee0960c | RegionOne | keystone     | identity     | True    | public    | http://keystone.openstack.svc.cluster.local:80/v3       |
| 9b3526e36307400b9accfc7cc834cf99 | RegionOne | keystone     | identity     | True    | admin     | http://keystone.openstack.svc.cluster.local:80/v3       |
+----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------------------+

Openstack services

ubuntu@node1:~$ openstack service list
+----------------------------------+----------+----------+
| ID                               | Name     | Type     |
+----------------------------------+----------+----------+
| 67cc6b945e934246b25d31a9374a64af | keystone | identity |
+----------------------------------+----------+----------+

5) Deploy Ceph for Tenant:

Script to update and execute: 030-tenant-ceph.sh

Make following changes to script: 1 Replace occurrence of ceph-fs-uuid.txt with tenant-ceph-fs-uuid.txt

2 Replace occurrence of ceph.yaml with tenant-ceph.yaml

3 For tenant Ceph, no need to deploy ceph-provisioners. Update script to for CHART in ceph-mon ceph-osd ceph-client; do

Update script's override section with following:

endpoints:
  identity:
    namespace: openstack
  object_store:
    namespace: openstack
  ceph_mon:
    namespace: tenant-ceph
    port:
      mon:
        default: 6790
  ceph_mgr:
    namespace: tenant-ceph
    port:
      mgr:
        default: 7001
      metrics:
        default: 9284
network:
  public: ${CEPH_PUBLIC_NETWORK}
  cluster: ${CEPH_CLUSTER_NETWORK}
deployment:
  storage_secrets: true
  ceph: true
  rbd_provisioner: false
  csi_rbd_provisioner: false
  cephfs_provisioner: false
  client_secrets: false
labels:
  mon:
    node_selector_key: ceph-mon-tenant
  osd:
    node_selector_key: ceph-osd-tenant
  rgw:
    node_selector_key: ceph-rgw-tenant
  mgr:
    node_selector_key: ceph-mgr-tenant
  job:
    node_selector_key: tenant-ceph-control-plane
storageclass:
  rbd:
    ceph_configmap_name: tenant-ceph-etc
    provision_storage_class: false
    name: tenant-rbd
    admin_secret_name: pvc-tenant-ceph-conf-combined-storageclass
    admin_secret_namespace: tenant-ceph
    user_secret_name: pvc-tenant-ceph-client-key
  cephfs:
    provision_storage_class: false
    name: cephfs
    user_secret_name: pvc-tenant-ceph-cephfs-client-key
    admin_secret_name: pvc-tenant-ceph-conf-combined-storageclass
    admin_secret_namespace: tenant-ceph
bootstrap:
  enabled: true
manifests:
  deployment_mds: false
ceph_mgr_modules_config:
  prometheus:
    server_port: 9284
monitoring:
  prometheus:
    enabled: true
    ceph_mgr:
      port: 9284
conf:
  ceph:
    global:
      fsid: ${CEPH_FS_ID}
  rgw_ks:
    enabled: true
  pool:
    crush:
      tunables: ${CRUSH_TUNABLES}
    target:
      osd: 3
      pg_per_osd: 100
  storage:
    osd:
      - data:
          type: directory
          location: /var/lib/openstack-helm/tenant-ceph/osd/osd-one
        journal:
          type: directory
          location: /var/lib/openstack-helm/tenant-ceph/osd/journal-one
    mon:
      directory: /var/lib/openstack-helm/tenant-ceph/mon

Note

- Port numbers for Ceph_Mon and Ceph_Mgr are different from default. - We are disabling rbd and cephfs provisioners. - Labels for mon, osd, rgw, mgr and job have been updated for tenant Ceph. - Under storageclass section, values for following have been updated: ceph_configmap_name, admin_secret_name, admin_secret_namespace, user_secret_name - Under storage: mon directory have been updated.

For Tenant Ceph, we will not be provisioning storage classes therefor, update script to not install ceph-provisioners chart as following.

for CHART in ceph-mon ceph-osd ceph-client; do

Execute script.

6) Enable Openstack namespace to use Tenant Ceph:

Script to update and execute: 040-tenant-ceph-ns-activate.sh

Update script as following:

...
tee /tmp/tenant-ceph-openstack-config.yaml <<EOF
endpoints:
  identity:
    namespace: openstack
  object_store:
    namespace: openstack
  ceph_mon:
    namespace: tenant-ceph
    port:
      mon:
        default: 6790
network:
  public: ${CEPH_PUBLIC_NETWORK}
  cluster: ${CEPH_CLUSTER_NETWORK}
deployment:
  storage_secrets: false
  ceph: false
  rbd_provisioner: false
  csi_rbd_provisioner: false
  cephfs_provisioner: false
  client_secrets: true
bootstrap:
  enabled: false
conf:
  rgw_ks:
    enabled: true
storageclass:
  rbd:
    ceph_configmap_name: tenant-ceph-etc
    provision_storage_class: false
    name: tenant-rbd
    admin_secret_name: pvc-tenant-ceph-conf-combined-storageclass
    admin_secret_namespace: tenant-ceph
    user_secret_name: pvc-tenant-ceph-client-key
  cephfs:
    provision_storage_class: false
    name: cephfs
    admin_secret_name: pvc-tenant-ceph-conf-combined-storageclass
    admin_secret_namespace: tenant-ceph
    user_secret_name: pvc-tenant-ceph-cephfs-client-key
EOF
helm upgrade --install tenant-ceph-openstack-config ./ceph-provisioners \
  --namespace=openstack \
  --values=/tmp/tenant-ceph-openstack-config.yaml \
  ${OSH_EXTRA_HELM_ARGS} \
  ${OSH_EXTRA_HELM_ARGS_CEPH_NS_ACTIVATE}

#NOTE: Wait for deploy
./tools/deployment/common/wait-for-pods.sh openstack

#NOTE: Validate Deployment info
helm status tenant-ceph-openstack-config

Execute script.

7) Tenant Ceph: Deploy Rados Gateway:

Script to update: 090-tenant-ceph-radosgateway.sh

Update script with following overrides:

tee /tmp/tenant-radosgw-openstack.yaml <<EOF
endpoints:
  identity:
    namespace: openstack
  object_store:
    namespace: openstack
  ceph_mon:
    namespace: tenant-ceph
    port:
      mon:
        default: 6790
network:
  public: ${CEPH_PUBLIC_NETWORK}
  cluster: ${CEPH_CLUSTER_NETWORK}
deployment:
  storage_secrets: false
  ceph: true
  rbd_provisioner: false
  csi_rbd_provisioner: false
  cephfs_provisioner: false
  client_secrets: false
bootstrap:
  enabled: false
conf:
  rgw_ks:
    enabled: true
secrets:
  keyrings:
    admin: pvc-tenant-ceph-client-key
    rgw: os-ceph-bootstrap-rgw-keyring
  identity:
    admin: ceph-keystone-admin
    swift: ceph-keystone-user
    user_rgw: ceph-keystone-user-rgw
ceph_client:
  configmap: tenant-ceph-etc
EOF
helm upgrade --install tenant-radosgw-openstack ./ceph-rgw \
  --namespace=openstack \
  --values=/tmp/tenant-radosgw-openstack.yaml \
  ${OSH_EXTRA_HELM_ARGS} \
  ${OSH_EXTRA_HELM_ARGS_HEAT}

#NOTE: Wait for deploy
./tools/deployment/common/wait-for-pods.sh openstack

#NOTE: Validate Deployment info
helm status tenant-radosgw-openstack

Execute script

+ openstack service list
+----------------------------------+----------+--------------+
| ID                               | Name     | Type         |
+----------------------------------+----------+--------------+
| 0eddeb6af4fd43ea8f73f63a1ae01438 | swift    | object-store |
| 67cc6b945e934246b25d31a9374a64af | keystone | identity     |
+----------------------------------+----------+--------------+
ubuntu@node1: openstack endpoint list
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------------------------+
| ID                               | Region    | Service Name | Service Type | Enabled | Interface | URL                                                                         |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------------------------+
| 265212a5856e4a0aba8eb294508279c7 | RegionOne | swift        | object-store | True    | admin     | http://ceph-rgw.openstack.svc.cluster.local:8088/swift/v1/KEY_$(tenant_id)s |
| 430174e280444598b676d503c5ed9799 | RegionOne | swift        | object-store | True    | internal  | http://ceph-rgw.openstack.svc.cluster.local:8088/swift/v1/KEY_$(tenant_id)s |
| 480cc7360752498e822cbbc7211d213a | RegionOne | keystone     | identity     | True    | internal  | http://keystone-api.openstack.svc.cluster.local:5000/v3                     |
| 8dfe4e4725b84e51a5eda564dee0960c | RegionOne | keystone     | identity     | True    | public    | http://keystone.openstack.svc.cluster.local:80/v3                           |
| 948552a0d90940f7944f8c2eba7ef462 | RegionOne | swift        | object-store | True    | public    | http://radosgw.openstack.svc.cluster.local:80/swift/v1/KEY_$(tenant_id)s    |
| 9b3526e36307400b9accfc7cc834cf99 | RegionOne | keystone     | identity     | True    | admin     | http://keystone.openstack.svc.cluster.local:80/v3                           |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------------------------+

Results from Step 5, 6, 7:

Storage on node1, node2, node3:

ubuntu@node1:~$ ls -l /var/lib/openstack-helm/
total 8
drwxr-xr-x 4 root root 4096 Aug 27 04:57 ceph
drwxr-xr-x 3 root root 4096 Aug 27 05:47 tenant-ceph

Storage on node4, node5, node6:

ubuntu@node6:~$ ls -l /var/lib/openstack-helm/
total 4
drwxr-xr-x 3 root root 4096 Aug 27 05:49 tenant-ceph

Ceph Status

ubuntu@node1: kubectl exec -n tenant-ceph ceph-mon-2g6km -- ceph -s
  cluster:
    id:     38339a5a-d976-49dd-88a0-2ac092c271c7
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum node3,node2,node1
    mgr: node2(active), standbys: node1
    osd: 3 osds: 3 up, 3 in
    rgw: 2 daemons active

  data:
    pools:   18 pools, 93 pgs
    objects: 193 objects, 37421 bytes
    usage:   33394 MB used, 199 GB / 232 GB avail
    pgs:     93 active+clean
ubuntu@node1: kubectl get cm -n openstack
NAME                                                 DATA      AGE
ceph-etc                                             1         2h
ceph-openstack-config-ceph-prov-bin-clients          2         2h
ceph-rgw-bin                                         5         3m
ceph-rgw-bin-ks                                      3         3m
ceph-rgw-etc                                         1         3m
tenant-ceph-etc                                      1         1h
tenant-ceph-openstack-config-ceph-prov-bin-clients   2         1h
tenant-radosgw-openstack-ceph-templates              1         3m
...
ubuntu@node1: kubectl get cm -n openstack ceph-rgw-etc -o yaml
apiVersion: v1
data:
  ceph.conf: |
    [global]
    cephx = true
    cephx_cluster_require_signatures = true
    cephx_require_signatures = false
    cephx_service_require_signatures = false
    mon_addr = :6790
    mon_host = ceph-mon.tenant-ceph.svc.cluster.local:6790
    [osd]
    cluster_network = 10.0.0.0/24
    ms_bind_port_max = 7100
    ms_bind_port_min = 6800
    osd_max_object_name_len = 256
    osd_mkfs_options_xfs = -f -i size=2048
    osd_mkfs_type = xfs
    public_network = 10.0.0.0/24
kind: ConfigMap
metadata:
  creationTimestamp: 2018-08-27T07:47:59Z
  name: ceph-rgw-etc
  namespace: openstack
  resourceVersion: "30058"
  selfLink: /api/v1/namespaces/openstack/configmaps/ceph-rgw-etc
  uid: 848df05c-a9cd-11e8-bb1d-fa163ec12213

Note

mon_addr and mon_host have non default mon port 6790.

ubuntu@node1: kubectl get secrets -n openstack
NAME                                                             TYPE                                  DATA      AGE
ceph-keystone-admin                                              Opaque                                8         4m
ceph-keystone-user                                               Opaque                                8         4m
ceph-keystone-user-rgw                                           Opaque                                8         4m
ceph-ks-endpoints-token-crnrr                                    kubernetes.io/service-account-token   3         4m
ceph-ks-service-token-9bnr8                                      kubernetes.io/service-account-token   3         4m
ceph-openstack-config-ceph-ns-key-cleaner-token-jj7n6            kubernetes.io/service-account-token   3         2h
ceph-openstack-config-ceph-ns-key-generator-token-5sqfw          kubernetes.io/service-account-token   3         2h
ceph-rgw-storage-init-token-mhqdw                                kubernetes.io/service-account-token   3         4m
ceph-rgw-token-9s6nd                                             kubernetes.io/service-account-token   3         4m
os-ceph-bootstrap-rgw-keyring                                    Opaque                                1         36m
pvc-ceph-client-key                                              kubernetes.io/rbd                     1         2h
pvc-tenant-ceph-client-key                                       kubernetes.io/rbd                     1         1h
swift-ks-user-token-9slvc                                        kubernetes.io/service-account-token   3         4m
tenant-ceph-openstack-config-ceph-ns-key-cleaner-token-r6v9v     kubernetes.io/service-account-token   3         1h
tenant-ceph-openstack-config-ceph-ns-key-generator-token-dt472   kubernetes.io/service-account-token   3         1h
...
ubuntu@node1: kubectl get svc -n tenant-ceph
NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
ceph-mgr              ClusterIP   10.107.183.4     <none>        7001/TCP,9284/TCP   2h
ceph-mon              ClusterIP   None             <none>        6790/TCP            2h
ceph-mon-discovery    ClusterIP   None             <none>        6790/TCP            2h
ingress               ClusterIP   10.109.105.140   <none>        80/TCP,443/TCP      3h
ingress-error-pages   ClusterIP   None             <none>        80/TCP              3h
ingress-exporter      ClusterIP   10.102.110.153   <none>        10254/TCP           3h
ubuntu@node1: kubectl get endpoints -n tenant-ceph
NAME                  ENDPOINTS                                                    AGE
ceph-mgr              10.0.0.12:9284,10.0.0.16:9284,10.0.0.12:7001 + 1 more...     2h
ceph-mon              10.0.0.12:6790,10.0.0.16:6790,10.0.0.9:6790                  2h
ceph-mon-discovery    10.0.0.12:6790,10.0.0.16:6790,10.0.0.9:6790                  2h
ingress               192.168.2.7:80,192.168.5.14:80,192.168.2.7:443 + 1 more...   3h
ingress-error-pages   192.168.2.8:8080,192.168.5.13:8080                           3h
ingress-exporter      192.168.2.7:10254,192.168.5.14:10254                         3h
ubuntu@node1: kubectl get endpoints -n openstack
NAME                          ENDPOINTS                                                               AGE
ceph-rgw                      192.168.2.42:8088,192.168.5.44:8088                                     20m
ingress                       192.168.2.4:80,192.168.5.5:80,192.168.2.4:443 + 1 more...               3h
ingress-error-pages           192.168.2.3:8080,192.168.5.6:8080                                       3h
ingress-exporter              192.168.2.4:10254,192.168.5.5:10254                                     3h
keystone                      192.168.2.4:80,192.168.5.5:80,192.168.2.4:443 + 1 more...               2h
keystone-api                  192.168.2.21:5000,192.168.5.24:5000                                     2h
mariadb                       192.168.2.17:3306                                                       2h
mariadb-discovery             192.168.5.22:4567,192.168.5.22:3306                                     2h
mariadb-ingress-error-pages   192.168.5.20:8080                                                       2h
mariadb-server                192.168.5.22:3306                                                       2h
memcached                     192.168.2.19:11211                                                      2h
rabbitmq                      192.168.0.196:15672,192.168.2.18:15672,192.168.5.23:15672 + 6 more...   2h
rabbitmq-dsv-7b1733           192.168.0.196:15672,192.168.2.18:15672,192.168.5.23:15672 + 6 more...   2h
rabbitmq-mgr-7b1733           192.168.2.4:80,192.168.5.5:80,192.168.2.4:443 + 1 more...               2h
radosgw                       192.168.2.4:80,192.168.5.5:80,192.168.2.4:443 + 1 more...               20m
ubuntu@node1: kubectl get svc -n openstack
NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                        AGE
ceph-rgw                      ClusterIP   10.102.173.130   <none>        8088/TCP                       20m
ingress                       ClusterIP   10.102.1.71      <none>        80/TCP,443/TCP                 3h
ingress-error-pages           ClusterIP   None             <none>        80/TCP                         3h
ingress-exporter              ClusterIP   10.105.29.29     <none>        10254/TCP                      3h
keystone                      ClusterIP   10.108.94.108    <none>        80/TCP,443/TCP                 2h
keystone-api                  ClusterIP   10.99.50.35      <none>        5000/TCP                       2h
mariadb                       ClusterIP   10.111.140.93    <none>        3306/TCP                       2h
mariadb-discovery             ClusterIP   None             <none>        3306/TCP,4567/TCP              2h
mariadb-ingress-error-pages   ClusterIP   None             <none>        80/TCP                         2h
mariadb-server                ClusterIP   10.101.237.241   <none>        3306/TCP                       2h
memcached                     ClusterIP   10.111.175.130   <none>        11211/TCP                      2h
rabbitmq                      ClusterIP   10.96.78.137     <none>        5672/TCP,25672/TCP,15672/TCP   2h
rabbitmq-dsv-7b1733           ClusterIP   None             <none>        5672/TCP,25672/TCP,15672/TCP   2h
rabbitmq-mgr-7b1733           ClusterIP   10.104.105.46    <none>        80/TCP,443/TCP                 2h
radosgw                       ClusterIP   10.101.237.167   <none>        80/TCP,443/TCP                 20m
ubuntu@node1: kubectl get storageclasses
NAME      PROVISIONER    AGE
general   ceph.com/rbd   1h

8) Deploy Glance:

Script to update and execute: 100-glance.sh

Update script overrides as following:

endpoints:
  object_store:
    namespace: tenant-ceph
  ceph_object_store:
    namespace: tenant-ceph
ceph_client:
  configmap: tenant-ceph-etc
  user_secret_name: tenant-pvc-ceph-client-key
ubuntu@node1: openstack service list
+----------------------------------+----------+--------------+
| ID                               | Name     | Type         |
+----------------------------------+----------+--------------+
| 0eddeb6af4fd43ea8f73f63a1ae01438 | swift    | object-store |
| 67cc6b945e934246b25d31a9374a64af | keystone | identity     |
| 81a61ec8eff74070bb3c2f0118c1bcd5 | glance   | image        |
+----------------------------------+----------+--------------+
ubuntu@node1: openstack endpoint list
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------------------------+
| ID                               | Region    | Service Name | Service Type | Enabled | Interface | URL                                                                         |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------------------------+
| 265212a5856e4a0aba8eb294508279c7 | RegionOne | swift        | object-store | True    | admin     | http://ceph-rgw.openstack.svc.cluster.local:8088/swift/v1/KEY_$(tenant_id)s |
| 3fd88bc6e4774ff78c94bfa8aaaec3cf | RegionOne | glance       | image        | True    | admin     | http://glance-api.openstack.svc.cluster.local:9292/                         |
| 430174e280444598b676d503c5ed9799 | RegionOne | swift        | object-store | True    | internal  | http://ceph-rgw.openstack.svc.cluster.local:8088/swift/v1/KEY_$(tenant_id)s |
| 47505d5186ab448e9213f67bc833d2f1 | RegionOne | glance       | image        | True    | public    | http://glance.openstack.svc.cluster.local:80/                               |
| 480cc7360752498e822cbbc7211d213a | RegionOne | keystone     | identity     | True    | internal  | http://keystone-api.openstack.svc.cluster.local:5000/v3                     |
| 8dfe4e4725b84e51a5eda564dee0960c | RegionOne | keystone     | identity     | True    | public    | http://keystone.openstack.svc.cluster.local:80/v3                           |
| 937c2eacce8b4159bf918f4005c2b0ab | RegionOne | glance       | image        | True    | internal  | http://glance-api.openstack.svc.cluster.local:9292/                         |
| 948552a0d90940f7944f8c2eba7ef462 | RegionOne | swift        | object-store | True    | public    | http://radosgw.openstack.svc.cluster.local:80/swift/v1/KEY_$(tenant_id)s    |
| 9b3526e36307400b9accfc7cc834cf99 | RegionOne | keystone     | identity     | True    | admin     | http://keystone.openstack.svc.cluster.local:80/v3                           |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------------------------+

Note

Above output shows http://ceph-rgw.openstack.svc.cluster.local which shows that swift is pointing to tenant-ceph.

9) Deploy Cinder:

Script to update and execute: 110-cinder.sh

Update script overrides as following:

backup:
  posix:
    volume:
      class_name: rbd-tenant
ceph_client:
  configmap: tenant-ceph-etc
  user_secret_name: pvc-tenant-ceph-client-key
+ OS_CLOUD=openstack_helm
+ openstack service list
+----------------------------------+----------+--------------+
| ID                               | Name     | Type         |
+----------------------------------+----------+--------------+
| 0eddeb6af4fd43ea8f73f63a1ae01438 | swift    | object-store |
| 66bd0179eada4ab8899a58356fd4d508 | cinder   | volume       |
| 67cc6b945e934246b25d31a9374a64af | keystone | identity     |
| 81a61ec8eff74070bb3c2f0118c1bcd5 | glance   | image        |
| c126046fc5ec4c52acfc8fee0e2f4dda | cinderv2 | volumev2     |
| f89b99a31a124b7790e3bb60387380b1 | cinderv3 | volumev3     |
+----------------------------------+----------+--------------+
+ sleep 30
+ openstack volume type list
+--------------------------------------+------+-----------+
| ID                                   | Name | Is Public |
+--------------------------------------+------+-----------+
| d1734540-38e7-4ef8-b74d-36a2c71df8e5 | rbd1 | True      |
+--------------------------------------+------+-----------+
+ helm test cinder --timeout 900
RUNNING: cinder-test
PASSED: cinder-test
ubuntu@node1: kubectl exec -n tenant-ceph ceph-mon-2g6km -- ceph osd lspools
1 rbd,2 cephfs_metadata,3 cephfs_data,4 .rgw.root,5 default.rgw.control,
6 default.rgw.data.root,7 default.rgw.gc,8 default.rgw.log,
9 default.rgw.intent-log,10 default.rgw.meta,
11 default.rgw.usage,12 default.rgw.users.keys,
13 default.rgw.users.email,14 default.rgw.users.swift,
15 default.rgw.users.uid,16 default.rgw.buckets.extra,
17 default.rgw.buckets.index,18 default.rgw.buckets.data,
19 cinder.volumes,

Note

Above output shows that tenant ceph now has 19 pools including one for Cinder.

ubuntu@node1: kubectl exec -n tenant-ceph ceph-mon-2g6km -- ceph -s
  cluster:
    id:     38339a5a-d976-49dd-88a0-2ac092c271c7
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum node3,node2,node1
    mgr: node2(active), standbys: node1
    osd: 3 osds: 3 up, 3 in
    rgw: 2 daemons active

  data:
    pools:   19 pools, 101 pgs
    objects: 233 objects, 52644 bytes
    usage:   33404 MB used, 199 GB / 232 GB avail
    pgs:     101 active+clean

  io:
    client:   27544 B/s rd, 0 B/s wr, 26 op/s rd, 17 op/s wr