1075 lines
49 KiB
ReStructuredText
1075 lines
49 KiB
ReStructuredText
================================
|
|
Deploying multuple Ceph clusters
|
|
================================
|
|
|
|
This guide shows how to setup multiple Ceph clusters. One Ceph cluster will be
|
|
used for k8s RBD storage and while other Ceph cluster will be for tenant facing
|
|
storage backend for Cinder and Glance.
|
|
|
|
Ceph Clusters:
|
|
==============
|
|
|
|
Ceph for RBD:
|
|
-------------
|
|
|
|
This Ceph cluster will be used for k8s RBD storage (pvc). This can be used by
|
|
entire Kubernetes cluster.
|
|
|
|
- k8s namespace: ceph
|
|
- mon endpoint port: 6789
|
|
- mgr endpoint port: 7000
|
|
- metric port: 9283
|
|
- storage classes: general (rbd based for pvc)
|
|
- no ceph-mds and ceph-rgw
|
|
|
|
Ceph for Tenant:
|
|
----------------
|
|
|
|
This Ceph cluster will be used by Cinder and Glance as storage backend.
|
|
|
|
- k8s namespace: tenant-ceph
|
|
- mon endpoint port: 6790
|
|
- mgr endpoint port: 7001
|
|
- metric port: 9284
|
|
- no storage classes
|
|
- no ceph-mds
|
|
|
|
Env Setup:
|
|
==========
|
|
6 VM based hosts (node1, node2, node3, node4, node5, node6)
|
|
|
|
k8s node labels:
|
|
----------------
|
|
``Ceph for RBD related labels:``
|
|
|
|
Labels assigned to nodes: node1, node2, node3:
|
|
|
|
openstack-control-plane=enabled,
|
|
ceph-mon=enabled,
|
|
ceph-mgr=enabled,
|
|
ceph-rgw=enabled,
|
|
ceph-mds=enabled,
|
|
ceph-osd=enabled
|
|
|
|
``Ceph for Tenant related labels:``
|
|
|
|
Labels assigned to nodes: node1, node2, node3:
|
|
|
|
tenant-ceph-control-plane=enabled,
|
|
ceph-mon-tenant=enabled,
|
|
ceph-mgr-tenant=enabled,
|
|
ceph-rgw-tenant=enabled
|
|
|
|
Labels assigned to nodes: node4, node5, node6:
|
|
|
|
openstack-data-plane=enabled,
|
|
openstack-compute-node=enabled,
|
|
ceph-osd-tenant=enabled,
|
|
openstack-data-plane=enabled
|
|
|
|
|
|
|
|
``k8s node list with labels``
|
|
After applying above labels, node labels should look like following.
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1:~$ kubectl get nodes --show-labels=true
|
|
NAME STATUS ROLES AGE VERSION LABELS
|
|
node1 Ready <none> 9m v1.10.6 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ceph-mds=enabled,ceph-mgr-tenant=enabled,ceph-mgr=enabled,ceph-mon-tenant=enabled,ceph-mon=enabled,ceph-osd=enabled,ceph-rgw-tenant=enabled,ceph-rgw=enabled,kubernetes.io/hostname=node1,linuxbridge=enabled,openstack-control-plane=enabled,openstack-helm-node-class=primary,openvswitch=enabled,tenant-ceph-control-plane=enabled
|
|
node2 Ready <none> 6m v1.10.6 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ceph-mds=enabled,ceph-mgr-tenant=enabled,ceph-mgr=enabled,ceph-mon-tenant=enabled,ceph-mon=enabled,ceph-osd=enabled,ceph-rgw-tenant=enabled,ceph-rgw=enabled,kubernetes.io/hostname=node2,linuxbridge=enabled,openstack-control-plane=enabled,openstack-helm-node-class=general,openvswitch=enabled,tenant-ceph-control-plane=enabled
|
|
node3 Ready <none> 6m v1.10.6 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ceph-mds=enabled,ceph-mgr-tenant=enabled,ceph-mgr=enabled,ceph-mon-tenant=enabled,ceph-mon=enabled,ceph-osd=enabled,ceph-rgw-tenant=enabled,ceph-rgw=enabled,kubernetes.io/hostname=node3,linuxbridge=enabled,openstack-control-plane=enabled,openstack-helm-node-class=general,openvswitch=enabled,tenant-ceph-control-plane=enabled
|
|
node4 Ready <none> 7m v1.10.6 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ceph-osd-tenant=enabled,kubernetes.io/hostname=node4,linuxbridge=enabled,openstack-compute-node=enabled,openstack-data-plane=enabled,openstack-helm-node-class=general,openvswitch=enabled
|
|
node5 Ready <none> 6m v1.10.6 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ceph-osd-tenant=enabled,kubernetes.io/hostname=node5,linuxbridge=enabled,openstack-compute-node=enabled,openstack-data-plane=enabled,openstack-helm-node-class=general,openvswitch=enabled
|
|
node6 Ready <none> 6m v1.10.6 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ceph-osd-tenant=enabled,kubernetes.io/hostname=node6,linuxbridge=enabled,openstack-compute-node=enabled,openstack-data-plane=enabled,openstack-helm-node-class=general,openvswitch=enabled
|
|
|
|
|
|
Test Steps:
|
|
===========
|
|
|
|
1) Prepare scripts:
|
|
-------------------
|
|
|
|
OpenStack-Helm multinode guide includes scripts which are used to specify
|
|
overrides and deploy charts.
|
|
|
|
Duplicate scripts as shows below for later use.
|
|
|
|
.. code-block:: console
|
|
|
|
cd tools/deployment/multinode/
|
|
cp 030-ceph.sh 030-tenant-ceph.sh
|
|
cp 040-ceph-ns-activate.sh 040-tenant-ceph-ns-activate.sh
|
|
cp 090-ceph-radosgateway.sh 090-tenant-ceph-radosgateway.sh
|
|
|
|
|
|
2) Deploy ingress chart:
|
|
------------------------
|
|
|
|
Script to update and execute: ``020-ingress.sh``
|
|
|
|
Update script to include namespace ``tenant-ceph`` as shown
|
|
below.
|
|
|
|
.. code-block:: yaml
|
|
|
|
for NAMESPACE in openstack ceph tenant-ceph; do
|
|
|
|
Execute script.
|
|
|
|
3) Deploy Ceph for RBD:
|
|
-----------------------
|
|
|
|
Script to update and execute: ``030-ceph.sh``
|
|
|
|
Update script with following overrides. Note: The original RBD provisioner
|
|
is now deprecated. The CSI RBD provisioner is selected here. If you prefer
|
|
the original non-CSI RBD provisioner, then set rbd_provisioner to true instead.
|
|
|
|
.. code-block:: yaml
|
|
|
|
deployment:
|
|
storage_secrets: true
|
|
ceph: true
|
|
rbd_provisioner: false
|
|
csi_rbd_provisioner: true
|
|
cephfs_provisioner: false
|
|
client_secrets: false
|
|
endpoints:
|
|
ceph_mon:
|
|
namespace: ceph
|
|
port:
|
|
mon:
|
|
default: 6789
|
|
ceph_mgr:
|
|
namespace: ceph
|
|
port:
|
|
mgr:
|
|
default: 7000
|
|
metrics:
|
|
default: 9283
|
|
manifests:
|
|
deployment_mds: false
|
|
bootstrap:
|
|
enabled: true
|
|
conf:
|
|
pool:
|
|
target:
|
|
osd: 3
|
|
storageclass:
|
|
rbd:
|
|
ceph_configmap_name: ceph-etc
|
|
cephfs:
|
|
provision_storage_class: false
|
|
ceph_mgr_modules_config:
|
|
prometheus:
|
|
server_port: 9283
|
|
monitoring:
|
|
prometheus:
|
|
enabled: true
|
|
ceph_mgr:
|
|
port: 9283
|
|
|
|
.. note::
|
|
``cephfs_provisioner: false`` and ``provision_storage_class: false`` are set
|
|
to false to disable cephfs.
|
|
``deployment_mds: false`` is set to disable ceph-mds
|
|
|
|
Execute script.
|
|
|
|
4) Deploy MariaDB, RabbitMQ, Memcached and Keystone:
|
|
----------------------------------------------------
|
|
|
|
Use default overrides and execute following scripts as per OSH guide steps:
|
|
|
|
- ``040-ceph-ns-activate.sh``
|
|
- ``050-mariadb.sh``
|
|
- ``060-rabbitmq.sh``
|
|
- ``070-memcached.sh``
|
|
- ``080-keystone.sh``
|
|
|
|
|
|
Result from Steps 2, 3, 4:
|
|
--------------------------
|
|
|
|
``Ceph Pods``
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1:~$ kubectl get pods -n ceph -o wide
|
|
NAME READY STATUS RESTARTS AGE IP NODE
|
|
ceph-bootstrap-g45qc 0/1 Completed 0 28m 192.168.5.16 node3
|
|
ceph-mds-keyring-generator-gsw4m 0/1 Completed 0 28m 192.168.2.11 node2
|
|
ceph-mgr-5746dd89db-mmrg4 1/1 Running 0 23m 10.0.0.12 node2
|
|
ceph-mgr-5746dd89db-q25lt 1/1 Running 0 23m 10.0.0.9 node3
|
|
ceph-mgr-keyring-generator-t4s8l 0/1 Completed 0 28m 192.168.2.9 node2
|
|
ceph-mon-6n4hk 1/1 Running 0 28m 10.0.0.9 node3
|
|
ceph-mon-b2d9w 1/1 Running 0 28m 10.0.0.12 node2
|
|
ceph-mon-check-d85994946-2dcpg 1/1 Running 0 28m 192.168.5.17 node3
|
|
ceph-mon-keyring-generator-rmvfz 0/1 Completed 0 28m 192.168.2.10 node2
|
|
ceph-mon-svkdl 1/1 Running 0 28m 10.0.0.16 node1
|
|
ceph-osd-default-83945928-2mhrj 1/1 Running 0 25m 10.0.0.9 node3
|
|
ceph-osd-default-83945928-gqbd9 1/1 Running 0 25m 10.0.0.16 node1
|
|
ceph-osd-default-83945928-krrl8 1/1 Running 0 25m 10.0.0.12 node2
|
|
ceph-osd-keyring-generator-zg8s5 0/1 Completed 0 28m 192.168.0.195 node1
|
|
ceph-rbd-pool-92nbv 0/1 Completed 0 23m 192.168.5.18 node3
|
|
ceph-rbd-provisioner-599895579c-jl6qk 1/1 Running 0 21m 192.168.2.15 node2
|
|
ceph-rbd-provisioner-599895579c-n4hbk 1/1 Running 0 21m 192.168.5.19 node3
|
|
ceph-rgw-keyring-generator-2wv4j 0/1 Completed 0 28m 192.168.5.15 node3
|
|
ceph-storage-keys-generator-8vzrx 0/1 Completed 0 28m 192.168.2.12 node2
|
|
ingress-796d8cf8d6-9khkm 1/1 Running 0 28m 192.168.2.6 node2
|
|
ingress-796d8cf8d6-nznvc 1/1 Running 0 28m 192.168.5.12 node3
|
|
ingress-error-pages-54454dc79b-bgc5m 1/1 Running 0 28m 192.168.2.5 node2
|
|
ingress-error-pages-54454dc79b-hwnv4 1/1 Running 0 28m 192.168.5.7 node3
|
|
|
|
``Openstack Pods:``
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1:~$ kubectl get pods -n openstack -o wide
|
|
NAME READY STATUS RESTARTS AGE IP NODE
|
|
ceph-openstack-config-ceph-ns-key-generator-mcxrs 0/1 Completed 0 11m 192.168.2.16 node2
|
|
ingress-7b4bc84cdd-7wslz 1/1 Running 0 30m 192.168.5.5 node3
|
|
ingress-7b4bc84cdd-z6t2z 1/1 Running 0 30m 192.168.2.4 node2
|
|
ingress-error-pages-586c7f86d6-7m58l 1/1 Running 0 30m 192.168.5.6 node3
|
|
ingress-error-pages-586c7f86d6-n9tzv 1/1 Running 0 30m 192.168.2.3 node2
|
|
keystone-api-7974676d5d-5k27d 1/1 Running 0 6m 192.168.5.24 node3
|
|
keystone-api-7974676d5d-cd9kv 1/1 Running 0 6m 192.168.2.21 node2
|
|
keystone-bootstrap-twfrj 0/1 Completed 0 6m 192.168.0.197 node1
|
|
keystone-credential-setup-txf5p 0/1 Completed 0 6m 192.168.5.25 node3
|
|
keystone-db-init-tjxgm 0/1 Completed 0 6m 192.168.2.20 node2
|
|
keystone-db-sync-zl9t4 0/1 Completed 0 6m 192.168.2.22 node2
|
|
keystone-domain-manage-thwdm 0/1 Completed 0 6m 192.168.0.198 node1
|
|
keystone-fernet-setup-qm424 0/1 Completed 0 6m 192.168.5.26 node3
|
|
keystone-rabbit-init-6699r 0/1 Completed 0 6m 192.168.2.23 node2
|
|
keystone-test 0/1 Completed 0 4m 192.168.3.3 node4
|
|
mariadb-ingress-84894687fd-wfc9b 1/1 Running 0 11m 192.168.2.17 node2
|
|
mariadb-ingress-error-pages-78fb865f84-bg8sg 1/1 Running 0 11m 192.168.5.20 node3
|
|
mariadb-server-0 1/1 Running 0 11m 192.168.5.22 node3
|
|
memcached-memcached-5db74ddfd5-m5gw2 1/1 Running 0 7m 192.168.2.19 node2
|
|
rabbitmq-rabbitmq-0 1/1 Running 0 8m 192.168.2.18 node2
|
|
rabbitmq-rabbitmq-1 1/1 Running 0 8m 192.168.5.23 node3
|
|
rabbitmq-rabbitmq-2 1/1 Running 0 8m 192.168.0.196 node1
|
|
|
|
``Ceph Status``
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1:~$ kubectl exec -n ceph ceph-mon-b2d9w -- ceph -s
|
|
cluster:
|
|
id: 3e53e3b7-e5d9-4bab-9701-134687f4954e
|
|
health: HEALTH_OK
|
|
|
|
services:
|
|
mon: 3 daemons, quorum node3,node2,node1
|
|
mgr: node3(active), standbys: node2
|
|
osd: 3 osds: 3 up, 3 in
|
|
|
|
data:
|
|
pools: 18 pools, 93 pgs
|
|
objects: 127 objects, 218 MB
|
|
usage: 46820 MB used, 186 GB / 232 GB avail
|
|
pgs: 93 active+clean
|
|
|
|
|
|
``Ceph ConfigMaps``
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1:~$ kubectl get cm -n ceph
|
|
NAME DATA AGE
|
|
ceph-client-bin 7 25m
|
|
ceph-client-etc 1 25m
|
|
ceph-etc 1 23m
|
|
ceph-mon-bin 10 29m
|
|
ceph-mon-etc 1 29m
|
|
ceph-osd-bin 7 27m
|
|
ceph-osd-default 1 27m
|
|
ceph-osd-etc 1 27m
|
|
ceph-provisioners-ceph-provisioners-bin 4 23m
|
|
ceph-templates 6 29m
|
|
ingress-bin 2 30m
|
|
ingress-ceph-nginx 0 30m
|
|
ingress-conf 3 30m
|
|
ingress-services-tcp 0 30m
|
|
ingress-services-udp 0 30m
|
|
|
|
|
|
``ceph-mon-etc (ceph.conf)``
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1:~$ kubectl get cm -n ceph ceph-mon-etc -o yaml
|
|
|
|
.. code-block:: yaml
|
|
|
|
apiVersion: v1
|
|
data:
|
|
ceph.conf: |
|
|
[global]
|
|
cephx = true
|
|
cephx_cluster_require_signatures = true
|
|
cephx_require_signatures = false
|
|
cephx_service_require_signatures = false
|
|
fsid = 3e53e3b7-e5d9-4bab-9701-134687f4954e
|
|
mon_addr = :6789
|
|
mon_host = ceph-mon-discovery.ceph.svc.cluster.local:6789
|
|
[osd]
|
|
cluster_network = 10.0.0.0/24
|
|
ms_bind_port_max = 7100
|
|
ms_bind_port_min = 6800
|
|
osd_max_object_name_len = 256
|
|
osd_mkfs_options_xfs = -f -i size=2048
|
|
osd_mkfs_type = xfs
|
|
public_network = 10.0.0.0/24
|
|
kind: ConfigMap
|
|
metadata:
|
|
creationTimestamp: 2018-08-27T04:55:32Z
|
|
name: ceph-mon-etc
|
|
namespace: ceph
|
|
resourceVersion: "3218"
|
|
selfLink: /api/v1/namespaces/ceph/configmaps/ceph-mon-etc
|
|
uid: 6d9fdcba-a9b5-11e8-bb1d-fa163ec12213
|
|
|
|
.. note::
|
|
Note that mon_addr and mon_host have default mon port 6789.
|
|
|
|
``k8s storageclass``
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1:~$ kubectl get storageclasses
|
|
NAME PROVISIONER AGE
|
|
general ceph.com/rbd 14m
|
|
|
|
``Ceph services``
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1:~$ kubectl get svc -n ceph
|
|
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
|
ceph-mgr ClusterIP 10.111.185.73 <none> 7000/TCP,9283/TCP 27m
|
|
ceph-mon ClusterIP None <none> 6789/TCP 31m
|
|
ceph-mon-discovery ClusterIP None <none> 6789/TCP 31m
|
|
ingress ClusterIP 10.100.23.32 <none> 80/TCP,443/TCP 32m
|
|
ingress-error-pages ClusterIP None <none> 80/TCP 32m
|
|
ingress-exporter ClusterIP 10.109.196.155 <none> 10254/TCP 32m
|
|
|
|
``Ceph endpoints``
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1:~$ kubectl get endpoints -n ceph
|
|
NAME ENDPOINTS AGE
|
|
ceph-mgr 10.0.0.12:9283,10.0.0.9:9283,10.0.0.12:7000 + 1 more... 27m
|
|
ceph-mon 10.0.0.12:6789,10.0.0.16:6789,10.0.0.9:6789 31m
|
|
ceph-mon-discovery 10.0.0.12:6789,10.0.0.16:6789,10.0.0.9:6789 31m
|
|
ingress 192.168.2.6:80,192.168.5.12:80,192.168.2.6:443 + 1 more... 32m
|
|
ingress-error-pages 192.168.2.5:8080,192.168.5.7:8080 32m
|
|
ingress-exporter 192.168.2.6:10254,192.168.5.12:10254 32m
|
|
|
|
``netstat ceph mon port``
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1: netstat -ntlp | grep 6789
|
|
(Not all processes could be identified, non-owned process info
|
|
will not be shown, you would have to be root to see it all.)
|
|
tcp 0 0 10.0.0.16:6789 0.0.0.0:* LISTEN -
|
|
|
|
ubuntu@node1: netstat -ntlp | grep 6790
|
|
(Not all processes could be identified, non-owned process info
|
|
will not be shown, you would have to be root to see it all.)
|
|
|
|
``Ceph secrets``
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1:~$ kubectl get secrets -n ceph
|
|
NAME TYPE DATA AGE
|
|
ceph-bootstrap-mds-keyring Opaque 1 34m
|
|
ceph-bootstrap-mgr-keyring Opaque 1 34m
|
|
ceph-bootstrap-osd-keyring Opaque 1 34m
|
|
ceph-bootstrap-rgw-keyring Opaque 1 34m
|
|
ceph-bootstrap-token-w2sqp kubernetes.io/service-account-token 3 34m
|
|
ceph-client-admin-keyring Opaque 1 34m
|
|
ceph-mds-keyring-generator-token-s9kst kubernetes.io/service-account-token 3 34m
|
|
ceph-mgr-keyring-generator-token-h5sw6 kubernetes.io/service-account-token 3 34m
|
|
ceph-mgr-token-hr88m kubernetes.io/service-account-token 3 30m
|
|
ceph-mon-check-token-bfvgk kubernetes.io/service-account-token 3 34m
|
|
ceph-mon-keyring Opaque 1 34m
|
|
ceph-mon-keyring-generator-token-5gs5q kubernetes.io/service-account-token 3 34m
|
|
ceph-mon-token-zsd6w kubernetes.io/service-account-token 3 34m
|
|
ceph-osd-keyring-generator-token-h97wb kubernetes.io/service-account-token 3 34m
|
|
ceph-osd-token-4wfm5 kubernetes.io/service-account-token 3 32m
|
|
ceph-provisioners-ceph-rbd-provisioner-token-f92tw kubernetes.io/service-account-token 3 28m
|
|
ceph-rbd-pool-token-p2nxt kubernetes.io/service-account-token 3 30m
|
|
ceph-rgw-keyring-generator-token-wmfx6 kubernetes.io/service-account-token 3 34m
|
|
ceph-storage-keys-generator-token-dq5ts kubernetes.io/service-account-token 3 34m
|
|
default-token-j8h48 kubernetes.io/service-account-token 3 35m
|
|
ingress-ceph-ingress-token-68rws kubernetes.io/service-account-token 3 35m
|
|
ingress-error-pages-token-mpvhm kubernetes.io/service-account-token 3 35m
|
|
pvc-ceph-conf-combined-storageclass kubernetes.io/rbd 1 34m
|
|
|
|
``Openstack secrets``
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1:~$ kubectl get secrets -n openstack
|
|
NAME TYPE DATA AGE
|
|
ceph-openstack-config-ceph-ns-key-cleaner-token-jj7n6 kubernetes.io/service-account-token 3 17m
|
|
ceph-openstack-config-ceph-ns-key-generator-token-5sqfw kubernetes.io/service-account-token 3 17m
|
|
default-token-r5knr kubernetes.io/service-account-token 3 35m
|
|
ingress-error-pages-token-xxjxt kubernetes.io/service-account-token 3 35m
|
|
ingress-openstack-ingress-token-hrvv8 kubernetes.io/service-account-token 3 35m
|
|
keystone-api-token-xwczg kubernetes.io/service-account-token 3 12m
|
|
keystone-bootstrap-token-dhnb6 kubernetes.io/service-account-token 3 12m
|
|
keystone-credential-keys Opaque 2 12m
|
|
keystone-credential-rotate-token-68lnk kubernetes.io/service-account-token 3 12m
|
|
keystone-credential-setup-token-b2smc kubernetes.io/service-account-token 3 12m
|
|
keystone-db-admin Opaque 1 12m
|
|
keystone-db-init-token-brzkj kubernetes.io/service-account-token 3 12m
|
|
keystone-db-sync-token-xzqj9 kubernetes.io/service-account-token 3 12m
|
|
keystone-db-user Opaque 1 12m
|
|
keystone-domain-manage-token-48gn5 kubernetes.io/service-account-token 3 12m
|
|
keystone-etc Opaque 9 12m
|
|
keystone-fernet-keys Opaque 2 12m
|
|
keystone-fernet-rotate-token-djtzb kubernetes.io/service-account-token 3 12m
|
|
keystone-fernet-setup-token-n9st2 kubernetes.io/service-account-token 3 12m
|
|
keystone-keystone-admin Opaque 8 12m
|
|
keystone-keystone-test Opaque 8 12m
|
|
keystone-rabbit-init-token-pt5b2 kubernetes.io/service-account-token 3 12m
|
|
keystone-rabbitmq-admin Opaque 1 12m
|
|
keystone-rabbitmq-user Opaque 1 12m
|
|
keystone-test-token-z8mb6 kubernetes.io/service-account-token 3 12m
|
|
mariadb-db-root-password Opaque 1 17m
|
|
mariadb-ingress-error-pages-token-cnrqp kubernetes.io/service-account-token 3 17m
|
|
mariadb-ingress-token-gfrg4 kubernetes.io/service-account-token 3 17m
|
|
mariadb-secrets Opaque 1 17m
|
|
mariadb-token-pr5lp kubernetes.io/service-account-token 3 17m
|
|
memcached-memcached-token-gq96p kubernetes.io/service-account-token 3 13m
|
|
pvc-ceph-client-key kubernetes.io/rbd 1 17m
|
|
rabbitmq-rabbitmq-token-5bj85 kubernetes.io/service-account-token 3 14m
|
|
rabbitmq-test-token-w4clj kubernetes.io/service-account-token 3 14m
|
|
|
|
``Openstack PV list``
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1:~$ kubectl get pv -n openstack
|
|
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
|
|
pvc-348f4c52-a9b8-11e8-bb1d-fa163ec12213 256Mi RWO Delete Bound openstack/rabbitmq-data-rabbitmq-rabbitmq-0 general 15m
|
|
pvc-4418c745-a9b8-11e8-bb1d-fa163ec12213 256Mi RWO Delete Bound openstack/rabbitmq-data-rabbitmq-rabbitmq-1 general 14m
|
|
pvc-524d4213-a9b8-11e8-bb1d-fa163ec12213 256Mi RWO Delete Bound openstack/rabbitmq-data-rabbitmq-rabbitmq-2 general 14m
|
|
pvc-da9c9dd2-a9b7-11e8-bb1d-fa163ec12213 5Gi RWO Delete Bound openstack/mysql-data-mariadb-server-0 general 17m
|
|
|
|
``Openstack endpoints``
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1:~$ openstack endpoint list
|
|
+----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------------------+
|
|
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
|
|
+----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------------------+
|
|
| 480cc7360752498e822cbbc7211d213a | RegionOne | keystone | identity | True | internal | http://keystone-api.openstack.svc.cluster.local:5000/v3 |
|
|
| 8dfe4e4725b84e51a5eda564dee0960c | RegionOne | keystone | identity | True | public | http://keystone.openstack.svc.cluster.local:80/v3 |
|
|
| 9b3526e36307400b9accfc7cc834cf99 | RegionOne | keystone | identity | True | admin | http://keystone.openstack.svc.cluster.local:80/v3 |
|
|
+----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------------------+
|
|
|
|
``Openstack services``
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1:~$ openstack service list
|
|
+----------------------------------+----------+----------+
|
|
| ID | Name | Type |
|
|
+----------------------------------+----------+----------+
|
|
| 67cc6b945e934246b25d31a9374a64af | keystone | identity |
|
|
+----------------------------------+----------+----------+
|
|
|
|
|
|
|
|
5) Deploy Ceph for Tenant:
|
|
--------------------------
|
|
|
|
Script to update and execute: ``030-tenant-ceph.sh``
|
|
|
|
Make following changes to script:
|
|
1 Replace occurrence of ``ceph-fs-uuid.txt`` with ``tenant-ceph-fs-uuid.txt``
|
|
|
|
2 Replace occurrence of ``ceph.yaml`` with ``tenant-ceph.yaml``
|
|
|
|
3 For tenant Ceph, no need to deploy ceph-provisioners. Update script
|
|
to ``for CHART in ceph-mon ceph-osd ceph-client; do``
|
|
|
|
|
|
Update script's override section with following:
|
|
|
|
|
|
.. code-block:: yaml
|
|
|
|
endpoints:
|
|
identity:
|
|
namespace: openstack
|
|
object_store:
|
|
namespace: openstack
|
|
ceph_mon:
|
|
namespace: tenant-ceph
|
|
port:
|
|
mon:
|
|
default: 6790
|
|
ceph_mgr:
|
|
namespace: tenant-ceph
|
|
port:
|
|
mgr:
|
|
default: 7001
|
|
metrics:
|
|
default: 9284
|
|
network:
|
|
public: ${CEPH_PUBLIC_NETWORK}
|
|
cluster: ${CEPH_CLUSTER_NETWORK}
|
|
deployment:
|
|
storage_secrets: true
|
|
ceph: true
|
|
rbd_provisioner: false
|
|
csi_rbd_provisioner: false
|
|
cephfs_provisioner: false
|
|
client_secrets: false
|
|
labels:
|
|
mon:
|
|
node_selector_key: ceph-mon-tenant
|
|
osd:
|
|
node_selector_key: ceph-osd-tenant
|
|
rgw:
|
|
node_selector_key: ceph-rgw-tenant
|
|
mgr:
|
|
node_selector_key: ceph-mgr-tenant
|
|
job:
|
|
node_selector_key: tenant-ceph-control-plane
|
|
storageclass:
|
|
rbd:
|
|
ceph_configmap_name: tenant-ceph-etc
|
|
provision_storage_class: false
|
|
name: tenant-rbd
|
|
admin_secret_name: pvc-tenant-ceph-conf-combined-storageclass
|
|
admin_secret_namespace: tenant-ceph
|
|
user_secret_name: pvc-tenant-ceph-client-key
|
|
cephfs:
|
|
provision_storage_class: false
|
|
name: cephfs
|
|
user_secret_name: pvc-tenant-ceph-cephfs-client-key
|
|
admin_secret_name: pvc-tenant-ceph-conf-combined-storageclass
|
|
admin_secret_namespace: tenant-ceph
|
|
bootstrap:
|
|
enabled: true
|
|
manifests:
|
|
deployment_mds: false
|
|
ceph_mgr_modules_config:
|
|
prometheus:
|
|
server_port: 9284
|
|
monitoring:
|
|
prometheus:
|
|
enabled: true
|
|
ceph_mgr:
|
|
port: 9284
|
|
conf:
|
|
ceph:
|
|
global:
|
|
fsid: ${CEPH_FS_ID}
|
|
rgw_ks:
|
|
enabled: true
|
|
pool:
|
|
crush:
|
|
tunables: ${CRUSH_TUNABLES}
|
|
target:
|
|
osd: 3
|
|
pg_per_osd: 100
|
|
storage:
|
|
osd:
|
|
- data:
|
|
type: directory
|
|
location: /var/lib/openstack-helm/tenant-ceph/osd/osd-one
|
|
journal:
|
|
type: directory
|
|
location: /var/lib/openstack-helm/tenant-ceph/osd/journal-one
|
|
mon:
|
|
directory: /var/lib/openstack-helm/tenant-ceph/mon
|
|
|
|
|
|
.. note::
|
|
- Port numbers for Ceph_Mon and Ceph_Mgr are different from default.
|
|
- We are disabling rbd and cephfs provisioners.
|
|
- Labels for mon, osd, rgw, mgr and job have been updated for tenant Ceph.
|
|
- Under storageclass section, values for following have been updated:
|
|
ceph_configmap_name, admin_secret_name, admin_secret_namespace, user_secret_name
|
|
- Under storage: mon directory have been updated.
|
|
|
|
For Tenant Ceph, we will not be provisioning storage classes therefor, update
|
|
script to not install ceph-provisioners chart as following.
|
|
|
|
``for CHART in ceph-mon ceph-osd ceph-client; do``
|
|
|
|
Execute script.
|
|
|
|
6) Enable Openstack namespace to use Tenant Ceph:
|
|
-------------------------------------------------
|
|
|
|
Script to update and execute: ``040-tenant-ceph-ns-activate.sh``
|
|
|
|
Update script as following:
|
|
|
|
.. code-block:: console
|
|
|
|
...
|
|
tee /tmp/tenant-ceph-openstack-config.yaml <<EOF
|
|
endpoints:
|
|
identity:
|
|
namespace: openstack
|
|
object_store:
|
|
namespace: openstack
|
|
ceph_mon:
|
|
namespace: tenant-ceph
|
|
port:
|
|
mon:
|
|
default: 6790
|
|
network:
|
|
public: ${CEPH_PUBLIC_NETWORK}
|
|
cluster: ${CEPH_CLUSTER_NETWORK}
|
|
deployment:
|
|
storage_secrets: false
|
|
ceph: false
|
|
rbd_provisioner: false
|
|
csi_rbd_provisioner: false
|
|
cephfs_provisioner: false
|
|
client_secrets: true
|
|
bootstrap:
|
|
enabled: false
|
|
conf:
|
|
rgw_ks:
|
|
enabled: true
|
|
storageclass:
|
|
rbd:
|
|
ceph_configmap_name: tenant-ceph-etc
|
|
provision_storage_class: false
|
|
name: tenant-rbd
|
|
admin_secret_name: pvc-tenant-ceph-conf-combined-storageclass
|
|
admin_secret_namespace: tenant-ceph
|
|
user_secret_name: pvc-tenant-ceph-client-key
|
|
cephfs:
|
|
provision_storage_class: false
|
|
name: cephfs
|
|
admin_secret_name: pvc-tenant-ceph-conf-combined-storageclass
|
|
admin_secret_namespace: tenant-ceph
|
|
user_secret_name: pvc-tenant-ceph-cephfs-client-key
|
|
EOF
|
|
helm upgrade --install tenant-ceph-openstack-config ./ceph-provisioners \
|
|
--namespace=openstack \
|
|
--values=/tmp/tenant-ceph-openstack-config.yaml \
|
|
${OSH_EXTRA_HELM_ARGS} \
|
|
${OSH_EXTRA_HELM_ARGS_CEPH_NS_ACTIVATE}
|
|
|
|
#NOTE: Wait for deploy
|
|
./tools/deployment/common/wait-for-pods.sh openstack
|
|
|
|
#NOTE: Validate Deployment info
|
|
helm status tenant-ceph-openstack-config
|
|
|
|
Execute script.
|
|
|
|
7) Tenant Ceph: Deploy Rados Gateway:
|
|
-------------------------------------
|
|
|
|
Script to update: ``090-tenant-ceph-radosgateway.sh``
|
|
|
|
Update script with following overrides:
|
|
|
|
.. code-block:: console
|
|
|
|
tee /tmp/tenant-radosgw-openstack.yaml <<EOF
|
|
endpoints:
|
|
identity:
|
|
namespace: openstack
|
|
object_store:
|
|
namespace: openstack
|
|
ceph_mon:
|
|
namespace: tenant-ceph
|
|
port:
|
|
mon:
|
|
default: 6790
|
|
network:
|
|
public: ${CEPH_PUBLIC_NETWORK}
|
|
cluster: ${CEPH_CLUSTER_NETWORK}
|
|
deployment:
|
|
storage_secrets: false
|
|
ceph: true
|
|
rbd_provisioner: false
|
|
csi_rbd_provisioner: false
|
|
cephfs_provisioner: false
|
|
client_secrets: false
|
|
bootstrap:
|
|
enabled: false
|
|
conf:
|
|
rgw_ks:
|
|
enabled: true
|
|
secrets:
|
|
keyrings:
|
|
admin: pvc-tenant-ceph-client-key
|
|
rgw: os-ceph-bootstrap-rgw-keyring
|
|
identity:
|
|
admin: ceph-keystone-admin
|
|
swift: ceph-keystone-user
|
|
user_rgw: ceph-keystone-user-rgw
|
|
ceph_client:
|
|
configmap: tenant-ceph-etc
|
|
EOF
|
|
helm upgrade --install tenant-radosgw-openstack ./ceph-rgw \
|
|
--namespace=openstack \
|
|
--values=/tmp/tenant-radosgw-openstack.yaml \
|
|
${OSH_EXTRA_HELM_ARGS} \
|
|
${OSH_EXTRA_HELM_ARGS_HEAT}
|
|
|
|
#NOTE: Wait for deploy
|
|
./tools/deployment/common/wait-for-pods.sh openstack
|
|
|
|
#NOTE: Validate Deployment info
|
|
helm status tenant-radosgw-openstack
|
|
|
|
|
|
Execute script
|
|
|
|
.. code-block:: console
|
|
|
|
+ openstack service list
|
|
+----------------------------------+----------+--------------+
|
|
| ID | Name | Type |
|
|
+----------------------------------+----------+--------------+
|
|
| 0eddeb6af4fd43ea8f73f63a1ae01438 | swift | object-store |
|
|
| 67cc6b945e934246b25d31a9374a64af | keystone | identity |
|
|
+----------------------------------+----------+--------------+
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1: openstack endpoint list
|
|
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------------------------+
|
|
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
|
|
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------------------------+
|
|
| 265212a5856e4a0aba8eb294508279c7 | RegionOne | swift | object-store | True | admin | http://ceph-rgw.openstack.svc.cluster.local:8088/swift/v1/KEY_$(tenant_id)s |
|
|
| 430174e280444598b676d503c5ed9799 | RegionOne | swift | object-store | True | internal | http://ceph-rgw.openstack.svc.cluster.local:8088/swift/v1/KEY_$(tenant_id)s |
|
|
| 480cc7360752498e822cbbc7211d213a | RegionOne | keystone | identity | True | internal | http://keystone-api.openstack.svc.cluster.local:5000/v3 |
|
|
| 8dfe4e4725b84e51a5eda564dee0960c | RegionOne | keystone | identity | True | public | http://keystone.openstack.svc.cluster.local:80/v3 |
|
|
| 948552a0d90940f7944f8c2eba7ef462 | RegionOne | swift | object-store | True | public | http://radosgw.openstack.svc.cluster.local:80/swift/v1/KEY_$(tenant_id)s |
|
|
| 9b3526e36307400b9accfc7cc834cf99 | RegionOne | keystone | identity | True | admin | http://keystone.openstack.svc.cluster.local:80/v3 |
|
|
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------------------------+
|
|
|
|
Results from Step 5, 6, 7:
|
|
--------------------------
|
|
|
|
``Storage on node1, node2, node3:``
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1:~$ ls -l /var/lib/openstack-helm/
|
|
total 8
|
|
drwxr-xr-x 4 root root 4096 Aug 27 04:57 ceph
|
|
drwxr-xr-x 3 root root 4096 Aug 27 05:47 tenant-ceph
|
|
|
|
``Storage on node4, node5, node6:``
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node6:~$ ls -l /var/lib/openstack-helm/
|
|
total 4
|
|
drwxr-xr-x 3 root root 4096 Aug 27 05:49 tenant-ceph
|
|
|
|
``Ceph Status``
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1: kubectl exec -n tenant-ceph ceph-mon-2g6km -- ceph -s
|
|
cluster:
|
|
id: 38339a5a-d976-49dd-88a0-2ac092c271c7
|
|
health: HEALTH_OK
|
|
|
|
services:
|
|
mon: 3 daemons, quorum node3,node2,node1
|
|
mgr: node2(active), standbys: node1
|
|
osd: 3 osds: 3 up, 3 in
|
|
rgw: 2 daemons active
|
|
|
|
data:
|
|
pools: 18 pools, 93 pgs
|
|
objects: 193 objects, 37421 bytes
|
|
usage: 33394 MB used, 199 GB / 232 GB avail
|
|
pgs: 93 active+clean
|
|
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1: kubectl get cm -n openstack
|
|
NAME DATA AGE
|
|
ceph-etc 1 2h
|
|
ceph-openstack-config-ceph-prov-bin-clients 2 2h
|
|
ceph-rgw-bin 5 3m
|
|
ceph-rgw-bin-ks 3 3m
|
|
ceph-rgw-etc 1 3m
|
|
tenant-ceph-etc 1 1h
|
|
tenant-ceph-openstack-config-ceph-prov-bin-clients 2 1h
|
|
tenant-radosgw-openstack-ceph-templates 1 3m
|
|
...
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1: kubectl get cm -n openstack ceph-rgw-etc -o yaml
|
|
|
|
.. code-block:: yaml
|
|
|
|
apiVersion: v1
|
|
data:
|
|
ceph.conf: |
|
|
[global]
|
|
cephx = true
|
|
cephx_cluster_require_signatures = true
|
|
cephx_require_signatures = false
|
|
cephx_service_require_signatures = false
|
|
mon_addr = :6790
|
|
mon_host = ceph-mon.tenant-ceph.svc.cluster.local:6790
|
|
[osd]
|
|
cluster_network = 10.0.0.0/24
|
|
ms_bind_port_max = 7100
|
|
ms_bind_port_min = 6800
|
|
osd_max_object_name_len = 256
|
|
osd_mkfs_options_xfs = -f -i size=2048
|
|
osd_mkfs_type = xfs
|
|
public_network = 10.0.0.0/24
|
|
kind: ConfigMap
|
|
metadata:
|
|
creationTimestamp: 2018-08-27T07:47:59Z
|
|
name: ceph-rgw-etc
|
|
namespace: openstack
|
|
resourceVersion: "30058"
|
|
selfLink: /api/v1/namespaces/openstack/configmaps/ceph-rgw-etc
|
|
uid: 848df05c-a9cd-11e8-bb1d-fa163ec12213
|
|
|
|
.. note::
|
|
mon_addr and mon_host have non default mon port 6790.
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1: kubectl get secrets -n openstack
|
|
NAME TYPE DATA AGE
|
|
ceph-keystone-admin Opaque 8 4m
|
|
ceph-keystone-user Opaque 8 4m
|
|
ceph-keystone-user-rgw Opaque 8 4m
|
|
ceph-ks-endpoints-token-crnrr kubernetes.io/service-account-token 3 4m
|
|
ceph-ks-service-token-9bnr8 kubernetes.io/service-account-token 3 4m
|
|
ceph-openstack-config-ceph-ns-key-cleaner-token-jj7n6 kubernetes.io/service-account-token 3 2h
|
|
ceph-openstack-config-ceph-ns-key-generator-token-5sqfw kubernetes.io/service-account-token 3 2h
|
|
ceph-rgw-storage-init-token-mhqdw kubernetes.io/service-account-token 3 4m
|
|
ceph-rgw-token-9s6nd kubernetes.io/service-account-token 3 4m
|
|
os-ceph-bootstrap-rgw-keyring Opaque 1 36m
|
|
pvc-ceph-client-key kubernetes.io/rbd 1 2h
|
|
pvc-tenant-ceph-client-key kubernetes.io/rbd 1 1h
|
|
swift-ks-user-token-9slvc kubernetes.io/service-account-token 3 4m
|
|
tenant-ceph-openstack-config-ceph-ns-key-cleaner-token-r6v9v kubernetes.io/service-account-token 3 1h
|
|
tenant-ceph-openstack-config-ceph-ns-key-generator-token-dt472 kubernetes.io/service-account-token 3 1h
|
|
...
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1: kubectl get svc -n tenant-ceph
|
|
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
|
ceph-mgr ClusterIP 10.107.183.4 <none> 7001/TCP,9284/TCP 2h
|
|
ceph-mon ClusterIP None <none> 6790/TCP 2h
|
|
ceph-mon-discovery ClusterIP None <none> 6790/TCP 2h
|
|
ingress ClusterIP 10.109.105.140 <none> 80/TCP,443/TCP 3h
|
|
ingress-error-pages ClusterIP None <none> 80/TCP 3h
|
|
ingress-exporter ClusterIP 10.102.110.153 <none> 10254/TCP 3h
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1: kubectl get endpoints -n tenant-ceph
|
|
NAME ENDPOINTS AGE
|
|
ceph-mgr 10.0.0.12:9284,10.0.0.16:9284,10.0.0.12:7001 + 1 more... 2h
|
|
ceph-mon 10.0.0.12:6790,10.0.0.16:6790,10.0.0.9:6790 2h
|
|
ceph-mon-discovery 10.0.0.12:6790,10.0.0.16:6790,10.0.0.9:6790 2h
|
|
ingress 192.168.2.7:80,192.168.5.14:80,192.168.2.7:443 + 1 more... 3h
|
|
ingress-error-pages 192.168.2.8:8080,192.168.5.13:8080 3h
|
|
ingress-exporter 192.168.2.7:10254,192.168.5.14:10254 3h
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1: kubectl get endpoints -n openstack
|
|
NAME ENDPOINTS AGE
|
|
ceph-rgw 192.168.2.42:8088,192.168.5.44:8088 20m
|
|
ingress 192.168.2.4:80,192.168.5.5:80,192.168.2.4:443 + 1 more... 3h
|
|
ingress-error-pages 192.168.2.3:8080,192.168.5.6:8080 3h
|
|
ingress-exporter 192.168.2.4:10254,192.168.5.5:10254 3h
|
|
keystone 192.168.2.4:80,192.168.5.5:80,192.168.2.4:443 + 1 more... 2h
|
|
keystone-api 192.168.2.21:5000,192.168.5.24:5000 2h
|
|
mariadb 192.168.2.17:3306 2h
|
|
mariadb-discovery 192.168.5.22:4567,192.168.5.22:3306 2h
|
|
mariadb-ingress-error-pages 192.168.5.20:8080 2h
|
|
mariadb-server 192.168.5.22:3306 2h
|
|
memcached 192.168.2.19:11211 2h
|
|
rabbitmq 192.168.0.196:15672,192.168.2.18:15672,192.168.5.23:15672 + 6 more... 2h
|
|
rabbitmq-dsv-7b1733 192.168.0.196:15672,192.168.2.18:15672,192.168.5.23:15672 + 6 more... 2h
|
|
rabbitmq-mgr-7b1733 192.168.2.4:80,192.168.5.5:80,192.168.2.4:443 + 1 more... 2h
|
|
radosgw 192.168.2.4:80,192.168.5.5:80,192.168.2.4:443 + 1 more... 20m
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1: kubectl get svc -n openstack
|
|
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
|
ceph-rgw ClusterIP 10.102.173.130 <none> 8088/TCP 20m
|
|
ingress ClusterIP 10.102.1.71 <none> 80/TCP,443/TCP 3h
|
|
ingress-error-pages ClusterIP None <none> 80/TCP 3h
|
|
ingress-exporter ClusterIP 10.105.29.29 <none> 10254/TCP 3h
|
|
keystone ClusterIP 10.108.94.108 <none> 80/TCP,443/TCP 2h
|
|
keystone-api ClusterIP 10.99.50.35 <none> 5000/TCP 2h
|
|
mariadb ClusterIP 10.111.140.93 <none> 3306/TCP 2h
|
|
mariadb-discovery ClusterIP None <none> 3306/TCP,4567/TCP 2h
|
|
mariadb-ingress-error-pages ClusterIP None <none> 80/TCP 2h
|
|
mariadb-server ClusterIP 10.101.237.241 <none> 3306/TCP 2h
|
|
memcached ClusterIP 10.111.175.130 <none> 11211/TCP 2h
|
|
rabbitmq ClusterIP 10.96.78.137 <none> 5672/TCP,25672/TCP,15672/TCP 2h
|
|
rabbitmq-dsv-7b1733 ClusterIP None <none> 5672/TCP,25672/TCP,15672/TCP 2h
|
|
rabbitmq-mgr-7b1733 ClusterIP 10.104.105.46 <none> 80/TCP,443/TCP 2h
|
|
radosgw ClusterIP 10.101.237.167 <none> 80/TCP,443/TCP 20m
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1: kubectl get storageclasses
|
|
NAME PROVISIONER AGE
|
|
general ceph.com/rbd 1h
|
|
|
|
|
|
8) Deploy Glance:
|
|
-----------------
|
|
|
|
Script to update and execute: ``100-glance.sh``
|
|
|
|
Update script overrides as following:
|
|
|
|
.. code-block:: yaml
|
|
|
|
endpoints:
|
|
object_store:
|
|
namespace: tenant-ceph
|
|
ceph_object_store:
|
|
namespace: tenant-ceph
|
|
ceph_client:
|
|
configmap: tenant-ceph-etc
|
|
user_secret_name: tenant-pvc-ceph-client-key
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1: openstack service list
|
|
+----------------------------------+----------+--------------+
|
|
| ID | Name | Type |
|
|
+----------------------------------+----------+--------------+
|
|
| 0eddeb6af4fd43ea8f73f63a1ae01438 | swift | object-store |
|
|
| 67cc6b945e934246b25d31a9374a64af | keystone | identity |
|
|
| 81a61ec8eff74070bb3c2f0118c1bcd5 | glance | image |
|
|
+----------------------------------+----------+--------------+
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1: openstack endpoint list
|
|
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------------------------+
|
|
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
|
|
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------------------------+
|
|
| 265212a5856e4a0aba8eb294508279c7 | RegionOne | swift | object-store | True | admin | http://ceph-rgw.openstack.svc.cluster.local:8088/swift/v1/KEY_$(tenant_id)s |
|
|
| 3fd88bc6e4774ff78c94bfa8aaaec3cf | RegionOne | glance | image | True | admin | http://glance-api.openstack.svc.cluster.local:9292/ |
|
|
| 430174e280444598b676d503c5ed9799 | RegionOne | swift | object-store | True | internal | http://ceph-rgw.openstack.svc.cluster.local:8088/swift/v1/KEY_$(tenant_id)s |
|
|
| 47505d5186ab448e9213f67bc833d2f1 | RegionOne | glance | image | True | public | http://glance.openstack.svc.cluster.local:80/ |
|
|
| 480cc7360752498e822cbbc7211d213a | RegionOne | keystone | identity | True | internal | http://keystone-api.openstack.svc.cluster.local:5000/v3 |
|
|
| 8dfe4e4725b84e51a5eda564dee0960c | RegionOne | keystone | identity | True | public | http://keystone.openstack.svc.cluster.local:80/v3 |
|
|
| 937c2eacce8b4159bf918f4005c2b0ab | RegionOne | glance | image | True | internal | http://glance-api.openstack.svc.cluster.local:9292/ |
|
|
| 948552a0d90940f7944f8c2eba7ef462 | RegionOne | swift | object-store | True | public | http://radosgw.openstack.svc.cluster.local:80/swift/v1/KEY_$(tenant_id)s |
|
|
| 9b3526e36307400b9accfc7cc834cf99 | RegionOne | keystone | identity | True | admin | http://keystone.openstack.svc.cluster.local:80/v3 |
|
|
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------------------------+
|
|
|
|
.. note::
|
|
Above output shows ``http://ceph-rgw.openstack.svc.cluster.local`` which shows
|
|
that swift is pointing to tenant-ceph.
|
|
|
|
9) Deploy Cinder:
|
|
-----------------
|
|
|
|
Script to update and execute: ``110-cinder.sh``
|
|
|
|
Update script overrides as following:
|
|
|
|
.. code-block:: yaml
|
|
|
|
backup:
|
|
posix:
|
|
volume:
|
|
class_name: rbd-tenant
|
|
ceph_client:
|
|
configmap: tenant-ceph-etc
|
|
user_secret_name: pvc-tenant-ceph-client-key
|
|
|
|
|
|
.. code-block:: console
|
|
|
|
+ OS_CLOUD=openstack_helm
|
|
+ openstack service list
|
|
+----------------------------------+----------+--------------+
|
|
| ID | Name | Type |
|
|
+----------------------------------+----------+--------------+
|
|
| 0eddeb6af4fd43ea8f73f63a1ae01438 | swift | object-store |
|
|
| 66bd0179eada4ab8899a58356fd4d508 | cinder | volume |
|
|
| 67cc6b945e934246b25d31a9374a64af | keystone | identity |
|
|
| 81a61ec8eff74070bb3c2f0118c1bcd5 | glance | image |
|
|
| c126046fc5ec4c52acfc8fee0e2f4dda | cinderv2 | volumev2 |
|
|
| f89b99a31a124b7790e3bb60387380b1 | cinderv3 | volumev3 |
|
|
+----------------------------------+----------+--------------+
|
|
+ sleep 30
|
|
+ openstack volume type list
|
|
+--------------------------------------+------+-----------+
|
|
| ID | Name | Is Public |
|
|
+--------------------------------------+------+-----------+
|
|
| d1734540-38e7-4ef8-b74d-36a2c71df8e5 | rbd1 | True |
|
|
+--------------------------------------+------+-----------+
|
|
+ helm test cinder --timeout 900
|
|
RUNNING: cinder-test
|
|
PASSED: cinder-test
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1: kubectl exec -n tenant-ceph ceph-mon-2g6km -- ceph osd lspools
|
|
1 rbd,2 cephfs_metadata,3 cephfs_data,4 .rgw.root,5 default.rgw.control,
|
|
6 default.rgw.data.root,7 default.rgw.gc,8 default.rgw.log,
|
|
9 default.rgw.intent-log,10 default.rgw.meta,
|
|
11 default.rgw.usage,12 default.rgw.users.keys,
|
|
13 default.rgw.users.email,14 default.rgw.users.swift,
|
|
15 default.rgw.users.uid,16 default.rgw.buckets.extra,
|
|
17 default.rgw.buckets.index,18 default.rgw.buckets.data,
|
|
19 cinder.volumes,
|
|
|
|
.. note::
|
|
Above output shows that tenant ceph now has 19 pools including one for Cinder.
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@node1: kubectl exec -n tenant-ceph ceph-mon-2g6km -- ceph -s
|
|
cluster:
|
|
id: 38339a5a-d976-49dd-88a0-2ac092c271c7
|
|
health: HEALTH_OK
|
|
|
|
services:
|
|
mon: 3 daemons, quorum node3,node2,node1
|
|
mgr: node2(active), standbys: node1
|
|
osd: 3 osds: 3 up, 3 in
|
|
rgw: 2 daemons active
|
|
|
|
data:
|
|
pools: 19 pools, 101 pgs
|
|
objects: 233 objects, 52644 bytes
|
|
usage: 33404 MB used, 199 GB / 232 GB avail
|
|
pgs: 101 active+clean
|
|
|
|
io:
|
|
client: 27544 B/s rd, 0 B/s wr, 26 op/s rd, 17 op/s wr
|