.. -*- coding: utf-8 -*- .. NOTE TO MAINTAINERS: use rst2html script to convert .rst to .html rst2html ./failure-domain.rst ./failure-domain.html open ./failure-domain.html ============================== Failure Domains in CRUSH Map ============================== .. contents:: .. sectnum:: Overview ======== The `CRUSH Map `__ in a Ceph cluster is best visualized as an inverted tree. The hierarchical layout describes the physical topology of the Ceph cluster. Through the physical topology, failure domains are conceptualized from the different branches in the inverted tree. CRUSH rules are created and map to failure domains with data placement policy to distribute the data. The internal nodes (non-leaves and non-root) in the hierarchy are identified as buckets. Each bucket is a hierarchical aggregation of storage locations and their assigned weights. These are the types defined by CRUSH as the supported buckets. :: # types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 region type 10 root This guide describes the host and rack buckets and their role in constructing a CRUSH Map with separate failure domains. Once a Ceph cluster is configured with the expected CRUSh Map and Rule, the PGs of the designated pool are verified with a script (**utils-checkPGs.py**) to ensure that the OSDs in all the PGs reside in separate failure domains. Ceph Environment ================ The ceph commands and scripts described in this write-up are executed as Linux user root on one of orchestration nodes and one of the ceph monitors deployed as kubernetes pods. The root user has the credential to execute all the ceph commands. On a kubernetes cluster, a separate namespace named **ceph** is configured for the ceph cluster. Include the **ceph** namespace in **kubectl** when executing this command. A kubernetes pod is a collection of docker containers sharing a network and mount namespace. It is the basic unit of deployment in the kubernetes cluster. The node in the kubernetes cluster where the orchestration operations are performed needs access to the **kubectl** command. In this guide, this node is referred to as the orchestration node. On this node, you can list all the pods that are deployed. To execute a command in a given pod, use **kubectl** to locate the name of the pod and switch to it to execute the command. Orchestration Node ------------------ To gain access to the kubernetes orchestration node, use your login credential and the authentication procedure assigned to you. For environments setup with SSH key-based access, your id_rsa.pub (generated through the ssh-keygen) public key should be in your ~/.ssh/authorized_keys file on the orchestration node. The kubernetes and ceph commands require the root login credential to execute. Your Linux login requires the *sudo* privilege to execute commands as user root. On the orchestration node, acquire the root's privilege with your Linux login through the *sudo* command. :: [orchestration]$ sudo -i : [orchestration]# Kubernetes Pods --------------- On the orchestration node, execute the **kubectl** command to list the specific set of pods with the **--selector** option. This **kubectl** command lists all the ceph monitor pods. :: [orchestration]# kubectl get pods -n ceph --selector component=mon NAME READY STATUS RESTARTS AGE ceph-mon-85mlt 2/2 Running 0 9d ceph-mon-9mpnb 2/2 Running 0 9d ceph-mon-rzzqr 2/2 Running 0 9d ceph-mon-snds8 2/2 Running 0 9d ceph-mon-snzwx 2/2 Running 0 9d The following **kubectl** command lists the Ceph OSD pods. :: [orchestration]# kubectl get pods -n ceph --selector component=osd NAME READY STATUS RESTARTS AGE ceph-osd-default-166a1044-95s74 2/2 Running 0 9d ceph-osd-default-166a1044-bglnm 2/2 Running 0 9d ceph-osd-default-166a1044-lq5qq 2/2 Running 0 9d ceph-osd-default-166a1044-lz6x6 2/2 Running 0 9d . . . To list all the pods in all the namespaces, execute this **kubectl** command. :: [orchestration]# kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE ceph ceph-bootstrap-rpzld 0/1 Completed 0 10d ceph ceph-cephfs-client-key-generator-pvzs6 0/1 Completed 0 10d ceph ceph-cephfs-provisioner-796668cd7-bn6mn 1/1 Running 0 10d Execute Commands in Pods ^^^^^^^^^^^^^^^^^^^^^^^^ To execute multiple commands in a pod, you can switch to the execution context of the pod with a /bin/bash session. :: [orchestration]# kubectl exec -it ceph-mon-85mlt -n ceph -- /bin/bash [ceph-mon]# ceph status cluster: id: 07c31d0f-bcc6-4db4-aadf-2d2a0f13edb8 health: HEALTH_OK services: mon: 5 daemons, quorum host1,host2,host3,host4,host5 mgr: host6(active), standbys: host1 mds: cephfs-1/1/1 up {0=mds-ceph-mds-7cb4f57cc-prh87=up:active}, 1 up:standby osd: 72 osds: 72 up, 72 in rgw: 2 daemons active data: pools: 20 pools, 3944 pgs objects: 86970 objects, 323 GB usage: 1350 GB used, 79077 GB / 80428 GB avail pgs: 3944 active+clean io: client: 981 kB/s wr, 0 op/s rd, 84 op/s wr To verify that you are executing within the context of a pod. Display the content of the */proc/self/cgroup* control group file. The *kubepods* output in the cgroup file shows that you're executing in a docker container of a pod. :: [ceph-mon]# cat /proc/self/cgroup 11:hugetlb:/kubepods/besteffort/podafb3689c-8c5b-11e8-be6a-246e96290f14/ff6cbc58348a44722ee6a493845b9c2903fabdce80d0902d217cc4d6962d7b53 . . . To exit the pod and resume the orchestration node's execution context. :: [ceph-mon]# exit [orchestration]# To verify that you are executing on the orchestration node's context, display the */proc/self/cgroup* control group file. You would not see the *kubepods* docker container in the output. :: [orchestration]# cat /proc/self/cgroup 11:blkio:/user.slice 10:freezer:/ 9:hugetlb:/ . . . It is also possible to run the ceph commands via the **kubectl exec** without switching to a pod's container. :: [orchestration]# kubectl exec ceph-mon-9mpnb -n ceph -- ceph status cluster: id: 07c31d0f-bcc6-4db4-aadf-2d2a0f13edb8 health: HEALTH_OK . . . Failure Domains =============== A failure domain provides the fault isolation for the data and it corresponds to a branch on the hierarchical topology. To protect against data loss, OSDs that are allocated to PGs should be chosen from different failure domains. Losing a branch takes down all the OSDs in that branch only and OSDs in the other branches are not effected. In a data center, baremetal hosts are typically installed in a rack (refrigerator size cabinet). Multiple racks with hosts in each rack are used to provision the OSDs running on each host. A rack is envisioned as a branch in the CRUSH topology. To provide data redundancy, ceph maintains multiple copies of the data. The total number of copies to store for each piece of data is determined by the ceph **osd_pool_default_size** ceph.conf parameter. With this parameter set to 3, each piece of the data has 3 copies that gets stored in a pool. Each copy is stored on different OSDs allocated from different failure domains. Host ---- Choosing host as the failure domain lacks all the protections against data loss. To illustrate, a Ceph cluster has been provisioned with six hosts and four OSDs on each host. The hosts are enclosed in respective racks where each rack contains two hosts. In the configuration of the Ceph cluster, without explicit instructions on where the host and rack buckets should be placed, Ceph would create a CRUSH map without the rack bucket. A CRUSH rule that get created uses the host as the failure domain. With the size (replica) of a pool set to 3, the OSDs in all the PGs are allocated from different hosts. :: root=default ├── host1 │   ├── osd.1 │   ├── osd.2 │   ├── osd.3 │   └── osd.4 ├── host2 │   ├── osd.5 │   ├── osd.6 │   ├── osd.7 │   └── osd.8 ├── host3 │   ├── osd.9 │   ├── osd.10 │   ├── osd.11 │   └── osd.12 ├── host4 │   ├── osd.13 │   ├── osd.14 │   ├── osd.15 │   └── osd.16 ├── host5 │   ├── osd.17 │   ├── osd.18 │   ├── osd.19 │   └── osd.20 └── host6 ├── osd.21 ├── osd.22 ├── osd.23 └── osd.24 On this ceph cluster, it has a CRUSH rule that uses the host as the failure domain. :: # ceph osd crush rule ls replicated_host # ceph osd crush rule dump replicated_host { "rule_id": 0, "rule_name": "replicated_host", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } Verify the CRUSH rule that is assigned to the ceph pool. In this example, the rbd pool is used. :: # ceph osd pool get rbd crush_rule crush_rule: replicated_host # ceph osd pool get rbd size size: 3 # ceph osd pool get rbd pg_num pg_num: 1024 To verify that the OSDs in all the PGs are allocated from different hosts, invoke the **utils-checkPGs.py** utility on the ceph pool. The offending PGs are printed to stdout. :: # /tmp/utils-checkPGs.py rbd Checking PGs in pool rbd ... Passed With host as the failure domain, quite possibly, some of the PGs might have OSDs allocated from different hosts that are located in the same rack. For example, one PG might have OSD numbers [1, 8, 13]. OSDs 1 and 8 are found on hosts located in rack1. When rack1 suffers a catastrophe failure, PGs with OSDs allocated from the hosts in rack1 would be severely degraded. Rack ---- Choosing rack as the failure domain provides better protection against data loss. To prevent PGs with OSDs allocated from hosts that are located in the same rack, configure the CRUSH hierarchy with the rack buckets. In each rack bucket, it contains the hosts that reside in the same physical rack. A CRUSH Rule is configured with rack as the failure domain. In the following hierarchical topology, the Ceph cluster was configured with three rack buckets. Each bucket has two hosts. In pools that were created with the CRUSH rule set to rack, the OSDs in all the PGs are allocated from the distinct rack. :: root=default ├── rack1 │   ├── host1 │   │   ├── osd.1 │   │   ├── osd.2 │   │   ├── osd.3 │   │   └── osd.4 │   └── host2 │   ├── osd.5 │   ├── osd.6 │   ├── osd.7 │   └── osd.8 ├── rack2 │   ├── host3 │   │   ├── osd.9 │   │   ├── osd.10 │   │   ├── osd.11 │   │   └── osd.12 │   └── host4 │   ├── osd.13 │   ├── osd.14 │   ├── osd.15 │   └── osd.16 └── rack3 ├── host5 │   ├── osd.17 │   ├── osd.18 │   ├── osd.19 │   └── osd.20 └── host6 ├── osd.21 ├── osd.22 ├── osd.23 └── osd.24 Verify the Ceph cluster has a CRUSH rule with rack as the failure domain. :: # ceph osd crush rule ls rack_replicated_rule # ceph osd crush rule dump rack_replicated_rule { "rule_id": 2, "rule_name": "rack_replicated_rule", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "rack" }, { "op": "emit" } ] } Create a ceph pool with its CRUSH rule set to the rack's rule. :: # ceph osd pool create rbd 2048 2048 replicated rack_replicated_rule pool 'rbd' created # ceph osd pool get rbd crush_rule crush_rule: rack_replicated_rule # ceph osd pool get rbd size size: 3 # ceph osd pool get rbd pg_num pg_num: 2048 Invoke the **utils-checkPGs.py** script on the pool to verify that there are no PGs with OSDs allocated from the same rack. The offending PGs are printed to stdout. :: # /tmp/utils-checkPGs.py rbd Checking PGs in pool rbd ... Passed CRUSH Map and Rule ================== On a properly configured Ceph cluster, there are different ways to view the CRUSH hierarchy. ceph CLI -------- Print to stdout the CRUSH hierarchy with the ceph CLI. :: root@host5:/# ceph osd crush tree ID CLASS WEIGHT TYPE NAME -1 78.47974 root default -15 26.15991 rack rack1 -2 13.07996 host host1 0 hdd 1.09000 osd.0 1 hdd 1.09000 osd.1 2 hdd 1.09000 osd.2 3 hdd 1.09000 osd.3 4 hdd 1.09000 osd.4 5 hdd 1.09000 osd.5 6 hdd 1.09000 osd.6 7 hdd 1.09000 osd.7 8 hdd 1.09000 osd.8 9 hdd 1.09000 osd.9 10 hdd 1.09000 osd.10 11 hdd 1.09000 osd.11 -5 13.07996 host host2 12 hdd 1.09000 osd.12 13 hdd 1.09000 osd.13 14 hdd 1.09000 osd.14 15 hdd 1.09000 osd.15 16 hdd 1.09000 osd.16 17 hdd 1.09000 osd.17 18 hdd 1.09000 osd.18 19 hdd 1.09000 osd.19 20 hdd 1.09000 osd.20 21 hdd 1.09000 osd.21 22 hdd 1.09000 osd.22 23 hdd 1.09000 osd.23 -16 26.15991 rack rack2 -13 13.07996 host host3 53 hdd 1.09000 osd.53 54 hdd 1.09000 osd.54 58 hdd 1.09000 osd.58 59 hdd 1.09000 osd.59 64 hdd 1.09000 osd.64 65 hdd 1.09000 osd.65 66 hdd 1.09000 osd.66 67 hdd 1.09000 osd.67 68 hdd 1.09000 osd.68 69 hdd 1.09000 osd.69 70 hdd 1.09000 osd.70 71 hdd 1.09000 osd.71 -9 13.07996 host host4 36 hdd 1.09000 osd.36 37 hdd 1.09000 osd.37 38 hdd 1.09000 osd.38 39 hdd 1.09000 osd.39 40 hdd 1.09000 osd.40 41 hdd 1.09000 osd.41 42 hdd 1.09000 osd.42 43 hdd 1.09000 osd.43 44 hdd 1.09000 osd.44 45 hdd 1.09000 osd.45 46 hdd 1.09000 osd.46 47 hdd 1.09000 osd.47 -17 26.15991 rack rack3 -11 13.07996 host host5 48 hdd 1.09000 osd.48 49 hdd 1.09000 osd.49 50 hdd 1.09000 osd.50 51 hdd 1.09000 osd.51 52 hdd 1.09000 osd.52 55 hdd 1.09000 osd.55 56 hdd 1.09000 osd.56 57 hdd 1.09000 osd.57 60 hdd 1.09000 osd.60 61 hdd 1.09000 osd.61 62 hdd 1.09000 osd.62 63 hdd 1.09000 osd.63 -7 13.07996 host host6 24 hdd 1.09000 osd.24 25 hdd 1.09000 osd.25 26 hdd 1.09000 osd.26 27 hdd 1.09000 osd.27 28 hdd 1.09000 osd.28 29 hdd 1.09000 osd.29 30 hdd 1.09000 osd.30 31 hdd 1.09000 osd.31 32 hdd 1.09000 osd.32 33 hdd 1.09000 osd.33 34 hdd 1.09000 osd.34 35 hdd 1.09000 osd.35 root@host5:/# To see weight and affinity of each OSD. :: root@host5:/# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 78.47974 root default -15 26.15991 rack rack1 -2 13.07996 host host1 0 hdd 1.09000 osd.0 up 1.00000 1.00000 1 hdd 1.09000 osd.1 up 1.00000 1.00000 2 hdd 1.09000 osd.2 up 1.00000 1.00000 3 hdd 1.09000 osd.3 up 1.00000 1.00000 4 hdd 1.09000 osd.4 up 1.00000 1.00000 5 hdd 1.09000 osd.5 up 1.00000 1.00000 6 hdd 1.09000 osd.6 up 1.00000 1.00000 7 hdd 1.09000 osd.7 up 1.00000 1.00000 8 hdd 1.09000 osd.8 up 1.00000 1.00000 9 hdd 1.09000 osd.9 up 1.00000 1.00000 10 hdd 1.09000 osd.10 up 1.00000 1.00000 11 hdd 1.09000 osd.11 up 1.00000 1.00000 -5 13.07996 host host2 12 hdd 1.09000 osd.12 up 1.00000 1.00000 13 hdd 1.09000 osd.13 up 1.00000 1.00000 14 hdd 1.09000 osd.14 up 1.00000 1.00000 15 hdd 1.09000 osd.15 up 1.00000 1.00000 16 hdd 1.09000 osd.16 up 1.00000 1.00000 17 hdd 1.09000 osd.17 up 1.00000 1.00000 18 hdd 1.09000 osd.18 up 1.00000 1.00000 19 hdd 1.09000 osd.19 up 1.00000 1.00000 20 hdd 1.09000 osd.20 up 1.00000 1.00000 21 hdd 1.09000 osd.21 up 1.00000 1.00000 22 hdd 1.09000 osd.22 up 1.00000 1.00000 23 hdd 1.09000 osd.23 up 1.00000 1.00000 crushtool CLI ------------- To extract the CRUSH Map from a running cluster and convert it into ascii text. :: # ceph osd getcrushmap -o /tmp/cm.bin 100 # crushtool -d /tmp/cm.bin -o /tmp/cm.rack.ascii # cat /tmp/cm.rack.ascii . . . # buckets host host1 { id -2 # do not change unnecessarily id -3 class hdd # do not change unnecessarily # weight 13.080 alg straw2 hash 0 # rjenkins1 item osd.0 weight 1.090 item osd.1 weight 1.090 item osd.2 weight 1.090 item osd.3 weight 1.090 item osd.4 weight 1.090 item osd.5 weight 1.090 item osd.6 weight 1.090 item osd.7 weight 1.090 item osd.8 weight 1.090 item osd.9 weight 1.090 item osd.10 weight 1.090 item osd.11 weight 1.090 } host host2 { id -5 # do not change unnecessarily id -6 class hdd # do not change unnecessarily # weight 13.080 alg straw2 hash 0 # rjenkins1 item osd.12 weight 1.090 item osd.13 weight 1.090 item osd.14 weight 1.090 item osd.15 weight 1.090 item osd.16 weight 1.090 item osd.18 weight 1.090 item osd.19 weight 1.090 item osd.17 weight 1.090 item osd.20 weight 1.090 item osd.21 weight 1.090 item osd.22 weight 1.090 item osd.23 weight 1.090 } rack rack1 { id -15 # do not change unnecessarily id -20 class hdd # do not change unnecessarily # weight 26.160 alg straw2 hash 0 # rjenkins1 item host1 weight 13.080 item host2 weight 13.080 } . . . root default { id -1 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 78.480 alg straw2 hash 0 # rjenkins1 item rack1 weight 26.160 item rack2 weight 26.160 item rack3 weight 26.160 } # rules rule replicated_rack { id 2 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type rack step emit } # end crush map The **utils-checkPGs.py** script can read the same data from memory and construct the failure domains with OSDs. Verify the OSDs in each PG against the constructed failure domains. Configure the Failure Domain in CRUSH Map ========================================= The Ceph ceph-osd, ceph-client and cinder charts accept configuration parameters to set the Failure Domain for CRUSH. The options available are **failure_domain**, **failure_domain_by_hostname**, **failure_domain_name** and **crush_rule** :: ceph-osd specific overrides failure_domain: Set the CRUSH bucket type for your OSD to reside in. (DEFAULT: "host") failure_domain_by_hostname: Specify the portion of the hostname to use for your failure domain bucket name. (DEFAULT: "false") failure_domain_name: Manually name the failure domain bucket name. This configuration option should only be used when using host based overrides. (DEFAULT: "false") :: ceph-client and cinder specific overrides crush_rule**: Set the crush rule for a pool (DEFAULT: "replicated_rule") An example of a lab enviroment had the following paramters set for the ceph yaml override file to apply a rack level failure domain within CRUSH. :: endpoints: identity: namespace: openstack object_store: namespace: ceph ceph_mon: namespace: ceph network: public: 10.0.0.0/24 cluster: 10.0.0.0/24 deployment: storage_secrets: true ceph: true rbd_provisioner: true cephfs_provisioner: true client_secrets: false rgw_keystone_user_and_endpoints: false bootstrap: enabled: true conf: ceph: global: fsid: 6c12a986-148d-45a7-9120-0cf0522ca5e0 rgw_ks: enabled: true pool: default: crush_rule: rack_replicated_rule crush: tunables: null target: # NOTE(portdirect): 5 nodes, with one osd per node osd: 18 pg_per_osd: 100 storage: osd: - data: type: block-logical location: /dev/vdb journal: type: block-logical location: /dev/vde1 - data: type: block-logical location: /dev/vdc journal: type: block-logical location: /dev/vde2 - data: type: block-logical location: /dev/vdd journal: type: block-logical location: /dev/vde3 overrides: ceph_osd: hosts: - name: osh-1 conf: storage: failure_domain: "rack" failure_domain_name: "rack1" - name: osh-2 conf: storage: failure_domain: "rack" failure_domain_name: "rack1" - name: osh-3 conf: storage: failure_domain: "rack" failure_domain_name: "rack2" - name: osh-4 conf: storage: failure_domain: "rack" failure_domain_name: "rack2" - name: osh-5 conf: storage: failure_domain: "rack" failure_domain_name: "rack3" - name: osh-6 conf: storage: failure_domain: "rack" failure_domain_name: "rack3" .. NOTE:: Note that the cinder chart will need an override configured to ensure the cinder pools in Ceph are using the correct **crush_rule**. :: pod: replicas: api: 2 volume: 1 scheduler: 1 backup: 1 conf: cinder: DEFAULT: backup_driver: cinder.backup.drivers.swift ceph: pools: backup: replicated: 3 crush_rule: rack_replicated_rule chunk_size: 8 volume: replicated: 3 crush_rule: rack_replicated_rule chunk_size: 8 The charts can be updated with these overrides pre or post deployment. If this is a post deployment change then the following steps will apply for a gate based openstack-helm deployment. :: cd /opt/openstack-helm helm upgrade --install ceph-osd ../openstack-helm-infra/ceph-osd --namespace=ceph --values=/tmp/ceph.yaml kubectl delete jobs/ceph-rbd-pool -n ceph helm upgrade --install ceph-client ../openstack-helm-infra/ceph-client --namespace=ceph --values=/tmp/ceph.yaml helm delete cinder --purge helm upgrade --install cinder ./cinder --namespace=openstack --values=/tmp/cinder.yaml .. NOTE:: There will be a brief interuption of I/O and a data movement of placement groups in Ceph while these changes are applied. The data movement operation can take several minutes to several days to complete. The utils-checkPGs.py Script ============================ The purpose of the **utils-checkPGs.py** script is to check whether a PG has OSDs allocated from the same failure domain. The violating PGs with their respective OSDs are printed to the stdout. In this example, a pool was created with the CRUSH rule set to the host failure domain. The ceph cluster was configured with the rack buckets. The CRUSH algorithm allocated the OSDs from different hosts in each PG. The rack buckets were ignored and thus the duplicate racks which get reported by the script. :: root@host5:/# /tmp/utils-checkPGs.py cmTestPool Checking PGs in pool cmTestPool ... Failed OSDs [44, 32, 53] in PG 20.a failed check in rack [u'rack2', u'rack2', u'rack2'] OSDs [61, 5, 12] in PG 20.19 failed check in rack [u'rack1', u'rack1', u'rack1'] OSDs [69, 9, 15] in PG 20.2a failed check in rack [u'rack1', u'rack1', u'rack1'] . . . .. NOTE:: The **utils-checkPGs.py** utility is executed on-demand. It is intended to be executed on one of the ceph-mon pods. If the **utils-checkPGs.py** script did not find any violation, it prints Passed. In this example, the ceph cluster was configured with the rack buckets. The rbd pool was created with its CRUSH rule set to the rack. The **utils-checkPGs.py** script did not find duplicate racks in PGs. :: root@host5:/# /tmp/utils-checkPGs.py rbd Checking PGs in pool rbd ... Passed Invoke the **utils-checkPGs.py** script with the --help option to get the script's usage. :: root@host5:/# /tmp/utils-checkPGs.py --help usage: utils-checkPGs.py [-h] PoolName [PoolName ...] Cross-check the OSDs assigned to the Placement Groups (PGs) of a ceph pool with the CRUSH topology. The cross-check compares the OSDs in a PG and verifies the OSDs reside in separate failure domains. PGs with OSDs in the same failure domain are flagged as violation. The offending PGs are printed to stdout. This CLI is executed on-demand on a ceph-mon pod. To invoke the CLI, you can specify one pool or list of pools to check. The special pool name All (or all) checks all the pools in the ceph cluster. positional arguments: PoolName List of pools (or All) to validate the PGs and OSDs mapping optional arguments: -h, --help show this help message and exit root@host5:/# The source for the **utils-checkPGs.py** script is available at **openstack-helm/ceph-mon/templates/bin/utils/_checkPGs.py.tpl**. Ceph Deployments ================ Through testing and verification, you derive at a CRUSH Map with the buckets that are deemed beneficial to your ceph cluster. Standardize on the verified CRUSH map to have the consistency in all the Ceph deployments across the data centers. Mimicking the hierarchy in your CRUSH Map with the physical hardware setup should provide the needed information on the topology layout. With the racks layout, each rack can store a replica of your data. To validate a ceph cluster with the number of replica that is based on the number of racks: #. The number of physical racks and the number of replicas are 3, respectively. Create a ceph pool with replica set to 3 and pg_num set to (# of OSDs * 50) / 3 and round the number to the next power-of-2. For example, if the calculation is 240, round it to 256. Assuming the pool you just created had 256 PGs. In each PG, verify the OSDs are chosen from the three racks, respectively. Use the **utils-checkPGs.py** script to verify the OSDs in all the PGs of the pool. #. The number of physical racks is 2 and the number of replica is 3. Create a ceph pool as described in the previous step. In the pool you created, in each PG, verify two of the OSDs are chosen from the two racks, respectively. The third OSD can come from one of the two racks but not from the same hosts as the other two OSDs. Data Movement ============= Changes to the CRUSH Map always trigger data movement. It is prudent that you plan accordingly when restructuring the CRUSH Map. Once started, the CRUSH Map restructuring runs to completion and can neither be stopped nor suspended. On a busy Ceph cluster with live transactions, it is always safer to use divide-and-conquer approach to complete small chunk of works in multiple sessions. Watch the progress of the data movement while the Ceph cluster re-balances itself. :: # watch ceph status cluster: id: 07c31d0f-bcc6-4db4-aadf-2d2a0f13edb8 health: HEALTH_WARN 137084/325509 objects misplaced (42.114%) Degraded data redundancy: 28/325509 objects degraded (0.009%), 15 pgs degraded services: mon: 5 daemons, quorum host1,host2,host3,host4,host5 mgr: host6(active), standbys: host1 mds: cephfs-1/1/1 up {0=mds-ceph-mds-7cb4f57cc-prh87=up:active}, 1 up:standby osd: 72 osds: 72 up, 72 in; 815 remapped pgs rgw: 2 daemons active data: pools: 19 pools, 2920 pgs objects: 105k objects, 408 GB usage: 1609 GB used, 78819 GB / 80428 GB avail pgs: 28/325509 objects degraded (0.009%) 137084/325509 objects misplaced (42.114%) 2085 active+clean 790 active+remapped+backfill_wait 22 active+remapped+backfilling 15 active+recovery_wait+degraded 4 active+recovery_wait+remapped 4 active+recovery_wait io: client: 11934 B/s rd, 3731 MB/s wr, 2 op/s rd, 228 kop/s wr recovery: 636 MB/s, 163 objects/s At the time this **ceph status** command was executed, the status's output showed that the ceph cluster was going through re-balancing. Among the overall 2920 pgs, 2085 of them are in **active+clean** state. The remaining pgs are either being remapped or recovered. As the ceph cluster continues its re-balance, the number of pgs in **active+clean** increases. :: # ceph status cluster: id: 07c31d0f-bcc6-4db4-aadf-2d2a0f13edb8 health: HEALTH_OK services: mon: 5 daemons, quorum host1,host2,host3,host4,host5 mgr: host6(active), standbys: host1 mds: cephfs-1/1/1 up {0=mds-ceph-mds-7cc55c9695-lj22d=up:active}, 1 up:standby osd: 72 osds: 72 up, 72 in rgw: 2 daemons active data: pools: 19 pools, 2920 pgs objects: 134k objects, 519 GB usage: 1933 GB used, 78494 GB / 80428 GB avail pgs: 2920 active+clean io: client: 1179 B/s rd, 971 kB/s wr, 1 op/s rd, 41 op/s wr When the overall number of pgs is equal to the number of **active+clean** pgs, the health of the ceph cluster changes to **HEALTH_OK** (assuming there are no other warning conditions).