877e0cd46f
* As of helm 2.0.0, it no longer seems necessary to copy a "globals.tpl" into each child chart, defines in child charts now seem to be reliably rendered in the parent. This was not working correctly in 2.0.0rc2 and so after a lot of testing, the globals define insertion has been removed from the Makefiles. Also, includes piped to b64encode are working now, meaning we no longer have to do Makefile magic to base64 encode successfully. That has been removed. Once .Files.Get works properly in a child chart context we can remove all Make help. * The openstack-base chart has been renamed to common to better reflect that it is common to everything in aic-helm, even non-openstack projects like ceph. All charts now include it as a requirement. * A first pass at a "cleaner" template directory approach has been applied to rabbitmq as a test chart. This allows files such as _start_rabbit to live in their raw form and organizes files by config vs script in 'etc' or 'bin' directories. If accepted, will apply to all other charts for consistency.
179 lines
6.1 KiB
Markdown
179 lines
6.1 KiB
Markdown
# aic-helm/ceph
|
|
|
|
This chart installs a working version of ceph. It is based on the ceph-docker work and follows closely with the setup [examples](https://github.com/ceph/ceph-docker/tree/master/examples/kubernetes) for kubernetes.
|
|
|
|
It attempts to simplify that process by wrapping up much of the setup into a helm-chart. A few items are still necessary, however until they can be refined:
|
|
|
|
### SkyDNS Resolution
|
|
|
|
The Ceph MONs are what clients talk to when mounting Ceph storage. Because Ceph MON IPs can change, we need a Kubernetes service to front them. Otherwise your clients will eventually stop working over time as MONs are rescheduled.
|
|
|
|
To get skyDNS resolution working, the resolv.conf on your nodes should look something like this:
|
|
|
|
```
|
|
domain <EXISTING_DOMAIN>
|
|
search <EXISTING_DOMAIN>
|
|
|
|
search svc.cluster.local #Your kubernetes cluster ip domain
|
|
|
|
nameserver 10.0.0.10 #The cluster IP of skyDNS
|
|
nameserver <EXISTING_RESOLVER_IP>
|
|
```
|
|
|
|
### Ceph and RBD utilities installed on the nodes
|
|
|
|
The Kubernetes kubelet shells out to system utilities to mount Ceph volumes. This means that every system must have these utilities installed. This requirement extends to the control plane, since there may be interactions between kube-controller-manager and the Ceph cluster.
|
|
|
|
For Debian-based distros:
|
|
|
|
```
|
|
apt-get install ceph-fs-common ceph-common
|
|
```
|
|
|
|
For Redhat-based distros:
|
|
|
|
```
|
|
yum install ceph
|
|
```
|
|
|
|
### Linux Kernel version 4.2.0 or newer
|
|
|
|
You'll need a newer kernel to use this. Kernel panics have been observed on older versions. Your kernel should also have RBD support.
|
|
|
|
This has been tested on:
|
|
|
|
- Ubuntu 15.10
|
|
|
|
This will not work on:
|
|
|
|
- Debian 8.5
|
|
|
|
|
|
### Override the default network settings
|
|
|
|
By default, `10.244.0.0/16` is used for the `cluster_network` and `public_network` in ceph.conf. To change these defaults, set the following environment variables according to your network requirements. These IPs should be set according to the range of your Pod IPs in your kubernetes cluster:
|
|
|
|
```
|
|
export osd_cluster_network=192.168.0.0/16
|
|
export osd_public_network=192.168.0.0/16
|
|
```
|
|
|
|
For a kubeadm installed weave cluster, you will likely want to run:
|
|
|
|
```
|
|
export osd_cluster_network=10.32.0.0/12
|
|
export osd_public_network=10.32.0.0/12
|
|
```
|
|
|
|
### Label your storage nodes
|
|
|
|
You must label your storage nodes in order to run Ceph pods on them.
|
|
|
|
```
|
|
kubectl label node <nodename> node-type=storage
|
|
```
|
|
|
|
If you want all nodes in your Kubernetes cluster to be a part of your Ceph cluster, label them all.
|
|
|
|
```
|
|
kubectl label nodes node-type=storage --all
|
|
```
|
|
|
|
### Quickstart
|
|
|
|
You will need to generate ceph keys and configuration. There is a simple to use utility that can do this quickly. Please note the generator utility (per ceph-docker) requires the sigil template framework: (https://github.com/gliderlabs/sigil) to be installed and on the current path.
|
|
|
|
```
|
|
cd common/utils/secret-generator
|
|
./generate_secrets.sh all `./generate_secrets.sh fsid`
|
|
cd ../../..
|
|
```
|
|
|
|
At this point, you're ready to generate base64 encoded files based on the secrets generated above. This is done automatically if you run make which rebuilds all charts.
|
|
|
|
```
|
|
make
|
|
```
|
|
|
|
You can also trigger it specifically:
|
|
|
|
```
|
|
make base64
|
|
make ceph
|
|
```
|
|
|
|
Finally, you can now deploy your ceph chart:
|
|
|
|
```
|
|
helm --debug install local/ceph --namespace=ceph
|
|
```
|
|
|
|
You should see a deployed/successful helm deployment:
|
|
|
|
```
|
|
# helm ls
|
|
NAME REVISION UPDATED STATUS CHART
|
|
saucy-elk 1 Thu Nov 17 13:43:27 2016 DEPLOYED ceph-0.1.0
|
|
```
|
|
|
|
as well as all kubernetes resources deployed into the ceph namespace:
|
|
|
|
```
|
|
# kubectl get all --namespace=ceph
|
|
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
|
svc/ceph-mon None <none> 6789/TCP 1h
|
|
svc/ceph-rgw 100.76.18.187 <pending> 80/TCP 1h
|
|
NAME READY STATUS RESTARTS AGE
|
|
po/ceph-mds-840702866-0n24u 1/1 Running 3 1h
|
|
po/ceph-mon-1870970076-7h5zw 1/1 Running 2 1h
|
|
po/ceph-mon-1870970076-d4uu2 1/1 Running 3 1h
|
|
po/ceph-mon-1870970076-s6d2p 1/1 Running 1 1h
|
|
po/ceph-mon-check-4116985937-ggv4m 1/1 Running 0 1h
|
|
po/ceph-osd-2m2mf 1/1 Running 2 1h
|
|
po/ceph-rgw-2085838073-02154 0/1 Pending 0 1h
|
|
po/ceph-rgw-2085838073-0d6z7 0/1 CrashLoopBackOff 21 1h
|
|
po/ceph-rgw-2085838073-3trec 0/1 Pending 0 1h
|
|
```
|
|
|
|
Note that the ceph-rgw image is crashing because of an issue processing the mon_host name 'ceph-mon' in ceph.conf. This is an upstream issue that needs to be worked but is not required to test ceph rbd or ceph filesystem functionality.
|
|
|
|
Finally, you can now test a ceph rbd volume:
|
|
|
|
```
|
|
export PODNAME=`kubectl get pods --selector="app=ceph,daemon=mon" --output=template --template="{{with index .items 0}}{{.metadata.name}}{{end}}" --namespace=ceph`
|
|
kubectl exec -it $PODNAME --namespace=ceph -- rbd create ceph-rbd-test --size 20G
|
|
kubectl exec -it $PODNAME --namespace=ceph -- rbd info ceph-rbd-test
|
|
```
|
|
|
|
If that works, you can create a container and attach it to that volume:
|
|
|
|
```
|
|
cd ceph/utils/test
|
|
kubectl create -f ceph-rbd-test.yaml --namespace=ceph
|
|
kubectl exec -it --namespace=ceph ceph-rbd-test -- df -h
|
|
```
|
|
|
|
### Cleanup
|
|
|
|
Always make sure to delete any test instances that have ceph volumes mounted before you delete your ceph cluster. Otherwise, kubelet may get stuck trying to unmount volumes which can only be recovered with a reboot. If you ran the tests above, this can be done with:
|
|
|
|
```
|
|
kubectl delete ceph-rbd-test --namespace=ceph
|
|
```
|
|
|
|
The easiest way to delete your environment is to delete the helm install:
|
|
|
|
```
|
|
# helm ls
|
|
NAME REVISION UPDATED STATUS CHART
|
|
saucy-elk 1 Thu Nov 17 13:43:27 2016 DEPLOYED ceph-0.1.0
|
|
|
|
# helm delete saucy-elk
|
|
```
|
|
|
|
And finally, because helm does not appear to cleanup all artifacts, you will want to delete the ceph namespace to remove any secrets helm installed:
|
|
|
|
```
|
|
kubectl delete namespace ceph
|
|
```
|