cc243499ec
There were few remaining README.md files. This commit converts them to RST. Change-Id: Ia0be0012fff33b9be5c9db3febc1e45a076701ec
220 lines
6.4 KiB
ReStructuredText
220 lines
6.4 KiB
ReStructuredText
openstack-helm/ceph
|
|
===================
|
|
|
|
This chart installs a working version of ceph. It is based on the
|
|
ceph-docker work and follows closely with the setup `examples
|
|
<https://github.com/ceph/ceph-docker/tree/master/examples/kubernetes>`__
|
|
for kubernetes.
|
|
|
|
It attempts to simplify that process by wrapping up much of the setup
|
|
into a helm-chart. A few items are still necessary, however until they
|
|
can be refined:
|
|
|
|
SkyDNS Resolution
|
|
~~~~~~~~~~~~~~~~~
|
|
|
|
The Ceph MONs are what clients talk to when mounting Ceph storage.
|
|
Because Ceph MON IPs can change, we need a Kubernetes service to front
|
|
them. Otherwise your clients will eventually stop working over time as
|
|
MONs are rescheduled.
|
|
|
|
To get skyDNS resolution working, the resolv.conf on your nodes should
|
|
look something like this:
|
|
|
|
::
|
|
|
|
domain <EXISTING_DOMAIN>
|
|
search <EXISTING_DOMAIN>
|
|
|
|
search svc.cluster.local #Your kubernetes cluster ip domain
|
|
|
|
nameserver 10.0.0.10 #The cluster IP of skyDNS
|
|
nameserver <EXISTING_RESOLVER_IP>
|
|
|
|
Ceph and RBD utilities installed on the nodes
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The Kubernetes kubelet shells out to system utilities to mount Ceph
|
|
volumes. This means that every system must have these utilities
|
|
installed. This requirement extends to the control plane, since there
|
|
may be interactions between kube-controller-manager and the Ceph
|
|
cluster.
|
|
|
|
For Debian-based distros:
|
|
|
|
::
|
|
|
|
apt-get install ceph-fs-common ceph-common
|
|
|
|
For Redhat-based distros:
|
|
|
|
::
|
|
|
|
yum install ceph
|
|
|
|
Linux Kernel version 4.2.0 or newer
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
You'll need a newer kernel to use this. Kernel panics have been observed
|
|
on older versions. Your kernel should also have RBD support.
|
|
|
|
This has been tested on:
|
|
|
|
* Ubuntu 15.10
|
|
|
|
This will not work on:
|
|
|
|
* Debian 8.5
|
|
|
|
Override the default network settings
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
By default, ``10.244.0.0/16`` is used for the ``cluster_network`` and
|
|
``public_network`` in ceph.conf. To change these defaults, set the
|
|
following environment variables according to your network requirements.
|
|
These IPs should be set according to the range of your Pod IPs in your
|
|
kubernetes cluster:
|
|
|
|
::
|
|
|
|
export osd_cluster_network=192.168.0.0/16
|
|
export osd_public_network=192.168.0.0/16
|
|
|
|
For a kubeadm installed weave cluster, you will likely want to run:
|
|
|
|
::
|
|
|
|
export osd_cluster_network=10.32.0.0/12
|
|
export osd_public_network=10.32.0.0/12
|
|
|
|
Label your storage nodes
|
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
You must label your storage nodes in order to run Ceph pods on them.
|
|
|
|
::
|
|
|
|
kubectl label node <nodename> node-type=storage
|
|
|
|
If you want all nodes in your Kubernetes cluster to be a part of your
|
|
Ceph cluster, label them all.
|
|
|
|
::
|
|
|
|
kubectl label nodes node-type=storage --all
|
|
|
|
Quickstart
|
|
~~~~~~~~~~
|
|
|
|
You will need to generate ceph keys and configuration. There is a simple
|
|
to use utility that can do this quickly. Please note the generator
|
|
utility (per ceph-docker) requires the `sigil template framework
|
|
<https://github.com/gliderlabs/sigil>`_ to be installed and on the current
|
|
path.
|
|
|
|
::
|
|
|
|
cd common/utils/secret-generator
|
|
./generate_secrets.sh all `./generate_secrets.sh fsid`
|
|
cd ../../..
|
|
|
|
At this point, you're ready to generate base64 encoded files based on
|
|
the secrets generated above. This is done automatically if you run make
|
|
which rebuilds all charts.
|
|
|
|
::
|
|
|
|
make
|
|
|
|
You can also trigger it specifically:
|
|
|
|
::
|
|
|
|
make base64
|
|
make ceph
|
|
|
|
Finally, you can now deploy your ceph chart:
|
|
|
|
::
|
|
|
|
helm --debug install local/ceph --namespace=ceph
|
|
|
|
You should see a deployed/successful helm deployment:
|
|
|
|
::
|
|
|
|
# helm ls
|
|
NAME REVISION UPDATED STATUS CHART
|
|
saucy-elk 1 Thu Nov 17 13:43:27 2016 DEPLOYED ceph-0.1.0
|
|
|
|
as well as all kubernetes resources deployed into the ceph namespace:
|
|
|
|
::
|
|
|
|
# kubectl get all --namespace=ceph
|
|
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
|
svc/ceph-mon None <none> 6789/TCP 1h
|
|
svc/ceph-rgw 100.76.18.187 <pending> 80/TCP 1h
|
|
NAME READY STATUS RESTARTS AGE
|
|
po/ceph-mds-840702866-0n24u 1/1 Running 3 1h
|
|
po/ceph-mon-1870970076-7h5zw 1/1 Running 2 1h
|
|
po/ceph-mon-1870970076-d4uu2 1/1 Running 3 1h
|
|
po/ceph-mon-1870970076-s6d2p 1/1 Running 1 1h
|
|
po/ceph-mon-check-4116985937-ggv4m 1/1 Running 0 1h
|
|
po/ceph-osd-2m2mf 1/1 Running 2 1h
|
|
po/ceph-rgw-2085838073-02154 0/1 Pending 0 1h
|
|
po/ceph-rgw-2085838073-0d6z7 0/1 CrashLoopBackOff 21 1h
|
|
po/ceph-rgw-2085838073-3trec 0/1 Pending 0 1h
|
|
|
|
Note that the ceph-rgw image is crashing because of an issue processing
|
|
the mon\_host name 'ceph-mon' in ceph.conf. This is an upstream issue
|
|
that needs to be worked but is not required to test ceph rbd or ceph
|
|
filesystem functionality.
|
|
|
|
Finally, you can now test a ceph rbd volume:
|
|
|
|
::
|
|
|
|
export PODNAME=`kubectl get pods --selector="app=ceph,daemon=mon" --output=template --template="{{with index .items 0}}{{.metadata.name}}{{end}}" --namespace=ceph`
|
|
kubectl exec -it $PODNAME --namespace=ceph -- rbd create ceph-rbd-test --size 20G
|
|
kubectl exec -it $PODNAME --namespace=ceph -- rbd info ceph-rbd-test
|
|
|
|
If that works, you can create a container and attach it to that volume:
|
|
|
|
::
|
|
|
|
cd ceph/utils/test
|
|
kubectl create -f ceph-rbd-test.yaml --namespace=ceph
|
|
kubectl exec -it --namespace=ceph ceph-rbd-test -- df -h
|
|
|
|
Cleanup
|
|
~~~~~~~
|
|
|
|
Always make sure to delete any test instances that have ceph volumes
|
|
mounted before you delete your ceph cluster. Otherwise, kubelet may get
|
|
stuck trying to unmount volumes which can only be recovered with a
|
|
reboot. If you ran the tests above, this can be done with:
|
|
|
|
::
|
|
|
|
kubectl delete ceph-rbd-test --namespace=ceph
|
|
|
|
The easiest way to delete your environment is to delete the helm
|
|
install:
|
|
|
|
::
|
|
|
|
# helm ls
|
|
NAME REVISION UPDATED STATUS CHART
|
|
saucy-elk 1 Thu Nov 17 13:43:27 2016 DEPLOYED ceph-0.1.0
|
|
|
|
# helm delete saucy-elk
|
|
|
|
And finally, because helm does not appear to cleanup all artifacts, you
|
|
will want to delete the ceph namespace to remove any secrets helm
|
|
installed:
|
|
|
|
::
|
|
|
|
kubectl delete namespace ceph
|