From 780afe54e47d3d4acdf9e49013ec3bd4b4f84e88 Mon Sep 17 00:00:00 2001 From: Steve Wilkerson Date: Thu, 29 Jun 2017 10:26:01 -0500 Subject: [PATCH] Add docs detailing testing and gating requirements Adds basic expectations and explanations of openstack-helms testing and gating approaches. This will continue to be expanded upon but serves to provide a base on which to build Change-Id: I689446b7124c4e11a92c73ef04e95d1840c6dc0a --- doc/source/gates.rst | 144 +++++++++++++++++++++++++++++++++++++++++ doc/source/index.rst | 2 + doc/source/testing.rst | 61 +++++++++++++++++ 3 files changed, 207 insertions(+) create mode 100644 doc/source/gates.rst create mode 100644 doc/source/testing.rst diff --git a/doc/source/gates.rst b/doc/source/gates.rst new file mode 100644 index 0000000000..343cdf51d9 --- /dev/null +++ b/doc/source/gates.rst @@ -0,0 +1,144 @@ +==================== +OpenStack-Helm Gates +==================== + +To facilitate ease of testing and debugging, information regarding gates and +their functionality can be found here. + +OpenStack-Helm's single node and multinode gates leverage the kubeadm-aio +environment created and maintained for use as a development environment. All +information regarding the kubeadm-aio environment can be found here_. + +.. _here: https://github.com/openstack/openstack-helm/tree/master/tools/kubeadm-aio + +Gate Checks +----------- + +OpenStack-Helm currently checks the following scenarios: + +- Testing any documentation changes and impacts +- Running Make on each chart, which lints and packages the charts. This gate + does not stand up a Kubernetes cluster +- Provisioning a single node cluster and deploying the OpenStack services. This + check is provided for: Ubuntu-1604, CentOS-7, and Fedora-25 +- Provisioning a multi-node Ubuntu-1604 cluster and deploying the OpenStack + services. This check is provided for both a two node cluster and a three + node cluster. + + +Gate Functions +-------------- + +To provide reusable components for gate functionality, functions have been +provided in the gates/funcs directory. These functions include: + +- Functions for common host preparation operations, found in common.sh +- Functions for Helm specific operations, found in helm.sh. These functions + include: installing Helm, serving a Helm repository locally, linting and + building all Helm charts, running Helm tests on a release, installing the + helm template plugin, and running the helm template plugin against a chart. +- Functions for Kubernetes specific operations, found in kube.sh. These + functions include: waiting for pods in a specific namespace to register as + ready, waiting for all nodes to register as ready, install the requirements + for the kubeadm-aio container used in the gates, building the kubeadm-aio + container, launching the kubeadm-aio container, and replacing the + kube-controller-manager with a specific image necessary for ceph functionality. +- Functions for network specific operations, found in network.sh. These + functions include: creating a backup of the host's resolv.conf file before + deploying the kubeadm environments, restoring the original resolv.conf + settings, creating a backup of the host's /etc/hosts file before adding the + hosts interface and address, and restoring the original /etc/hosts file. +- Functions for OpenStack specific operations, found in openstack.sh. These + functions include: waiting for a successful ping, and waiting for a booted + virtual machine's status to return as ACTIVE. + +Any additional functions required for testing new charts or improving the gate +workflow should be placed in the appropriate location. + + +Gate Output +----------- + +To provide meaningful output from the gates, all information pertaining to the +components of the cluster and workflow are output to the logs directory inside +each gate. The contents of the log directory are as follows: + +- The dry-runs directory contains the rendered output of Helm dry-run installs + on each of the OpenStack service charts. This gives visibility into the + manifests created by the templates with the supplied values. When the dry-run + gate fails, the reason should be apparent in the dry-runs output. The logs + found here are helpful in identifying issues resulting from using helm-toolkit + functions incorrectly or other rendering issues with gotpl. +- The k8s directory contains the logs and output of the Kubernetes objects. It + includes: pods, nodes, secrets, services, namespaces, configmaps, deployments, + daemonsets, and statefulsets. Descriptions for the state of all resources + during execution are found here, and this information can prove valuable when + debugging issues raised during a check. When a single node or multi-node + check fails, this is the first place to look. The logs found here are helpful + when the templates render correctly, but the services are not functioning + correctly, whether due to service configuration issues or issues with the + pods themselves. +- The nodes directory contains information about the node the gate tests are + running on in openstack-infra. This includes: the network interfaces, the + contents of iptables, the host's resolv.conf, and the kernel IP routing table. + These logs can be helpful when trying to identify issues with host networking + or other issues at the node level. + + +Adding Services +--------------- + +As charts for additional services are added to OpenStack-Helm, they should be +included in the gates. Adding new services to the gates allows a chart +developer and the review team to identify any potential issues associated with +a new service. All services are currently launched in the gate via +basic_launch.sh. Any new services are added to this file, and the required +format for a basic service is: + +:: + + helm install --namespace=openstack ${WORK_DIR}/mistral --name=mistral + +Some services in the gate require specific overrides to the default values in +the chart's values.yaml file. If a service requires multiple overrides to +function in the gate, the service should include a separate values.yaml file +placed in the tools/overrides/mvp directory. The .yaml MVP files +provide a configuration file to use for overriding default configuration values +in the chart's values.yaml as an alternative to overriding individual values +during installation. A chart that requires a MVP overrides file +requires the following format: + +:: + + helm install --namespace=openstack ${WORK_DIR}/cinder --name=cinder \ + --values=${WORK_DIR}/tools/overrides/mvp/cinder.yaml + + +Adding Tests +------------ + +As new charts are developed and the services are added to the gate, an +associated Helm test should be introduced to the gates. The appropriate place +for executing these tests is in basic_launch.sh, and must be placed after the +entry for installing the service and any associated overrides. Any tests that +use the Rally testing framework should leverage the helm_test_deployment +function in the aforementioned funcs/helm.sh file. For example, a Helm test for +Mistral might look like: + +:: + + helm_test_deployment mistral 600 + +This results in the gate running the following: + +:: + + helm test --timeout 600 mistral + mkdir -p mistral/rally + kubectl logs -n openstack mistral-rally-test > logs/rally/mistral + kubectl delete -n openstack pod mistral-rally-test + +Any tests that do not use the Rally testing framework would need to be handled +in the appropriate manner in basic_launch.sh. This would ideally result in new +functions that could be reused, or expansion of the gate scripts to include +scenarios beyond basic service launches. diff --git a/doc/source/index.rst b/doc/source/index.rst index 2991385a30..aad5e0e5f0 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -16,6 +16,8 @@ Contents: devref/index contributing troubleshooting/index + testing + gates Indices and tables ================== diff --git a/doc/source/testing.rst b/doc/source/testing.rst new file mode 100644 index 0000000000..8c82b9f639 --- /dev/null +++ b/doc/source/testing.rst @@ -0,0 +1,61 @@ +======= +Testing +======= + +Helm Tests +---------- + +Every OpenStack-Helm chart should include any required Helm tests necessary to +provide a sanity check for the OpenStack service. Information on using the Helm +testing framework can be found in the Helm repository_. Currently, the Rally +testing framework is used to provide these checks for the core services. The +Keystone Helm test template can be used as a reference, and can be found here_. + +.. _repository: https://github.com/kubernetes/helm/blob/master/docs/chart_tests.md + +.. _here: https://github.com/openstack/openstack-helm/blob/master/keystone/templates/pod-rally-test.yaml + + +Testing Expectations +-------------------- + +Any templates for Helm tests submitted should follow the philosophies applied in +the other templates. These include: use of overrides where appropriate, use of +endpoint lookups and other common functionality in helm-toolkit, and mounting +any required scripting templates via the configmap-bin template for the service +chart. If Rally tests are not appropriate or adequate for a service chart, any +additional tests should be documented appropriately and adhere to the same +expectations. + + +Running Tests +------------- + +Any Helm tests associated with a chart can be run by executing: + +:: + + helm test + +The output of the Helm tests can be seen by looking at the logs of the pod +created by the Helm tests. These logs can be viewed with: + +:: + + kubectl logs -n + +Additional information on Helm tests for OpenStack-Helm and how to execute +these tests locally via the scripts used in the gate can be found in the +gates_ directory. + +.. _gates: https://github.com/openstack/openstack-helm/blob/master/tools/gate/ + + +Adding Tests +------------ + +All tests should be added to the gates during development, and are required for +any new service charts prior to merging. All Helm tests are added to the gates +in the basic_launch.sh_ file. + +.. _basic_launch.sh: https://github.com/openstack/openstack-helm/blob/master/tools/gate/basic_launch.sh