Add docs detailing testing and gating requirements
Adds basic expectations and explanations of openstack-helms testing and gating approaches. This will continue to be expanded upon but serves to provide a base on which to build Change-Id: I689446b7124c4e11a92c73ef04e95d1840c6dc0a
This commit is contained in:
Steve Wilkerson
parent
8ac7ec6525
commit
780afe54e4
144
doc/source/gates.rst
Normal file
144
doc/source/gates.rst
Normal file
@ -0,0 +1,144 @@
|
|||||||
|
====================
|
||||||
|
OpenStack-Helm Gates
|
||||||
|
====================
|
||||||
|
|
||||||
|
To facilitate ease of testing and debugging, information regarding gates and
|
||||||
|
their functionality can be found here.
|
||||||
|
|
||||||
|
OpenStack-Helm's single node and multinode gates leverage the kubeadm-aio
|
||||||
|
environment created and maintained for use as a development environment. All
|
||||||
|
information regarding the kubeadm-aio environment can be found here_.
|
||||||
|
|
||||||
|
.. _here: https://github.com/openstack/openstack-helm/tree/master/tools/kubeadm-aio
|
||||||
|
|
||||||
|
Gate Checks
|
||||||
|
-----------
|
||||||
|
|
||||||
|
OpenStack-Helm currently checks the following scenarios:
|
||||||
|
|
||||||
|
- Testing any documentation changes and impacts
|
||||||
|
- Running Make on each chart, which lints and packages the charts. This gate
|
||||||
|
does not stand up a Kubernetes cluster
|
||||||
|
- Provisioning a single node cluster and deploying the OpenStack services. This
|
||||||
|
check is provided for: Ubuntu-1604, CentOS-7, and Fedora-25
|
||||||
|
- Provisioning a multi-node Ubuntu-1604 cluster and deploying the OpenStack
|
||||||
|
services. This check is provided for both a two node cluster and a three
|
||||||
|
node cluster.
|
||||||
|
|
||||||
|
|
||||||
|
Gate Functions
|
||||||
|
--------------
|
||||||
|
|
||||||
|
To provide reusable components for gate functionality, functions have been
|
||||||
|
provided in the gates/funcs directory. These functions include:
|
||||||
|
|
||||||
|
- Functions for common host preparation operations, found in common.sh
|
||||||
|
- Functions for Helm specific operations, found in helm.sh. These functions
|
||||||
|
include: installing Helm, serving a Helm repository locally, linting and
|
||||||
|
building all Helm charts, running Helm tests on a release, installing the
|
||||||
|
helm template plugin, and running the helm template plugin against a chart.
|
||||||
|
- Functions for Kubernetes specific operations, found in kube.sh. These
|
||||||
|
functions include: waiting for pods in a specific namespace to register as
|
||||||
|
ready, waiting for all nodes to register as ready, install the requirements
|
||||||
|
for the kubeadm-aio container used in the gates, building the kubeadm-aio
|
||||||
|
container, launching the kubeadm-aio container, and replacing the
|
||||||
|
kube-controller-manager with a specific image necessary for ceph functionality.
|
||||||
|
- Functions for network specific operations, found in network.sh. These
|
||||||
|
functions include: creating a backup of the host's resolv.conf file before
|
||||||
|
deploying the kubeadm environments, restoring the original resolv.conf
|
||||||
|
settings, creating a backup of the host's /etc/hosts file before adding the
|
||||||
|
hosts interface and address, and restoring the original /etc/hosts file.
|
||||||
|
- Functions for OpenStack specific operations, found in openstack.sh. These
|
||||||
|
functions include: waiting for a successful ping, and waiting for a booted
|
||||||
|
virtual machine's status to return as ACTIVE.
|
||||||
|
|
||||||
|
Any additional functions required for testing new charts or improving the gate
|
||||||
|
workflow should be placed in the appropriate location.
|
||||||
|
|
||||||
|
|
||||||
|
Gate Output
|
||||||
|
-----------
|
||||||
|
|
||||||
|
To provide meaningful output from the gates, all information pertaining to the
|
||||||
|
components of the cluster and workflow are output to the logs directory inside
|
||||||
|
each gate. The contents of the log directory are as follows:
|
||||||
|
|
||||||
|
- The dry-runs directory contains the rendered output of Helm dry-run installs
|
||||||
|
on each of the OpenStack service charts. This gives visibility into the
|
||||||
|
manifests created by the templates with the supplied values. When the dry-run
|
||||||
|
gate fails, the reason should be apparent in the dry-runs output. The logs
|
||||||
|
found here are helpful in identifying issues resulting from using helm-toolkit
|
||||||
|
functions incorrectly or other rendering issues with gotpl.
|
||||||
|
- The k8s directory contains the logs and output of the Kubernetes objects. It
|
||||||
|
includes: pods, nodes, secrets, services, namespaces, configmaps, deployments,
|
||||||
|
daemonsets, and statefulsets. Descriptions for the state of all resources
|
||||||
|
during execution are found here, and this information can prove valuable when
|
||||||
|
debugging issues raised during a check. When a single node or multi-node
|
||||||
|
check fails, this is the first place to look. The logs found here are helpful
|
||||||
|
when the templates render correctly, but the services are not functioning
|
||||||
|
correctly, whether due to service configuration issues or issues with the
|
||||||
|
pods themselves.
|
||||||
|
- The nodes directory contains information about the node the gate tests are
|
||||||
|
running on in openstack-infra. This includes: the network interfaces, the
|
||||||
|
contents of iptables, the host's resolv.conf, and the kernel IP routing table.
|
||||||
|
These logs can be helpful when trying to identify issues with host networking
|
||||||
|
or other issues at the node level.
|
||||||
|
|
||||||
|
|
||||||
|
Adding Services
|
||||||
|
---------------
|
||||||
|
|
||||||
|
As charts for additional services are added to OpenStack-Helm, they should be
|
||||||
|
included in the gates. Adding new services to the gates allows a chart
|
||||||
|
developer and the review team to identify any potential issues associated with
|
||||||
|
a new service. All services are currently launched in the gate via
|
||||||
|
basic_launch.sh. Any new services are added to this file, and the required
|
||||||
|
format for a basic service is:
|
||||||
|
|
||||||
|
::
|
||||||
|
|
||||||
|
helm install --namespace=openstack ${WORK_DIR}/mistral --name=mistral
|
||||||
|
|
||||||
|
Some services in the gate require specific overrides to the default values in
|
||||||
|
the chart's values.yaml file. If a service requires multiple overrides to
|
||||||
|
function in the gate, the service should include a separate values.yaml file
|
||||||
|
placed in the tools/overrides/mvp directory. The <service>.yaml MVP files
|
||||||
|
provide a configuration file to use for overriding default configuration values
|
||||||
|
in the chart's values.yaml as an alternative to overriding individual values
|
||||||
|
during installation. A chart that requires a MVP overrides file
|
||||||
|
requires the following format:
|
||||||
|
|
||||||
|
::
|
||||||
|
|
||||||
|
helm install --namespace=openstack ${WORK_DIR}/cinder --name=cinder \
|
||||||
|
--values=${WORK_DIR}/tools/overrides/mvp/cinder.yaml
|
||||||
|
|
||||||
|
|
||||||
|
Adding Tests
|
||||||
|
------------
|
||||||
|
|
||||||
|
As new charts are developed and the services are added to the gate, an
|
||||||
|
associated Helm test should be introduced to the gates. The appropriate place
|
||||||
|
for executing these tests is in basic_launch.sh, and must be placed after the
|
||||||
|
entry for installing the service and any associated overrides. Any tests that
|
||||||
|
use the Rally testing framework should leverage the helm_test_deployment
|
||||||
|
function in the aforementioned funcs/helm.sh file. For example, a Helm test for
|
||||||
|
Mistral might look like:
|
||||||
|
|
||||||
|
::
|
||||||
|
|
||||||
|
helm_test_deployment mistral 600
|
||||||
|
|
||||||
|
This results in the gate running the following:
|
||||||
|
|
||||||
|
::
|
||||||
|
|
||||||
|
helm test --timeout 600 mistral
|
||||||
|
mkdir -p mistral/rally
|
||||||
|
kubectl logs -n openstack mistral-rally-test > logs/rally/mistral
|
||||||
|
kubectl delete -n openstack pod mistral-rally-test
|
||||||
|
|
||||||
|
Any tests that do not use the Rally testing framework would need to be handled
|
||||||
|
in the appropriate manner in basic_launch.sh. This would ideally result in new
|
||||||
|
functions that could be reused, or expansion of the gate scripts to include
|
||||||
|
scenarios beyond basic service launches.
|
||||||
@ -16,6 +16,8 @@ Contents:
|
|||||||
devref/index
|
devref/index
|
||||||
contributing
|
contributing
|
||||||
troubleshooting/index
|
troubleshooting/index
|
||||||
|
testing
|
||||||
|
gates
|
||||||
|
|
||||||
Indices and tables
|
Indices and tables
|
||||||
==================
|
==================
|
||||||
|
|||||||
61
doc/source/testing.rst
Normal file
61
doc/source/testing.rst
Normal file
@ -0,0 +1,61 @@
|
|||||||
|
=======
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
Helm Tests
|
||||||
|
----------
|
||||||
|
|
||||||
|
Every OpenStack-Helm chart should include any required Helm tests necessary to
|
||||||
|
provide a sanity check for the OpenStack service. Information on using the Helm
|
||||||
|
testing framework can be found in the Helm repository_. Currently, the Rally
|
||||||
|
testing framework is used to provide these checks for the core services. The
|
||||||
|
Keystone Helm test template can be used as a reference, and can be found here_.
|
||||||
|
|
||||||
|
.. _repository: https://github.com/kubernetes/helm/blob/master/docs/chart_tests.md
|
||||||
|
|
||||||
|
.. _here: https://github.com/openstack/openstack-helm/blob/master/keystone/templates/pod-rally-test.yaml
|
||||||
|
|
||||||
|
|
||||||
|
Testing Expectations
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
Any templates for Helm tests submitted should follow the philosophies applied in
|
||||||
|
the other templates. These include: use of overrides where appropriate, use of
|
||||||
|
endpoint lookups and other common functionality in helm-toolkit, and mounting
|
||||||
|
any required scripting templates via the configmap-bin template for the service
|
||||||
|
chart. If Rally tests are not appropriate or adequate for a service chart, any
|
||||||
|
additional tests should be documented appropriately and adhere to the same
|
||||||
|
expectations.
|
||||||
|
|
||||||
|
|
||||||
|
Running Tests
|
||||||
|
-------------
|
||||||
|
|
||||||
|
Any Helm tests associated with a chart can be run by executing:
|
||||||
|
|
||||||
|
::
|
||||||
|
|
||||||
|
helm test <helm-release-name>
|
||||||
|
|
||||||
|
The output of the Helm tests can be seen by looking at the logs of the pod
|
||||||
|
created by the Helm tests. These logs can be viewed with:
|
||||||
|
|
||||||
|
::
|
||||||
|
|
||||||
|
kubectl logs <test-pod-name> -n <namespace>
|
||||||
|
|
||||||
|
Additional information on Helm tests for OpenStack-Helm and how to execute
|
||||||
|
these tests locally via the scripts used in the gate can be found in the
|
||||||
|
gates_ directory.
|
||||||
|
|
||||||
|
.. _gates: https://github.com/openstack/openstack-helm/blob/master/tools/gate/
|
||||||
|
|
||||||
|
|
||||||
|
Adding Tests
|
||||||
|
------------
|
||||||
|
|
||||||
|
All tests should be added to the gates during development, and are required for
|
||||||
|
any new service charts prior to merging. All Helm tests are added to the gates
|
||||||
|
in the basic_launch.sh_ file.
|
||||||
|
|
||||||
|
.. _basic_launch.sh: https://github.com/openstack/openstack-helm/blob/master/tools/gate/basic_launch.sh
|
||||||
Loading…
Reference in New Issue
Block a user