Perform puppet module functional testing
This spec proposes using beaker-rspec as the testing framework for the infra puppet modules, as agreed in the Puppet Testing infra session at the Vancouver summit. Change-Id: Ia6d1ac79461b221b2fedc6a7a903d65f38dbf299
This commit is contained in:
Clark Boylan
James E. Blair
parent
0dae151f4d
commit
e67c3d153e
@ -43,6 +43,7 @@ permits.
|
||||
specs/nodepool-workers
|
||||
specs/public_hiera
|
||||
specs/shade
|
||||
specs/puppet-module-functional-testing
|
||||
specs/storyboard_integration_tests
|
||||
specs/storyboard_story_tags
|
||||
specs/storyboard_subscription_pub_sub
|
||||
|
||||
240
specs/puppet-module-functional-testing.rst
Normal file
240
specs/puppet-module-functional-testing.rst
Normal file
@ -0,0 +1,240 @@
|
||||
::
|
||||
|
||||
Copyright 2015 OpenStack Foundation
|
||||
|
||||
This work is licensed under a Creative Commons Attribution 3.0
|
||||
Unported License.
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
================================
|
||||
Puppet Module Functional Testing
|
||||
================================
|
||||
|
||||
https://storyboard.openstack.org/#!/story/2000250
|
||||
|
||||
Perform functional testing of our puppet modules and the services they
|
||||
deploy to assert that they work for both us and others.
|
||||
|
||||
Problem Description
|
||||
===================
|
||||
|
||||
Today our puppet module and deployed service testing is very rudimentary.
|
||||
We do linting of our puppet modules and noop runs to make sure that the
|
||||
puppet parser is happy with them. This catches quite a few issues but does
|
||||
nothing to assert that the puppet modules actually deploy a working service
|
||||
when run in non noop mode.
|
||||
|
||||
We should be testing that changes to the puppet modules do not break
|
||||
existing interfaces that we and others use. At a high level we can do this
|
||||
by running puppet apply (not in noop mode) against a module then assert
|
||||
things about the resulting state both for the running service and changes
|
||||
puppet should be making.
|
||||
|
||||
Proposed Change
|
||||
===============
|
||||
|
||||
The framework proposed to assist with functional testing is Beaker-rspec
|
||||
<https://github.com/puppetlabs/beaker-rspec>. Beaker is an all-in-one testing
|
||||
harness that spins up a virtual machine, provisions it, copies puppet manifests
|
||||
onto it, runs puppet apply with detailed exit codes, and asserts state.
|
||||
Beaker-rspec is a layer on top of Beaker that provides rspec (a ruby testing
|
||||
DSL) syntax to describe behavior. Beaker-rspec is the puppet module testing
|
||||
framework that is recommended by Puppet Labs and widely adopted in the puppet
|
||||
community. The OpenStack Puppet Modules are currently using this framework, so
|
||||
the implementation is largely already in place.
|
||||
|
||||
Because beaker is an all-in-one tool that controls the management and
|
||||
provisioning of virtual machines, it has the advantage that it is easy for
|
||||
developers to run tests locally using its built-in support for vagrant.
|
||||
However, this monolithic design is also problematic in the openstack-infra CI
|
||||
environment because it insists on managing several steps for which we already
|
||||
have existing infrastructure. However, workarounds have been developed and the
|
||||
OpenStack Puppet Modules are using these workarounds in their functional
|
||||
testing jobs (currently non-voting).
|
||||
|
||||
Keep in mind that this does not aim to provide integration testing. For
|
||||
example when testing the puppet-jenkins module we would not also have a
|
||||
Zuul service talking to it, but may consider triggering a Jenkins job
|
||||
via Gearman directly. Functional testing for many services may not be possible
|
||||
without complicated mocks for dependency services. In cases like this we should
|
||||
focus on asserting the configured state is correct. The functional testing step
|
||||
will be followed by an integration testing step, so it is sufficient at this
|
||||
stage to just test characteristics of some of the installations instead of the
|
||||
running services themselves.
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
* `Rspec-system <https://github.com/puppetlabs/rspec-system>`_.
|
||||
Rspec-system is the predecessor to beaker-rspec and is largely comparable to
|
||||
it. However, it is no longer maintained.
|
||||
|
||||
* `Envassert <https://pypi.python.org/pypi/envassert>`_.
|
||||
This is similar to serverspec, but it is written in python and depends
|
||||
on fabric. I would expect us to use envassert in a python unittest test
|
||||
runner so that we can do a simple `tox -efunc` after the puppet run
|
||||
completes and have that run our tests. Probably the only real issue
|
||||
here is that it relies on fabric which some may see as competing with
|
||||
puppet and ansible. To an extent this is probably true as you basically
|
||||
have to run fabric around envassert to get all the environment asserting
|
||||
goodness.
|
||||
|
||||
* Python unittest tests.
|
||||
We could just use a python test runner then write our own python tests
|
||||
using the unittest framework. This would be most like the existing tests
|
||||
that we run today, but would also likely require some lib building on our
|
||||
part. Anything that requires OS agnosticism will have to be built in
|
||||
somehow, though the majority of test cases can probably be checked with
|
||||
the os, subprocess, and requests modules.
|
||||
|
||||
* Collection of shell scripts
|
||||
Basically what it says, run a collections of shell scripts in some order
|
||||
(possibly via run parts or xargs), then check they all return 0. This is
|
||||
the simplest option and likely quickest to bootstrap, but also has almost
|
||||
no existing structure we can build around. It will likely lead to
|
||||
reinventing utility functions in each puppet module and tests that are not
|
||||
very similar to each other. Long term the cost for this is very high.
|
||||
|
||||
* We can perform rspec unittesting of the puppet modules instead of doing
|
||||
functional testing. This can have limited value in some cases, such as:
|
||||
|
||||
* checking that everything actually parses & compiles
|
||||
* conditional logic testing
|
||||
* template output & logic testing
|
||||
|
||||
Consensus seems to be that the value here is limited, and not worth the
|
||||
effort when compared to functional testing.
|
||||
|
||||
* Rely on integration testing instead. There is a spec up for integration testing.
|
||||
I think tempest has taught as an important lesson about integration testing.
|
||||
It is very useful for making sure that the interactions between services
|
||||
remain sane, but it is not so good at providing an easy to debug environment
|
||||
when something breaks at a functional level. We should learn from tempest
|
||||
and do both integration testing and functional testing. They complement
|
||||
each other and one is not a substitute for the others.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
nibalizer
|
||||
|
||||
Other assignees:
|
||||
crinkle
|
||||
Other volunteers?
|
||||
|
||||
Gerrit Topic
|
||||
------------
|
||||
|
||||
Use Gerrit topic "puppet-func-testing" for all patches related to this spec.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git-review -t puppet-func-testing
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
#. Add Gemfiles to the puppet modules. Jenkins use the Gemfile to install the required gems, namely beaker-rspec.
|
||||
|
||||
#. Add nodepool and vagrant "nodesets" to the puppet modules. Beaker nodesets
|
||||
are YAML config files that tell Beaker how to manage and provision the node it
|
||||
is going to test. A nodepool nodeset contains the workarounds that allow us to
|
||||
tell beaker to manage and provision localhost instead of an external node. On
|
||||
an Ubuntu platform, this nodeset will look like this::
|
||||
|
||||
HOSTS:
|
||||
ubuntu-14.04-amd64:
|
||||
roles:
|
||||
- master
|
||||
platform: ubuntu-14.04-amd64
|
||||
hypervisor : none
|
||||
ip: 127.0.0.1
|
||||
CONFIG:
|
||||
type: foss
|
||||
set_env: false
|
||||
|
||||
Even though nodepool provisions the platform, beaker requires us to specify a
|
||||
valid platform so that it can do provisioning of its own. A centos 7 node will
|
||||
specify "el-7-x86_64" as its platform.
|
||||
|
||||
"Hypervisor" is the term beaker uses for a virtual machine management API such
|
||||
as vagrant, OpenStack, Amazon, libvirt, etc. Setting it to "none" skips this part.
|
||||
|
||||
Beaker is heavily reliant on creating SSH connections with its host under
|
||||
test, so we must specify an IP of 127.0.0.1 so that it does not try to assign
|
||||
an alternate IP address.
|
||||
|
||||
The set_env option prevents beaker from modifying the sshd_config on the
|
||||
node, since the infra jobs already carefully manage this.
|
||||
|
||||
The vagrant nodesets can be copied directly from one of the Puppet Labs
|
||||
modules with no modification.
|
||||
|
||||
#. Add spec/spec_helper_acceptance.rb to the puppet modules to control
|
||||
provisioning steps, such as installing puppet and other modules. This can be
|
||||
largely inspired by the Puppet Labs modules and the OpenStack Puppet
|
||||
Modules, but will be customized to work in the nodepool environment. We need
|
||||
to investigate the possibility of using zuul-cloner here to help with
|
||||
inter-dependent changes.
|
||||
|
||||
#. Add tests and puppet manifests. Tests are written in the Rspec DSL. The
|
||||
puppet manifests will be stored as fixtures, separate from the tests. This
|
||||
will be beneficial if we decide to replace beaker with an alternate testing
|
||||
framework, as the important part is in the fixtures.
|
||||
|
||||
#. Add jobs in JJB to run the tests, following the example of the jobs already
|
||||
in place for the OpenStack Puppet Modules.
|
||||
|
||||
#. In the future, it may be beneficial to create a new "hypervisor" for beaker
|
||||
to get better support for the hack that we're doing here.
|
||||
|
||||
#. Create and maintain a wiki page for issues we have with beaker and
|
||||
beaker-rspec, both for the purposes of helping the maintainers improve the
|
||||
tools as well as to help compare an alternate tool in the future.
|
||||
|
||||
Repositories
|
||||
------------
|
||||
|
||||
No new repositories necessary. We will update project-config and the existing
|
||||
puppet modules.
|
||||
|
||||
Servers
|
||||
-------
|
||||
|
||||
The existing devstack-* test slaves should be perfect platforms to run
|
||||
this testing on.
|
||||
|
||||
DNS Entries
|
||||
-----------
|
||||
|
||||
None
|
||||
|
||||
Documentation
|
||||
-------------
|
||||
|
||||
We will need to update the per module developer documentation to teach
|
||||
developers how to run the tests locally and how to add new tests when
|
||||
they make changes.
|
||||
|
||||
Security
|
||||
--------
|
||||
|
||||
Since beaker will be run within a single nodepool node, this does not pose any
|
||||
additional risk on top of what we already assume given that we run arbitrary
|
||||
code on our nodes.
|
||||
|
||||
Testing
|
||||
-------
|
||||
|
||||
These changes should be self tested by the new tests that we are adding.
|
||||
So add the new tests and let them run.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
The beaker-rspec gem and its dependencies will be installed via `bundle
|
||||
install` using each module's Gemfile.
|
||||
Loading…
Reference in New Issue
Block a user