From e67c3d153e611337a664184acbf380d2ed705228 Mon Sep 17 00:00:00 2001 From: Clark Boylan Date: Wed, 29 Apr 2015 17:50:40 -0700 Subject: [PATCH] Perform puppet module functional testing This spec proposes using beaker-rspec as the testing framework for the infra puppet modules, as agreed in the Puppet Testing infra session at the Vancouver summit. Change-Id: Ia6d1ac79461b221b2fedc6a7a903d65f38dbf299 --- doc/source/index.rst | 1 + specs/puppet-module-functional-testing.rst | 240 +++++++++++++++++++++ 2 files changed, 241 insertions(+) create mode 100644 specs/puppet-module-functional-testing.rst diff --git a/doc/source/index.rst b/doc/source/index.rst index ceb3d79..4da496c 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -43,6 +43,7 @@ permits. specs/nodepool-workers specs/public_hiera specs/shade + specs/puppet-module-functional-testing specs/storyboard_integration_tests specs/storyboard_story_tags specs/storyboard_subscription_pub_sub diff --git a/specs/puppet-module-functional-testing.rst b/specs/puppet-module-functional-testing.rst new file mode 100644 index 0000000..d1e2c7a --- /dev/null +++ b/specs/puppet-module-functional-testing.rst @@ -0,0 +1,240 @@ +:: + + Copyright 2015 OpenStack Foundation + + This work is licensed under a Creative Commons Attribution 3.0 + Unported License. + http://creativecommons.org/licenses/by/3.0/legalcode + +================================ +Puppet Module Functional Testing +================================ + +https://storyboard.openstack.org/#!/story/2000250 + +Perform functional testing of our puppet modules and the services they +deploy to assert that they work for both us and others. + +Problem Description +=================== + +Today our puppet module and deployed service testing is very rudimentary. +We do linting of our puppet modules and noop runs to make sure that the +puppet parser is happy with them. This catches quite a few issues but does +nothing to assert that the puppet modules actually deploy a working service +when run in non noop mode. + +We should be testing that changes to the puppet modules do not break +existing interfaces that we and others use. At a high level we can do this +by running puppet apply (not in noop mode) against a module then assert +things about the resulting state both for the running service and changes +puppet should be making. + +Proposed Change +=============== + +The framework proposed to assist with functional testing is Beaker-rspec +. Beaker is an all-in-one testing +harness that spins up a virtual machine, provisions it, copies puppet manifests +onto it, runs puppet apply with detailed exit codes, and asserts state. +Beaker-rspec is a layer on top of Beaker that provides rspec (a ruby testing +DSL) syntax to describe behavior. Beaker-rspec is the puppet module testing +framework that is recommended by Puppet Labs and widely adopted in the puppet +community. The OpenStack Puppet Modules are currently using this framework, so +the implementation is largely already in place. + +Because beaker is an all-in-one tool that controls the management and +provisioning of virtual machines, it has the advantage that it is easy for +developers to run tests locally using its built-in support for vagrant. +However, this monolithic design is also problematic in the openstack-infra CI +environment because it insists on managing several steps for which we already +have existing infrastructure. However, workarounds have been developed and the +OpenStack Puppet Modules are using these workarounds in their functional +testing jobs (currently non-voting). + +Keep in mind that this does not aim to provide integration testing. For +example when testing the puppet-jenkins module we would not also have a +Zuul service talking to it, but may consider triggering a Jenkins job +via Gearman directly. Functional testing for many services may not be possible +without complicated mocks for dependency services. In cases like this we should +focus on asserting the configured state is correct. The functional testing step +will be followed by an integration testing step, so it is sufficient at this +stage to just test characteristics of some of the installations instead of the +running services themselves. + +Alternatives +------------ + +* `Rspec-system `_. + Rspec-system is the predecessor to beaker-rspec and is largely comparable to + it. However, it is no longer maintained. + +* `Envassert `_. + This is similar to serverspec, but it is written in python and depends + on fabric. I would expect us to use envassert in a python unittest test + runner so that we can do a simple `tox -efunc` after the puppet run + completes and have that run our tests. Probably the only real issue + here is that it relies on fabric which some may see as competing with + puppet and ansible. To an extent this is probably true as you basically + have to run fabric around envassert to get all the environment asserting + goodness. + +* Python unittest tests. + We could just use a python test runner then write our own python tests + using the unittest framework. This would be most like the existing tests + that we run today, but would also likely require some lib building on our + part. Anything that requires OS agnosticism will have to be built in + somehow, though the majority of test cases can probably be checked with + the os, subprocess, and requests modules. + +* Collection of shell scripts + Basically what it says, run a collections of shell scripts in some order + (possibly via run parts or xargs), then check they all return 0. This is + the simplest option and likely quickest to bootstrap, but also has almost + no existing structure we can build around. It will likely lead to + reinventing utility functions in each puppet module and tests that are not + very similar to each other. Long term the cost for this is very high. + +* We can perform rspec unittesting of the puppet modules instead of doing + functional testing. This can have limited value in some cases, such as: + + * checking that everything actually parses & compiles + * conditional logic testing + * template output & logic testing + + Consensus seems to be that the value here is limited, and not worth the + effort when compared to functional testing. + +* Rely on integration testing instead. There is a spec up for integration testing. + I think tempest has taught as an important lesson about integration testing. + It is very useful for making sure that the interactions between services + remain sane, but it is not so good at providing an easy to debug environment + when something breaks at a functional level. We should learn from tempest + and do both integration testing and functional testing. They complement + each other and one is not a substitute for the others. + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + nibalizer + +Other assignees: + crinkle + Other volunteers? + +Gerrit Topic +------------ + +Use Gerrit topic "puppet-func-testing" for all patches related to this spec. + +.. code-block:: bash + + git-review -t puppet-func-testing + +Work Items +---------- + +#. Add Gemfiles to the puppet modules. Jenkins use the Gemfile to install the required gems, namely beaker-rspec. + +#. Add nodepool and vagrant "nodesets" to the puppet modules. Beaker nodesets + are YAML config files that tell Beaker how to manage and provision the node it + is going to test. A nodepool nodeset contains the workarounds that allow us to + tell beaker to manage and provision localhost instead of an external node. On + an Ubuntu platform, this nodeset will look like this:: + + HOSTS: + ubuntu-14.04-amd64: + roles: + - master + platform: ubuntu-14.04-amd64 + hypervisor : none + ip: 127.0.0.1 + CONFIG: + type: foss + set_env: false + + Even though nodepool provisions the platform, beaker requires us to specify a + valid platform so that it can do provisioning of its own. A centos 7 node will + specify "el-7-x86_64" as its platform. + + "Hypervisor" is the term beaker uses for a virtual machine management API such + as vagrant, OpenStack, Amazon, libvirt, etc. Setting it to "none" skips this part. + + Beaker is heavily reliant on creating SSH connections with its host under + test, so we must specify an IP of 127.0.0.1 so that it does not try to assign + an alternate IP address. + + The set_env option prevents beaker from modifying the sshd_config on the + node, since the infra jobs already carefully manage this. + + The vagrant nodesets can be copied directly from one of the Puppet Labs + modules with no modification. + +#. Add spec/spec_helper_acceptance.rb to the puppet modules to control + provisioning steps, such as installing puppet and other modules. This can be + largely inspired by the Puppet Labs modules and the OpenStack Puppet + Modules, but will be customized to work in the nodepool environment. We need + to investigate the possibility of using zuul-cloner here to help with + inter-dependent changes. + +#. Add tests and puppet manifests. Tests are written in the Rspec DSL. The + puppet manifests will be stored as fixtures, separate from the tests. This + will be beneficial if we decide to replace beaker with an alternate testing + framework, as the important part is in the fixtures. + +#. Add jobs in JJB to run the tests, following the example of the jobs already + in place for the OpenStack Puppet Modules. + +#. In the future, it may be beneficial to create a new "hypervisor" for beaker + to get better support for the hack that we're doing here. + +#. Create and maintain a wiki page for issues we have with beaker and + beaker-rspec, both for the purposes of helping the maintainers improve the + tools as well as to help compare an alternate tool in the future. + +Repositories +------------ + +No new repositories necessary. We will update project-config and the existing +puppet modules. + +Servers +------- + +The existing devstack-* test slaves should be perfect platforms to run +this testing on. + +DNS Entries +----------- + +None + +Documentation +------------- + +We will need to update the per module developer documentation to teach +developers how to run the tests locally and how to add new tests when +they make changes. + +Security +-------- + +Since beaker will be run within a single nodepool node, this does not pose any +additional risk on top of what we already assume given that we run arbitrary +code on our nodes. + +Testing +------- + +These changes should be self tested by the new tests that we are adding. +So add the new tests and let them run. + +Dependencies +============ + +The beaker-rspec gem and its dependencies will be installed via `bundle +install` using each module's Gemfile.