Explicitly document requirements for 3rd party testing

As we are moving towards requiring more and more third party testing, we
should make sure we have a central and explicit list of requirements.
These are the requirements to have a voting third party testing system,
not the requirements of what the system should be running (to count a
driver as fully tested).

Change-Id: Ic1e1cdf9af9cd21e29a0e768d5d78ae4f84108fb
This commit is contained in:
Joe Gordon 2013-12-20 10:40:13 -08:00
parent 4de3c41e50
commit 971446f5ab

View File

@ -14,6 +14,38 @@ Smokestack reads the Gerrit event stream and runs it's own tests on the commits.
If one of the tests fails it will publish information and links to the failure If one of the tests fails it will publish information and links to the failure
on the review in Gerrit. on the review in Gerrit.
Requirements
------------
* Until a third party testing system operates in a stable fashion, third
party tests can comment on patches but not vote on them.
* A system can also be set up to only do '+1' reviews and leave all the
'-1's to be manually confirmed.
* The maintainers are responsible for re-triggering tests when their third
party testing system breaks.
* Support recheck to request re-running a test.
* Support the following syntaxes ``recheck no bug`` and ``recheck bug ###``.
* Recheck means recheck everything. A single recheck comment should
re-trigger all testing systems.
* Publish who the maintainers of the third party testing system are, and make
them available for support as needed. Maintainers are encouraged to be
in IRC regularly to make it faster to contact them.
* All CI comments must contain a link to a contact page with the details.
* Include a public link to all test artifacts to make debugging failed tests
easier. This should include:
* Environment details
* Test configuration
* Skipped tests
* logs should include a trace of the commands used
* OpenStack logs
* Tempest logs (including ``testr_results.html.gz``)
Reading the Event Stream Reading the Event Stream
------------------------ ------------------------