openinfra/interop
Code Issues Proposed changes

Add Guidance on Modifying Tests, Filing Flags

Browse Source 
It's now come up a couple of times that we've found people modifying
tests in order to pass them.  Although seems obvious that not testing
things the same way across all clouds voids any guarantee of
interoperability, we should make this explit in the guidance we provide.
It's been proposed that one reason folks may be modifying tests is that
they don't clearly understand what they need to do in order to file a
flag request when they run into a problem, so they try to fix what they
think is wrong with the test instead.

This patch makes three changes to help address the issue:

1.) It adds a section to the 2016.01 guidance (which is currently
linked to from http://openstack.org/interop) to make it explicitly
clear that tests shouldn't be modified.

2.) Assuming that the interop website will be updated to point to the
most recent Guideline at some point, it creates the same guidance for
2016.08.

3.) It ammends HACKING to include simple instructions and an example
of how to file a flag request.  The individual guidelines' guidance
files also point to HACKING so folks know to look there.

Change-Id: I2f57a852da3181714e87d8e689dd5a1cb33cb417
This commit is contained in:
Mark T. Voelker 2016-10-05 16:23:32 -04:00
parent 0abe4b8c15
commit 35d8985013
3 changed files with 175 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
2016.01/procedure.rst
View File

@ -17,8 +17,10 @@ The test names of the capabilities are derived from a recent release of
Tempest, from the time of capability approval. Keep in mind that Tempest
is under active development, and tests may move. If you're not seeing
full coverage, please consider reverting back to a Tempest sha that more
closely matches the capability release date. Please contact Chris Hoge
<interop@openstack.org> for assistance if needed.
closely matches the capability release date. The git SHA of Tempest that was
known to be working at the time the Guideline was approved is listed in the
Guideline JSON document itself (just search for "git-sha"). Please contact
Chris Hoge <interop@openstack.org> for assistance if needed.
It's important to run a recent version of Tempest, as major bugs related to
network provisioning have been fixed. Some tests are still flagged due to
@ -30,7 +32,20 @@ expectation that the capabilities covered by the tests are available.
In addition to testing required capabilities, we are also interested
in collecting data on which API tests are being passed by production clouds.
This information will be very useful in determining which capabilities will be
used to define future releases.
used to define future releases. For that reason, we ask that you run all
tests rather than just the required subset when submitting results to
the OpenStack Foundation.
It is important to note that you MUST NOT modify the Tempest tests in any
way. Modifying the tests means that Capability being tested is validated
in a different way on your cloud than it is on other clouds, which voids
any guarantee of interoperability. If you're having problems passing
all required tests, please contact Chris Hoge <interop@openstack.org>
for assistance or consider filing a request to have the tests flagged.
Please refer to `HACKING <../HACKING.rst>`_ for information on valid
reasons to flag a test and how to file a flag request. Results from
modified tests cannot be accpeted as valid for trademark licensing
purposes.
DefCore Recommended Test Procedure

99
2016.08/procedure.rst Executable file
View File

@ -0,0 +1,99 @@
2016.08 Interoperability Guideline Testing
==========================================
Testing against 2016.08 Capabilities
-------------------------------------
https://git.openstack.org/cgit/openstack/defcore/tree/2016.08.json
Tempest can be run standalone, or under a test runner such as refstack-client
or rally. If only testing against required capabilities, you can use the
--load-list argument and a file containing a list of the required tests. If
run with the refstack-client, test output will be parsed to list only
passing tests in a JSON formatted file. We recommend running under
refstack-client.
The test names of the capabilities are derived from a recent release of
Tempest, from the time of capability approval. Keep in mind that Tempest
is under active development, and tests may move. If you're not seeing
full coverage, please consider reverting back to a Tempest sha that more
closely matches the capability release date. The git SHA of Tempest that was
known to be working at the time the Guideline was approved is listed in the
Guideline JSON document itself (just search for "git-sha"). Please contact
Chris Hoge <interop@openstack.org> for assistance if needed.
It's important to run a recent version of Tempest, as major bugs related to
network provisioning have been fixed. Some tests are still flagged due to
outstanding bugs in the Tempest library, particularly tests that require SSH.
We are working on correcting these bugs upstream. Please note that
although some tests are flagged because of bugs, there is still an
expectation that the capabilities covered by the tests are available.
In addition to testing required capabilities, we are also interested
in collecting data on which API tests are being passed by production clouds.
This information will be very useful in determining which capabilities will be
used to define future releases. For that reason, we ask that you run all
tests rather than just the required subset when submitting results to
the OpenStack Foundation.
It is important to note that you MUST NOT modify the Tempest tests in any
way. Modifying the tests means that Capability being tested is validated
in a different way on your cloud than it is on other clouds, which voids
any guarantee of interoperability. If you're having problems passing
all required tests, please contact Chris Hoge <interop@openstack.org>
for assistance or consider filing a request to have the tests flagged.
Please refer to `HACKING <../HACKING.rst>`_ for information on valid
reasons to flag a test and how to file a flag request. Results from
modified tests cannot be accpeted as valid for trademark licensing
purposes.
Recommended Test Procedure
The following procedure is recommended, but not required for testing.
This procedure assumes you're running a Linux test platform (Ubuntu 14.04
or CentOS 7 have been verified) with administrator privileges.
* Download the RefStack client:
``git clone https://git.openstack.org/openstack/refstack-client``
* In the refstack-client directory, install tempest and required dependencies.
You may specify a specific tag of tempest with the -t option.
``./setup_env``
* Optionally, download a list of test from the RefStack site. We strongly
encourage you to run the full set of api tests, as this not only qualifies
you for the trademark but also gives the Interop Working Group (formerly
the DefCore Committee) feedback on deployed capabilities to help us
determine future guidelines.
https://refstack.openstack.org/api/v1/guidelines/2016.08/tests?type=required
* Configure tempest.conf for your cloud. If you need assistance in common
parameters or settings contact interop@openstack.org. The recommended
configuration is to use one non-admin account, defined in account.yaml
with dynamic credentials disabled. More information is available in the
configuration guide at
http://docs.openstack.org/developer/tempest/configuration.html
* You can run within the refstack, from the refstack-client directory either
against all api tests or against the downloaded test list.
``./refstack-client test -c ~/tempest.conf``
``./refstack-client test -c ~/tempest.conf --test-list <test-list-file-name>``
* Review the test results, and when you're satisfied, upload it to RefStack server
then send them to interop@openstack.org.
``./refstack-client upload <Path of results file>``
* The results are stored in a JSON file in the directory. You can also check your
result on the RefStack server https://refstack.openstack.org:
``.tempest/.testrepository``
* Every effort should be made to pass all of the required tests, but you
will want to compare any lists of failed tests to the list of flagged tests.
The refstack server will automatically grade tests results, taking
into account flagged tests.

58
HACKING.rst
View File

@ -105,3 +105,61 @@ Valid reasons for flagging a test are limited to the following:
- [D405] The test reflects an implementation choice that is not widely
deployed even if the Capability is widely deployed.
- [D406] Test removed permanently from test suite.
To file a flag request for a test, follow these steps:
#. Make sure you have a development environment set up accoring to the
`Developer's Guide
<http://docs.openstack.org/infra/manual/developers.html>`_.
#. Clone the defcore repo by running:
.. code-block:: console
$ git clone https://git.openstack.org/openstack/defcore.git
#. Open the relevant Guideline or Guidelines in your favorite text
editor and locate the tests you want to flag.
#. Add a flag block to the test using the rationale above. For
example, if I want to flag this test:
.. code-block:: json
"tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filtered_by_ip": {
"idempotent_id": "id-43a1242e-7b31-48d1-88f2-3f72aa9f2077"
},
I might modify the section to read:
.. code-block:: json
"tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filtered_by_ip": {
"idempotent_id": "id-43a1242e-7b31-48d1-88f2-3f72aa9f2077",
"flagged": {
"date": "2016-06-13",
"reason": "[D401] This test is skipped by Tempest due to bug https://bugs.launchpad.net/tempest/+bug/1600349.",
"action": "Remove flag when Tempest fix is in place."
}
},
#. Once you've saved your changes, commit the chage locally:
.. code-block:: console
$ git commit -a
Be sure to write a useful commit message explaning your change.
#. Push the change to gerrit, where your request can be reviewed:
.. code-block:: console
$ git review
Please note that if you're having trouble setting up a development
environment, you can also talk to the OpenStack Foundation's
Interoperability Engineer <interop@openstack.org> or contact the
Interop Working Group (formerly DefCore Committee) via email
<defcore-committee@lists.openstack.org>. We'll be happy to try to
assist you or help file a request on your behalf.