openinfra/interop
Code Issues Proposed changes

Merge "Make procedure.rst from 2016.08 more generic"

Browse Source
This commit is contained in:
Zuul 2021-06-21 22:08:15 +00:00 committed by Gerrit Code Review
commit 407352c872
1 changed files with 68 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

68
doc/source/process/procedure.rst Executable file
View File

@ -0,0 +1,68 @@
Interoperability Guideline Testing
==================================
Testing against 2017.01 until present Capabilities
--------------------------------------------------
Tempest can be run standalone, or under a test runner such as refstack-client.
We recommend running under refstack-client.
The test names of the capabilities are derived from a recent release of
Tempest, from the time of capability approval. Keep in mind that Tempest
is under active development, and tests may move. If you're not seeing
full coverage, please consider reverting back to a Tempest SHA that more
closely matches the capability release date. The git SHA of Tempest that was
known to be working at the time the Guideline was approved is listed in the
Guideline JSON document itself (just search for "git-sha"). Please contact
RefStack team at <openstack-discuss@lists.openstack.org> for assistance if
needed. Use [interop] and/or [refstack] tags in the subject of the email.
It's important to run a recent version of Tempest, as some bugs might have
been fixed just recently. refstack-client uses Tempest from a specific commit
(see setup_env script or README), which is intentionally older than master to
avoid compatibility issues with older stable releases of OpenStack.
Network provisioning have been fixed. Some tests are still flagged due to
outstanding bugs in the Tempest library, particularly tests that require SSH.
We are working on correcting these bugs upstream. Please note that
although some tests are flagged because of bugs, there is still an
expectation that the capabilities covered by the tests are available.
In addition to testing required capabilities, we are also interested
in collecting data on which API tests are being passed by production clouds.
This information will be very useful in determining which capabilities will be
used to define future releases. For that reason, we ask that you run all
tests rather than just the required subset when submitting results to
the OpenStack Foundation.
It is important to note that you MUST NOT modify the Tempest tests in any
way. Modifying the tests means that Capability being tested is validated
in a different way on your cloud than it is on other clouds, which voids
any guarantee of interoperability. If you're having problems passing
all required tests, please contact RefStack team at
<openstack-discuss@lists.openstack.org> for assistance or consider filing a
request to have the tests flagged.
Please refer to `HACKING <../HACKING.rst>`_ for information on valid
reasons to flag a test and how to file a flag request. Results from
modified tests cannot be accepted as valid for trademark licensing
purposes.
Recommended Test Procedure
##########################
Follow steps mentioned in the refstack-client's README.rst:
https://opendev.org/openstack/refstack-client
* Follow 'Environment setup' section to clone and install refstack-client
* Follow steps in 'Usage' section in order to run tests against a desired
OpenStack Marketing program.
* Review the test results, and when you're satisfied, upload it to RefStack
server (steps in the 'Usage' section) then send them to interop@openstack.org
* Every effort should be made to pass all of the required tests, but you
will want to compare any lists of failed tests to the list of flagged tests.
The refstack server will automatically grade tests results, taking
into account flagged tests.