diff --git a/doc/source/process/procedure.rst b/doc/source/process/procedure.rst new file mode 100755 index 00000000..994c4233 --- /dev/null +++ b/doc/source/process/procedure.rst @@ -0,0 +1,68 @@ +Interoperability Guideline Testing +================================== + +Testing against 2017.01 until present Capabilities +-------------------------------------------------- + +Tempest can be run standalone, or under a test runner such as refstack-client. +We recommend running under refstack-client. + +The test names of the capabilities are derived from a recent release of +Tempest, from the time of capability approval. Keep in mind that Tempest +is under active development, and tests may move. If you're not seeing +full coverage, please consider reverting back to a Tempest SHA that more +closely matches the capability release date. The git SHA of Tempest that was +known to be working at the time the Guideline was approved is listed in the +Guideline JSON document itself (just search for "git-sha"). Please contact +RefStack team at for assistance if +needed. Use [interop] and/or [refstack] tags in the subject of the email. + +It's important to run a recent version of Tempest, as some bugs might have +been fixed just recently. refstack-client uses Tempest from a specific commit +(see setup_env script or README), which is intentionally older than master to +avoid compatibility issues with older stable releases of OpenStack. + +Network provisioning have been fixed. Some tests are still flagged due to +outstanding bugs in the Tempest library, particularly tests that require SSH. +We are working on correcting these bugs upstream. Please note that +although some tests are flagged because of bugs, there is still an +expectation that the capabilities covered by the tests are available. + +In addition to testing required capabilities, we are also interested +in collecting data on which API tests are being passed by production clouds. +This information will be very useful in determining which capabilities will be +used to define future releases. For that reason, we ask that you run all +tests rather than just the required subset when submitting results to +the OpenStack Foundation. + +It is important to note that you MUST NOT modify the Tempest tests in any +way. Modifying the tests means that Capability being tested is validated +in a different way on your cloud than it is on other clouds, which voids +any guarantee of interoperability. If you're having problems passing +all required tests, please contact RefStack team at + for assistance or consider filing a +request to have the tests flagged. + +Please refer to `HACKING <../HACKING.rst>`_ for information on valid +reasons to flag a test and how to file a flag request. Results from +modified tests cannot be accepted as valid for trademark licensing +purposes. + +Recommended Test Procedure +########################## + +Follow steps mentioned in the refstack-client's README.rst: +https://opendev.org/openstack/refstack-client + +* Follow 'Environment setup' section to clone and install refstack-client + +* Follow steps in 'Usage' section in order to run tests against a desired + OpenStack Marketing program. + +* Review the test results, and when you're satisfied, upload it to RefStack + server (steps in the 'Usage' section) then send them to interop@openstack.org + +* Every effort should be made to pass all of the required tests, but you + will want to compare any lists of failed tests to the list of flagged tests. + The refstack server will automatically grade tests results, taking + into account flagged tests.