d8fb682a2b
At the Board of Directors meeting on April 24, 2016 the Board of Directors indicated that as DefCore has evolved it's focus on interoperability and it's working structure, it's name may be a source of some confusion to those outside of the community or who are new to the community. The Board made an informal request that the DefCore Committee consider changing it's name to more clearly reflect it's focus and structure. This patch is the first step in that process. It updates references in various documents to change the name "DefCore Committee" to "Interop Working Group" as agreed at the summer 2016 DefCore Committee Sprint [1]. It should be noted that this patch should be considered a work in progress to generate discussion until the new name is approved by the DefCore Committee and the Board of Directors. Should we elect to go forward with the new name, some other actions will also need to be taken, including but not limited to: 1. We will need to consider updating references on the OpenStack wiki. 2. We will need to consider updating the name of our IRC channel, mailing list, Launchpad project, and git repository. Most of these changes will need to be carefully coordinated with the OpenStack Infrastructure team. 3. We will need to take into account external resources that point to DefCore artifacts, such as Foundation-maintained websites (such as: http://www.openstack.org/interop ). 4. We will need to coordinate with RefStack to minimize impact. 5. We will need to clearly communicate the name change to the rest of the community. Note also that I've intentionally left many historical documents that have been superceded (such as the 2015A process docs, Guidelines that are no longer used, etc) in tact. There seemed little value in spending time on them and cluttering the patch with them since they're now obsolete. [1] https://etherpad.openstack.org/p/DefCoreSummer2016Sprint Change-Id: I79d337c193e75c54d49f1d847468f6347e2ef2b3
81 lines
3.2 KiB
ReStructuredText
81 lines
3.2 KiB
ReStructuredText
======================
|
|
Feature Description
|
|
======================
|
|
|
|
This test specification describes interoperability testing as it is understood
|
|
by the Interop Working Group. An important goal of interoperability is that
|
|
end users of OpenStack should be able to expect certain APIs to be reliable
|
|
across clouds.
|
|
|
|
The spec aims to define how interoperability should be tested for OpenStack.
|
|
It provides guidelines for evaluating current tests and the need for new test
|
|
cases.
|
|
|
|
This document is a declaration of intent and looks at where interoperability
|
|
testing is heading, not just where it is today.
|
|
|
|
What is tested
|
|
==============
|
|
|
|
The focus of this test specification is functional and behavioral testing
|
|
of the OpenStack REST API of the OpenStack projects.
|
|
|
|
What is not tested
|
|
==================
|
|
|
|
- OpenStack clients
|
|
- Internal APIs
|
|
- Python APIs
|
|
- Performance testing
|
|
|
|
Interoperability Tests
|
|
======================
|
|
|
|
Goals
|
|
#. The only requirement to run interop functional tests should be an endpoint
|
|
and credentials necessary to run the tests. Tests should ideally discover
|
|
the methods it needs to use to set up whatever is necessary to exercise the
|
|
API under test. For example: a test for image-delete needs to have an image
|
|
in the cloud to delete. Ideally, the test should discover what method it needs
|
|
to use to put an image in the cloud (image upload, image import, etc). Tests
|
|
must favor direct API calls where they can. Discoverability for multiple
|
|
implementations remains an open issue.
|
|
|
|
#. The test should favor minimization. Test only the capability under
|
|
test with as little set up as possible, using as few other resources
|
|
as possible. For example, tests that don't actually need multiple user
|
|
credentials or admin privileges shouldn't require them.
|
|
|
|
Guidelines
|
|
#. Interop API functional tests should target and validate one
|
|
capability. Some capabilities may require discovery calls
|
|
and these should be included also in the test. Each capability
|
|
should have at least one interop API functional test.
|
|
|
|
#. Interop API functional tests should not use non-required
|
|
capabilities.
|
|
|
|
#. Interop scenario tests may use several capabilities to test end
|
|
to end functionality and none of the capabilities used may be
|
|
non-required capabilities.
|
|
|
|
#. Tests must be reliable; race conditions must be avoided; tests
|
|
should not be time dependent. If any of these problems occur the
|
|
test will be flagged and either fixed or removed from the guidelines.
|
|
|
|
#. Tests must be able to run consistently across all the releases
|
|
covered by the Guidelines which are considered valid by the OpenStack
|
|
Foundation for the OpenStack Powered program.
|
|
|
|
#. Tests should not require more credentials than required to test the
|
|
capability itself.
|
|
|
|
#. Tests must not be harmful to the end user's environment, that is no
|
|
changing credentials nor destroying user's data nor altering any
|
|
other user resource other than those specifically created as part
|
|
of the test.
|
|
|
|
#. Test cases must setup any data they need to run and clean up after
|
|
themselves. It is not acceptable to corrupt the user's environment
|
|
or leave any test data or test configuration behind.
|