openinfra/refstack-client
Code Issues Proposed changes
8f8293611d
refstack-client/specs/newton/approved/refstack-waiver.rst
Luz Cazares d1bfd90593 Add support for new doc PTI jobs 
Implement optional cleanup to support the new PTI jobs
for documentation which won't use tox anymore.

Docs tox environment will still available for developers but
it is no loger used in the gate to construct the documentation.

See http://lists.openstack.org/pipermail/openstack-dev/2017-December/125710.html

Change-Id: Ib26b711365389c59658f9d7f6cafd5acb736dfce
2017-12-18 15:35:32 -08:00

265 lines
7.0 KiB
ReStructuredText
Raw Blame History

=============================================================
Defcore waiver for additional properties on Nova API response
=============================================================
Launchpad blueprint: https://blueprints.launchpad.net/refstack/+spec/refstack-waiver
Defcore waiver: https://review.openstack.org/#/c/333067/
Defcore committee approved above waiver which allows vendors who are using the
Nova 2.0 API with additional properties to disable strict response checking
when testing products for the OpenStack Powered program in 2016.
This spec defines the changes needed for refstack-client to optionally bypass
Tempest strict validation.
Problem description
===================
Vendors need an automated way to apply the waiver. The proposed method is to
run Tempest from the RefStack client, identify tests that fail because of
strict response checking, and rerun those tests with strict checking disabled.
APIs and test cases using the waiver must be clearly identified.
Proposed change
===============
1. Workflow
- Vendor run Tempest suite as usual: via refstack-client test, ostestr, testr
or with any other test runner. Some test cases failed due to additional
properties in Nova response.
- Vendor have the subunit test results file from the Tempest test execution.
- Vendor have the Tempest configuration file.
- Vendor rerun failed test cases by running ``refstack-client bypass-extras``
command. Command identifies failed test cases, disables Tempest strict
validations, rerun test cases, and enables strict validations again.
.. code-block:: bash
$ refstack-client bypass-extras --subunit-file <results> --conf-file <tempest-conf-file>
- Output of bypass-extras command is a zip bundle containing the following
files:
- tests_list - List of failed test cases due to additional properties.
- patched_schemas - List of tempest schemas which value was set to True
(to allow additional properties).
- api_details - API call details from each failed test case (due to
additional properties).
- rerun_test_results - The subunit result file for the re-run test cases.
- combined_test_results.json - The Refstack JSON file with the combined
passed TCs from both initial and rerun subunit files.
2. Implement "bypass-extras" Refstack command:
Assume Tempest test suite was run independently.
Subunit test results and Tempest configuration files are available.
*bypass-extras* command is the helper tool for vendors to bypass the strict
validation of additional properties in Tempest. Process steps and
implementation details are explained on step 3.
.. code-block:: bash
$ refstack-client --help
usage: refstack-client [-h] <ARG> ...
...
bypass-extras Apply Defcore waiver to identify additional properties
on Nova API response. Re-runs failed test cases
without Tempest strict response validations.
$ refstack-client bypass-extras --help
usage: refstack-client [-h] <ARG> ...
To see help on specific argument, do:
refstack-client <ARG> -h waiver
[-h] [-s | -v] [-y] [--url URL] [-k] [-i PRIV_KEY] file
optional arguments:
-h, --help Show this help message and exit
-s, --silent Suppress output except warnings and errors.
-v, --verbose Show verbose output.
-y Assume Yes to all prompt queries
--subunit-file Path to subunit test result file.
-c, --conf-file Path of the Tempest configuration file to use.
3. Flow for ``bypass-extras`` command.
Having as input a subunit test results file and a Tempest configuration file:
3.1 Find failed test cases and its details
Integrate code from find_additional_properties.py into Refstack-client to
analyze subunit stream (from input results file). Find failed test cases
due to additional properties in the response. Reconstruct the tempest schema
causing the test case failure. Run subunit-describe-calls
filter command to get test cases API call details.
Input: subunit-results
Output files:
- tests_list - List of failed test cases due to additional properties.
- patched_schemas - List of tempest schemas causing errors
- api_details - API calls from each test case.
3.2 Patch Tempest:
Create patch for .tempest virtual environment which lives under refstack-client
installation.
- Modify tempest/lib/api_schema/response/compute/v2_1/__init__.py:
- Import module where schema lives.
- Set schema addtionalProperties key to True so that additional properties
are accepted - bypass strict validation.
3.3 Rerun failed test cases using patched refstack-client .tempest environment
Use tests_list as withelist for ostestr in order to re-run failed test cases.
.. code-block:: bash
ostestr --serial -w test_list
Input: test_list and conf-file files.
Output: rerun_test_results subunit file
3.4 Remove Tempest patch
Regardless of previous steps outcome, unpatching Tempest step will be
attempted.
Clean __init__.py by opening with access mode 'w'
3.5 Create refstack JSON format files
Transform subunit result files - The one provided as input and the rerun
test results - into a combined refstack JSON format.
Input: initial_results_file, rerun_test_results files.
Output: combined_test_results.json files.
3.6 Create zip bundle
Alternatives
------------
- Add additional property to Tempest config file
- For Tempest patch
Comment the validate_response call by looking into the service_client.py file
for the corresponding method (search through code files).
- Manual process
Products applying for the OpenStack Powered Trademark in 2016 may
request the waiver by submitting subunit data from their Tempest run
that can be manually analyzed by the `find_additional_properties.py` script
from the DefCore repository. This script will identify tests that
failed because of additional properties. The vendor will then need
to manually modify tempest-lib to remove additional checks on the impacted
APIs.
Data model impact
-----------------
None
REST API impact
---------------
None
Security impact
---------------
None
Notifications impact
--------------------
None
Other end user impact
---------------------
None
Performance Impact
------------------
None
Other deployer impact
---------------------
None
Developer impact
----------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
luz-cazares
Other contributors:
Chris Hoge
Work Items
----------
- Add *bypass-extras* command to refstack-client.
- Integrate find_additional_properties code
- Method to call subunit-describe-calls filter
- Implement JSON schema gathering from test exception
- Implement Tempest patch
- Implement Tempest unpatch
- Create zip bundle with file results
- Add *bypass-extras* command usage documentation
Dependencies
============
None
Testing
=======
Add unit testing for the new command, verify expected outcomes are met.
Documentation Impact
====================
Add refstack-client bypass-extras usage information under refstack-client/
README.rst
References
==========
None
View Git Blame Copy Permalink