7e8af456c4
Test results are validated using jsonschema. Actual schema can be retrieved from /v1/results/schema endpoint. https://github.com/stackforge/refstack/blob/master/specs/proposed/refstack-org-test-result-json-schema.rst Change-Id: I05e5fa3b84c60925f162985cd7d51b14d69b19d8
177 lines
4.8 KiB
ReStructuredText
177 lines
4.8 KiB
ReStructuredText
==========================================
|
|
Example Spec - The title of your blueprint
|
|
==========================================
|
|
|
|
https://blueprints.launchpad.net/refstack/+spec/test-result-json-schema
|
|
|
|
Refstack accepts connections from the refstac-client to post test results
|
|
for storage and scoring. Test results are in json and follow a simple
|
|
format. However there is no defined schema for the test results, allowing
|
|
for a greater possibility of invalid or malicious data to be injected
|
|
into the refstack database. This spec proposes the creation of a json
|
|
schema that can be used by client developers to generate their own valid
|
|
test results. The spec will also be used by the server to validate
|
|
incoming test results.
|
|
|
|
Problem description
|
|
===================
|
|
|
|
* Refstack accepts test results in json format. However, there is no
|
|
explicit definition of the structure of this json data.
|
|
|
|
* Data sent to the Refstack server is not validated against any schema,
|
|
increasing the possibility of invalid or malicious data.
|
|
|
|
* Clients developers do not have a specific schema to develop against.
|
|
|
|
|
|
Proposed change
|
|
===============
|
|
|
|
This change proposes the following additions:
|
|
|
|
* A json schema file to capture the format of the test results.
|
|
|
|
* Validation of incoming results against the json schema in Refstack.
|
|
|
|
Alternatives
|
|
------------
|
|
|
|
Another alternative is manually writing code to validate test results.
|
|
This schema does not preclue that alternative, but by creating and
|
|
validating against a schema we take advantage of existing libraries
|
|
and reduce the possiblity of introducing unintended parsing errors.
|
|
|
|
Data model impact
|
|
-----------------
|
|
|
|
This proposed change has two impacts on the data model:
|
|
|
|
* It introduces a schema that must be maintained and versioned
|
|
against the current data model.
|
|
|
|
* It adds an additional level of validation to the 'create' operation
|
|
of the CRUD model. Since running a test is a long operation this
|
|
addition level of complexity will not have an impact on application
|
|
performance.
|
|
|
|
* No database migrations will be necessary.
|
|
|
|
* A static page to serve the schema will be necessary.
|
|
|
|
REST API impact
|
|
---------------
|
|
|
|
The v1 api must be modified to validate test results. The modification
|
|
will result in an addition to behavior on the API with no changes
|
|
to the user-facing endpoint.
|
|
|
|
**description:** Receive tempest test result from a remote test runner.
|
|
This function expects json formated pass results.
|
|
Update to the api described at:
|
|
https://github.com/stackforge/refstack/blob/master/specs/approved/api-v1.md
|
|
|
|
**url:** post /v1/results
|
|
|
|
**failed response:** http:400 - Malformed data.
|
|
|
|
{
|
|
'message': 'Malformed json data, see /v1/results/schema'
|
|
}
|
|
|
|
**url:** get /v1/results/schema
|
|
|
|
**valid response:** http:200 schema.json file
|
|
|
|
No invalid responses. No accepted parameters.
|
|
|
|
Security impact
|
|
---------------
|
|
|
|
This change is intented to improve the security of the application
|
|
by introducing data validation. No data-changing apis are
|
|
introduced.
|
|
|
|
Notifications impact
|
|
--------------------
|
|
|
|
No known notification impacts.
|
|
|
|
Other end user impact
|
|
---------------------
|
|
|
|
End users may experience client failures if their client does not produce
|
|
valid json.
|
|
|
|
Performance Impact
|
|
------------------
|
|
|
|
This change introduces a validation step to the POST process. The additional
|
|
time used to validate the data is very small compared to the time taken
|
|
to generate data through cloud testing. The timing impact is negligible
|
|
in the context of the upload use case.
|
|
|
|
Other deployer impact
|
|
---------------------
|
|
|
|
This change will take immediate effect after being merged.
|
|
|
|
Developer impact
|
|
----------------
|
|
|
|
No additional developer impact.
|
|
|
|
Implementation
|
|
==============
|
|
|
|
This change will be implemented as a validation funtion in the API POST
|
|
pipeline. It will essentially be middleware that takes the input data,
|
|
validates the results, and sends back a positive or a negative result.
|
|
If the result is negative, the 400 response will be returned.
|
|
If the result is positive, the data processing will continue as normal.
|
|
|
|
Assignee(s)
|
|
-----------
|
|
|
|
Primary assignee:
|
|
hogepodge
|
|
|
|
Work Items
|
|
----------
|
|
|
|
* Write json schema
|
|
* Write json schema GET endpoint
|
|
* Implement validation on results POST endpoint.
|
|
* Implement unit tests for validation and endpoint.
|
|
|
|
Dependencies
|
|
============
|
|
|
|
No additional dependencies will be added.
|
|
|
|
Testing
|
|
=======
|
|
|
|
To the TestRefStackApi class the following tests will be added:
|
|
* test_results_valid_data
|
|
* test_results_invalid_data
|
|
|
|
These results will confirm both positive (200) and negative (400) results.
|
|
|
|
To the unit tests the validator function will be tested:
|
|
* test_valid_data
|
|
* test_invalid_data
|
|
* test_empty_data
|
|
|
|
These results will provide three modes of schema validation.
|
|
|
|
Documentation Impact
|
|
====================
|
|
|
|
Documentation will be updated to link to current schema.
|
|
|
|
References
|
|
==========
|
|
|
|
No additional references.
|