[Spec] Update and move spec about new output format
Spec 'improve_scenario_output_format.rst' is implemented, so it is moved to doc/specs/implemented/ Also there were some updates to this spec to conform improvements that are actually done: * Each output data set has 'description' text passed to Scenario.add_output() while adding the output. Description is displayed in HTML report under chart title. * Method Scenario.add_output() has changed (schema verification is added) * Subtab 'Detailed' is renamed to 'Per iteration' Change-Id: Iddf0b0f5edea8002493d47131994f46202363ef5
This commit is contained in:
Alexander Maretskiy
parent
07d277c2f3
commit
821e4e6b32
324
doc/specs/implemented/improve_scenario_output_format.rst
Normal file
324
doc/specs/implemented/improve_scenario_output_format.rst
Normal file
@ -0,0 +1,324 @@
|
|||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
..
|
||||||
|
This template should be in ReSTructured text. The filename in the git
|
||||||
|
repository should match the launchpad URL, for example a URL of
|
||||||
|
https://blueprints.launchpad.net/heat/+spec/awesome-thing should be named
|
||||||
|
awesome-thing.rst . Please do not delete any of the sections in this
|
||||||
|
template. If you have nothing to say for a whole section, just write: None
|
||||||
|
For help with syntax, see http://sphinx-doc.org/rest.html
|
||||||
|
To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html
|
||||||
|
|
||||||
|
=======================================
|
||||||
|
Improvements for scenario output format
|
||||||
|
=======================================
|
||||||
|
|
||||||
|
Current implementation of how scenario saves output data is limited and
|
||||||
|
does not meet the needs - it neither allows having more than one data set,
|
||||||
|
nor saving custom data structures by each iteration. There is simply a dict
|
||||||
|
with int values.
|
||||||
|
|
||||||
|
This specification proposes how this can be significantly improved.
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
At first, let's clarify types of desired output.
|
||||||
|
|
||||||
|
Output divides on two main types: additive and complete.
|
||||||
|
|
||||||
|
*Additive output* requires processing and representation for the whole
|
||||||
|
scenario. For example each iteration has duration - this additive data can
|
||||||
|
be taken from each iteration and analyzed how it changes during the
|
||||||
|
scenario execution.
|
||||||
|
|
||||||
|
*Complete output* data is completely created by iteration and does not require
|
||||||
|
extra processing. It is related to this specific iteration only.
|
||||||
|
|
||||||
|
Currently scenario can just return a single dict with int values - this is an
|
||||||
|
additive data only, and it is stored in iteration results according to
|
||||||
|
this schema:
|
||||||
|
|
||||||
|
.. code-block::
|
||||||
|
|
||||||
|
"result": {
|
||||||
|
...
|
||||||
|
"scenario_output": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"data": {
|
||||||
|
"type": "object"
|
||||||
|
},
|
||||||
|
"errors": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"required": ["data", "errors"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Here are main issues:
|
||||||
|
|
||||||
|
* single data set - this does not allow to split data (if required) among
|
||||||
|
different sources. For example scenario runs two (or more) third-party
|
||||||
|
tools or scripts but has to put all data into single dict
|
||||||
|
|
||||||
|
* output is additive only - so its representation makes sense only after
|
||||||
|
putting data from all iterations together. Scenario iteration can not
|
||||||
|
save its own data list that can be processed independently from another
|
||||||
|
iterations.
|
||||||
|
|
||||||
|
* there is no specific data for HTML report generation like chart title
|
||||||
|
and chart type, so report uses hardcoded values.
|
||||||
|
|
||||||
|
As result, HTML report can represent output by a single chart of single type:
|
||||||
|
|
||||||
|
.. code-block::
|
||||||
|
|
||||||
|
.--------.
|
||||||
|
| Output |
|
||||||
|
-----' '-----------
|
||||||
|
Scenario output
|
||||||
|
--------------------
|
||||||
|
| |
|
||||||
|
| SINGLE StackedArea |
|
||||||
|
| |
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
Scenario should have ability to save arbitrary number of both additive
|
||||||
|
and complete output data. This data should include titles and instructions
|
||||||
|
how to be processed and displayed in HTML report.
|
||||||
|
|
||||||
|
Here is proposed iterations results structure for output data:
|
||||||
|
|
||||||
|
.. code-block::
|
||||||
|
|
||||||
|
|
||||||
|
"result": {
|
||||||
|
...
|
||||||
|
"output": {
|
||||||
|
"additive": [
|
||||||
|
# Each iteration duplicates "title", "description", "chart" and
|
||||||
|
# items keys, however this seems to be less evil than keeping
|
||||||
|
# aggregated metadata on upper level of task results schema.
|
||||||
|
# "chart" is required by HTML report and should be a name of
|
||||||
|
# existent Chart subclass that is responsible for processing
|
||||||
|
# and displaying the data
|
||||||
|
{"title": "How some durations changes during the scenario",
|
||||||
|
"description": "Some details explaind here",
|
||||||
|
"chart": "OutputStackedAreaChart",
|
||||||
|
"items": [[<str key>, <float value>], ...] # Additive data
|
||||||
|
},
|
||||||
|
... # More data if required
|
||||||
|
],
|
||||||
|
"complete": [
|
||||||
|
# Complete data from this specific iteration.
|
||||||
|
# "widget" is required by HTML report and should be a name
|
||||||
|
# of chart widget (see details below) that responsible for
|
||||||
|
# displaying data. We do not need to specify "chart" here
|
||||||
|
# because this data does not require processing - it is
|
||||||
|
# already processed and represents a result of Chart.render()
|
||||||
|
{"title": "Interesting data from specific iteration",
|
||||||
|
"description": "Some details explaind here",
|
||||||
|
"widget": "StackedArea",
|
||||||
|
"data": [
|
||||||
|
[
|
||||||
|
<str key>,
|
||||||
|
[[<float X pos>, <float Y value>], ...]
|
||||||
|
],
|
||||||
|
...
|
||||||
|
]
|
||||||
|
},
|
||||||
|
... # More data if required
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
**NOTES**:
|
||||||
|
|
||||||
|
* for backward compatibility, data from deprecated "scenario_output" should
|
||||||
|
be transformed into "output/data/additive[0]" on-the-fly (for example
|
||||||
|
if we load task results from file)
|
||||||
|
|
||||||
|
* as you can see, there is no container *output/errors* - that is because
|
||||||
|
value of *errors* is not used at all and not required (there is another
|
||||||
|
container for errors in iteration results)
|
||||||
|
|
||||||
|
How scenario saves output data
|
||||||
|
------------------------------
|
||||||
|
|
||||||
|
Scenario should be extended with method *add_output()*:
|
||||||
|
|
||||||
|
.. code-block::
|
||||||
|
|
||||||
|
class Scenario(...):
|
||||||
|
|
||||||
|
def __init__(self, context=None):
|
||||||
|
...
|
||||||
|
self._output = {"additive": [], "complete": []}
|
||||||
|
|
||||||
|
...
|
||||||
|
|
||||||
|
def add_output(self, additive=None, complete=None):
|
||||||
|
"""Add iteration values for additive output.
|
||||||
|
|
||||||
|
:param additive: dict with additive output
|
||||||
|
:param complete: dict with complete output
|
||||||
|
:raises RallyException: When additive or complete has wrong format
|
||||||
|
"""
|
||||||
|
for key, value in (("additive", additive), ("complete", complete)):
|
||||||
|
if value:
|
||||||
|
try:
|
||||||
|
jsonschema.validate(
|
||||||
|
value, task.OUTPUT_SCHEMA["properties"][key]["items"])
|
||||||
|
self._output[key].append(value)
|
||||||
|
except jsonschema.ValidationError:
|
||||||
|
raise exceptions.RallyException(
|
||||||
|
"%s output has wrong format" % key.capitalize())
|
||||||
|
|
||||||
|
|
||||||
|
Here is an example how scenario can save different output:
|
||||||
|
|
||||||
|
.. code-block::
|
||||||
|
|
||||||
|
class SomePlugin(Scenario):
|
||||||
|
|
||||||
|
def specific_scenario(self):
|
||||||
|
...
|
||||||
|
|
||||||
|
self.add_output(additive={"title": "Foo data",
|
||||||
|
"description": "Some words about Foo",
|
||||||
|
"chart": "OutputStackedAreaChart",
|
||||||
|
"items": [["foo 1", 12], ["foo 2", 34]]})
|
||||||
|
self.add_output(additive={"title": "Bar data",
|
||||||
|
"description": "Some words about Bar",
|
||||||
|
"chart": "OutputAvgChart",
|
||||||
|
"items": [["bar 1", 56], ["bar 2", 78]]})
|
||||||
|
self.add_output(complete={"title": "Complete data",
|
||||||
|
"description": "Some details here",
|
||||||
|
"widget": "StackedArea",
|
||||||
|
"data": [["foo key", [ ... ]], ... ]})
|
||||||
|
self.add_output(complete={"title": "Another data",
|
||||||
|
"description": "Some details here",
|
||||||
|
"widget": "Pie",
|
||||||
|
"data": [["bar key", [ ... ]], ... ]})
|
||||||
|
self.add_output(complete={"title": "Yet another data",
|
||||||
|
"description": "Some details here",
|
||||||
|
"widget": "Table",
|
||||||
|
"data": [["spam key", [ ... ]], ... ]})
|
||||||
|
|
||||||
|
Displaying scenario output in HTML report
|
||||||
|
-----------------------------------------
|
||||||
|
|
||||||
|
The following changes are planned for HTML report and charts classes:
|
||||||
|
|
||||||
|
* rename tab *Output* to *Scenario Data*
|
||||||
|
* implement subtabs under *Scenario Data*: *Aggregated* and *Per iteration*
|
||||||
|
* *Aggregated* subtab shows charts with additive data
|
||||||
|
* *Per iteration* subtab shows charts with complete data, for each iteration
|
||||||
|
* Both subtabs (as well as parent tab) are shown only if there is
|
||||||
|
something to display
|
||||||
|
* add base class OutputChart and generic charts classes for processing
|
||||||
|
output data: OutputStackedAreaChart, OutputAvgChart, OutputStatsTable
|
||||||
|
* add optional *title* and *description* arguments to OutputChart.__init__()
|
||||||
|
so title and description - this is important for custom charts
|
||||||
|
* add *WIDGET* property to each OutputChart subclass to bind it to specific
|
||||||
|
chart widget (StackedArea, Pie, Table). For example, AvgChart will be
|
||||||
|
bound to "Pie". This will allow defining both how to process and how
|
||||||
|
to display some data simply by single class name
|
||||||
|
* update return value format of OutputChart.render() with title and widget:
|
||||||
|
{"title": <str>, "description": <str>, "widget": <str>, "data": [...]}
|
||||||
|
|
||||||
|
UI sketch for active "Aggregated" subtab:
|
||||||
|
|
||||||
|
.. code-block::
|
||||||
|
|
||||||
|
.---------------.
|
||||||
|
| Scenario Data |
|
||||||
|
----' '-------------------
|
||||||
|
Aggregated Per iteration
|
||||||
|
-------------
|
||||||
|
<Custom chart title>
|
||||||
|
<Here is a description text>
|
||||||
|
----------------------------
|
||||||
|
| |
|
||||||
|
| Any available chart widget |
|
||||||
|
| |
|
||||||
|
----------------------------
|
||||||
|
|
||||||
|
<Custom chart title>
|
||||||
|
<Here is a description text>
|
||||||
|
----------------------------
|
||||||
|
| |
|
||||||
|
| Any available chart widget |
|
||||||
|
| |
|
||||||
|
----------------------------
|
||||||
|
|
||||||
|
[... more charts]
|
||||||
|
|
||||||
|
UI sketch for active "Per iteration" subtab, let it be iteration 5
|
||||||
|
selected by dropdown:
|
||||||
|
|
||||||
|
.. code-block::
|
||||||
|
|
||||||
|
.---------------.
|
||||||
|
| Scenario Data |
|
||||||
|
----' '-------------------
|
||||||
|
Aggregated Per iteration
|
||||||
|
----------
|
||||||
|
|
||||||
|
[iteration 5]
|
||||||
|
|
||||||
|
<Custom chart title>
|
||||||
|
<Here is a description text>
|
||||||
|
----------------------------
|
||||||
|
| |
|
||||||
|
| Any available chart widget |
|
||||||
|
| |
|
||||||
|
----------------------------
|
||||||
|
|
||||||
|
<Custom chart title>
|
||||||
|
<Here is a description text>
|
||||||
|
----------------------------
|
||||||
|
| |
|
||||||
|
| Any available chart widget |
|
||||||
|
| |
|
||||||
|
----------------------------
|
||||||
|
|
||||||
|
[... more charts]
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
* amaretskiy <amaretskiy@mirantis.com>
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
* Update task results schema with *output* container
|
||||||
|
* Extend Scenario with method *add_output()*
|
||||||
|
* Bound Chart subclasses to specific charts widgets
|
||||||
|
* Add generic Charts subclasses for output data
|
||||||
|
* Changes in HTML report related to *Output* tab
|
||||||
|
* Add scenario with example output data
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
None
|
||||||
Loading…
x
Reference in New Issue
Block a user