6f350c38f1
In the common.streaming_algorithms module, we add the base for the streaming
algorithms that take values from the input stream and compute certain
quantities needed for the benchmark results processing. These algorithms
should use constant memory and be able to scale.
Two streaming algorithms for computing mean and standard deviation are implemented.
There is also a new "max_outliers" SLA that checks for the maximum number of
outliers based on the mean and standard deviation of the durations, computed
using these new streaming algorithms. By default, the outliers are detected
only when the total number of iterations reaches 10 (can be configured).
Example:
3.1 4.2 3.6 4.5 2.8 3.3 4.1 3.8 4.3 2.9 10.2 11.2 3.4
has 2 outliers (10.2 and 11.2), so:
{"outliers": {"max": 2}} -> PASS
{"outliers": {"max": 1}} -> FAIL
Bonus:
* Add gate testing of different SLAs with the Dummy scenario
* Add samples for all SLAs
* Fix detailed message for max_avg_duration
Change-Id: I7c4f77c418c7b61f71b43216110fa4c7aaccc2f5
25 lines
605 B
JSON
25 lines
605 B
JSON
{
|
|
"KeystoneBasic.create_delete_user": [
|
|
{
|
|
"args": {
|
|
"name_length": 10
|
|
},
|
|
"runner": {
|
|
"type": "constant",
|
|
"times": 100,
|
|
"concurrency": 10
|
|
},
|
|
"sla": {
|
|
"max_seconds_per_iteration": 4.0,
|
|
"failure_rate": {"max": 1},
|
|
"max_avg_duration": 3.0,
|
|
"outliers": {
|
|
"max": 1,
|
|
"min_iterations": 10,
|
|
"sigmas": 10
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|