Define performance test suite framework
This specification will describe the scope of the performance framework, use cases and how it can scale by new tests added by the community. Change-Id: I844e1619772789edf571a598ce0b6e4561e4e94b Signed-off-by: VictorRodriguez <victor.rodriguez.bahena@intel.com>
This commit is contained in:
VictorRodriguez
parent
4cafd9dd1f
commit
7c1936321d
268
specs/2019.03/approved/testing-2006406-performance-framework.rst
Normal file
268
specs/2019.03/approved/testing-2006406-performance-framework.rst
Normal file
@ -0,0 +1,268 @@
|
||||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License. http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
..
|
||||
|
||||
============================================
|
||||
StarlingX: Performance measurement framework
|
||||
============================================
|
||||
|
||||
Storyboard: https://storyboard.openstack.org/#!/story/2006406
|
||||
|
||||
The measurement of performance metrics in edge cloud systems is a key factor in
|
||||
quality assurance. Having a common understanding and a practical framework to
|
||||
run the performance tests are important and necessary for users and developers
|
||||
in the community. This specification will describe the scope of the performance
|
||||
framework, use cases and how it can scale by new tests developed by the
|
||||
community or imported from existing test performance frameworks.
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
The deployment of StarlingX could happen in a wide variability of hardware and
|
||||
network environments, these differences could result in a difference in
|
||||
performance metrics. As of today, the community does not have a consolidated
|
||||
performance framework to measure the performance metrics with specific and
|
||||
well-defined test cases.
|
||||
|
||||
Use Cases
|
||||
---------
|
||||
|
||||
Developers who made some fairly significant changes in codebase want to ensure
|
||||
no performance regression brought by the changes. Developers who did some
|
||||
performance improvements need to measure the performance result with a
|
||||
consolidating framework valid across the community. Some of the community
|
||||
members want to measure out the performance items on StarlingX and promote
|
||||
StarlingX advantages in terms of performance which are relevant to an edge
|
||||
solution. Potential users want to evaluate StarlingX as one of their Edge
|
||||
solution candidates, so offering such a performance framework will be a
|
||||
positive factor of contributing to the evaluation process.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
The proposed change is to create a set of scripts and documentation under the
|
||||
starlingx testing repository (https://opendev.org/starlingx/test) that anyone
|
||||
can use to measure the metric they need under their hardware and software
|
||||
environment.
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
The community can use existing performance test frameworks, Some of those are:
|
||||
|
||||
* Rally:
|
||||
|
||||
OpenStack project, dedicated to the performance analysis and benchmarking
|
||||
system of individual OpenStack components
|
||||
|
||||
* Kubernetes perf-tests:
|
||||
|
||||
An open source project dedicated to Kubernetes-related performance test tools
|
||||
|
||||
* Yardstick by OPNFV:
|
||||
|
||||
The Yardstick concept decomposes typical virtual network function work-load
|
||||
performance metrics into several characteristics/performance vectors, which
|
||||
each of them can be represented by distinct test-cases.
|
||||
|
||||
The disadvantage of all these frameworks is that they coexist in separate
|
||||
projects. Users need to go and find the test cases anywhere on the internet, is
|
||||
not a centralized and scalable solution. The StarlingX Performance measurement
|
||||
framework proposed will allow the importing of existing test cases to be
|
||||
re-used in a centralized system easy to use for the community.
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
None
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
None
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
None
|
||||
|
||||
Other end-user impact
|
||||
---------------------
|
||||
|
||||
End users will interact with this framework by command line on the controller
|
||||
system by the performance-test-runner.py script, which is the entry point to
|
||||
select and configure the test case. An example of to launch a test case might
|
||||
be:
|
||||
|
||||
::
|
||||
./performance-test-runner.py --test failed_vm_detection --compute=0
|
||||
|
||||
This will generate results of multiple runs in CSV format easy to post-process.
|
||||
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
Running this performance framework or test cases, there shouldn’t be a
|
||||
performance impact or obvious overhead on the system.
|
||||
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
None, there will be no impact on how to deploy and configure StarlingX
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
There will be a good impact on the developers. Since they will be able to have
|
||||
a consolidated and well aligned framework to measure the performance impact of
|
||||
their code/configuration changes.
|
||||
|
||||
Developers can't improve something if they don’t know it needs improvement.
|
||||
That’s where this performance testing framework comes in. It gives the
|
||||
developers tools to detect how fast, stable, and scalable their StarlingX, is
|
||||
so they can decide if it needs to improve or not.
|
||||
|
||||
Upgrade impact
|
||||
--------------
|
||||
None
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
The test framework will have the next components on the implementation
|
||||
|
||||
* Common scripts:
|
||||
|
||||
This directory will contain scripts to capture key timestamps or collect other
|
||||
key data along with some use cases so that developers can measure out the time
|
||||
(latency) or other performance data (such as throughput, system CPU utilization
|
||||
and so on). Any common script that any other test case could use should be in
|
||||
this directory.
|
||||
|
||||
Despite the existence of a directory for common scripts, each one of the test
|
||||
cases is responsible for its own metrics. One example is the time measurement
|
||||
of test cases that run in more than one node. For these kinds of test cases
|
||||
will be necessary to use synchronization protocols like the Network Time
|
||||
Protocol (NTP). The Network Time Protocol (NTP) is a networking protocol for
|
||||
clock synchronization between computer systems. NTP is intended to synchronize
|
||||
all participating computers to within a few milliseconds of Coordinated
|
||||
Universal Time (UTC).In some other cases, it might not be necessary to have a
|
||||
protocol for clock synchronization.
|
||||
|
||||
Most of the details for wether to re-use a common script form the common
|
||||
directory or handle the measurement by the test itself will be left to the test
|
||||
implementation
|
||||
|
||||
* Performance-test-runner.py:
|
||||
|
||||
The main command-line interface to execute the test cases on the system under
|
||||
test
|
||||
|
||||
* Test cases directory:
|
||||
|
||||
This directory can contain wrapper scripts to existing upstream performance
|
||||
test cases as well as new test cases specific for Starlting X.
|
||||
|
||||
A view of the directory layout will look like:
|
||||
|
||||
::
|
||||
performance/
|
||||
├── common
|
||||
│ ├── latency_metrics.py
|
||||
│ ├── network_generator.py
|
||||
│ ├── ntp_metrics.py
|
||||
│ └── statistics.py
|
||||
└── tests_cases
|
||||
│ ├── failed_compute_detection.py
|
||||
│ ├── failed_control_detection.py
|
||||
│ ├── failed_network_detection.py
|
||||
│ ├── failed_vm_detection.py
|
||||
│ ├── neutron_test_case_1.py
|
||||
│ ├── opnfv_test_case_1.py
|
||||
│ ├── opnfv_test_case_2.py
|
||||
│ ├── rally_test_case_1.py
|
||||
│ ├── rally_test_case_2.py
|
||||
│
|
||||
└── performance-test-runner.py
|
||||
|
||||
The goal is that anyone on the StarlingX community can either define
|
||||
performance tests cases or re use existing on other projects and create
|
||||
scripts to measure in a scalable and repeatable framework.
|
||||
|
||||
In the future, it might be possible to evaluate how to connect some basic
|
||||
performance tests cases to zuul to detect regressions on commits to be merged.
|
||||
This will require to set up a robust infrastructure to run the sanity
|
||||
performance test cases for each commit to be merged.
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Wang Hai Tao
|
||||
Elio Martinez
|
||||
Juan Carlos Alonzo
|
||||
Victor Rodriguez
|
||||
|
||||
|
||||
Repos Impacted
|
||||
--------------
|
||||
|
||||
https://opendev.org/starlingx/test
|
||||
|
||||
Work Items
|
||||
----------
|
||||
* Develop automated scripts for core test cases, such as:
|
||||
* Detection of failed VM
|
||||
* Detection failed compute node
|
||||
* Controller/Node fail detect/recovery
|
||||
* Detection of network link fail
|
||||
* DPDK Live Migrate Latency
|
||||
* Avg/Max host/guest Latency (Cyclictest)
|
||||
* Swact Time
|
||||
|
||||
* Develop performance-test-runner.py that call each test case script
|
||||
* Integrate performance-test-runner.py with pytest framework
|
||||
* Implement call clone of opnfv test cases
|
||||
* Implement execution of opnfv test case
|
||||
* Implement monitoring pipeline to automatically detect changes in upstream
|
||||
* Automate weekly mail with list of upstream performance test cases that change
|
||||
* Implement automatic test cases documentation based on pydoc
|
||||
* Document framework on StarlingX wiki
|
||||
* Send regular status to ML and demos on community call for feedback
|
||||
|
||||
Dependencies
|
||||
============
|
||||
None
|
||||
|
||||
Testing
|
||||
=======
|
||||
None
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
The performance test cases will be documented with pydoc
|
||||
(https://docs.python.org/3.0/library/pydoc.html). The pydoc module
|
||||
automatically generates documentation from Python modules. The documentation
|
||||
can be presented as pages of text on the console, served to a Web browser, or
|
||||
saved to HTML files.
|
||||
|
||||
The goal is to generate automatically the documentation for the end-user
|
||||
testing guide. This methodology will catch when the community adds or modify a
|
||||
new performance test case. At the same time if a standard config option changes
|
||||
or is deprecated the test framework documentation will be updated to reflect
|
||||
the change.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
|
||||
History
|
||||
=======
|
||||
|
||||
.. list-table:: Revisions
|
||||
:header-rows: 1
|
||||
|
||||
* - Release Name
|
||||
- Description
|
||||
* - Stein
|
||||
- Introduced
|
||||
|
||||
Loading…
Reference in New Issue
Block a user