Add test plan for OpenStack API performance metrics
The test plan defines set of metrics that can be collected for OpenStack API operations. Change-Id: I20899adff29578253be7a1d9440d937797a9f400
This commit is contained in:
Ilya Shakhat
parent
20b0943204
commit
20d28676ec
250
doc/source/test_plans/openstack_api_metrics/plan.rst
Normal file
250
doc/source/test_plans/openstack_api_metrics/plan.rst
Normal file
@ -0,0 +1,250 @@
|
||||
.. _openstack_api_performance_metrics_test_plan:
|
||||
|
||||
=================================
|
||||
OpenStack API Performance Metrics
|
||||
=================================
|
||||
|
||||
:status: **draft**
|
||||
:version: 1.0
|
||||
|
||||
:Abstract:
|
||||
|
||||
This test plan defines performance metrics for OpenStack API and the way
|
||||
to measure them.
|
||||
|
||||
:Conventions:
|
||||
- **Operation Duration** - how long does it take to perform a single
|
||||
operation.
|
||||
- **Operation Throughput** - how many operations can be done in one second in
|
||||
average.
|
||||
- **Concurrency** - how many parallel operations can be run when operation
|
||||
throughput reaches the maximum.
|
||||
- **Scale Impact** - comparison of operation metrics when number of objects
|
||||
is high versus low.
|
||||
|
||||
|
||||
Test Plan
|
||||
=========
|
||||
|
||||
This test plan defines set of performance metrics for OpenStack API. This
|
||||
metrics can be used to compare different cloud implementations and for
|
||||
performance tuning.
|
||||
|
||||
This test plan can be used to answer the following questions:
|
||||
* How long does it take to perform a particular operation? (*e.g. duration of
|
||||
Neutron net_create operation*)
|
||||
* How many concurrent operation can be run in parallel without degradation?
|
||||
(*e.g. can one do 10 Neutron net_create operation in parallel or better do
|
||||
them one-by-one*)
|
||||
* How many particular operations can OpenStack cloud process in a second?
|
||||
(*e.g. find out whether one can do 100 Neutron net_create ops per second or
|
||||
not*)
|
||||
* What is the impact of having many objects in the cloud? How the performance
|
||||
degrades? (*e.g. will the cloud be slower when there are thousands of
|
||||
objects and how slower will it be*)
|
||||
|
||||
Test Environment
|
||||
----------------
|
||||
|
||||
Preparation
|
||||
^^^^^^^^^^^
|
||||
|
||||
This test plan is executed against existing OpenStack cloud.
|
||||
|
||||
Measurements can be done with the tool that can:
|
||||
* report duration of single operations;
|
||||
* execute operations one-by-one and in a configurable number of concurrent
|
||||
threads.
|
||||
|
||||
Environment description
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The environment description includes hardware specification of servers,
|
||||
network parameters, operation system and OpenStack deployment characteristics.
|
||||
|
||||
Hardware
|
||||
~~~~~~~~
|
||||
|
||||
This section contains list of all types of hardware nodes.
|
||||
|
||||
+-----------+-------+----------------------------------------------------+
|
||||
| Parameter | Value | Comments |
|
||||
+-----------+-------+----------------------------------------------------+
|
||||
| model | | e.g. Supermicro X9SRD-F |
|
||||
+-----------+-------+----------------------------------------------------+
|
||||
| CPU | | e.g. 6 x Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz |
|
||||
+-----------+-------+----------------------------------------------------+
|
||||
| role | | e.g. compute or network |
|
||||
+-----------+-------+----------------------------------------------------+
|
||||
|
||||
Network
|
||||
~~~~~~~
|
||||
|
||||
This section contains list of interfaces and network parameters.
|
||||
For complicated cases this section may include topology diagram and switch
|
||||
parameters.
|
||||
|
||||
+------------------+-------+-------------------------+
|
||||
| Parameter | Value | Comments |
|
||||
+------------------+-------+-------------------------+
|
||||
| network role | | e.g. provider or public |
|
||||
+------------------+-------+-------------------------+
|
||||
| card model | | e.g. Intel |
|
||||
+------------------+-------+-------------------------+
|
||||
| driver | | e.g. ixgbe |
|
||||
+------------------+-------+-------------------------+
|
||||
| speed | | e.g. 10G or 1G |
|
||||
+------------------+-------+-------------------------+
|
||||
| MTU | | e.g. 9000 |
|
||||
+------------------+-------+-------------------------+
|
||||
| offloading modes | | e.g. default |
|
||||
+------------------+-------+-------------------------+
|
||||
|
||||
Software
|
||||
~~~~~~~~
|
||||
|
||||
This section describes installed software.
|
||||
|
||||
+-----------------+-------+---------------------------+
|
||||
| Parameter | Value | Comments |
|
||||
+-----------------+-------+---------------------------+
|
||||
| OS | | e.g. Ubuntu 14.04.3 |
|
||||
+-----------------+-------+---------------------------+
|
||||
| OpenStack | | e.g. Liberty |
|
||||
+-----------------+-------+---------------------------+
|
||||
| Hypervisor | | e.g. KVM |
|
||||
+-----------------+-------+---------------------------+
|
||||
| Neutron plugin | | e.g. ML2 + OVS |
|
||||
+-----------------+-------+---------------------------+
|
||||
| L2 segmentation | | e.g. VLAN or VxLAN or GRE |
|
||||
+-----------------+-------+---------------------------+
|
||||
| virtual routers | | e.g. legacy or HA or DVR |
|
||||
+-----------------+-------+---------------------------+
|
||||
|
||||
|
||||
Test Case: Operation Performance Measurements
|
||||
---------------------------------------------
|
||||
|
||||
Description
|
||||
^^^^^^^^^^^
|
||||
|
||||
The test case is performed by running a specific OpenStack operation. Every
|
||||
operation is executed several times to collect more reliable statistical data.
|
||||
|
||||
|
||||
Parameters
|
||||
^^^^^^^^^^
|
||||
|
||||
The only parameter is the operation being tested.
|
||||
|
||||
List of performance metrics
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
|
||||
*
|
||||
- Priority
|
||||
- Value
|
||||
- Measurement Unit
|
||||
- Description
|
||||
*
|
||||
- 1
|
||||
- Duration median
|
||||
- ms
|
||||
- Median of operation durations measured when operations are performed
|
||||
one-by-one in 1 thread
|
||||
*
|
||||
- 1
|
||||
- Duration 95% percentile
|
||||
- ms
|
||||
- 95% percentile of operation durations measured when operations are
|
||||
performed one-by-one in 1 thread
|
||||
*
|
||||
- 2
|
||||
- Duration 99% percentile
|
||||
- ms
|
||||
- 99% percentile of operation durations measured when operations are
|
||||
performed one-by-one in 1 thread
|
||||
*
|
||||
- 1
|
||||
- Concurrency
|
||||
- count
|
||||
- How many operations can be processed in parallel without significant
|
||||
degradation of duration
|
||||
*
|
||||
- 1
|
||||
- Throughput
|
||||
- operations per second
|
||||
- How many operations can be processed in one second
|
||||
*
|
||||
- 1
|
||||
- Scale impact
|
||||
- %
|
||||
- Performance degradation measured as ratio of operation duration when
|
||||
number of objects is 1k versus when number of objects is low.
|
||||
|
||||
|
||||
Tools
|
||||
=====
|
||||
|
||||
Rally
|
||||
-----
|
||||
|
||||
This test plan can be executed with `Rally`_ tool. Rally can report
|
||||
duration of individual operations and can be configured to perform operations
|
||||
in multiple parallel threads.
|
||||
|
||||
Rally scenario execution also involves creation/deletion of additional objects
|
||||
(like tenants, users) and cleaning of resources created by scenario. All this
|
||||
consumes extra time, so it makes sense to run measurements not one-by-one, but
|
||||
grouped by resource type. E.g. instead of having 4 separate scenarios for
|
||||
create, get, list and delete operations have 1 that calls these operations
|
||||
sequentially.
|
||||
|
||||
Scenarios
|
||||
^^^^^^^^^
|
||||
|
||||
To perform measurements we will need 2 types of scenarios:
|
||||
* **cyclic** - sequence of `create`, `get`, `list` and `delete`
|
||||
operations; total number of objects is not increased.
|
||||
* **accumulative** - sequence of `create`, `get` and `list` operations;
|
||||
total number of objects is increasing.
|
||||
|
||||
Duration metrics
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
Duration metrics are collected with help of cyclic scenario.
|
||||
|
||||
Actions:
|
||||
#. Set concurrency in 1 thread.
|
||||
#. Run scenario N times, where N is large enough to make a good sample.
|
||||
Collect list of operation durations.
|
||||
#. For every operation calculate median and percentiles.
|
||||
|
||||
|
||||
Concurrency and throughput metrics
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
These metrics are collected with help of cyclic scenarios.
|
||||
|
||||
Actions:
|
||||
#. Start with concurrency in 1 thread.
|
||||
#. Run scenario N times, where N is large enough to make a good sample.
|
||||
Collect list of operation durations.
|
||||
#. Calculate throughput (divide number of operations on total duration).
|
||||
|
||||
Scale impact metrics
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
These metrics are collected with help of accumulative scenarios.
|
||||
|
||||
Actions:
|
||||
#. Set concurrency in 1 thread.
|
||||
#. Run scenario until desired number of objects reached (e.g. 1 thousand).
|
||||
#. Calculate mean for first 50 objects and for last 50.
|
||||
#. Calculate the ratio between means.
|
||||
|
||||
.. references:
|
||||
|
||||
.. _Rally: http://rally.readthedocs.io/
|
||||
Loading…
Reference in New Issue
Block a user