openstack/performance-docs
Code Issues Proposed changes

Add Openstack control plane performance test plan

Browse Source 
Change-Id: Ib775f7086abf7aa5173b37746bbbad7868631182
This commit is contained in:
David Burnazyan 2016-07-29 00:11:46 +03:00
parent 83edf5219d
commit 79c6683809
6 changed files with 690 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

366
doc/source/test_plans/control_plane/plan.rst Normal file
View File

@ -0,0 +1,366 @@
.. _openstack_control_plane_performance_test_plan:
=============================================
OpenStack control plane performance test plan
=============================================
:status: **ready**
:version: 1.0
:Abstract:
This test plan aims to provide set of tests to identify Control Plane
performance against given OpenStack cloud using simple minimalistic set of
Rally tests.
Test Plan
=========
This test plan describes several test cases
that can cover almost all most important in terms of performance basic cloud
operations e.g. VMs creation, work with the security groups, authentication
and other operations.
Test Environment
----------------
Preparation
^^^^^^^^^^^
This test plan is performed either against existing OpenStack cloud with
pre-installed Rally framework or can be executed via Rally from very beginning
including deployment of the OpenStack cloud. As an option verification (Tempest
testing) can be run prior the benchmarking (scenarios to be described in this
document).
Environment description
^^^^^^^^^^^^^^^^^^^^^^^
The environment description includes hardware specification of servers,
network parameters, operation system and OpenStack deployment characteristics.
Hardware
~~~~~~~~
This section contains list of all types of hardware nodes.
+-----------+-------+----------------------------------------------------+
| Parameter | Value | Comments |
+-----------+-------+----------------------------------------------------+
| model | | e.g. Supermicro X9SRD-F |
+-----------+-------+----------------------------------------------------+
| CPU | | e.g. 6 x Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz |
+-----------+-------+----------------------------------------------------+
| role | | e.g. compute or network |
+-----------+-------+----------------------------------------------------+
Network
~~~~~~~
This section contains list of interfaces and network parameters.
For complicated cases this section may include topology diagram and switch
parameters.
+------------------+-------+-------------------------+
| Parameter | Value | Comments |
+------------------+-------+-------------------------+
| network role | | e.g. provider or public |
+------------------+-------+-------------------------+
| card model | | e.g. Intel |
+------------------+-------+-------------------------+
| driver | | e.g. ixgbe |
+------------------+-------+-------------------------+
| speed | | e.g. 10G or 1G |
+------------------+-------+-------------------------+
| MTU | | e.g. 9000 |
+------------------+-------+-------------------------+
| offloading modes | | e.g. default |
+------------------+-------+-------------------------+
Software
~~~~~~~~
This section describes installed software.
+-----------------+-------+---------------------------+
| Parameter | Value | Comments |
+-----------------+-------+---------------------------+
| OS | | e.g. Ubuntu 14.04.3 |
+-----------------+-------+---------------------------+
| OpenStack | | e.g. Mitaka |
+-----------------+-------+---------------------------+
| Hypervisor | | e.g. KVM |
+-----------------+-------+---------------------------+
| Neutron plugin | | e.g. ML2 + OVS |
+-----------------+-------+---------------------------+
| L2 segmentation | | e.g. VLAN / VxLAN / GRE |
+-----------------+-------+---------------------------+
| virtual routers | | e.g. HA / DVR |
+-----------------+-------+---------------------------+
Test tool
---------
**Rally** is a benchmarking tool that was designed specifically for OpenStack
API testing. To make this possible, **Rally** automates and unifies multi-node
OpenStack deployment, cloud verification, benchmarking & profiling. This is a
simple way to check cloud workability and performance of control plane
operations running on it.
Test Case 1: Boot, attach, migrate and delete server with security groups
-------------------------------------------------------------------------
Description
^^^^^^^^^^^
The most user-facing control plane operation is new virtual machine creation.
At the same time security groups management is very time consuming operation
in case of lots VMs attached to the same security group, therefore it's vital
to understand these operations performance. Special Rally plugin can be written
for this purpose.
Parameters
^^^^^^^^^^
+-------------------------+-----------------------------------------+
|Name | Description |
+=========================+=========================================+
|IMAGE | Image from which boot server |
+-------------------------+-----------------------------------------+
|FLAVOR | Flavor type from which boot server |
+-------------------------+-----------------------------------------+
|SEC_GROUP_COUNT | Count of security groups |
| | to be created in one iteration |
+-------------------------+-----------------------------------------+
|RULES_PER_SECURITY_GROUP | Count of rules to be added to |
| | each security group |
+-------------------------+-----------------------------------------+
|VOLUME_SIZE size | Size of volume to be created in Cinder |
+-------------------------+-----------------------------------------+
|CONCURRENCY | Amount of parallel executors |
+-------------------------+-----------------------------------------+
|ITERATIONS | Total amount of iterations processed by |
| | all executors |
+-------------------------+-----------------------------------------+
List of performance metrics
^^^^^^^^^^^^^^^^^^^^^^^^^^^
+----------+-------+-------------------+---------------------------+
| Priority | Value | Measurement Units | Description |
+==========+=======+===================+===========================+
| 1 | | sec | Time of atomic operations |
+----------+-------+-------------------+---------------------------+
Measuring performance values
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1. Successively create SEC_GROUP_COUNT security groups through Nova API.
Duration of this step represent time that control plane process
create_SEC_GROUP_COUNT_security_groups atomic operation.
2. Successively create RULES_PER_SECURITY_GROUP rules for security
groups through Nova API.
3. Create server with FLAVOR flavor from IMAGE image through Nova API
4. Create VOLUME_SIZE GB size volume through Cinder API.
5. Attach created volume to server.
6. Migrate server to pre-chosen host through Nova API.
7. List all security groups through Nova API.
8. Detach volume from server through Nova API.
9. Delete server through Nova API.
10. Delete volume through Cinder API.
11. Successively delete SEC_GROUP_COUNT security group through Nova API.
This 11 steps executed successively in CONCURRENCY parallel executors.
One cycle of this 11 steps is called as iteration.
ITERATIONS is a total amount of iterations which was processed by executors.
At the end of this test case you should calculate average, 90% percentile,
50% percentile, minimum and maximum for each step. You need to fill the
following tables with calculated values:
Cinder
------
+---------------+-------+--------+--------+-------+-------+
| Operation | Mean | 90%ile | 50%ile | Max | Min |
| | (sec) | (sec) | (sec) | (sec) | (sec) |
+===============+=======+========+========+=======+=======+
| create_volume | | | | | |
+---------------+-------+--------+--------+-------+-------+
| delete_volume | | | | | |
+---------------+-------+--------+--------+-------+-------+
Neutron
-------
+--------------------------+------+--------+--------+-------+-------+
| Operation | Mean | 90%ile | 50%ile | Max | Min |
| | (sec)| (sec) | (sec) | (sec) | (sec) |
+==========================+======+========+========+=======+=======+
| create_N_security_groups | | | | | |
+--------------------------+------+--------+--------+-------+-------+
| delete_N_security_groups | | | | | |
+--------------------------+------+--------+--------+-------+-------+
| create_M_rules | | | | | |
+--------------------------+------+--------+--------+-------+-------+
| delete_M_rules | | | | | |
+--------------------------+------+--------+--------+-------+-------+
.. note::
Change operation name to appropriate regarding SEC_GROUP_COUNT and
RULES_PER_SECURITY_GROUP values.
Nova
----
+---------------+------+--------+--------+-------+-------+
| Operation | Mean | 90%ile | 50%ile | Max | Min |
| | (sec)| (sec) | (sec) | (sec) | (sec) |
+===============+======+========+========+=======+=======+
| create_server | | | | | |
+---------------+------+--------+--------+-------+-------+
| attach_volume | | | | | |
+---------------+------+--------+--------+-------+-------+
| live_migrate | | | | | |
+---------------+------+--------+--------+-------+-------+
| detach_volume | | | | | |
+---------------+------+--------+--------+-------+-------+
| delete_server | | | | | |
+---------------+------+--------+--------+-------+-------+
Example of Rally scenario configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. literalinclude:: test_plans/boot_attach_migrate_delete_server_with_sec_groups.json
:language: bash
This scenario use custom Rally plugin: :download:`nova_performance.py <plugins/nova_performance.py>`
Test Case 2: Create and delete image
------------------------------------
Description
^^^^^^^^^^^
To cover Glance control plane operations simple crete and delete image scenario
can be used.
Parameters
^^^^^^^^^^
+-----------------+-----------------------------------------+
|Name | Description |
+=================+=========================================+
|IMAGE | Image to upload to glance |
+-----------------+-----------------------------------------+
|CONTAINER_FORMAT | Container format to create |
+-----------------+-----------------------------------------+
|DISK_FORMAT | Disk format to create |
+-----------------+-----------------------------------------+
|CONCURRENCY | Amount of parallel executors |
+-----------------+-----------------------------------------+
|ITERATIONS | Total amount of iterations processed by |
| | all executors |
+-----------------+-----------------------------------------+
List of performance metrics
^^^^^^^^^^^^^^^^^^^^^^^^^^^
+----------+-------+-------------------+---------------------------+
| Priority | Value | Measurement Units | Description |
+==========+=======+===================+===========================+
| 1 | | sec | Time of atomic operations |
+----------+-------+-------------------+---------------------------+
Measuring performance values
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1. Create image from IMAGE with CONTAINER_FORMAT container format and
DISK_FORMAT disk format through Glance API.
2. Delete image from Glance through Glance API.
This 2 steps executed successively in CONCURRENCY parallel executors.
One cycle of this 2 steps is called as iteration.
ITERATIONS is a total amount of iterations which was processed by executors.
At the end of this test case you should calculate average, 90% percentile,
50% percentile, minimum and maximum for each step. You need to fill the
following tables with calculated values:
Glance
------
+--------------+------+--------+--------+-------+-------+
| Operation | Mean | 90%ile | 50%ile | Max | Min |
| | (sec)| (sec) | (sec) | (sec) | (sec) |
+==============+======+========+========+=======+=======+
| create_image | | | | | |
+--------------+------+--------+--------+-------+-------+
| delete_image | | | | | |
+--------------+------+--------+--------+-------+-------+
Example of Rally scenario configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. literalinclude:: test_plans/create_and_delete_image.json
:language: bash
Test case 3: Keystone authentication
------------------------------------
Description
^^^^^^^^^^^
To cover Keystone control plane operations simple authenticate
scenario can be used.
Parameters
^^^^^^^^^^
+------------------+-----------------------------------------+
| Name | Description |
+==================+=========================================+
| RPS | Generated load |
+------------------+-----------------------------------------+
| ITERATIONS | Total amount of iterations processed by |
| | all executors |
+------------------+-----------------------------------------+
Measuring performance values
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1. Authenticate in Keystone through Keystone API.
This step executed in parallel on multiple executors to generate
RPS load.
Execution of this step is called as iteration.
ITERATIONS is a total amount of iterations which was processed by executors.
At the end of this test case you should calculate average, 90% percentile,
50% percentile, minimum and maximum for each step. You need to fill the
following tables with calculated values:
Keystone
--------
+--------------+------+--------+--------+-------+-------+
| Operation | Mean | 90%ile | 50%ile | Max | Min |
| | (sec)| (sec) | (sec) | (sec) | (sec) |
+==============+======+========+========+=======+=======+
| authenticate | | | | | |
+--------------+------+--------+--------+-------+-------+
List of performance metrics
^^^^^^^^^^^^^^^^^^^^^^^^^^^
+----------+-------+-------------------+---------------------------+
| Priority | Value | Measurement Units | Description |
+==========+=======+===================+===========================+
| 1 | | sec | Time of atomic operations |
+----------+-------+-------------------+---------------------------+
Example of Rally scenario configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. literalinclude:: test_plans/keystone_authenticate.json
:language: bash

213
doc/source/test_plans/control_plane/plugins/nova_performance.py Normal file
View File

@ -0,0 +1,213 @@
# Copyright 2016: Mirantis Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from rally import consts
from rally.plugins.openstack import scenario
from rally.plugins.openstack.scenarios.cinder import utils as cinder_utils
from rally.plugins.openstack.scenarios.nova import utils
from rally.task import types
from rally.task import validation
class NovaPerformancePlugin(utils.NovaScenario, cinder_utils.CinderScenario):
"""boot_attach_live_migrate_and_delete_server_with_secgroups"""
@types.convert(image={"type": "glance_image"},
flavor={"type": "nova_flavor"})
@validation.image_valid_on_flavor("flavor", "image")
@validation.required_parameters("security_group_count",
"rules_per_security_group")
@validation.required_contexts("network")
@validation.required_services(consts.Service.NOVA)
@validation.required_openstack(users=True)
@scenario.configure(context={"cleanup": ["cinder", "nova"]})
def boot_attach_live_migrate_and_delete_server_with_secgroups(
self, image, flavor,
volume_size,
security_group_count,
rules_per_security_group,
do_delete=True,
do_migration=False,
block_migration=False,
disk_over_commit=False,
min_sleep=0, max_sleep=0,
boot_server_kwargs=None,
create_volume_kwargs=None
):
if boot_server_kwargs is None:
boot_server_kwargs = {}
if create_volume_kwargs is None:
create_volume_kwargs = {}
security_groups = self._create_security_groups(
security_group_count)
self._create_rules_for_security_group(security_groups,
rules_per_security_group)
secgroups_names = [sg.name for sg in security_groups]
server = self._boot_server(image, flavor,
security_groups=secgroups_names,
**boot_server_kwargs)
volume = self._create_volume(volume_size, **create_volume_kwargs)
self._attach_volume(server, volume)
if do_migration:
new_host = self._find_host_to_migrate(server)
self._live_migrate(server, new_host,
block_migration, disk_over_commit)
self.sleep_between(min_sleep, max_sleep)
self._list_security_groups()
if do_delete:
self._detach_volume(server, volume)
self._delete_server(server)
self._delete_volume(volume)
self._delete_security_groups(security_groups)
"""boot_attach_and_delete_server_with_secgroups"""
@types.convert(image={"type": "glance_image"},
flavor={"type": "nova_flavor"})
@validation.image_valid_on_flavor("flavor", "image")
@validation.required_parameters("security_group_count",
"rules_per_security_group")
@validation.required_contexts("network")
@validation.required_services(consts.Service.NOVA)
@validation.required_openstack(users=True)
@scenario.configure(context={"cleanup": ["cinder", "nova"]})
def boot_attach_and_delete_server_with_secgroups(
self, image, flavor,
volume_size,
security_group_count,
rules_per_security_group,
do_delete=True,
boot_server_kwargs=None,
create_volume_kwargs=None
):
if boot_server_kwargs is None:
boot_server_kwargs = {}
if create_volume_kwargs is None:
create_volume_kwargs = {}
security_groups = self._create_security_groups(
security_group_count)
self._create_rules_for_security_group(security_groups,
rules_per_security_group)
secgroups_names = [sg.name for sg in security_groups]
server = self._boot_server(image, flavor,
security_groups=secgroups_names,
**boot_server_kwargs)
volume = self._create_volume(volume_size, **create_volume_kwargs)
self._attach_volume(server, volume)
self._list_security_groups()
if do_delete:
self._detach_volume(server, volume)
self._delete_server(server)
self._delete_volume(volume)
self._delete_security_groups(security_groups)
"""boot_live_migrate_and_delete_server_with_secgroups"""
@types.convert(image={"type": "glance_image"},
flavor={"type": "nova_flavor"})
@validation.image_valid_on_flavor("flavor", "image")
@validation.required_parameters("security_group_count",
"rules_per_security_group")
@validation.required_contexts("network")
@validation.required_services(consts.Service.NOVA)
@validation.required_openstack(users=True)
@scenario.configure(context={"cleanup": ["nova"]})
def boot_live_migrate_and_delete_server_with_secgroups(
self, image, flavor,
security_group_count,
rules_per_security_group,
do_delete=True,
do_migration=False,
block_migration=False,
disk_over_commit=False,
min_sleep=0, max_sleep=0,
boot_server_kwargs=None
):
if boot_server_kwargs is None:
boot_server_kwargs = {}
security_groups = self._create_security_groups(
security_group_count)
self._create_rules_for_security_group(security_groups,
rules_per_security_group)
secgroups_names = [sg.name for sg in security_groups]
server = self._boot_server(image, flavor,
security_groups=secgroups_names,
**boot_server_kwargs)
if do_migration:
new_host = self._find_host_to_migrate(server)
self._live_migrate(server, new_host,
block_migration, disk_over_commit)
self.sleep_between(min_sleep, max_sleep)
self._list_security_groups()
if do_delete:
self._delete_server(server)
self._delete_security_groups(security_groups)
"""boot_attach_live_migrate_and_delete_server"""
@types.convert(image={"type": "glance_image"},
flavor={"type": "nova_flavor"})
@validation.image_valid_on_flavor("flavor", "image")
@validation.required_services(consts.Service.NOVA)
@validation.required_openstack(users=True)
@scenario.configure(context={"cleanup": ["cinder", "nova"]})
def boot_attach_live_migrate_and_delete_server(
self, image, flavor,
volume_size,
do_delete=True,
do_migration=False,
block_migration=False,
disk_over_commit=False,
min_sleep=0, max_sleep=0,
boot_server_kwargs=None,
create_volume_kwargs=None
):
if boot_server_kwargs is None:
boot_server_kwargs = {}
if create_volume_kwargs is None:
create_volume_kwargs = {}
server = self._boot_server(image, flavor,
**boot_server_kwargs)
volume = self._create_volume(volume_size, **create_volume_kwargs)
self._attach_volume(server, volume)
if do_migration:
new_host = self._find_host_to_migrate(server)
self._live_migrate(server, new_host,
block_migration, disk_over_commit)
self.sleep_between(min_sleep, max_sleep)
if do_delete:
self._detach_volume(server, volume)
self._delete_server(server)
self._delete_volume(volume)

63
doc/source/test_plans/control_plane/test_plans/boot_attach_migrate_delete_server_with_sec_groups.json Normal file
View File

@ -0,0 +1,63 @@
{% set flavor_name = flavor_name or "m1.tiny" %}
{% set image_name = image_name or "^(cirros.*uec|TestVM)$" %}
{
"NovaPerformancePlugin.boot_attach_and_delete_server_with_secgroups": [
{
"args": {
"flavor": {
"name": "{{flavor_name}}"
},
"image": {
"name": "{{image_name}}"
},
"security_group_count": 2,
"rules_per_security_group": 10,
"boot_server_kwargs": { "auto_assign_nic" : true },
"create_volume_kwargs": {},
"volume_size": 1,
"do_delete": true
},
"runner": {
"type": "constant",
"times": {{ 2 * compute }},
"concurrency": {{concurrency}}
},
"context": {
"users": {
"tenants": 1,
"users_per_tenant": 10
},
"quotas": {
"neutron": {
"network": -1,
"security_group": -1,
"security_group_rule": -1,
"port": -1,
"subnet": -1,
"router": -1
},
"nova": {
"instances": -1,
"cores": -1,
"ram": -1,
"floating_ips": -1,
"security_groups": -1,
"security_group_rules": -1
},
"cinder": {
"volumes": -1,
"gigabytes": -1,
"snapshots": -1
}
},
"network": {
"start_cidr": "{{ "100.1.0.0/25" if gre_enabled else "1.0.0.0/25" }}",
"networks_per_tenant": 10
}
},
"sla": {
"failure_rate": { "max": 0 }
}
}
]
}

26
doc/source/test_plans/control_plane/test_plans/create_and_delete_image.json Normal file
View File

@ -0,0 +1,26 @@
{% set http_server_with_glance_images = http_server_with_glance_images or "127.0.0.1" %}
{
"GlanceImages.create_and_delete_image": [
{
"args": {
"image_location": "http://{{http_server_with_glance_images}}/images/glance_1g_image.raw",
"container_format": "bare",
"disk_format": "raw"
},
"runner": {
"type": "constant",
"times": 400,
"concurrency": 50
},
"context": {
"users": {
"tenants": 2,
"users_per_tenant": 3
}
},
"sla": {
"failure_rate": { "max": 0 }
}
}
]
}

20
doc/source/test_plans/control_plane/test_plans/keystone_authenticate.json Normal file
View File

@ -0,0 +1,20 @@
{
"Authenticate.keystone": [
{
"runner": {
"type": "rps",
"times": 120000,
"rps": 180
},
"context": {
"users": {
"tenants": 10,
"users_per_tenant": 10
}
},
"sla": {
"failure_rate": { "max": 0 }
}
}
]
}

2
doc/source/test_plans/index.rst
View File

@ -22,3 +22,5 @@ Test Plans
openstack_load/plan openstack_load/plan
1000_nodes/plan 1000_nodes/plan
reliability/plan reliability/plan
control_plane/plan