diff --git a/doc/source/test_plans/control_plane/plan.rst b/doc/source/test_plans/control_plane/plan.rst new file mode 100644 index 0000000..e3b598f --- /dev/null +++ b/doc/source/test_plans/control_plane/plan.rst @@ -0,0 +1,366 @@ +.. _openstack_control_plane_performance_test_plan: + +============================================= +OpenStack control plane performance test plan +============================================= + +:status: **ready** +:version: 1.0 + +:Abstract: + + This test plan aims to provide set of tests to identify Control Plane + performance against given OpenStack cloud using simple minimalistic set of + Rally tests. + +Test Plan +========= +This test plan describes several test cases +that can cover almost all most important in terms of performance basic cloud +operations e.g. VMs creation, work with the security groups, authentication +and other operations. + +Test Environment +---------------- + +Preparation +^^^^^^^^^^^ + +This test plan is performed either against existing OpenStack cloud with +pre-installed Rally framework or can be executed via Rally from very beginning +including deployment of the OpenStack cloud. As an option verification (Tempest +testing) can be run prior the benchmarking (scenarios to be described in this +document). + +Environment description +^^^^^^^^^^^^^^^^^^^^^^^ + +The environment description includes hardware specification of servers, +network parameters, operation system and OpenStack deployment characteristics. + +Hardware +~~~~~~~~ + +This section contains list of all types of hardware nodes. + ++-----------+-------+----------------------------------------------------+ +| Parameter | Value | Comments | ++-----------+-------+----------------------------------------------------+ +| model | | e.g. Supermicro X9SRD-F | ++-----------+-------+----------------------------------------------------+ +| CPU | | e.g. 6 x Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz | ++-----------+-------+----------------------------------------------------+ +| role | | e.g. compute or network | ++-----------+-------+----------------------------------------------------+ + +Network +~~~~~~~ + +This section contains list of interfaces and network parameters. +For complicated cases this section may include topology diagram and switch +parameters. + ++------------------+-------+-------------------------+ +| Parameter | Value | Comments | ++------------------+-------+-------------------------+ +| network role | | e.g. provider or public | ++------------------+-------+-------------------------+ +| card model | | e.g. Intel | ++------------------+-------+-------------------------+ +| driver | | e.g. ixgbe | ++------------------+-------+-------------------------+ +| speed | | e.g. 10G or 1G | ++------------------+-------+-------------------------+ +| MTU | | e.g. 9000 | ++------------------+-------+-------------------------+ +| offloading modes | | e.g. default | ++------------------+-------+-------------------------+ + +Software +~~~~~~~~ + +This section describes installed software. + ++-----------------+-------+---------------------------+ +| Parameter | Value | Comments | ++-----------------+-------+---------------------------+ +| OS | | e.g. Ubuntu 14.04.3 | ++-----------------+-------+---------------------------+ +| OpenStack | | e.g. Mitaka | ++-----------------+-------+---------------------------+ +| Hypervisor | | e.g. KVM | ++-----------------+-------+---------------------------+ +| Neutron plugin | | e.g. ML2 + OVS | ++-----------------+-------+---------------------------+ +| L2 segmentation | | e.g. VLAN / VxLAN / GRE | ++-----------------+-------+---------------------------+ +| virtual routers | | e.g. HA / DVR | ++-----------------+-------+---------------------------+ + +Test tool +--------- + +**Rally** is a benchmarking tool that was designed specifically for OpenStack +API testing. To make this possible, **Rally** automates and unifies multi-node +OpenStack deployment, cloud verification, benchmarking & profiling. This is a +simple way to check cloud workability and performance of control plane +operations running on it. + +Test Case 1: Boot, attach, migrate and delete server with security groups +------------------------------------------------------------------------- + +Description +^^^^^^^^^^^ + +The most user-facing control plane operation is new virtual machine creation. +At the same time security groups management is very time consuming operation +in case of lots VMs attached to the same security group, therefore it's vital +to understand these operations performance. Special Rally plugin can be written +for this purpose. + +Parameters +^^^^^^^^^^ + ++-------------------------+-----------------------------------------+ +|Name | Description | ++=========================+=========================================+ +|IMAGE | Image from which boot server | ++-------------------------+-----------------------------------------+ +|FLAVOR | Flavor type from which boot server | ++-------------------------+-----------------------------------------+ +|SEC_GROUP_COUNT | Count of security groups | +| | to be created in one iteration | ++-------------------------+-----------------------------------------+ +|RULES_PER_SECURITY_GROUP | Count of rules to be added to | +| | each security group | ++-------------------------+-----------------------------------------+ +|VOLUME_SIZE size | Size of volume to be created in Cinder | ++-------------------------+-----------------------------------------+ +|CONCURRENCY | Amount of parallel executors | ++-------------------------+-----------------------------------------+ +|ITERATIONS | Total amount of iterations processed by | +| | all executors | ++-------------------------+-----------------------------------------+ + + +List of performance metrics +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + ++----------+-------+-------------------+---------------------------+ +| Priority | Value | Measurement Units | Description | ++==========+=======+===================+===========================+ +| 1 | | sec | Time of atomic operations | ++----------+-------+-------------------+---------------------------+ + +Measuring performance values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. Successively create SEC_GROUP_COUNT security groups through Nova API. + Duration of this step represent time that control plane process + create_SEC_GROUP_COUNT_security_groups atomic operation. +2. Successively create RULES_PER_SECURITY_GROUP rules for security + groups through Nova API. +3. Create server with FLAVOR flavor from IMAGE image through Nova API +4. Create VOLUME_SIZE GB size volume through Cinder API. +5. Attach created volume to server. +6. Migrate server to pre-chosen host through Nova API. +7. List all security groups through Nova API. +8. Detach volume from server through Nova API. +9. Delete server through Nova API. +10. Delete volume through Cinder API. +11. Successively delete SEC_GROUP_COUNT security group through Nova API. + +This 11 steps executed successively in CONCURRENCY parallel executors. +One cycle of this 11 steps is called as iteration. +ITERATIONS is a total amount of iterations which was processed by executors. + +At the end of this test case you should calculate average, 90% percentile, +50% percentile, minimum and maximum for each step. You need to fill the +following tables with calculated values: + +Cinder +------ + ++---------------+-------+--------+--------+-------+-------+ +| Operation | Mean | 90%ile | 50%ile | Max | Min | +| | (sec) | (sec) | (sec) | (sec) | (sec) | ++===============+=======+========+========+=======+=======+ +| create_volume | | | | | | ++---------------+-------+--------+--------+-------+-------+ +| delete_volume | | | | | | ++---------------+-------+--------+--------+-------+-------+ + +Neutron +------- + ++--------------------------+------+--------+--------+-------+-------+ +| Operation | Mean | 90%ile | 50%ile | Max | Min | +| | (sec)| (sec) | (sec) | (sec) | (sec) | ++==========================+======+========+========+=======+=======+ +| create_N_security_groups | | | | | | ++--------------------------+------+--------+--------+-------+-------+ +| delete_N_security_groups | | | | | | ++--------------------------+------+--------+--------+-------+-------+ +| create_M_rules | | | | | | ++--------------------------+------+--------+--------+-------+-------+ +| delete_M_rules | | | | | | ++--------------------------+------+--------+--------+-------+-------+ + +.. note:: + Change operation name to appropriate regarding SEC_GROUP_COUNT and + RULES_PER_SECURITY_GROUP values. + +Nova +---- + ++---------------+------+--------+--------+-------+-------+ +| Operation | Mean | 90%ile | 50%ile | Max | Min | +| | (sec)| (sec) | (sec) | (sec) | (sec) | ++===============+======+========+========+=======+=======+ +| create_server | | | | | | ++---------------+------+--------+--------+-------+-------+ +| attach_volume | | | | | | ++---------------+------+--------+--------+-------+-------+ +| live_migrate | | | | | | ++---------------+------+--------+--------+-------+-------+ +| detach_volume | | | | | | ++---------------+------+--------+--------+-------+-------+ +| delete_server | | | | | | ++---------------+------+--------+--------+-------+-------+ + +Example of Rally scenario configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. literalinclude:: test_plans/boot_attach_migrate_delete_server_with_sec_groups.json + :language: bash + +This scenario use custom Rally plugin: :download:`nova_performance.py ` + +Test Case 2: Create and delete image +------------------------------------ + +Description +^^^^^^^^^^^ + +To cover Glance control plane operations simple crete and delete image scenario +can be used. + +Parameters +^^^^^^^^^^ + ++-----------------+-----------------------------------------+ +|Name | Description | ++=================+=========================================+ +|IMAGE | Image to upload to glance | ++-----------------+-----------------------------------------+ +|CONTAINER_FORMAT | Container format to create | ++-----------------+-----------------------------------------+ +|DISK_FORMAT | Disk format to create | ++-----------------+-----------------------------------------+ +|CONCURRENCY | Amount of parallel executors | ++-----------------+-----------------------------------------+ +|ITERATIONS | Total amount of iterations processed by | +| | all executors | ++-----------------+-----------------------------------------+ + +List of performance metrics +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + ++----------+-------+-------------------+---------------------------+ +| Priority | Value | Measurement Units | Description | ++==========+=======+===================+===========================+ +| 1 | | sec | Time of atomic operations | ++----------+-------+-------------------+---------------------------+ + +Measuring performance values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. Create image from IMAGE with CONTAINER_FORMAT container format and + DISK_FORMAT disk format through Glance API. +2. Delete image from Glance through Glance API. + +This 2 steps executed successively in CONCURRENCY parallel executors. +One cycle of this 2 steps is called as iteration. +ITERATIONS is a total amount of iterations which was processed by executors. + +At the end of this test case you should calculate average, 90% percentile, +50% percentile, minimum and maximum for each step. You need to fill the +following tables with calculated values: + +Glance +------ + ++--------------+------+--------+--------+-------+-------+ +| Operation | Mean | 90%ile | 50%ile | Max | Min | +| | (sec)| (sec) | (sec) | (sec) | (sec) | ++==============+======+========+========+=======+=======+ +| create_image | | | | | | ++--------------+------+--------+--------+-------+-------+ +| delete_image | | | | | | ++--------------+------+--------+--------+-------+-------+ + +Example of Rally scenario configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. literalinclude:: test_plans/create_and_delete_image.json + :language: bash + +Test case 3: Keystone authentication +------------------------------------ + +Description +^^^^^^^^^^^ + +To cover Keystone control plane operations simple authenticate +scenario can be used. + +Parameters +^^^^^^^^^^ + ++------------------+-----------------------------------------+ +| Name | Description | ++==================+=========================================+ +| RPS | Generated load | ++------------------+-----------------------------------------+ +| ITERATIONS | Total amount of iterations processed by | +| | all executors | ++------------------+-----------------------------------------+ + +Measuring performance values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. Authenticate in Keystone through Keystone API. + +This step executed in parallel on multiple executors to generate +RPS load. +Execution of this step is called as iteration. +ITERATIONS is a total amount of iterations which was processed by executors. + +At the end of this test case you should calculate average, 90% percentile, +50% percentile, minimum and maximum for each step. You need to fill the +following tables with calculated values: + +Keystone +-------- + ++--------------+------+--------+--------+-------+-------+ +| Operation | Mean | 90%ile | 50%ile | Max | Min | +| | (sec)| (sec) | (sec) | (sec) | (sec) | ++==============+======+========+========+=======+=======+ +| authenticate | | | | | | ++--------------+------+--------+--------+-------+-------+ + +List of performance metrics +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + ++----------+-------+-------------------+---------------------------+ +| Priority | Value | Measurement Units | Description | ++==========+=======+===================+===========================+ +| 1 | | sec | Time of atomic operations | ++----------+-------+-------------------+---------------------------+ + +Example of Rally scenario configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. literalinclude:: test_plans/keystone_authenticate.json + :language: bash diff --git a/doc/source/test_plans/control_plane/plugins/nova_performance.py b/doc/source/test_plans/control_plane/plugins/nova_performance.py new file mode 100644 index 0000000..64c6912 --- /dev/null +++ b/doc/source/test_plans/control_plane/plugins/nova_performance.py @@ -0,0 +1,213 @@ +# Copyright 2016: Mirantis Inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +from rally import consts +from rally.plugins.openstack import scenario +from rally.plugins.openstack.scenarios.cinder import utils as cinder_utils +from rally.plugins.openstack.scenarios.nova import utils +from rally.task import types +from rally.task import validation + + +class NovaPerformancePlugin(utils.NovaScenario, cinder_utils.CinderScenario): + """boot_attach_live_migrate_and_delete_server_with_secgroups""" + @types.convert(image={"type": "glance_image"}, + flavor={"type": "nova_flavor"}) + @validation.image_valid_on_flavor("flavor", "image") + @validation.required_parameters("security_group_count", + "rules_per_security_group") + @validation.required_contexts("network") + @validation.required_services(consts.Service.NOVA) + @validation.required_openstack(users=True) + @scenario.configure(context={"cleanup": ["cinder", "nova"]}) + def boot_attach_live_migrate_and_delete_server_with_secgroups( + self, image, flavor, + volume_size, + security_group_count, + rules_per_security_group, + do_delete=True, + do_migration=False, + block_migration=False, + disk_over_commit=False, + min_sleep=0, max_sleep=0, + boot_server_kwargs=None, + create_volume_kwargs=None + ): + + if boot_server_kwargs is None: + boot_server_kwargs = {} + if create_volume_kwargs is None: + create_volume_kwargs = {} + + security_groups = self._create_security_groups( + security_group_count) + self._create_rules_for_security_group(security_groups, + rules_per_security_group) + + secgroups_names = [sg.name for sg in security_groups] + server = self._boot_server(image, flavor, + security_groups=secgroups_names, + **boot_server_kwargs) + + volume = self._create_volume(volume_size, **create_volume_kwargs) + self._attach_volume(server, volume) + + if do_migration: + new_host = self._find_host_to_migrate(server) + self._live_migrate(server, new_host, + block_migration, disk_over_commit) + self.sleep_between(min_sleep, max_sleep) + + self._list_security_groups() + + if do_delete: + self._detach_volume(server, volume) + self._delete_server(server) + self._delete_volume(volume) + self._delete_security_groups(security_groups) + + """boot_attach_and_delete_server_with_secgroups""" + @types.convert(image={"type": "glance_image"}, + flavor={"type": "nova_flavor"}) + @validation.image_valid_on_flavor("flavor", "image") + @validation.required_parameters("security_group_count", + "rules_per_security_group") + @validation.required_contexts("network") + @validation.required_services(consts.Service.NOVA) + @validation.required_openstack(users=True) + @scenario.configure(context={"cleanup": ["cinder", "nova"]}) + def boot_attach_and_delete_server_with_secgroups( + self, image, flavor, + volume_size, + security_group_count, + rules_per_security_group, + do_delete=True, + boot_server_kwargs=None, + create_volume_kwargs=None + ): + + if boot_server_kwargs is None: + boot_server_kwargs = {} + if create_volume_kwargs is None: + create_volume_kwargs = {} + + security_groups = self._create_security_groups( + security_group_count) + self._create_rules_for_security_group(security_groups, + rules_per_security_group) + + secgroups_names = [sg.name for sg in security_groups] + server = self._boot_server(image, flavor, + security_groups=secgroups_names, + **boot_server_kwargs) + + volume = self._create_volume(volume_size, **create_volume_kwargs) + self._attach_volume(server, volume) + + self._list_security_groups() + + if do_delete: + self._detach_volume(server, volume) + self._delete_server(server) + self._delete_volume(volume) + self._delete_security_groups(security_groups) + + """boot_live_migrate_and_delete_server_with_secgroups""" + @types.convert(image={"type": "glance_image"}, + flavor={"type": "nova_flavor"}) + @validation.image_valid_on_flavor("flavor", "image") + @validation.required_parameters("security_group_count", + "rules_per_security_group") + @validation.required_contexts("network") + @validation.required_services(consts.Service.NOVA) + @validation.required_openstack(users=True) + @scenario.configure(context={"cleanup": ["nova"]}) + def boot_live_migrate_and_delete_server_with_secgroups( + self, image, flavor, + security_group_count, + rules_per_security_group, + do_delete=True, + do_migration=False, + block_migration=False, + disk_over_commit=False, + min_sleep=0, max_sleep=0, + boot_server_kwargs=None + ): + + if boot_server_kwargs is None: + boot_server_kwargs = {} + + security_groups = self._create_security_groups( + security_group_count) + self._create_rules_for_security_group(security_groups, + rules_per_security_group) + + secgroups_names = [sg.name for sg in security_groups] + server = self._boot_server(image, flavor, + security_groups=secgroups_names, + **boot_server_kwargs) + + if do_migration: + new_host = self._find_host_to_migrate(server) + self._live_migrate(server, new_host, + block_migration, disk_over_commit) + self.sleep_between(min_sleep, max_sleep) + + self._list_security_groups() + + if do_delete: + self._delete_server(server) + self._delete_security_groups(security_groups) + + """boot_attach_live_migrate_and_delete_server""" + @types.convert(image={"type": "glance_image"}, + flavor={"type": "nova_flavor"}) + @validation.image_valid_on_flavor("flavor", "image") + @validation.required_services(consts.Service.NOVA) + @validation.required_openstack(users=True) + @scenario.configure(context={"cleanup": ["cinder", "nova"]}) + def boot_attach_live_migrate_and_delete_server( + self, image, flavor, + volume_size, + do_delete=True, + do_migration=False, + block_migration=False, + disk_over_commit=False, + min_sleep=0, max_sleep=0, + boot_server_kwargs=None, + create_volume_kwargs=None + ): + + if boot_server_kwargs is None: + boot_server_kwargs = {} + if create_volume_kwargs is None: + create_volume_kwargs = {} + + server = self._boot_server(image, flavor, + **boot_server_kwargs) + + volume = self._create_volume(volume_size, **create_volume_kwargs) + self._attach_volume(server, volume) + + if do_migration: + new_host = self._find_host_to_migrate(server) + self._live_migrate(server, new_host, + block_migration, disk_over_commit) + self.sleep_between(min_sleep, max_sleep) + + if do_delete: + self._detach_volume(server, volume) + self._delete_server(server) + self._delete_volume(volume) diff --git a/doc/source/test_plans/control_plane/test_plans/boot_attach_migrate_delete_server_with_sec_groups.json b/doc/source/test_plans/control_plane/test_plans/boot_attach_migrate_delete_server_with_sec_groups.json new file mode 100644 index 0000000..8334c7c --- /dev/null +++ b/doc/source/test_plans/control_plane/test_plans/boot_attach_migrate_delete_server_with_sec_groups.json @@ -0,0 +1,63 @@ +{% set flavor_name = flavor_name or "m1.tiny" %} +{% set image_name = image_name or "^(cirros.*uec|TestVM)$" %} +{ + "NovaPerformancePlugin.boot_attach_and_delete_server_with_secgroups": [ + { + "args": { + "flavor": { + "name": "{{flavor_name}}" + }, + "image": { + "name": "{{image_name}}" + }, + "security_group_count": 2, + "rules_per_security_group": 10, + "boot_server_kwargs": { "auto_assign_nic" : true }, + "create_volume_kwargs": {}, + "volume_size": 1, + "do_delete": true + }, + "runner": { + "type": "constant", + "times": {{ 2 * compute }}, + "concurrency": {{concurrency}} + }, + "context": { + "users": { + "tenants": 1, + "users_per_tenant": 10 + }, + "quotas": { + "neutron": { + "network": -1, + "security_group": -1, + "security_group_rule": -1, + "port": -1, + "subnet": -1, + "router": -1 + }, + "nova": { + "instances": -1, + "cores": -1, + "ram": -1, + "floating_ips": -1, + "security_groups": -1, + "security_group_rules": -1 + }, + "cinder": { + "volumes": -1, + "gigabytes": -1, + "snapshots": -1 + } + }, + "network": { + "start_cidr": "{{ "100.1.0.0/25" if gre_enabled else "1.0.0.0/25" }}", + "networks_per_tenant": 10 + } + }, + "sla": { + "failure_rate": { "max": 0 } + } + } + ] +} \ No newline at end of file diff --git a/doc/source/test_plans/control_plane/test_plans/create_and_delete_image.json b/doc/source/test_plans/control_plane/test_plans/create_and_delete_image.json new file mode 100644 index 0000000..f5cc88b --- /dev/null +++ b/doc/source/test_plans/control_plane/test_plans/create_and_delete_image.json @@ -0,0 +1,26 @@ +{% set http_server_with_glance_images = http_server_with_glance_images or "127.0.0.1" %} +{ + "GlanceImages.create_and_delete_image": [ + { + "args": { + "image_location": "http://{{http_server_with_glance_images}}/images/glance_1g_image.raw", + "container_format": "bare", + "disk_format": "raw" + }, + "runner": { + "type": "constant", + "times": 400, + "concurrency": 50 + }, + "context": { + "users": { + "tenants": 2, + "users_per_tenant": 3 + } + }, + "sla": { + "failure_rate": { "max": 0 } + } + } + ] +} \ No newline at end of file diff --git a/doc/source/test_plans/control_plane/test_plans/keystone_authenticate.json b/doc/source/test_plans/control_plane/test_plans/keystone_authenticate.json new file mode 100644 index 0000000..b03e60c --- /dev/null +++ b/doc/source/test_plans/control_plane/test_plans/keystone_authenticate.json @@ -0,0 +1,20 @@ +{ + "Authenticate.keystone": [ + { + "runner": { + "type": "rps", + "times": 120000, + "rps": 180 + }, + "context": { + "users": { + "tenants": 10, + "users_per_tenant": 10 + } + }, + "sla": { + "failure_rate": { "max": 0 } + } + } + ] +} \ No newline at end of file diff --git a/doc/source/test_plans/index.rst b/doc/source/test_plans/index.rst index de20ab2..64079f1 100644 --- a/doc/source/test_plans/index.rst +++ b/doc/source/test_plans/index.rst @@ -22,3 +22,5 @@ Test Plans openstack_load/plan 1000_nodes/plan reliability/plan + control_plane/plan +