diff --git a/doc/source/test_plans/neutron_control_plane/plan.rst b/doc/source/test_plans/neutron_control_plane/plan.rst new file mode 100644 index 0000000..5b8bc40 --- /dev/null +++ b/doc/source/test_plans/neutron_control_plane/plan.rst @@ -0,0 +1,221 @@ +.. _openstack_neutron_control_plane_performance_test_plan: + +================================================================== +OpenStack Networking (Neutron) control plane performance test plan +================================================================== + +:status: **ready** +:version: 1.0 + +:Abstract: + + This test plan aims to provide set of tests to identify OpenStack Networking + (aka Neutron) Control Plane performance against given OpenStack cloud using + Rally tests. + +Test Plan +========= +This test plan describes several Rally test cases that can cover almost +all most important in terms of networking performance basic cloud +operations e.g. routers, security groups and other objects management. + +Test Environment +---------------- + +Preparation +^^^^^^^^^^^ + +This test plan is performed either against existing OpenStack cloud with +pre-installed Rally framework or can be executed via Rally from very beginning +including deployment of the OpenStack cloud. As an option verification (Tempest +testing) can be run prior the benchmarking (scenarios to be described in this +document). + +Environment description +^^^^^^^^^^^^^^^^^^^^^^^ + +The environment description includes hardware specification of servers, +network parameters, operation system and OpenStack deployment characteristics. + +Hardware +~~~~~~~~ + +This section contains list of all types of hardware nodes. + ++-----------+-------+----------------------------------------------------+ +| Parameter | Value | Comments | ++-----------+-------+----------------------------------------------------+ +| model | | e.g. Supermicro X9SRD-F | ++-----------+-------+----------------------------------------------------+ +| CPU | | e.g. 6 x Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz | ++-----------+-------+----------------------------------------------------+ +| role | | e.g. compute or network | ++-----------+-------+----------------------------------------------------+ + +Network +~~~~~~~ + +This section contains list of interfaces and network parameters. +For complicated cases this section may include topology diagram and switch +parameters. + ++------------------+-------+-------------------------+ +| Parameter | Value | Comments | ++------------------+-------+-------------------------+ +| network role | | e.g. provider or public | ++------------------+-------+-------------------------+ +| card model | | e.g. Intel | ++------------------+-------+-------------------------+ +| driver | | e.g. ixgbe | ++------------------+-------+-------------------------+ +| speed | | e.g. 10G or 1G | ++------------------+-------+-------------------------+ +| MTU | | e.g. 9000 | ++------------------+-------+-------------------------+ +| offloading modes | | e.g. default | ++------------------+-------+-------------------------+ + +Software +~~~~~~~~ + +This section describes installed software. + ++-----------------+-------+---------------------------+ +| Parameter | Value | Comments | ++-----------------+-------+---------------------------+ +| OS | | e.g. Ubuntu 14.04.3 | ++-----------------+-------+---------------------------+ +| OpenStack | | e.g. Mitaka | ++-----------------+-------+---------------------------+ +| Hypervisor | | e.g. KVM | ++-----------------+-------+---------------------------+ +| Neutron plugin | | e.g. ML2 + OVS | ++-----------------+-------+---------------------------+ +| L2 segmentation | | e.g. VLAN / VxLAN / GRE | ++-----------------+-------+---------------------------+ +| virtual routers | | e.g. HA / DVR | ++-----------------+-------+---------------------------+ + +Test tool +--------- + +**Rally** is a benchmarking tool that was designed specifically for OpenStack +API testing. To make this possible, **Rally** automates and unifies multi-node +OpenStack deployment, cloud verification, benchmarking & profiling. This is a +simple way to check cloud workability and performance of control plane +operations running on it. + +Test Case 1: Basic Neutron test suite +------------------------------------- + +Description +^^^^^^^^^^^ + +This test suite is combined from default Rally Neutron test cases with `default +configuration`_. It is most useful for validating cloud operability. The +following Rally test scenarios need to be executed: + +* create-and-list-floating-ips +* create-and-list-networks +* create-and-list-ports +* create-and-list-routers +* create-and-list-security-groups +* create-and-list-subnets +* create-and-delete-floating-ips +* create-and-delete-networks +* create-and-delete-ports +* create-and-delete-routers +* create-and-delete-security-groups +* create-and-delete-subnets +* create-and-update-networks +* create-and-update-ports +* create-and-update-routers +* create-and-update-security-groups +* create-and-update-subnets + +.. _default configuration: https://github.com/openstack/rally/tree/master/samples/tasks/scenarios/neutron + +List of performance metrics +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + ++----------+-------+-------------------+---------------------------+ +| Priority | Value | Measurement Units | Description | ++==========+=======+===================+===========================+ +| 1 | | sec | Time of atomic operations | ++----------+-------+-------------------+---------------------------+ + +Test Case 2: Stressful Neutron test suite +----------------------------------------- + +Description +^^^^^^^^^^^ + +This test case the same set of scenarios that were mentioned in +`Test Case 1: Basic Neutron test suite`_ can be used, the difference is in +increased number of iterations and concurrency that create sufficient load on +Neutron control plane. To stress OpenStack networking control plane 50-100 +concurrency can be used with 2000-5000 iterations in total. We can advice to +focus on the following Rally test cases (that cover most interesting and most +stressed parts of OpenStack Networking): + +* create-and-list-networks +* create-and-list-ports +* create-and-list-routers +* create-and-list-security-groups +* create-and-list-subnets +* boot-and-list-server +* boot-and-delete-server-with-secgroups +* boot-runcommand-delete + +List of performance metrics +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + ++----------+-------+-------------------+---------------------------+ +| Priority | Value | Measurement Units | Description | ++==========+=======+===================+===========================+ +| 1 | | sec | Time of atomic operations | ++----------+-------+-------------------+---------------------------+ + +Test case 3: Neutron scalability test with many networks +-------------------------------------------------------- + +Description +^^^^^^^^^^^ + +The aim of this test is to create a large number of networks, subnets, routers +and security groups with rules per tenant. Each network has a single VM. For +example 100 networks (each with a subnet, router and a VM) can be created per +each iteration (up to 20 iterations in total). + +List of performance metrics +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + ++----------+-------+-------------------+---------------------------+ +| Priority | Value | Measurement Units | Description | ++==========+=======+===================+===========================+ +| 1 | | sec | Time of atomic operations | ++----------+-------+-------------------+---------------------------+ + +Test case 4: Neutron scalability test with many servers +------------------------------------------------------- + +The outline of this test is almost the same as of +`Test case 3: Neutron scalability test with many networks`_. The main +difference is that during each scenario iteration this test creates huge enough +number of VMs (e.g. 100) per a single network, hence it is possible to check +the case with many number of ports per subnet. + +List of performance metrics +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + ++----------+-------+-------------------+---------------------------+ +| Priority | Value | Measurement Units | Description | ++==========+=======+===================+===========================+ +| 1 | | sec | Time of atomic operations | ++----------+-------+-------------------+---------------------------+ + +Reports +======= + +Test plan execution reports: + * :ref:`openstack_neutron_control_plane_performance_report` diff --git a/doc/source/test_results/neutron_control_plane/configs/Network_Scheme.png b/doc/source/test_results/neutron_control_plane/configs/Network_Scheme.png new file mode 100644 index 0000000..5c800df Binary files /dev/null and b/doc/source/test_results/neutron_control_plane/configs/Network_Scheme.png differ diff --git a/doc/source/test_results/neutron_control_plane/index.rst b/doc/source/test_results/neutron_control_plane/index.rst new file mode 100644 index 0000000..06f6c71 --- /dev/null +++ b/doc/source/test_results/neutron_control_plane/index.rst @@ -0,0 +1,571 @@ + +.. _openstack_neutron_control_plane_performance_report: + +OpenStack Networking (Neutron) control plane performance report for 400 nodes +***************************************************************************** + +:Abstract: + + This document includes OpenStack Networking (aka Neutron) control plane + performance test results against two OpenStack environments: 200 nodes and + 378 nodes. All tests have been performed regarding + :ref:`openstack_neutron_control_plane_performance_test_plan` + + +Environment description +======================= + +Lab A (200 nodes) +----------------- + +3 controllers, 196 computes, 1 node for Grafana/Prometheus + +Hardware configuration of each server +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. table:: Description of controller servers + + +-------+----------------+---------------------------------+ + |server |vendor,model |Supermicro MBD-X10DRI | + +-------+----------------+---------------------------------+ + |CPU |vendor,model |Intel Xeon E5-2650v3 | + | +----------------+---------------------------------+ + | |processor_count |2 | + | +----------------+---------------------------------+ + | |core_count |10 | + | +----------------+---------------------------------+ + | |frequency_MHz |2300 | + +-------+----------------+---------------------------------+ + |RAM |vendor,model |8x Samsung M393A2G40DB0-CPB | + | +----------------+---------------------------------+ + | |amount_MB |2097152 | + +-------+----------------+---------------------------------+ + |NETWORK|vendor,model |Intel,I350 Dual Port | + | +----------------+---------------------------------+ + | |bandwidth |1G | + | +----------------+---------------------------------+ + | |vendor,model |Intel,82599ES Dual Port | + | +----------------+---------------------------------+ + | |bandwidth |10G | + +-------+----------------+---------------------------------+ + |STORAGE|vendor,model |Intel SSD DC S3500 Series | + | +----------------+---------------------------------+ + | |SSD/HDD |SSD | + | +----------------+---------------------------------+ + | |size |240GB | + | +----------------+---------------------------------+ + | |vendor,model |2x WD WD5003AZEX | + | +----------------+---------------------------------+ + | |SSD/HDD |HDD | + | +----------------+---------------------------------+ + | |size |500GB | + +-------+----------------+---------------------------------+ + +.. table:: Description of compute servers + + +-------+----------------+---------------------------------+ + |server |vendor,model |SUPERMICRO 5037MR-H8TRF | + +-------+----------------+---------------------------------+ + |CPU |vendor,model |INTEL XEON Ivy Bridge 6C E5-2620 | + | +----------------+---------------------------------+ + | |processor_count |1 | + | +----------------+---------------------------------+ + | |core_count |6 | + | +----------------+---------------------------------+ + | |frequency_MHz |2100 | + +-------+----------------+---------------------------------+ + |RAM |vendor,model |4x Samsung DDRIII 8GB DDR3-1866 | + | +----------------+---------------------------------+ + | |amount_MB |32768 | + +-------+----------------+---------------------------------+ + |NETWORK|vendor,model |AOC-STGN-i2S - 2-port | + | +----------------+---------------------------------+ + | |bandwidth |10G | + +-------+----------------+---------------------------------+ + |STORAGE|vendor,model |Intel SSD DC S3500 Series | + | +----------------+---------------------------------+ + | |SSD/HDD |SSD | + | +----------------+---------------------------------+ + | |size |80GB | + | +----------------+---------------------------------+ + | |vendor,model |1x WD Scorpio Black BP WD7500BPKT| + | +----------------+---------------------------------+ + | |SSD/HDD |HDD | + | +----------------+---------------------------------+ + | |size |750GB | + +-------+----------------+---------------------------------+ + +Lab B (378 nodes) +----------------- + +Environment contains 4 types of servers: + +- rally node +- controller node +- compute-osd node +- compute node + +.. table:: Amount of servers each role (node types are listed below) + + +------------+--------------+------+ + |Role |Servers count |Type | + +============+==============+======+ + |rally |1 |1 or 2| + +------------+--------------+------+ + |controller |3 |1 or 2| + +------------+--------------+------+ + |compute |291 |1 or 2| + +------------+--------------+------+ + |compute-osd |34 |3 | + +------------+--------------+------+ + |compute-osd |49 |1 | + +------------+--------------+------+ + +Hardware configuration of each server +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +All servers have 3 types of configuration describing in table below + +.. table:: Description of servers hardware type 1 + + +-------+----------------+---------------------------------+ + |server |vendor,model |Dell PowerEdge R630 | + +-------+----------------+---------------------------------+ + |CPU |vendor,model |Intel,E5-2680 v3 | + | +----------------+---------------------------------+ + | |processor_count |2 | + | +----------------+---------------------------------+ + | |core_count |12 | + | +----------------+---------------------------------+ + | |frequency_MHz |2500 | + +-------+----------------+---------------------------------+ + |RAM |vendor,model |Samsung, M393A2G40DB0-CPB | + | +----------------+---------------------------------+ + | |amount_MB |262144 | + +-------+----------------+---------------------------------+ + |NETWORK|interface_name s|eno1, eno2 | + | +----------------+---------------------------------+ + | |vendor,model |Intel,X710 Dual Port | + | +----------------+---------------------------------+ + | |bandwidth |10G | + | +----------------+---------------------------------+ + | |interface_names |enp3s0f0, enp3s0f1 | + | +----------------+---------------------------------+ + | |vendor,model |Intel,X710 Dual Port | + | +----------------+---------------------------------+ + | |bandwidth |10G | + +-------+----------------+---------------------------------+ + |STORAGE|dev_name |/dev/sda | + | +----------------+---------------------------------+ + | |vendor,model | | raid1 - Dell, PERC H730P Mini | + | | | | 2 disks Intel S3610 | + | +----------------+---------------------------------+ + | |SSD/HDD |SSD | + | +----------------+---------------------------------+ + | |size | 3,6TB | + +-------+----------------+---------------------------------+ + +.. table:: Description of servers hardware type 2 + + +-------+----------------+-------------------------------+ + |server |vendor,model |Lenovo ThinkServer RD550 | + +-------+----------------+-------------------------------+ + |CPU |vendor,model |Intel,E5-2680 v3 | + | +----------------+-------------------------------+ + | |processor_count |2 | + | +----------------+-------------------------------+ + | |core_count |12 | + | +----------------+-------------------------------+ + | |frequency_MHz |2500 | + +-------+----------------+-------------------------------+ + |RAM |vendor,model |Samsung, M393A2G40DB0-CPB | + | +----------------+-------------------------------+ + | |amount_MB |262144 | + +-------+----------------+-------------------------------+ + |NETWORK|interface_name s|enp3s0f0, enp3s0f1 | + | +----------------+-------------------------------+ + | |vendor,model |Intel,X710 Dual Port | + | +----------------+-------------------------------+ + | |bandwidth |10G | + | +----------------+-------------------------------+ + | |interface_names |ens2f0, ens2f1 | + | +----------------+-------------------------------+ + | |vendor,model |Intel,X710 Dual Port | + | +----------------+-------------------------------+ + | |bandwidth |10G | + +-------+----------------+-------------------------------+ + |STORAGE|dev_name |/dev/sda | + | +----------------+-------------------------------+ + | |vendor,model | | raid1 - Lenovo 720ix | + | | | | 2 disks Intel S3610 | + | +----------------+-------------------------------+ + | |SSD/HDD |SSD | + | +----------------+-------------------------------+ + | |size |799GB | + +-------+----------------+-------------------------------+ + + .. table:: Description of servers hardware type 3 + + +-------+----------------+-------------------------------+ + |server |vendor,model |Lenovo ThinkServer RD650 | + +-------+----------------+-------------------------------+ + |CPU |vendor,model |Intel,E5-2670 v3 | + | +----------------+-------------------------------+ + | |processor_count |2 | + | +----------------+-------------------------------+ + | |core_count |12 | + | +----------------+-------------------------------+ + | |frequency_MHz |2500 | + +-------+----------------+-------------------------------+ + |RAM |vendor,model |Samsung, M393A2G40DB0-CPB | + | +----------------+-------------------------------+ + | |amount_MB |131916 | + +-------+----------------+-------------------------------+ + |NETWORK|interface_name s|enp3s0f0, enp3s0f1 | + | +----------------+-------------------------------+ + | |vendor,model |Intel,X710 Dual Port | + | +----------------+-------------------------------+ + | |bandwidth |10G | + | +----------------+-------------------------------+ + | |interface_names |ens2f0, ens2f1 | + | +----------------+-------------------------------+ + | |vendor,model |Intel,X710 Dual Port | + | +----------------+-------------------------------+ + | |bandwidth |10G | + +-------+----------------+-------------------------------+ + |STORAGE|dev_name |/dev/sda | + | +----------------+-------------------------------+ + | |vendor,model | | raid1 - Lenovo 720ix | + | | | | 2 disks Intel S3610 | + | +----------------+-------------------------------+ + | |SSD/HDD |SSD | + | +----------------+-------------------------------+ + | |size |799GB | + | +----------------+-------------------------------+ + | |dev_name |/dev/sdb | + | +----------------+-------------------------------+ + | |vendor,model | | raid10 - Lenovo 720ix | + | | | | 10 disks 2T | + | +----------------+-------------------------------+ + | |SSD/HDD |HDD | + | +----------------+-------------------------------+ + | |size |9999GB | + +-------+----------------+-------------------------------+ + +Network configuration of each server +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +All servers have same network configuration: + +.. image:: configs/Network_Scheme.png + :alt: Network Scheme of the environment + :width: 650px + +Software configuration on servers with controller, compute and compute-osd roles +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. table:: Services on servers by role + + +------------+----------------------------+ + |Role |Service name | + +============+============================+ + |controller || horizon | + | || keystone | + | || nova-api | + | || nava-scheduler | + | || nova-cert | + | || nova-conductor | + | || nova-consoleauth | + | || nova-consoleproxy | + | || cinder-api | + | || cinder-backup | + | || cinder-scheduler | + | || cinder-volume | + | || glance-api | + | || glance-glare | + | || glance-registry | + | || neutron-dhcp-agent | + | || neutron-l3-agent | + | || neutron-metadata-agent | + | || neutron-openvswitch-agent | + | || neutron-server | + | || heat-api | + | || heat-api-cfn | + | || heat-api-cloudwatch | + | || ceph-mon | + | || rados-gw | + | || memcached | + | || rabbitmq_server | + | || mysqld | + | || galera | + | || corosync | + | || pacemaker | + | || haproxy | + +------------+----------------------------+ + |compute || nova-compute | + | || neutron-l3-agent | + | || neutron-metadata-agent | + | || neutron-openvswitch-agent | + +------------+----------------------------+ + |compute-osd || nova-compute | + | || neutron-l3-agent | + | || neutron-metadata-agent | + | || neutron-openvswitch-agent | + | || ceph-osd | + +------------+----------------------------+ + +.. table:: Software version on servers with controller, compute and compute-osd roles + + +------------+-------------------+ + |Software |Version | + +============+===================+ + |OpenStack |Mitaka | + +------------+-------------------+ + |Ceph |Hammer | + +------------+-------------------+ + |Ubuntu |Ubuntu 14.04.3 LTS | + +------------+-------------------+ + +You can find outputs of some commands and /etc folder in the following archives: + +| :download:`controller-1.tar.gz ` +| :download:`controller-2.tar.gz ` +| :download:`controller-3.tar.gz ` +| :download:`compute-1.tar.gz ` +| :download:`compute-osd-1.tar.gz ` + +Software configuration on servers with Rally role +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +On this server should be installed Rally. How to do it you can find in +`Rally installation documentation`_ + +.. table:: Software version on server with Rally role + + +------------+-------------------+ + |Software |Version | + +============+===================+ + |Rally |0.5.0 | + +------------+-------------------+ + |Ubuntu |Ubuntu 14.04.3 LTS | + +------------+-------------------+ + +Test results +============ + +Test Case 1: Basic Neutron test suite +------------------------------------- + +The following list of tests were run with the `default configuration`_ against +`Lab A (200 nodes)`_: + +* create-and-list-floating-ips +* create-and-list-networks +* create-and-list-ports +* create-and-list-routers +* create-and-list-security-groups +* create-and-list-subnets +* create-and-delete-floating-ips +* create-and-delete-networks +* create-and-delete-ports +* create-and-delete-routers +* create-and-delete-security-groups +* create-and-delete-subnets +* create-and-update-networks +* create-and-update-ports +* create-and-update-routers +* create-and-update-security-groups +* create-and-update-subnets + +The time needed for each scenario can be comparatively presented using the +following chart: + +.. image:: reports/basic_neutron.png + :alt: Basic Neutron scenarios timings comparison + :width: 650px + +To overview extended information please download the following report: +:download:`basic_neutron.html ` + +.. _default configuration: https://github.com/openstack/rally/tree/master/samples/tasks/scenarios/neutron + +Test Case 2: Stressful Neutron test suite +----------------------------------------- + +The following list of tests were run against both `Lab A (200 nodes)`_ and +`Lab B (378 nodes)`_: + +* create-and-list-networks +* create-and-list-ports +* create-and-list-routers +* create-and-list-security-groups +* create-and-list-subnets +* boot-and-list-server +* boot-and-delete-server-with-secgroups +* boot-runcommand-delete + +Here is short representation of the collected results: + ++--------------------------------------+-----------------------+-------------------------+---------------------------------------------+ +|Scenario |Iterations/concurrency |Time, sec |Errors | +| +-----------+-----------+------------+------------+----------------------+----------------------+ +| |Lab A |Lab B |Lab A |Lab B |Lab A |Lab B | ++======================================+===========+===========+============+============+======================+======================+ +|create-and-list-networks |3000/50 |5000/50 |avg 2.375 |avg 3.654 |1 |6 | +| | | |max 7.904 |max 11.669 |Internal server error |Internal server error | +| | | | | |while processing your |while processing your | +| | | | | |request |request | ++--------------------------------------+-----------+-----------+------------+------------+----------------------+----------------------+ +|create-and-list-ports |1000/50 |2000/50 |avg 123.97 |avg 99.274 |1 |0 | +| | | |max 277.977 |max 270.84 |Internal server error | | +| | | | | |while processing your | | +| | | | | |request | | ++--------------------------------------+-----------+-----------+------------+------------+----------------------+----------------------+ +|create-and-list-routers |2000/50 |2000/50 |avg 15.59 |avg 12.94 |0 |0 | +| | | |max 29.006 |max 19.398 | | | ++--------------------------------------+-----------+-----------+------------+------------+----------------------+----------------------+ +|create-and-list-security-groups |50/1 |1000/50 |avg 210.706 |avg 68.712 |0 |0 | +| | | |max 210.706 |max 169.315 | | | ++--------------------------------------+-----------+-----------+------------+------------+----------------------+----------------------+ +|create-and-list-subnets |2000/50 |2000/50 |avg 25.973 |avg 17.415 |1 |0 | +| | | |max 64.553 |max 50.415 |Internal server error | | +| | | | | |while processing your | | +| | | | | |request | | ++--------------------------------------+-----------+-----------+------------+------------+----------------------+----------------------+ +|boot-and-list-server |4975/50 |1000/50 |avg 21.445 |avg 14.375 |0 |0 | +| | | |max 40.736 |max 25.21 | | | ++--------------------------------------+-----------+-----------+------------+------------+----------------------+----------------------+ +|boot-and-delete-server-with-secgroups |4975/200 |1000/100 |avg 190.772 |avg 65.651 |394 |0 | +| | | |max 443.518 |max 95.651 |Server has ERROR | | +| | | | | |status; The server | | +| | | | | |didn't respond in | | +| | | | | |time. | | ++--------------------------------------+-----------+-----------+------------+------------+----------------------+----------------------+ +|boot-runcommand-delete |2000/15 |3000/50 |avg 28.39 |avg 28.587 |34 |1 | +| | | |max 35.756 |max 85.659 |Rally tired waiting |Resource to |s_rally_b58e9bd | +| | | | | |become ('ICMP UP'), |e_Y369JdPf> has ERROR | +| | | | | |current status |status. Deadlock | +| | | | | |('ICMP DOWN') |found when trying to | +| | | | | | |get lock. | ++--------------------------------------+-----------+-----------+------------+------------+----------------------+----------------------+ + +During execution of Rally were filed and fixed bugs affecting +`boot-and-delete-server-with-secgroups` and `boot-runcommand-delete` scenarios +on Lab A: + +* `Bug LP #1610303 l2pop mech fails to update_port_postcommit on a loaded cluster`_ , + fix - https://review.openstack.org/353835 +* `Bug LP #1614452 Port create time grows at scale due to dvr arp update`_ , + fix - https://review.openstack.org/357052 + +With these fixes applied on Lab B mentioned Rally scenarios passed +*successfully*. + +Other bugs that were faced: + +* `Bug LP #1595837 Neutron failed to create network`_ +* `Bug LP #1610153 nova list can sometimes return 404`_ + +Observed trends +~~~~~~~~~~~~~~~ + +Create and list networks: the total time spent on each iteration grows linearly + +.. image:: reports/create_and_list_networks.png + :alt: Create and list networks trend + :width: 650px + +Create and list routers: router list operation time gradually grows from 0.12 +to 1.5 sec (2000 iterations). + +.. image:: reports/create_and_list_routers_list.png + :alt: Create and list routers trend (list operation) + :width: 650px + +Create and list routers: total load duration remains line-rate + +.. image:: reports/create_and_list_routers.png + :alt: Create and list routers trend (total time) + :width: 650px + +Create and list subnets: subnet list operation time increases ~ after 1750 +iterations (4.5 sec at 1700th iteration to 10.48 at 1800th iteration). + +.. image:: reports/create_and_list_subnets_list.png + :alt: Create and list subnets trend (list operation) + :width: 650px + +Create and list subnets: creating subnets has time peaks after 1750 iterations + +.. image:: reports/create_and_list_subnets_create.png + :alt: Create and list subnets trend (create operation) + :width: 650px + +Create and list security groups: secgroup list operation exposes the most rapid +growth rate with time increasing from 0.548 sec in first iteration to over 10 +sec in last iterations + +.. image:: reports/create_and_list_secgroup_list.png + :alt: Create and list security groups trend (list operation) + :width: 650px + +More details can be found in original Rally report: +:download:`stress_neutron.html ` + +Test case 3: Neutron scalability test with many networks +-------------------------------------------------------- + +In our tests 100 networks (each with a subnet, router and a VM) were created +per each iteration. + ++-----------------------+--------------+--------------+-------------------------------+ +|Iterations/concurrency |Avg time, sec |Max time, sec |Errors | ++=======================+==============+==============+===============================+ +|10/1 |1237.389 |1294.549 |0 | ++-----------------------+--------------+--------------+-------------------------------+ +|20/3 |1298.611 |1425.878 |1 | +| | | |HTTPConnectionPool Read time | +| | | |out | ++-----------------------+--------------+--------------+-------------------------------+ + +Load graph for run with 20 iterations/concurrency 3: + +.. image:: reports/scale_neutron_networks.png + :alt: Neutron scalability test with many networks trend + :width: 650px + +More details can be found in original Rally report: +:download:`scale_neutron_networks.html ` + +Test case 4: Neutron scalability test with many servers +------------------------------------------------------- + +During each iteration this test creates huge number of VMs (100 in our case) +per a single network, hence it is possible to check the case with many number +of ports per subnet. + ++-----------------------+--------------+--------------+-------------------------------+ +|Iterations/concurrency |Avg time, sec |Max time, sec |Errors | ++=======================+==============+==============+===============================+ +|10/1 |100.422 |104.315 |0 | ++-----------------------+--------------+--------------+-------------------------------+ +|20/3 |119.767 |147.107 |0 | ++-----------------------+--------------+--------------+-------------------------------+ + +Load graph for run with 20 iterations/concurrency 3: + +.. image:: reports/scale_neutron_servers.png + :alt: Neutron scalability test with many servers trend + :width: 650px + +More details can be found in original Rally report: +:download:`scale_neutron_servers.html ` + +.. references: + +.. _Rally installation documentation: https://rally.readthedocs.io/en/latest/install.html +.. _Bug LP #1610303 l2pop mech fails to update_port_postcommit on a loaded cluster: https://bugs.launchpad.net/neutron/+bug/1610303 +.. _Bug LP #1614452 Port create time grows at scale due to dvr arp update: https://bugs.launchpad.net/neutron/+bug/1614452 +.. _Bug LP #1595837 Neutron failed to create network: https://bugs.launchpad.net/mos/+bug/1595837 +.. _Bug LP #1610153 nova list can sometimes return 404: https://bugs.launchpad.net/nova/+bug/1610153 \ No newline at end of file diff --git a/doc/source/test_results/neutron_control_plane/reports/basic_neutron.html b/doc/source/test_results/neutron_control_plane/reports/basic_neutron.html new file mode 100644 index 0000000..f958c9e --- /dev/null +++ b/doc/source/test_results/neutron_control_plane/reports/basic_neutron.html @@ -0,0 +1,864 @@ + + + + + + Rally | Rally Task Report + + + + + + + + + + + + + +
+
+ Rally  + task results +
+
+ +
+ + + + +
+
+ + +
+
+ + +
+
+ +
+ +
+

Task overview

+ + + + + + + + + +
+ Scenario + + + + + + Load duration (s) + + + + + + Full duration (s) + + + + + + Iterations + + + + + + Runner + + + + + + Errors + + + + + + Success (SLA) + + + + +
{{sc.ref}} + {{sc.load_duration | number:3}} + {{sc.full_duration | number:3}} + {{sc.iterations_count}} + {{sc.runner}} + {{sc.errors.length}} + + + +
+
+ +
+

Input file

+
{{source}}
+
+ +
+

{{scenario.cls}}.{{scenario.name}} ({{scenario.full_duration | number:3}}s)

+
    +
  • +
    {{t.name}}
    +
  • +
    +
+
+ + + + + + + + + + +
+ +
+
+ + +
+ + + + \ No newline at end of file diff --git a/doc/source/test_results/neutron_control_plane/reports/basic_neutron.png b/doc/source/test_results/neutron_control_plane/reports/basic_neutron.png new file mode 100644 index 0000000..dc412e2 Binary files /dev/null and b/doc/source/test_results/neutron_control_plane/reports/basic_neutron.png differ diff --git a/doc/source/test_results/neutron_control_plane/reports/create_and_list_networks.png b/doc/source/test_results/neutron_control_plane/reports/create_and_list_networks.png new file mode 100644 index 0000000..788bc07 Binary files /dev/null and b/doc/source/test_results/neutron_control_plane/reports/create_and_list_networks.png differ diff --git a/doc/source/test_results/neutron_control_plane/reports/create_and_list_routers.png b/doc/source/test_results/neutron_control_plane/reports/create_and_list_routers.png new file mode 100644 index 0000000..40766ff Binary files /dev/null and b/doc/source/test_results/neutron_control_plane/reports/create_and_list_routers.png differ diff --git a/doc/source/test_results/neutron_control_plane/reports/create_and_list_routers_list.png b/doc/source/test_results/neutron_control_plane/reports/create_and_list_routers_list.png new file mode 100644 index 0000000..d61e0cd Binary files /dev/null and b/doc/source/test_results/neutron_control_plane/reports/create_and_list_routers_list.png differ diff --git a/doc/source/test_results/neutron_control_plane/reports/create_and_list_secgroup_list.png b/doc/source/test_results/neutron_control_plane/reports/create_and_list_secgroup_list.png new file mode 100644 index 0000000..77beceb Binary files /dev/null and b/doc/source/test_results/neutron_control_plane/reports/create_and_list_secgroup_list.png differ diff --git a/doc/source/test_results/neutron_control_plane/reports/create_and_list_subnets_create.png b/doc/source/test_results/neutron_control_plane/reports/create_and_list_subnets_create.png new file mode 100644 index 0000000..e21607a Binary files /dev/null and b/doc/source/test_results/neutron_control_plane/reports/create_and_list_subnets_create.png differ diff --git a/doc/source/test_results/neutron_control_plane/reports/create_and_list_subnets_list.png b/doc/source/test_results/neutron_control_plane/reports/create_and_list_subnets_list.png new file mode 100644 index 0000000..eccc3ca Binary files /dev/null and b/doc/source/test_results/neutron_control_plane/reports/create_and_list_subnets_list.png differ diff --git a/doc/source/test_results/neutron_control_plane/reports/scale_neutron_networks.html b/doc/source/test_results/neutron_control_plane/reports/scale_neutron_networks.html new file mode 100644 index 0000000..d964a98 --- /dev/null +++ b/doc/source/test_results/neutron_control_plane/reports/scale_neutron_networks.html @@ -0,0 +1,873 @@ + + + + + + + Rally | Rally Task Report + + + + + + + + + + + + + +
+
+ Rally  + task results +
+
+ +
+ + + + +
+
+ + +
+
+ + +
+
+ +
+ +
+

Task overview

+ + + + + + + + + +
+ Scenario + + + + + + Load duration (s) + + + + + + Full duration (s) + + + + + + Iterations + + + + + + Runner + + + + + + Errors + + + + + + Success (SLA) + + + + +
{{sc.ref}} + {{sc.load_duration | number:3}} + {{sc.full_duration | number:3}} + {{sc.iterations_count}} + {{sc.runner}} + {{sc.errors.length}} + + + +
+
+ +
+

Input file

+
{{source}}
+
+ +
+

{{scenario.cls}}.{{scenario.name}} ({{scenario.full_duration | number:3}}s)

+
    +
  • +
    {{t.name}}
    +
  • +
    +
+
+ + + + + + + + + + +
+ +
+
+ + +
+ + + + \ No newline at end of file diff --git a/doc/source/test_results/neutron_control_plane/reports/scale_neutron_networks.png b/doc/source/test_results/neutron_control_plane/reports/scale_neutron_networks.png new file mode 100644 index 0000000..db79cb1 Binary files /dev/null and b/doc/source/test_results/neutron_control_plane/reports/scale_neutron_networks.png differ diff --git a/doc/source/test_results/neutron_control_plane/reports/scale_neutron_servers.html b/doc/source/test_results/neutron_control_plane/reports/scale_neutron_servers.html new file mode 100644 index 0000000..0ca5655 --- /dev/null +++ b/doc/source/test_results/neutron_control_plane/reports/scale_neutron_servers.html @@ -0,0 +1,873 @@ + + + + + + + Rally | Rally Task Report + + + + + + + + + + + + + +
+
+ Rally  + task results +
+
+ +
+ + + + +
+
+ + +
+
+ + +
+
+ +
+ +
+

Task overview

+ + + + + + + + + +
+ Scenario + + + + + + Load duration (s) + + + + + + Full duration (s) + + + + + + Iterations + + + + + + Runner + + + + + + Errors + + + + + + Success (SLA) + + + + +
{{sc.ref}} + {{sc.load_duration | number:3}} + {{sc.full_duration | number:3}} + {{sc.iterations_count}} + {{sc.runner}} + {{sc.errors.length}} + + + +
+
+ +
+

Input file

+
{{source}}
+
+ +
+

{{scenario.cls}}.{{scenario.name}} ({{scenario.full_duration | number:3}}s)

+
    +
  • +
    {{t.name}}
    +
  • +
    +
+
+ + + + + + + + + + +
+ +
+
+ + +
+ + + + \ No newline at end of file diff --git a/doc/source/test_results/neutron_control_plane/reports/scale_neutron_servers.png b/doc/source/test_results/neutron_control_plane/reports/scale_neutron_servers.png new file mode 100644 index 0000000..baeee9b Binary files /dev/null and b/doc/source/test_results/neutron_control_plane/reports/scale_neutron_servers.png differ diff --git a/doc/source/test_results/neutron_control_plane/reports/stress_neutron.html b/doc/source/test_results/neutron_control_plane/reports/stress_neutron.html new file mode 100644 index 0000000..e476c83 --- /dev/null +++ b/doc/source/test_results/neutron_control_plane/reports/stress_neutron.html @@ -0,0 +1,873 @@ + + + + + + + Rally | Rally Task Report + + + + + + + + + + + + + +
+
+ Rally  + task results +
+
+ +
+ + + + +
+
+ + +
+
+ + +
+
+ +
+ +
+

Task overview

+ + + + + + + + + +
+ Scenario + + + + + + Load duration (s) + + + + + + Full duration (s) + + + + + + Iterations + + + + + + Runner + + + + + + Errors + + + + + + Success (SLA) + + + + +
{{sc.ref}} + {{sc.load_duration | number:3}} + {{sc.full_duration | number:3}} + {{sc.iterations_count}} + {{sc.runner}} + {{sc.errors.length}} + + + +
+
+ +
+

Input file

+
{{source}}
+
+ +
+

{{scenario.cls}}.{{scenario.name}} ({{scenario.full_duration | number:3}}s)

+
    +
  • +
    {{t.name}}
    +
  • +
    +
+
+ + + + + + + + + + +
+ +
+
+ + +
+ + + + \ No newline at end of file