Publish test plan and test results for Neutron resource density

Change-Id: Icaa3bb0756dd26c298c35acf2fd6ffd336c247ef

doc/source/test_plans/neutron_features/index.rst

@@ -9,4 +9,5 @@ Neutron features test plans
 .. toctree::
     :maxdepth: 3
 
-    l3_ha/plan
+    l3_ha/plan
+    resource_density/plan

doc/source/test_plans/neutron_features/resource_density/plan.rst

@@ -0,0 +1,141 @@
+.. _neutron_resource_density_test_plan:
+
+============================================
+OpenStack Neutron Resource Density Test Plan
+============================================
+
+:status: **draft**
+:version: 1.0
+
+:Abstract:
+  Is data-plane performance affected by existence of other OpenStack resources?
+
+
+
+Test Plan
+=========
+
+The goal of this test plan is to investigate whether existing OpenStack
+resources affect data-plane performance.
+
+Out of all resources the following may theoretically affect performance:
+  * instances running on the same compute host - because of CPU consumption,
+    additional network namespaces, iptables, OVS ports and flows;
+  * routers - because of Linux network and OVS resources;
+  * security groups and rules - because of iptables rules.
+
+
+Test Environment
+----------------
+
+Preparation
+^^^^^^^^^^^
+
+This test plan is performed against existing OpenStack cloud.
+
+
+Environment description
+^^^^^^^^^^^^^^^^^^^^^^^
+
+The environment description includes hardware specification of servers,
+network parameters, operation system and OpenStack deployment characteristics.
+
+Hardware
+~~~~~~~~
+
+This section contains list of all types of hardware nodes.
+
++-----------+-------+----------------------------------------------------+
+| Parameter | Value | Comments                                           |
++-----------+-------+----------------------------------------------------+
+| model     |       | e.g. Supermicro X9SRD-F                            |
++-----------+-------+----------------------------------------------------+
+| CPU       |       | e.g. 6 x Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz |
++-----------+-------+----------------------------------------------------+
+| role      |       | e.g. compute or network                            |
++-----------+-------+----------------------------------------------------+
+
+Network
+~~~~~~~
+
+This section contains list of interfaces and network parameters.
+For complicated cases this section may include topology diagram and switch
+parameters.
+
++------------------+-------+-------------------------+
+| Parameter        | Value | Comments                |
++------------------+-------+-------------------------+
+| network role     |       | e.g. provider or public |
++------------------+-------+-------------------------+
+| card model       |       | e.g. Intel              |
++------------------+-------+-------------------------+
+| driver           |       | e.g. ixgbe              |
++------------------+-------+-------------------------+
+| speed            |       | e.g. 10G or 1G          |
++------------------+-------+-------------------------+
+| MTU              |       | e.g. 9000               |
++------------------+-------+-------------------------+
+| offloading modes |       | e.g. default            |
++------------------+-------+-------------------------+
+
+Software
+~~~~~~~~
+
+This section describes installed software.
+
++-----------------+-------+---------------------------+
+| Parameter       | Value | Comments                  |
++-----------------+-------+---------------------------+
+| OS              |       | e.g. Ubuntu 14.04.3       |
++-----------------+-------+---------------------------+
+| OpenStack       |       | e.g. Liberty              |
++-----------------+-------+---------------------------+
+| Hypervisor      |       | e.g. KVM                  |
++-----------------+-------+---------------------------+
+| Neutron plugin  |       | e.g. ML2 + OVS            |
++-----------------+-------+---------------------------+
+| L2 segmentation |       | e.g. VLAN or VxLAN or GRE |
++-----------------+-------+---------------------------+
+| virtual routers |       | HA                        |
++-----------------+-------+---------------------------+
+
+Test Case: Data-plane performance measurements
+----------------------------------------------
+
+Description
+^^^^^^^^^^^
+
+Measurements are performed between 2 instances running on different compute
+nodes. One of instances has floating IP assigned and thus is reachable from
+outside. We are interested in following metrics: TCP max throughput and
+UDP top packets throughput (for 64-byte packets).
+
+TCP throughput is measured with `flent`_::
+
+    flent -H <destination> -f stats tcp_download
+
+UDP throughput is measured with `iperf3`_::
+
+    iperf3 -c <destination> -u -l 64 -b 0 -t 20
+
+List of performance metrics
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+========  ===============  =================  ======================================
+Priority  Value            Measurement Units  Description
+========  ===============  =================  ======================================
+1         Latency          ms                 The network latency
+1         TCP bandwidth    Mbits/s            TCP network bandwidth
+1         UDP bandwidth    packets per sec    Number of UDP packets of 32 bytes size
+========  ===============  =================  ======================================
+
+Reports
+=======
+
+Test plan execution reports:
+ * :ref:`neutron_neutron_resource_test_report`
+
+.. references:
+
+.. _flent: http://flent.org/
+.. _iperf3: http://iperf.fr/

doc/source/test_results/neutron_features/index.rst

@@ -11,4 +11,4 @@ Neutron features scale testing
 
     l3_ha/test_results_liberty
     l3_ha/test_results_mitaka
-
+    resource_density/index

doc/source/test_results/neutron_features/resource_density/index.rst

@@ -0,0 +1,216 @@
+.. _neutron_neutron_resource_test_report:
+
+==============================================
+OpenStack Neutron Resource Density Test Report
+==============================================
+
+This report is generated for :ref:`neutron_resource_density_test_plan`.
+
+Environment description
+=======================
+
+Cluster description
+-------------------
+* 3 controllers
+* 2 compute nodes
+
+Software versions
+-----------------
+
+**OpenStack/System**:
+  Fuel/MOS 9.0, Ubuntu 14.04, Linux kernel 3.13, OVS 2.4.1
+**Networking**
+  Neutron ML2 + OVS plugin, DVR, L2pop, MTU 1500
+
+Hardware configuration of each server
+-------------------------------------
+
+Description of servers hardware
+
+**Compute Vendor**:
+    HP ProLiant DL380 Gen9,
+**CPU**
+    2 x Intel(R) Xeon(R) CPU E5-2680 v3 @2.50GHz (48 cores)
+**RAM**:
+    256 Gb
+**NIC**
+    2 x Intel Corporation Ethernet 10G 2P X710
+
+
+Report
+======
+
+Preparation
+-----------
+
+Measurements are performed between 2 instances deployed in L3 east-west
+topology. To prepare:
+ #. Create new Neutron router. Plug into external network.
+ #. Create 2 Neutron networks, e.g. `east` and `west`. Plug them into
+    the router.
+ #. Boot 2 instances (Ubuntu Cloud Image can be used). Each instance should be
+    in its own network and on its own compute node. Public key may be required
+    to login into instance (in case of using Ubuntu).
+ #. Assign floating IP to one of instances.
+
+The traffic is sent between 2 instances, from the one with floating IP to the
+other. Phisically packets will go from one compute node to the other directly.
+
+Measurements
+------------
+
+Baseline
+^^^^^^^^
+
+We start with measuring baseline numbers. During this test only 2 instances
+exist.
+
+ ================= ============================= ==========================
+  Latency, ms [σ]   TCP throughput, Mbits/s [σ]   UDP throughput, kpps [σ]
+ ================= ============================= ==========================
+  1.8 [0.6]         7445 [850]                    132 [6.5]
+ ================= ============================= ==========================
+
+
+10 instances per compute in separate network
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+There are 20 micro instances (10 per compute node). Each instance is
+Cirros-based with 1 VCPU and 64MB RAM. Instances are plugged into separate
+network.
+
+ ================= ============================= ==========================
+  Latency, ms [σ]   TCP throughput, Mbits/s [σ]   UDP throughput, kpps [σ]
+ ================= ============================= ==========================
+  2.0 [0.7]         6500 [807]                    106 [4.5]
+ ================= ============================= ==========================
+
+30 instances per compute in separate network
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+There are 60 micro instances (30 per compute node). Each instance is
+Cirros-based with 1 VCPU and 64MB RAM. Instances are plugged into separate
+network.
+
+ ================= ============================= ==========================
+  Latency, ms [σ]   TCP throughput, Mbits/s [σ]   UDP throughput, kpps [σ]
+ ================= ============================= ==========================
+  2.2 [0.8]         6650 [600]                    88 [3.0]
+ ================= ============================= ==========================
+
+10 instances per compute in tester network
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+There are 20 micro instances (10 per compute node). Each instance is
+Cirros-based with 1 VCPU and 64MB RAM. Instances are plugged into `east`
+network.
+
+ ================= ============================= ==========================
+  Latency, ms [σ]   TCP throughput, Mbits/s [σ]   UDP throughput, kpps [σ]
+ ================= ============================= ==========================
+  2.7 [1.0]         4930 [499]                    101 [3.6]
+ ================= ============================= ==========================
+
+30 instances per compute in tester network
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+There are 60 micro instances (30 per compute node). Each instance is
+Cirros-based with 1 VCPU and 64MB RAM. Instances are plugged into `east`
+network.
+
+ ================= ============================= ==========================
+  Latency, ms [σ]   TCP throughput, Mbits/s [σ]   UDP throughput, kpps [σ]
+ ================= ============================= ==========================
+  5.4 [2.1]         2470 [130]                    88 [2.3]
+ ================= ============================= ==========================
+
+10 instances and 10 routers per compute
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+There are 20 micro instances (10 per compute node). Each instance is
+Cirros-based with 1 VCPU and 64MB RAM. Each instance is plugged into its own
+network, which in turn is plugged into its own router.
+
+ ================= ============================= ==========================
+  Latency, ms [σ]   TCP throughput, Mbits/s [σ]   UDP throughput, kpps [σ]
+ ================= ============================= ==========================
+  2.3 [0.8]         5690 [630]                    107 [5.0]
+ ================= ============================= ==========================
+
+30 instances and 30 routers per compute
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+There are 60 micro instances (30 per compute node). Each instance is
+Cirros-based with 1 VCPU and 64MB RAM. Each instance is plugged into its own
+network, which in turn is plugged into its own router.
+
+ ================= ============================= ==========================
+  Latency, ms [σ]   TCP throughput, Mbits/s [σ]   UDP throughput, kpps [σ]
+ ================= ============================= ==========================
+  2.2 [0.8]         5890 [707]                    85 [1.9]
+ ================= ============================= ==========================
+
+30 instances and 30 routers per compute, QEMU paused
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+There are 60 micro instances (30 per compute node). Each instance is
+Cirros-based with 1 VCPU and 64MB RAM. Each instance is plugged into its own
+network, which in turn is plugged into its own router. QEMU processes are
+paused to reduce noise load on CPU.
+
+ ================= ============================= ==========================
+  Latency, ms [σ]   TCP throughput, Mbits/s [σ]   UDP throughput, kpps [σ]
+ ================= ============================= ==========================
+  2.0 [0.7]         6640 [623]                    103 [4.6]
+ ================= ============================= ==========================
+
+30 instances, 30 routers and 500 SG rules per instance, QEMU paused
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+There are 60 micro instances (30 per compute node). Each instance is
+Cirros-based with 1 VCPU and 64MB RAM. Each instance is plugged into its own
+network, which in turn is plugged into its own router. QEMU processes are
+paused to reduce noise load on CPU. All instances have security group which
+contains 500 rules (this results in ~17k iptables rules)
+
+ ================= ============================= ==========================
+  Latency, ms [σ]   TCP throughput, Mbits/s [σ]   UDP throughput, kpps [σ]
+ ================= ============================= ==========================
+  2.0 [0.7]         6658 [576]                    97 [1.0]
+ ================= ============================= ==========================
+
+1000 security group rules
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In this case we assign security group containing 1000 rules to test instances.
+There is no measurable difference from the baseline data.
+
+ ================= ============================= ==========================
+  Latency, ms [σ]   TCP throughput, Mbits/s [σ]   UDP throughput, kpps [σ]
+ ================= ============================= ==========================
+  1.8 [0.6]         7039 [653]                    130 [2.0]
+ ================= ============================= ==========================
+
+Conclusion
+^^^^^^^^^^
+
+It appears that two things have most of impact:
+ * neighbours, which produce slight CPU load
+ * OVS flows: the more flows, the less performance. Adding 30 instances to
+   compute node results in 180 new flows.
+
+.. image:: chart_latency.png
+
+The worst case for latency is when instances are co-located in the same network
+as the tester. While presence of security group rules have no measurable effect
+at all.
+
+.. image:: chart_tcp.png
+
+The worst case again when instances are co-located in the same network. CPU
+noise also affects - the throughput reduces on ~10%.
+
+.. image:: chart_udp.png
+
+For UDP metric the most impact is introduced by CPU noise and presence of OVS
+rules. With security group rules the numbers are the same as in the baseline.

doc/source/test_results/neutron_features/resource_density/router_vm.hot

@@ -0,0 +1,41 @@
+heat_template_version: 2013-05-23
+
+resources:
+  east_private_net:
+    type: OS::Neutron::Net
+    properties:
+      name: my_net_east
+
+  east_private_subnet:
+    type: OS::Neutron::Subnet
+    properties:
+      network_id: { get_resource: east_private_net }
+      cidr: 10.1.0.0/16
+
+  router:
+    type: OS::Neutron::Router
+    properties:
+      external_gateway_info:
+        network: admin_floating_net
+
+  router_interface:
+    type: OS::Neutron::RouterInterface
+    properties:
+      router_id: { get_resource: router }
+      subnet_id: { get_resource: east_private_subnet }
+
+  server_1:
+    type: OS::Nova::Server
+    properties:
+      name: server_1
+      image: TestVM
+      flavor: m1.micro
+      networks:
+        - port: { get_resource: port_1 }
+
+  port_1:
+    type: OS::Neutron::Port
+    properties:
+      network_id: { get_resource: east_private_net }
+      fixed_ips:
+        - subnet_id: { get_resource: east_private_subnet }