openstack/performance-docs
Code Issues Proposed changes

Extend test plan and results for OpenStack networking

Browse Source 
Add test cases to measure throughput for cases when instances are
located on the same compute node.

Change-Id: I50156b6b0ac4c66fd3481e897785bcf771f8cf55
This commit is contained in:
Ilya Shakhat 2016-07-13 16:03:57 +03:00
parent b665acbc71
commit 83edf5219d
12 changed files with 439 additions and 32 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

112
doc/source/test_plans/tenant_networking/plan.rst
View File

@ -105,16 +105,43 @@ This section describes installed software.
| virtual routers | | e.g. legacy or HA or DVR |
+-----------------+-------+---------------------------+
.. _openstack_tenant_networking_test_plan_l2_dense:
Test Case 1: single node L2 instance-to-instance performance
------------------------------------------------------------
Description
^^^^^^^^^^^
This test case is executed on a single pair of instances launched on *one
compute node* and plugged into the same tenant network. The measurement is
done during 60 seconds, average values are calculated.
List of performance metrics
^^^^^^^^^^^^^^^^^^^^^^^^^^^
======== ================ ================= ===========================================
Priority Value Measurement Units Description
======== ================ ================= ===========================================
1 Latency ms The network latency
1 TCP bandwidth Mbits/s TCP network bandwidth
1 UDP bandwidth packets per sec Number of UDP packets with 32 bytes payload
2 UDP delay jitter ms Packet delay variation
2 UDP packet loss % Percentage of lost UDP packets
2 TCP retransmits packets per sec Number of retransmitted TCP packets
======== ================ ================= ===========================================
.. _openstack_tenant_networking_test_plan_l2:
Test Case 1: L2 instance-to-instance performance
Test Case 2: L2 instance-to-instance performance
------------------------------------------------
Description
^^^^^^^^^^^
This test case is executed on a single pair of instances launched on different
compute nodes and plugged into the same tenant network. The measurement is
This test case is executed on a single pair of instances launched on *different
compute nodes* and plugged into the same tenant network. The measurement is
done during 60 seconds, average values are calculated.
List of performance metrics
@ -134,7 +161,7 @@ Priority Value Measurement Units Description
.. _openstack_tenant_networking_test_plan_l2_concurrent:
Test Case 2: L2 concurrent performance
Test Case 3: L2 concurrent performance
--------------------------------------
Description
@ -166,16 +193,44 @@ Priority Value Measurement Units Description
======== ================ ================= ===========================================
.. _openstack_tenant_networking_test_plan_l3_east_west_dense:
Test Case 4: single node L3 east-west instance-to-instance performance
----------------------------------------------------------------------
Description
^^^^^^^^^^^
This test case is executed on a single pair of instances launched on *one
compute node* and plugged into different tenant networks. Networks are
connected by a single router. The measurement is done during 60 seconds,
average values are calculated.
List of performance metrics
^^^^^^^^^^^^^^^^^^^^^^^^^^^
======== ================ ================= ===========================================
Priority Value Measurement Units Description
======== ================ ================= ===========================================
1 Latency ms The network latency
1 TCP bandwidth Mbits/s TCP network bandwidth
1 UDP bandwidth packets per sec Number of UDP packets with 32 bytes payload
2 UDP delay jitter ms Packet delay variation
2 UDP packet loss % Percentage of lost UDP packets
2 TCP retransmits packets per sec Number of retransmitted TCP packets
======== ================ ================= ===========================================
.. _openstack_tenant_networking_test_plan_l3_east_west:
Test Case 3: L3 east-west instance-to-instance performance
Test Case 5: L3 east-west instance-to-instance performance
----------------------------------------------------------
Description
^^^^^^^^^^^
This test case is executed on a single pair of instances launched on different
compute nodes and plugged into different tenant networks. Networks are
This test case is executed on a single pair of instances launched on *different
compute nodes* and plugged into different tenant networks. Networks are
connected by a single router. The measurement is done during 60 seconds,
average values are calculated.
@ -196,7 +251,7 @@ Priority Value Measurement Units Description
.. _openstack_tenant_networking_test_plan_l3_east_west_concurrent:
Test Case 4: L3 east-west concurrent performance
Test Case 6: L3 east-west concurrent performance
------------------------------------------------
Description
@ -230,16 +285,45 @@ Priority Value Measurement Units Description
======== ================ ================= ===========================================
.. _openstack_tenant_networking_test_plan_l3_north_south_dense:
Test Case 7: single node L3 north-south instance-to-instance performance
------------------------------------------------------------------------
Description
^^^^^^^^^^^
This test case is executed on a single pair of instances launched on *one
compute node* and plugged into different tenant networks. Each networks is
connected to own router, thus traffic goes via external network. The
destination instance is reached by floating IP.
The measurement is done during 60 seconds, average values are calculated.
List of performance metrics
^^^^^^^^^^^^^^^^^^^^^^^^^^^
======== ================ ================= ===========================================
Priority Value Measurement Units Description
======== ================ ================= ===========================================
1 Latency ms The network latency
1 TCP bandwidth Mbits/s TCP network bandwidth
1 UDP bandwidth packets per sec Number of UDP packets with 32 bytes payload
2 UDP delay jitter ms Packet delay variation
2 UDP packet loss % Percentage of lost UDP packets
2 TCP retransmits packets per sec Number of retransmitted TCP packets
======== ================ ================= ===========================================
.. _openstack_tenant_networking_test_plan_l3_north_south:
Test Case 5: L3 north-south instance-to-instance performance
Test Case 8: L3 north-south instance-to-instance performance
------------------------------------------------------------
Description
^^^^^^^^^^^
This test case is executed on a single pair of instances launched on different
compute nodes and plugged into different tenant networks. Each networks is
This test case is executed on a single pair of instances launched on *different
compute nodes* and plugged into different tenant networks. Each networks is
connected to own router, thus traffic goes via external network. The
destination instance is reached by floating IP.
The measurement is done during 60 seconds, average values are calculated.
@ -261,7 +345,7 @@ Priority Value Measurement Units Description
.. _openstack_tenant_networking_test_plan_l3_north_south_concurrent:
Test Case 6: L3 north-south concurrent performance
Test Case 9: L3 north-south concurrent performance
--------------------------------------------------
Description
@ -298,8 +382,8 @@ Priority Value Measurement Units Description
.. _openstack_tenant_networking_test_plan_qos:
Test Case 7: Neutron QoS testing
--------------------------------
Test Case 10: Neutron QoS testing
---------------------------------
Description
^^^^^^^^^^^

2
doc/source/test_results/tenant_networking/neutron_vxlan_dvr_200/index.rst
View File

@ -38,8 +38,10 @@ Reports
.. toctree::
:maxdepth: 2
perf_l2_dense/index
perf_l2/index
full_l2/index
perf_l3_east_west_dense/index
perf_l3_east_west/index
full_l3_east_west/index
perf_l3_north_south/index

22
doc/source/test_results/tenant_networking/neutron_vxlan_dvr_200/perf_l2/index.rst
View File

@ -62,7 +62,7 @@ Ping
**Stats**:
============= ======== ======== ========
Metric Min Avg Max
Metric Min Avg Max
============= ======== ======== ========
ping_icmp, ms 0.27 0.56 1.55
============= ======== ======== ========
@ -70,7 +70,7 @@ ping_icmp, ms 0.27 0.56 1.55
**SLA**:
========================= =========== ================== ========
Expression Concurrency Node Result
Expression Concurrency Node Result
========================= =========== ================== ========
stats.ping_icmp.avg < 2.0 1 node-25.domain.tld OK
========================= =========== ================== ========
@ -94,7 +94,7 @@ TCP
**Stats**:
================= ======== ======== ========
Metric Min Avg Max
Metric Min Avg Max
================= ======== ======== ========
bandwidth, Mbit/s 5559.96 6875.94 7930.98
retransmits 26
@ -103,7 +103,7 @@ retransmits 26
**SLA**:
========================== =========== ================== ========
Expression Concurrency Node Result
Expression Concurrency Node Result
========================== =========== ================== ========
stats.bandwidth.avg > 5000 1 node-25.domain.tld OK
stats.retransmits.max < 10 1 node-25.domain.tld FAIL
@ -129,16 +129,18 @@ UDP
**Stats**:
============ ======== ======== ========
Metric Min Avg Max
============ ======== ======== ========
packets, pps 184290 206668 223370
============ ======== ======== ========
============ ========= ========= =========
Metric Min Avg Max
============ ========= ========= =========
loss, % 0.019
jitter, ms 0.003
packets, pps 184290 206668 223370
============ ========= ========= =========
**SLA**:
========================== =========== ================== ========
Expression Concurrency Node Result
Expression Concurrency Node Result
========================== =========== ================== ========
stats.packets.avg > 100000 1 node-25.domain.tld OK
========================== =========== ================== ========

4
doc/source/test_results/tenant_networking/neutron_vxlan_dvr_200/perf_l2_dense/23d521f3-d5eb-444a-9dc2-d05ee6e9461d.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 16 KiB

4
doc/source/test_results/tenant_networking/neutron_vxlan_dvr_200/perf_l2_dense/4f806542-e1fc-4d45-a23f-9c10e77270b3.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 28 KiB

4
doc/source/test_results/tenant_networking/neutron_vxlan_dvr_200/perf_l2_dense/94599950-242d-4c96-8c5b-cd725edf1970.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 45 KiB

146
doc/source/test_results/tenant_networking/neutron_vxlan_dvr_200/perf_l2_dense/index.rst Normal file
View File

@ -0,0 +1,146 @@
.. _openstack_l2_performance_dense:
OpenStack L2 Performance within single compute node
***************************************************
In this scenario Shaker launches 1 pair of instances in the same tenant
network. Both instances are hosted on the same compute node. The traffic goes
within the tenant network (L2 domain).
**Scenario**:
.. code-block:: yaml
deployment:
accommodation:
- pair
- double_room
- compute_nodes: 1
template: l2.hot
description: In this scenario Shaker launches 1 pair of instances in the same tenant
network. Both instances are hosted on the same compute node. The traffic goes within
the tenant network (L2 domain).
execution:
tests:
- class: flent
method: ping
sla:
- '[type == ''agent''] >> (stats.ping_icmp.avg < 2.0)'
time: 10
title: Ping
- class: iperf3
sla:
- '[type == ''agent''] >> (stats.bandwidth.avg > 5000)'
- '[type == ''agent''] >> (stats.retransmits.max < 10)'
title: TCP
- bandwidth: 0
class: iperf3
datagram_size: 32
sla:
- '[type == ''agent''] >> (stats.packets.avg > 100000)'
title: UDP
udp: true
title: OpenStack L2 Performance
Ping
====
**Test Specification**:
.. code-block:: yaml
class: flent
method: ping
sla:
- '[type == ''agent''] >> (stats.ping_icmp.avg < 2.0)'
time: 10
title: Ping
.. image:: 23d521f3-d5eb-444a-9dc2-d05ee6e9461d.*
**Stats**:
============= ========= ========= =========
Metric Min Avg Max
============= ========= ========= =========
ping_icmp, ms 0.200 0.317 0.666
============= ========= ========= =========
**SLA**:
========================= =========== ================== =========
Expression Concurrency Node Result
========================= =========== ================== =========
stats.ping_icmp.avg < 2.0 1 node-17.domain.tld OK
========================= =========== ================== =========
TCP
===
**Test Specification**:
.. code-block:: yaml
class: iperf3
interval: 1
sla:
- '[type == ''agent''] >> (stats.bandwidth.avg > 5000)'
- '[type == ''agent''] >> (stats.retransmits.max < 10)'
title: TCP
.. image:: 94599950-242d-4c96-8c5b-cd725edf1970.*
**Stats**:
================= ========= ========= =========
Metric Min Avg Max
================= ========= ========= =========
bandwidth, Mbit/s 13009.930 16494.578 24087.429
retransmits
================= ========= ========= =========
**SLA**:
========================== =========== ================== =========
Expression Concurrency Node Result
========================== =========== ================== =========
stats.bandwidth.avg > 5000 1 node-17.domain.tld OK
stats.retransmits.max < 10 1 node-17.domain.tld OK
========================== =========== ================== =========
UDP
===
**Test Specification**:
.. code-block:: yaml
bandwidth: 0
class: iperf3
datagram_size: 32
interval: 1
sla:
- '[type == ''agent''] >> (stats.packets.avg > 100000)'
title: UDP
udp: true
.. image:: 4f806542-e1fc-4d45-a23f-9c10e77270b3.*
**Stats**:
============ ========= ========= =========
Metric Min Avg Max
============ ========= ========= =========
loss, % 0.000
jitter, ms 0.004
packets, pps 160720 166747 179430
============ ========= ========= =========
**SLA**:
========================== =========== ================== =========
Expression Concurrency Node Result
========================== =========== ================== =========
stats.packets.avg > 100000 1 node-17.domain.tld OK
========================== =========== ================== =========

18
doc/source/test_results/tenant_networking/neutron_vxlan_dvr_200/perf_l3_east_west/index.rst
View File

@ -71,7 +71,7 @@ ping_icmp, ms 0.54 0.71 1.24
**SLA**:
========================= =========== =================== ========
Expression Concurrency Node Result
Expression Concurrency Node Result
========================= =========== =================== ========
stats.ping_icmp.avg < 2.0 1 node-164.domain.tld OK
========================= =========== =================== ========
@ -95,7 +95,7 @@ TCP
**Stats**:
================= ======== ======== ========
Metric Min Avg Max
Metric Min Avg Max
================= ======== ======== ========
bandwidth, Mbit/s 5405.46 5812.68 7309.81
retransmits 13
@ -104,7 +104,7 @@ retransmits 13
**SLA**:
========================== =========== =================== ========
Expression Concurrency Node Result
Expression Concurrency Node Result
========================== =========== =================== ========
stats.bandwidth.avg > 5000 1 node-164.domain.tld OK
stats.retransmits.max < 10 1 node-164.domain.tld FAIL
@ -130,11 +130,13 @@ UDP
**Stats**:
============ ======== ======== ========
Metric Min Avg Max
============ ======== ======== ========
packets, pps 138870 153702 165970
============ ======== ======== ========
============ ========= ========= =========
Metric Min Avg Max
============ ========= ========= =========
loss, % 0.005
jitter, ms 0.004
packets, pps 138870 153702 165970
============ ========= ========= =========
**SLA**:

4
doc/source/test_results/tenant_networking/neutron_vxlan_dvr_200/perf_l3_east_west_dense/42f40fed-1f50-41cb-9bf7-8f7043164832.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 16 KiB

4
doc/source/test_results/tenant_networking/neutron_vxlan_dvr_200/perf_l3_east_west_dense/81bd6b54-c9ef-46b8-8325-b13004b9d651.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 29 KiB

4
doc/source/test_results/tenant_networking/neutron_vxlan_dvr_200/perf_l3_east_west_dense/c83bcfac-5181-4dd7-a808-250f230e144b.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 45 KiB

147
doc/source/test_results/tenant_networking/neutron_vxlan_dvr_200/perf_l3_east_west_dense/index.rst Normal file
View File

@ -0,0 +1,147 @@
.. _openstack_l3_east_west_performance_dense:
OpenStack L3 East-West Performance within single compute node
*************************************************************
In this scenario Shaker launches 1 pair of instances, both are hosted on the
same compute node. Instances are connected to one of 2 tenant networks, which
plugged into single router. The traffic goes from one network to the other (L3
east-west).
**Scenario**:
.. code-block:: yaml
deployment:
accommodation:
- pair
- double_room
- compute_nodes: 1
template: l3_east_west.hot
description: In this scenario Shaker launches 1 pair of instances, both are hosted on the
same compute node. Instances are connected to one of 2 tenant networks, which
plugged into single router. The traffic goes from one network to the other (L3 east-west).
execution:
tests:
- class: flent
method: ping
sla:
- '[type == ''agent''] >> (stats.ping_icmp.avg < 2.0)'
time: 10
title: Ping
- class: iperf3
sla:
- '[type == ''agent''] >> (stats.bandwidth.avg > 5000)'
- '[type == ''agent''] >> (stats.retransmits.max < 10)'
title: TCP
- bandwidth: 0
class: iperf3
datagram_size: 32
sla:
- '[type == ''agent''] >> (stats.packets.avg > 100000)'
title: UDP
udp: true
title: OpenStack L3 East-West Performance
Ping
====
**Test Specification**:
.. code-block:: yaml
class: flent
method: ping
sla:
- '[type == ''agent''] >> (stats.ping_icmp.avg < 2.0)'
time: 10
title: Ping
.. image:: 42f40fed-1f50-41cb-9bf7-8f7043164832.*
**Stats**:
============= ========= ========= =========
Metric Min Avg Max
============= ========= ========= =========
ping_icmp, ms 0.215 0.346 0.582
============= ========= ========= =========
**SLA**:
========================= =========== ================= =========
Expression Concurrency Node Result
========================= =========== ================= =========
stats.ping_icmp.avg < 2.0 1 node-7.domain.tld OK
========================= =========== ================= =========
TCP
===
**Test Specification**:
.. code-block:: yaml
class: iperf3
interval: 1
sla:
- '[type == ''agent''] >> (stats.bandwidth.avg > 5000)'
- '[type == ''agent''] >> (stats.retransmits.max < 10)'
title: TCP
.. image:: c83bcfac-5181-4dd7-a808-250f230e144b.*
**Stats**:
================= ========= ========= =========
Metric Min Avg Max
================= ========= ========= =========
bandwidth, Mbit/s 12611.771 16735.987 20540.142
retransmits
================= ========= ========= =========
**SLA**:
========================== =========== ================= =========
Expression Concurrency Node Result
========================== =========== ================= =========
stats.bandwidth.avg > 5000 1 node-7.domain.tld OK
stats.retransmits.max < 10 1 node-7.domain.tld OK
========================== =========== ================= =========
UDP
===
**Test Specification**:
.. code-block:: yaml
bandwidth: 0
class: iperf3
datagram_size: 32
interval: 1
sla:
- '[type == ''agent''] >> (stats.packets.avg > 100000)'
title: UDP
udp: true
.. image:: 81bd6b54-c9ef-46b8-8325-b13004b9d651.*
**Stats**:
============ ========= ========= =========
Metric Min Avg Max
============ ========= ========= =========
loss, % 0.000
jitter, ms 0.007
packets, pps 121030 131170 136500
============ ========= ========= =========
**SLA**:
========================== =========== ================= =========
Expression Concurrency Node Result
========================== =========== ================= =========
stats.packets.avg > 100000 1 node-7.domain.tld OK
========================== =========== ================= =========