openstack/performance-docs
Code Issues Proposed changes

Add Fuel CCP upgrade testing

Browse Source 
Add Fuel CCP upgrade test plan and results

Change-Id: Ib68d19575967b8792a3ae693c6c06481e360d210
This commit is contained in:
Sergey Galkin 2017-03-29 18:07:22 +04:00
parent 2544c1e125
commit 6cbbd5acfd
13 changed files with 1754 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

189
doc/source/test_plans/fuel_ccp_upgrade/plan.rst Normal file
View File

@ -0,0 +1,189 @@
.. _fuel_ccp_upgrade_test_plan:
==================================================================
Fuel Containerized Control Plane upgradability performance testing
==================================================================
:status: **ready**
:version: 1.0
:Abstract:
This test plan aims to provide set of tests to identify OpenStack
performance against given containerized OpenStack cloud (installed
on the top of pre-deployed Kubernetes cluster) using simple minimalistic set
of Rally tests during upgrade from Mitaka to Newton.
Test Plan
=========
This document is inspired by
:ref:`openstack_control_plane_performance_test_plan`, and aims to cover
baseline cloud operations and extend this test suite to verify containerized
deployment approach. As :ref:`openstack_control_plane_performance_test_plan`
this test plan covers basic cloud operation VMs creation, as well as
long-running test suites to verify cloud stability during update Open
Stack from Mitaka to Newton.
Test Environment
----------------
Preparation
^^^^^^^^^^^
This test plan is performed against existing OpenStack cloud installed on top
of pre-deployed Kubernetes cluster with `fuel-ccp`_ tool with pre-installed Rally
framework.
.. _fuel-ccp: http://fuel-ccp.readthedocs.io/en/latest/
Environment description
^^^^^^^^^^^^^^^^^^^^^^^
The environment description includes hardware specification of servers,
network parameters, operation system and OpenStack deployment characteristics.
Hardware
~~~~~~~~
This section contains list of all types of hardware nodes (table below is
an example).
+-----------+-------+----------------------------------------------------+
| Parameter | Value | Comments |
+-----------+-------+----------------------------------------------------+
| model | | e.g. Supermicro X9SRD-F |
+-----------+-------+----------------------------------------------------+
| CPU | | e.g. 6 x Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz |
+-----------+-------+----------------------------------------------------+
| role | | e.g. compute or network |
+-----------+-------+----------------------------------------------------+
Network
~~~~~~~
This section contains list of interfaces and network parameters. For
complicated cases this section may include topology diagram and switch
parameters (table below is an example).
+------------------+-------+-------------------------+
| Parameter | Value | Comments |
+------------------+-------+-------------------------+
| network role | | e.g. provider or public |
+------------------+-------+-------------------------+
| card model | | e.g. Intel |
+------------------+-------+-------------------------+
| driver | | e.g. ixgbe |
+------------------+-------+-------------------------+
| speed | | e.g. 10G or 1G |
+------------------+-------+-------------------------+
| MTU | | e.g. 9000 |
+------------------+-------+-------------------------+
| offloading modes | | e.g. default |
+------------------+-------+-------------------------+
Software
~~~~~~~~
This section describes installed software (table below is an example).
+-----------------+-------+---------------------------+
| Parameter | Value | Comments |
+-----------------+-------+---------------------------+
| OS | | e.g. Ubuntu 14.04.3 |
+-----------------+-------+---------------------------+
| OpenStack | | e.g. Mitaka |
+-----------------+-------+---------------------------+
| Hypervisor | | e.g. KVM |
+-----------------+-------+---------------------------+
| Neutron plugin | | e.g. ML2 + OVS |
+-----------------+-------+---------------------------+
| L2 segmentation | | e.g. VLAN / VxLAN / GRE |
+-----------------+-------+---------------------------+
| virtual routers | | e.g. HA / DVR |
+-----------------+-------+---------------------------+
Test tool
---------
**Rally** is a benchmarking tool that was designed specifically for OpenStack
API testing. To make this possible, **Rally** automates and unifies multi-node
OpenStack deployment, cloud verification, benchmarking & profiling. This is a
simple way to check cloud workability and performance of control plane
operations running on it.
Test Case 1: Boot and delete server during Open Stack update
------------------------------------------------------------
Description
^^^^^^^^^^^
The most user-facing control plane operation is new virtual machine creation.
This scenario covers the most basic OpenStack server creation to present the
baseline numbers for Nova (OpenStack Compute) control plane.
Parameters
^^^^^^^^^^
+-------------------------+-----------------------------------------+
|Name | Description |
+=========================+=========================================+
|IMAGE | Image from which boot server |
+-------------------------+-----------------------------------------+
|FLAVOR | Flavor type from which boot server |
+-------------------------+-----------------------------------------+
|ASSIGN_NIC | Bool, whether or not to auto assign |
| | NICs in Rally scenario |
+-------------------------+-----------------------------------------+
|CONCURRENCY | Amount of parallel executors |
+-------------------------+-----------------------------------------+
|ITERATIONS | Total amount of iterations processed by |
| | all executors |
+-------------------------+-----------------------------------------+
List of performance metrics
^^^^^^^^^^^^^^^^^^^^^^^^^^^
+----------+-------+-------------------+---------------------------+
| Priority | Value | Measurement Units | Description |
+==========+=======+===================+===========================+
| 1 | | sec | Time of atomic operations |
+----------+-------+-------------------+---------------------------+
Measuring performance values
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1. Create server with FLAVOR flavor from IMAGE image through Nova API
2. Delete server through Nova API.
These 2 steps executed successively in CONCURRENCY parallel executors.
ASSIGN_NIC parameter reflects Rally scenario configuration whether to assign
NIC to the booted server in automatic fashion.
One cycle of these 2 steps is called an iteration.
ITERATIONS is a total amount of iterations which was processed by executors.
During this testing
At the end of this test case you should calculate average, 90% percentile,
50% percentile, minimum and maximum for each step. You need to fill the
following tables with calculated values:
Nova
----
+---------------+------+--------+--------+-------+-------+
| Operation | Mean | 90%ile | 50%ile | Max | Min |
| | (sec)| (sec) | (sec) | (sec) | (sec) |
+===============+======+========+========+=======+=======+
| create_server | | | | | |
+---------------+------+--------+--------+-------+-------+
| delete_server | | | | | |
+---------------+------+--------+--------+-------+-------+
Example of Rally scenario configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. literalinclude:: test_plans/021-nova-boot-and-delete-baseline.yaml
:language: bash

BIN
doc/source/test_results/fuel_ccp_upgrade/configs/Network_Scheme.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 9.7 KiB

16
doc/source/test_results/fuel_ccp_upgrade/configs/ccp.yaml Normal file
View File

@ -0,0 +1,16 @@
builder:
push: true
no_cache: false
registry:
address: "172.20.9.32:5000/env-1"
repositories:
skip_empty: True
kubernetes:
server: http://127.0.0.1:8080
image_pull_policy: Always
---
!include
- versions.yaml
- topology.yaml
- configs.yaml
- repos.yaml

63
doc/source/test_results/fuel_ccp_upgrade/configs/configs.yaml Normal file
View File

@ -0,0 +1,63 @@
configs:
private_interface: p1p1.602
public_interface: p1p1.602
ingress:
enabled: true
glance:
tls:
enabled: false
bootstrap:
enable: true
neutron:
tls:
enabled: false
physnets:
- name: "physnet1"
bridge_name: "br-ex"
interface: "p1p1"
flat: true
vlan_range: false
dpdk: false
bootstrap:
internal:
enable: true
external:
enable: true
net_name: ext-net
subnet_name: ext-subnet
physnet: physnet1
network: 10.144.0.0/12
gateway: 10.144.0.1
nameserver: 10.144.0.1
pool:
start: 10.144.1.0
end: 10.159.255.250
keystone:
debug: true
security:
tls:
create_certificates: false
etcd:
tls:
enabled: false
db:
tls:
enabled: false
rabbitmq:
tls:
enabled: false
keystone:
tls:
enabled: false
cinder:
tls:
enabled: false
heat:
tls:
enabled: false
nova:
tls:
enabled: false
horizon:
tls:
enabled: false

34
doc/source/test_results/fuel_ccp_upgrade/configs/create_and_delete.yaml Normal file
View File

@ -0,0 +1,34 @@
---
NovaServers.boot_and_delete_server:
-
args:
flavor:
name: "m1.tiny"
image:
name: "cirros"
runner:
type: "constant"
times: 1530
concurrency: 5
context:
users:
tenants: 2
users_per_tenant: 2
quotas:
nova:
instances: -1
cores: -1
ram: -1
floating_ips: -1
security_groups: -1
security_group_rules: -1
neutron:
network: -1
subnet: -1
port: -1
network:
start_cidr: {{ "100.1.0.0/21" if gre_enabled else "1.0.0.0/21" }}
networks_per_tenant: 1
sla:
failure_rate:
max: 0

16
doc/source/test_results/fuel_ccp_upgrade/configs/rally_deployment.json Normal file
View File

@ -0,0 +1,16 @@
{
"admin": {
"password": "password",
"project_domain_name": "default",
"project_name": "admin",
"user_domain_name": "default",
"username": "admin"
},
"auth_url": "https://identity.external:8443/v3",
"endpoint": null,
"endpoint_type": null,
"https_cacert": "",
"https_insecure": true,
"region_name": "",
"type": "ExistingCloud"
}

38
doc/source/test_results/fuel_ccp_upgrade/configs/repos.yaml Normal file
View File

@ -0,0 +1,38 @@
repositories:
repos:
- git_url: https://git.openstack.org/openstack/fuel-ccp-ceph
name: fuel-ccp-ceph
- git_url: https://git.openstack.org/openstack/fuel-ccp-debian-base
name: fuel-ccp-debian-base
- git_url: https://git.openstack.org/openstack/fuel-ccp-entrypoint
name: fuel-ccp-entrypoint
- git_url: https://git.openstack.org/openstack/fuel-ccp-etcd
name: fuel-ccp-etcd
- git_url: https://git.openstack.org/openstack/fuel-ccp-galera
name: fuel-ccp-galera
- git_url: https://git.openstack.org/openstack/fuel-ccp-glance
name: fuel-ccp-glance
- git_url: https://git.openstack.org/openstack/fuel-ccp-heat
name: fuel-ccp-heat
- git_url: https://git.openstack.org/openstack/fuel-ccp-horizon
name: fuel-ccp-horizon
- git_url: https://git.openstack.org/openstack/fuel-ccp-ironic
name: fuel-ccp-ironic
- git_url: https://git.openstack.org/openstack/fuel-ccp-keystone
name: fuel-ccp-keystone
- git_url: https://git.openstack.org/openstack/fuel-ccp-mariadb
name: fuel-ccp-mariadb
- git_url: https://git.openstack.org/openstack/fuel-ccp-memcached
name: fuel-ccp-memcached
- git_url: https://git.openstack.org/openstack/fuel-ccp-murano
name: fuel-ccp-murano
- git_url: https://git.openstack.org/openstack/fuel-ccp-neutron
name: fuel-ccp-neutron
- git_url: https://git.openstack.org/openstack/fuel-ccp-nginx
name: fuel-ccp-nginx
- git_url: https://git.openstack.org/openstack/fuel-ccp-nova
name: fuel-ccp-nova
- git_url: https://git.openstack.org/openstack/fuel-ccp-openstack-base
name: fuel-ccp-openstack-base
- git_url: https://git.openstack.org/openstack/fuel-ccp-rabbitmq
name: fuel-ccp-rabbitmq

92
doc/source/test_results/fuel_ccp_upgrade/configs/topology.yaml Normal file
View File

@ -0,0 +1,92 @@
nodes:
# node[1-3]: ETCD
# node[4-5]: Kubernetes master
node([6|7|8])$: # 6-8
roles:
- controller
- openvswitch
- messaging
- db
node9$:
roles:
- etc
- backups
node[1-9][0-9]$: # 10-99
roles:
- compute
- openvswitch
node(1[0-4][0-9])$: # 100-149
roles:
- compute
- openvswitch
node(15[0-3])$: # 150 - 153
roles:
- compute
- openvswitch
replicas:
database: 3
etcd: 1
rpc: 3
notifications: 3
glance-api: 1
glance-registry: 1
keystone: 3
nova-api: 3
nova-scheduler: 3
nova-conductor: 3
neutron-server: 3
neutron-metadata-agent: 3
horizon: 3
heat-api: 3
heat-engine: 3
roles:
db:
- database
messaging:
- rpc
- notifications
controller:
- etcd
- glance-api
- glance-registry
- heat-api-cfn
- heat-api
- heat-engine
- horizon
- keystone
- memcached
- neutron-dhcp-agent
- neutron-l3-agent
- neutron-metadata-agent
- neutron-server
- nova-api
- nova-conductor
- nova-consoleauth
- nova-novncproxy
- nova-scheduler
compute:
- nova-compute
- nova-libvirt
openvswitch:
- neutron-openvswitch-agent
- openvswitch-db
- openvswitch-vswitchd
etc:
- etcd
backups:
- backup
services:
database:
service_def: galera
rpc:
service_def: rabbitmq
notifications:
service_def: rabbitmq

39
doc/source/test_results/fuel_ccp_upgrade/configs/versions-mitaka.yaml Normal file
View File

@ -0,0 +1,39 @@
images:
tag: mitaka
sources:
openstack/cinder:
git_ref: stable/mitaka
git_url: https://github.com/openstack/cinder.git
openstack/glance:
git_ref: stable/mitaka
git_url: https://github.com/openstack/glance.git
openstack/heat:
git_ref: stable/mitaka
git_url: https://github.com/openstack/heat.git
openstack/horizon:
git_ref: stable/mitaka
git_url: https://github.com/openstack/horizon.git
openstack/keystone:
git_ref: stable/mitaka
git_url: https://github.com/openstack/keystone.git
openstack/neutron:
git_ref: stable/mitaka
git_url: https://github.com/openstack/neutron.git
openstack/nova:
git_ref: stable/mitaka
git_url: https://github.com/openstack/nova.git
openstack/requirements:
git_ref: stable/mitaka
git_url: https://git.openstack.org/openstack/requirements.git
openstack/sahara-dashboard:
git_ref: stable/mitaka
git_url: https://git.openstack.org/openstack/sahara-dashboard.git
openstack/murano-dashboard:
git_url: http://git.openstack.org/openstack/murano-dashboard.git
git_ref: stable/mitaka
openstack/searchlight-ui:
git_url: https://git.openstack.org/openstack/searchlight-ui.git
git_ref: stable/mitaka
openstack/designate-dashboard:
git_url: https://git.openstack.org/openstack/designate-dashboard.git
git_ref: stable/mitaka

80
doc/source/test_results/fuel_ccp_upgrade/configs/versions-newton.yaml Normal file
View File

@ -0,0 +1,80 @@
images:
tag: mitaka
image_specs:
keystone:
tag: newton
horizon:
tag: newton
nova-upgrade:
tag: newton
nova-api:
tag: newton
nova-conductor:
tag: newton
nova-consoleauth:
tag: newton
nova-novncproxy:
tag: newton
nova-scheduler:
tag: newton
nova-compute:
tag: newton
nova-libvirt:
tag: newton
neutron-dhcp-agent:
tag: newton
neutron-l3-agent:
tag: newton
neutron-metadata-agent:
tag: newton
neutron-server:
tag: newton
neutron-openvswitch-agent:
tag: newton
glance-api:
tag: newton
glance-registry:
tag: newton
glance-upgrade:
tag: newton
sources:
openstack/cinder:
git_ref: stable/newton
git_url: https://github.com/openstack/cinder.git
openstack/glance:
git_ref: stable/newton
git_url: https://github.com/openstack/glance.git
openstack/heat:
git_ref: stable/newton
git_url: https://github.com/openstack/heat.git
openstack/horizon:
git_ref: stable/newton
git_url: https://github.com/openstack/horizon.git
openstack/keystone:
git_ref: stable/newton
git_url: https://github.com/openstack/keystone.git
openstack/neutron:
git_ref: stable/newton
git_url: https://github.com/openstack/neutron.git
openstack/nova:
git_ref: stable/newton
git_url: https://github.com/openstack/nova.git
openstack/requirements:
git_ref: stable/newton
git_url: https://git.openstack.org/openstack/requirements.git
openstack/sahara-dashboard:
git_ref: stable/newton
git_url: https://git.openstack.org/openstack/sahara-dashboard.git
openstack/murano-dashboard:
git_url: http://git.openstack.org/openstack/murano-dashboard.git
git_ref: stable/newton
openstack/searchlight-ui:
git_url: https://git.openstack.org/openstack/searchlight-ui.git
git_ref: stable/newton
openstack/designate-dashboard:
git_url: https://git.openstack.org/openstack/designate-dashboard.git
git_ref: stable/newton

144
doc/source/test_results/fuel_ccp_upgrade/index.rst Normal file
View File

@ -0,0 +1,144 @@
.. _`fuel_ccp_upgrade_test_report`:
==================================================================
Fuel Containerized Control Plane performance report during upgrade
==================================================================
:Abstract:
This document includes Fuel CCP control plane performance test results for
various environments (from 50 to 350 nodes). All tests have been performed
regarding :ref:`fuel_ccp_upgrade_test_plan`
Environment description
=======================
For Kubernetes pre-deployment `Kargo tool`_ was used. More information about
*fuel-ccp* and how it can be installed can be found in
`official documentation`_.
.. _Kargo tool: https://github.com/kubespray/kargo
.. _official documentation: http://fuel-ccp.readthedocs.io/en/latest/
Hardware configuration of each server
-------------------------------------
All servers have configuration describing in table below
.. table:: Description of servers hardware
+-------+----------------+-------------------------------+
|server |vendor,model |HP,DL380 Gen9 |
+-------+----------------+-------------------------------+
|CPU |vendor,model |Intel,E5-2680 v3 |
| +----------------+-------------------------------+
| |processor_count |2 |
| +----------------+-------------------------------+
| |core_count |12 |
| +----------------+-------------------------------+
| |frequency_MHz |2500 |
+-------+----------------+-------------------------------+
|RAM |vendor,model |HP,752369-081 |
| +----------------+-------------------------------+
| |amount_MB |262144 |
+-------+----------------+-------------------------------+
|NETWORK|interface_name |p1p1 |
| +----------------+-------------------------------+
| |vendor,model |Intel,X710 Dual Port |
| +----------------+-------------------------------+
| |bandwidth |10G |
+-------+----------------+-------------------------------+
|STORAGE|dev_name |/dev/sda |
| +----------------+-------------------------------+
| |vendor,model | | raid10 - HP P840 |
| | | | 12 disks EH0600JEDHE |
| +----------------+-------------------------------+
| |SSD/HDD |HDD |
| +----------------+-------------------------------+
| |size | 3,6TB |
+-------+----------------+-------------------------------+
Network configuration of each server
------------------------------------
All servers have same network configuration:
.. image:: configs/Network_Scheme.png
:alt: Network Scheme of the environment
:width: 650px
Fuel CCP configuration
----------------------
CCP main configuration
:download:`ccp.yaml <configs/ccp.yaml>`
CCP Cluster configuration
:download:`configs.yaml <configs/configs.yaml>`
CCP Cluster topology
:download:`topology.yaml <configs/topology.yaml>`
CCP repositories configuration
:download:`repos.yaml <configs/repos.yaml>`
CCP versions configuration for Mitaka
:download:`versions-mitaka.yaml <configs/versions-mitaka.yaml>`
CCP versions configuration for upgrade from Mitaka to Newton
:download:`versions-newton.yaml <configs/versions-newton.yaml>`
Rally configuration and scenario
--------------------------------
Rally deployment
:download:`rally_deployment.json <configs/rally_deployment.json>`
Rally scenario
:download:`create_and_delete.yaml <configs/create_and_delete.yaml>`
Upgrade and tests procedure
---------------------------
* Install fuel-ccp
* Rename ``ccp.yaml`` to ``/root/.ccp.ymal``
* Copy all yaml files to ``/root/``
* Rename ``versions-mitaka.yaml`` to ``versions.yaml``
* Execute ``ccp deploy`` and wait ok status for all services
* Create rally deployment with ``rally_deployment.json``
* Create role for rally ``openstack --insecure role create member``
* Start rally tests ``rally task start --task-args 'gre_enabled: true' ./create_and_delete.yaml``
* Rename ``versions-newton.yaml`` to ``versions.yaml``
* Execute ``ccp deploy`` and wait ok status for all services
* Wait tests results from rally
Test results
============
Test Case 1: Boot and delete server
-----------------------------------
The following set of results is dedicated to investigate how Nova installed
against Kubernetes cluster via fuel-ccp tool is behaving during update from
Mitaka to Newton. Upgrade started on 100 iteration and finished on the 156
iteration.
149 nodes OpenStack cluster, concurrency 5, 1530 iterations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NovaServers.boot_and_delete_server scenario in
:download:`upgrade_newton_to_mitaka.html <reports/upgrade_newton_to_mitaka.html>`
.. image:: reports/upgrade_newton_to_mitaka.png
:alt: Boot and delete servers Rally scenario (200 nodes)
:width: 650px
+----------------------+-----------+-----------+-----------+-----------+-----------+
| Operation | Median| 90%ile | 95%ile | Max | Min |
| | (sec) | (sec) | (sec) | (sec) | (sec) |
+======================+===========+===========+===========+===========+===========+
| boot_server | 11.879 | 16.753 | 18.708 | 48.84 | 7.563 |
+----------------------+-----------+-----------+-----------+-----------+-----------+
| delete_server | 2.593 | 4.754 | 4.898 | 6.829 | 2.313 |
+----------------------+-----------+-----------+-----------+-----------+-----------+
The control plane downtime of keystone upgrades will be avoided in the feature
versions of fuel-ccp

1043
doc/source/test_results/fuel_ccp_upgrade/reports/upgrade_newton_to_mitaka.html Normal file
View File

File diff suppressed because one or more lines are too long

BIN
doc/source/test_results/fuel_ccp_upgrade/reports/upgrade_newton_to_mitaka.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 36 KiB