x/browbeat

Charles Short 2ba39b30ab Refresh collectd for "train"

This commit does several things at once:

- Use ansible_distribution_major_version to detect which version of the
  EPEL repository. So we dont have to hard code the URL for either epel7
  or epel 8.
- Remove "stein" workaround for colelctd-openstack role. The "stein"
  workaround has been removed in favor of running the collectd daemon
  in a podman container.
- Drop opendaylight support for collectd since it is no longer
  suupported.
- Add the collectd playbook so we can run collectd in a centos 7
  container going forward for "train". This commit still needs
  to be tested on "stein" but it will probably work anyways.
- Add browbeat-containers to tox.ini for flake8
- Simplify detection of docker or podman for older versions of OSP.
(sai)
- Fixed typo from compute_compute to collectd_compute that caused failures on computes
- clear graphite_host in install/group_vars/all.yml
- Move container DockerFiles into brwobeat tree
- Conditionally copy required Dockerfiles to node instead of git clone
- Fix up some log file paths
- Use Docker/Podman depending on release
- Provide single interface(collectd.yml) which has container and baremetal playbooks
- Introduce variable collectd_container in install/group_vars/all
- remove unneeded selinux rebaelling (already running as priveleged) when running container
- remove unneed hostfs mount
- collectd container logs to file instead of STDOUT for easier debug
- add collectd-ping package to collectd-openstack Dockerfile
- Improve docs to reflect changes
- dynamically set rabbitmq and swift paths as well for tail plugin

Co-Authored-By: Sai Sindhur Malleni <smalleni@redhat.com>

Change-Id: I627a696f6f1240d96a0e1d85c26d59bbbfae2b1b
Signed-off-by: Charles Short <chucks@redhat.com>
Signed-off-by: Sai Sindhur Malleni <smalleni@redhat.com>

2019-11-05 08:08:37 -05:00

26 KiB

Raw Blame History

Usage

Run Browbeat performance tests from Undercloud

$ ssh undercloud-root
[root@ospd ~]# su - stack
[stack@ospd ~]$ cd browbeat/
[stack@ospd browbeat]$ . .browbeat-venv/bin/activate
(browbeat-venv)[stack@ospd browbeat]$ vi browbeat-config.yaml # Edit browbeat-config.yaml to control how many stress tests are run.
(browbeat-venv)[stack@ospd browbeat]$ ./browbeat.py <workload> #perfkit, rally, shaker or "all"

Running PerfKitBenchmarker

Note: PerfKitBenchmarker is disabled for Stein+ due to the lack of python3: support.

Many benchmarks work out of the box with Browbeat. You must ensure that your network is setup correctly to run those benchmarks. Currently tested benchmarks include: aerospike, bonnie++, cluster_boot, copy_throughput(cp,dd,scp), fio, iperf, mesh_network, mongodb_ycsb, netperf, object_storage_service, ping, scimark2, and sysbench_oltp.

To run Browbeat's PerfKit Benchmarks, you can start by viewing the tested benchmark's configuration in conf/browbeat-perfkit-complete.yaml. You must add them to your specific Browbeat config yaml file or enable/disable the benchmarks you wish to run in the default config file (browbeat-config.yaml). There are many flags exposed in the configuration files to tune how those benchmarks run. Additional flags are exposed in the source code of PerfKitBenchmarker available on the Google Cloud Github.

Example running only PerfKitBenchmarker benchmarks with Browbeat from browbeat-config.yaml:

(browbeat-venv)[stack@ospd browbeat]$ ./browbeat.py perfkit -s browbeat-config.yaml

Running Shaker

Running Shaker requires the shaker image to be built, which in turn requires instances to be able to access the internet. The playbooks for this installation have been described in the installation documentation but for the sake of convenience they are being mentioned here as well.

$ ansible-playbook -i hosts install/shaker_build.yml

Note

The playbook to setup networking is provided as an example only and might not work for you based on your underlay/overlay network setup. In such cases, the exercise of setting up networking for instances to be able to access the internet is left to the user.

Once the shaker image is built, you can run Shaker via Browbeat by filling in a few configuration options in the configuration file. The meaning of each option is summarized below:

shaker:

enabled: Boolean true or false, enable shaker or not
server: IP address of the shaker-server for agent to talk to (undercloud IP by default)
port: Port to connect to the shaker-server (undercloud port 5555 by default)
flavor: OpenStack instance flavor you want to use
join_timeout: Timeout in seconds for agents to join
sleep_before: Time in seconds to sleep before executing a scenario
sleep_after: Time in seconds to sleep after executing a scenario
shaker_region: OpenStack region you want to use
external_host: IP of a server for external tests (should have browbeat/util/shaker-external.sh executed on it previously and have iptables/firewalld/selinux allowing connections on the ports used by network testing tools netperf and iperf)

scenarios: List of scenarios you want to run

- name: Name for the scenario. It is used to create directories/files accordingly
enabled: Boolean true or false depending on whether or not you want to execute the scenario
density: Number of instances
compute: Number of compute nodes across which to spawn instances
placement: single_room would mean one instance per compute node and double_room would give you two instances per compute node
progression: null means all agents are involved, linear means execution starts with one agent and increases linearly, quadratic would result in quadratic growth in number of agents participating in the test concurrently
time: Time in seconds you want each test in the scenario file to run
file: The base shaker scenario file to use to override options (this would depend on whether you want to run L2, L3 E-W or L3 N-S tests and also on the class of tool you want to use such as flent or iperf3)

To analyze results sent to Elasticsearch (you must have Elasticsearch enabled and the IP of the Elasticsearch host provided in the browbeat configuration file), you can use the following playbook to setup some prebuilt dashboards for you:

$ ansible-playbook -i hosts install/kibana-visuals.yml

Alternatively you can create your own visualizations of specific shaker runs using some simple searches such as:

shaker_uuid: 97092334-34e8-446c-87d6-6a0f361b9aa8 AND record.concurrency: 1 AND result.result_type: bandwidth
shaker_uuid: c918a263-3b0b-409b-8cf8-22dfaeeaf33e AND record.concurrency:1 AND record.test:Bi-Directional

Interpreting Browbeat Results

By default results for each test will be placed in a timestamped folder results/ inside your Browbeat folder. Each run folder will contain output files from the various workloads and benchmarks that ran during that Browbeat run, as well as a report card that summarizes the results of the tests.

Browbeat for the most part tries to restrict itself to running tests, it will only exit with a nonzero return code if a workload failed to run. If, for example, Rally where to run but not be able to boot any instances on your cloud Browbeat would return with RC 0 without any complaints, only by looking into the Rally results for that Browbeat run would you determine that your cloud had a problem that made benchmarking it impossible.

Likewise if Rally manages to run at a snails pace, Browbeat will still exit without complaint. Be aware of this when running Browbeat and take the time to either view the contents of the results folder after a run. Or setup Elasticsearch and Kibana to view them more easily.

Working with Multiple Clouds

If you are running playbooks from your local machine you can run against more than one cloud at the same time. To do this, you should create a directory per-cloud and clone Browbeat into that specific directory:

[browbeat@laptop ~]$ mkdir cloud01; cd cloud01
[browbeat@laptop cloud01]$ git clone git@github.com:openstack/browbeat.git
...
[browbeat@laptop cloud01]$ cd browbeat/ansible
[browbeat@laptop ansible]$ ./generate_tripleo_hostfile.sh -t <cloud01-ip-address>
[browbeat@laptop ansible]$ ansible-playbook -i hosts (Your playbook you wish to run...)
[browbeat@laptop ansible]$ ssh -F ssh-config overcloud-controller-0  # Takes you to first controller

Repeat the above steps for as many clouds as you have to run playbooks against your clouds.

Compare software-metadata from two different runs

Browbeat's metadata is great to help build visuals in Kibana by querying on specific metadata fields, but sometimes we need to see what the difference between two builds might be. Kibana doesn't have a good way to show this, so we added an option to Browbeat CLI to query ElasticSearch.

To use :

$ python browbeat.py --compare software-metadata --uuid "browbeat-uuid-1" "browbeat-uuid-2"

Real world use-case, we had two builds in our CI that used the exact same DLRN hash, however the later build had a 10x performance hit for two Neutron operations, router-create and add-interface-to-router. Given we had exactly the same DLRN hash, the only difference could be how things were configured. Using this new code, we could quickly identify the difference -- TripleO enabled l3_ha.

Below is an example output of comparing metadata:

+-------------------------------------------------------------------------------------------------------------------------------------+
Host                 | Service              | Option               | Key                  | Old Value            | New Value
+-------------------------------------------------------------------------------------------------------------------------------------+
overcloud-controller-2 | nova                 | conductor            | workers              | 0                    | 12
overcloud-controller-2 | nova                 | DEFAULT              | metadata_workers     | 0                    | 12
overcloud-controller-2 | nova                 | DEFAULT              | my_ip                | 172.16.0.23          | 172.16.0.16
overcloud-controller-2 | nova                 | DEFAULT              | enabled_apis         | osapi_compute,metadata | metadata
overcloud-controller-2 | nova                 | DEFAULT              | osapi_compute_workers | 0                    | 12
overcloud-controller-2 | nova                 | neutron              | region_name          | RegionOne            | regionOne
overcloud-controller-2 | neutron-plugin       | ovs                  | local_ip             | 172.17.0.11          | 172.17.0.16
overcloud-controller-2 | neutron-plugin       | securitygroup        | firewall_driver      | openvswitch          | iptables_hybrid
overcloud-controller-2 | heat                 | DEFAULT              | num_engine_workers   | 0                    | 16
overcloud-controller-2 | keystone             | admin_workers        | processes            | 32                   |
overcloud-controller-2 | keystone             | admin_workers        | threads              | 1                    |
overcloud-controller-2 | keystone             | eventlet_server      | admin_workers        | 8                    | 12
overcloud-controller-2 | keystone             | eventlet_server      | public_workers       | 8                    | 12
overcloud-controller-2 | keystone             | oslo_messaging_notifications | driver               | messaging            | messagingv2
overcloud-controller-2 | keystone             | main_workers         | processes            | 32                   |
overcloud-controller-2 | keystone             | main_workers         | threads              | 1                    |
overcloud-controller-2 | keystone             | token                | provider             | uuid                 | fernet
overcloud-controller-2 | rabbitmq             | DEFAULT              | file                 | 65436                |
overcloud-controller-2 | mysql                | DEFAULT              | max                  | 4096                 |
overcloud-controller-2 | cinder               | DEFAULT              | exec_dirs            | /sbin,/usr/sbin,/bin,/usr/bin | /sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin,/usr/lpp/mmfs/bin
overcloud-controller-2 | cinder               | DEFAULT              | osapi_volume_workers | 32                   | 12
overcloud-controller-2 | glance               | DEFAULT              | bind_port            | 9191                 | 9292
overcloud-controller-2 | glance               | DEFAULT              | workers              | 32                   | 12
overcloud-controller-2 | glance               | DEFAULT              | log_file             | /var/log/glance/registry.log | /var/log/glance/cache.log
overcloud-controller-2 | glance               | ref1                 | auth_version         | 2                    | 3
overcloud-controller-2 | glance               | glance_store         | stores               | glance.store.http.Store,glance.store.swift.Store | http,swift
overcloud-controller-2 | glance               | glance_store         | os_region_name       | RegionOne            | regionOne
overcloud-controller-2 | gnocchi              | metricd              | workers              | 8                    | 12
overcloud-controller-2 | gnocchi              | storage              | swift_auth_version   | 2                    | 3
overcloud-controller-2 | neutron              | DEFAULT              | global_physnet_mtu   | 1496                 | 1500
overcloud-controller-2 | neutron              | DEFAULT              | rpc_workers          | 32                   | 12
overcloud-controller-2 | neutron              | DEFAULT              | api_workers          | 32                   | 12
overcloud-controller-1 | nova                 | conductor            | workers              | 0                    | 12
overcloud-controller-1 | nova                 | DEFAULT              | metadata_workers     | 0                    | 12
overcloud-controller-1 | nova                 | DEFAULT              | my_ip                | 172.16.0.11          | 172.16.0.23
overcloud-controller-1 | nova                 | DEFAULT              | enabled_apis         | osapi_compute,metadata | metadata
overcloud-controller-1 | nova                 | DEFAULT              | osapi_compute_workers | 0                    | 12
overcloud-controller-1 | nova                 | neutron              | region_name          | RegionOne            | regionOne
overcloud-controller-1 | neutron-plugin       | ovs                  | local_ip             | 172.17.0.15          | 172.17.0.11
overcloud-controller-1 | neutron-plugin       | securitygroup        | firewall_driver      | openvswitch          | iptables_hybrid
overcloud-controller-1 | heat                 | DEFAULT              | num_engine_workers   | 0                    | 16
overcloud-controller-1 | keystone             | admin_workers        | processes            | 32                   |
overcloud-controller-1 | keystone             | admin_workers        | threads              | 1                    |
overcloud-controller-1 | keystone             | eventlet_server      | admin_workers        | 8                    | 12
overcloud-controller-1 | keystone             | eventlet_server      | public_workers       | 8                    | 12
overcloud-controller-1 | keystone             | oslo_messaging_notifications | driver               | messaging            | messagingv2
overcloud-controller-1 | keystone             | main_workers         | processes            | 32                   |
overcloud-controller-1 | keystone             | main_workers         | threads              | 1                    |
overcloud-controller-1 | keystone             | token                | provider             | uuid                 | fernet
overcloud-controller-1 | rabbitmq             | DEFAULT              | file                 | 65436                |
overcloud-controller-1 | mysql                | DEFAULT              | max                  | 4096                 |
overcloud-controller-1 | cinder               | DEFAULT              | exec_dirs            | /sbin,/usr/sbin,/bin,/usr/bin | /sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin,/usr/lpp/mmfs/bin
overcloud-controller-1 | cinder               | DEFAULT              | osapi_volume_workers | 32                   | 12
overcloud-controller-1 | glance               | DEFAULT              | bind_port            | 9191                 | 9292
overcloud-controller-1 | glance               | DEFAULT              | workers              | 32                   | 12
overcloud-controller-1 | glance               | DEFAULT              | log_file             | /var/log/glance/registry.log | /var/log/glance/cache.log
overcloud-controller-1 | glance               | ref1                 | auth_version         | 2                    | 3
overcloud-controller-1 | glance               | glance_store         | stores               | glance.store.http.Store,glance.store.swift.Store | http,swift
overcloud-controller-1 | glance               | glance_store         | os_region_name       | RegionOne            | regionOne
overcloud-controller-1 | gnocchi              | metricd              | workers              | 8                    | 12
overcloud-controller-1 | gnocchi              | storage              | swift_auth_version   | 2                    | 3
overcloud-controller-1 | neutron              | DEFAULT              | global_physnet_mtu   | 1496                 | 1500
overcloud-controller-1 | neutron              | DEFAULT              | rpc_workers          | 32                   | 12
overcloud-controller-1 | neutron              | DEFAULT              | api_workers          | 32                   | 12
overcloud-controller-0 | nova                 | conductor            | workers              | 0                    | 12
overcloud-controller-0 | nova                 | DEFAULT              | metadata_workers     | 0                    | 12
overcloud-controller-0 | nova                 | DEFAULT              | my_ip                | 172.16.0.15          | 172.16.0.10
overcloud-controller-0 | nova                 | DEFAULT              | enabled_apis         | osapi_compute,metadata | metadata
overcloud-controller-0 | nova                 | DEFAULT              | osapi_compute_workers | 0                    | 12
overcloud-controller-0 | nova                 | neutron              | region_name          | RegionOne            | regionOne
overcloud-controller-0 | neutron-plugin       | ovs                  | local_ip             | 172.17.0.10          | 172.17.0.18
overcloud-controller-0 | neutron-plugin       | securitygroup        | firewall_driver      | openvswitch          | iptables_hybrid
overcloud-controller-0 | heat                 | DEFAULT              | num_engine_workers   | 0                    | 16
overcloud-controller-0 | keystone             | admin_workers        | processes            | 32                   |
overcloud-controller-0 | keystone             | admin_workers        | threads              | 1                    |
overcloud-controller-0 | keystone             | eventlet_server      | admin_workers        | 8                    | 12
overcloud-controller-0 | keystone             | eventlet_server      | public_workers       | 8                    | 12
overcloud-controller-0 | keystone             | oslo_messaging_notifications | driver               | messaging            | messagingv2
overcloud-controller-0 | keystone             | main_workers         | processes            | 32                   |
overcloud-controller-0 | keystone             | main_workers         | threads              | 1                    |
overcloud-controller-0 | keystone             | token                | provider             | uuid                 | fernet
overcloud-controller-0 | rabbitmq             | DEFAULT              | file                 | 65436                |
overcloud-controller-0 | mysql                | DEFAULT              | max                  | 4096                 |
overcloud-controller-0 | cinder               | DEFAULT              | exec_dirs            | /sbin,/usr/sbin,/bin,/usr/bin | /sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin,/usr/lpp/mmfs/bin
overcloud-controller-0 | cinder               | DEFAULT              | osapi_volume_workers | 32                   | 12
overcloud-controller-0 | glance               | DEFAULT              | bind_port            | 9191                 | 9292
overcloud-controller-0 | glance               | DEFAULT              | workers              | 32                   | 12
overcloud-controller-0 | glance               | DEFAULT              | log_file             | /var/log/glance/registry.log | /var/log/glance/cache.log
overcloud-controller-0 | glance               | ref1                 | auth_version         | 2                    | 3
overcloud-controller-0 | glance               | glance_store         | stores               | glance.store.http.Store,glance.store.swift.Store | http,swift
overcloud-controller-0 | glance               | glance_store         | os_region_name       | RegionOne            | regionOne
overcloud-controller-0 | gnocchi              | metricd              | workers              | 8                    | 12
overcloud-controller-0 | gnocchi              | storage              | swift_auth_version   | 2                    | 3
overcloud-controller-0 | neutron              | DEFAULT              | global_physnet_mtu   | 1496                 | 1500
overcloud-controller-0 | neutron              | DEFAULT              | rpc_workers          | 32                   | 12
overcloud-controller-0 | neutron              | DEFAULT              | api_workers          | 32                   | 12
+-------------------------------------------------------------------------------------------------------------------------------------+

Compare performance of two different runs

Using the CLI the user can determine, run to run performance differences. This is a good tool for spot checking performance of an OpenStack release.

You'll need to install extra dependencies for browbeat insights, which will provide additional modules needed for providing insights.

To install :

$ source browbeat/.browbeat-venv/bin/activate
$ pip install .[insights]