x/browbeat
Code Issues Proposed changes
03be425102
browbeat/doc/source/installation.rst
Sai Sindhur Malleni d090700eeb Update docs for grafana-dashboards playbook 
Grafyaml needs to be installed for dahboard uploading to work. So,
the browbeat installation playbook needs to be run before uploading dashboards.
This commit adds text in docs to emphasize the order in which playbooks are run.

Change-Id: Iec154f600db156907a2bf78fcb1c71b4bceb5469
2018-10-10 09:20:59 -04:00

715 lines
22 KiB
ReStructuredText
Raw Blame History

============
Installation
============
Browbeat is currently installed via an ansible playbook. In a Tripleo
environment it can be installed directly on the Undercloud or a separate
machine. The installation can be run from either your local machine or
directly on the machine you want Browbeat installed on.
Install Browbeat on Undercloud
------------------------------
This is usually the easiest installation due to many requirements are satisfied
on the Undercloud. In some cases it may not be desired to install Browbeat on
the Undercloud (Ex. Limited Resource requirements or Non-Tripleo installed
cloud)
Requirements
~~~~~~~~~~~~
Hardware
* Undercloud Machine (Baremetal or Virtual Machine)
Networking
* Access to Public API endpoints
* Access to Keystone Admin Endpoint
.. note:: For tripleo, public API endpoints are located on the External
Network by default. The Keystone Admin Endpoint is deployed on the ctlplane
network by default. These networking requirements should be validated before
attempting an installation.
On the Undercloud
~~~~~~~~~~~~~~~~~
::
$ ssh undercloud-root
[root@undercloud ~]# su - stack
[stack@undercloud ~]$ git clone https://github.com/openstack/browbeat.git
[stack@undercloud ~]$ source stackrc
[stack@undercloud ~]$ cd browbeat/ansible
[stack@undercloud ansible]$ ./bootstrap.py tripleo
[stack@undercloud ansible]$ sudo easy_install pip
[stack@undercloud ansible]$ sudo pip install ansible
[stack@undercloud ansible]$ vi install/group_vars/all.yml # Make sure to edit the dns_server to the correct ip address
[stack@undercloud ansible]$ ansible-playbook -i hosts install/browbeat.yml
[stack@undercloud ansible]$ ansible-playbook -i hosts install/shaker_build.yml
.. note:: Your default network might not work for you depending on your
underlay/overlay network setup. In such cases, user needs to create
appropriate networks for instances to allow them to reach the
internet. Some useful documentation can be found at:
https://access.redhat.com/documentation/en/red-hat-openstack-platform/11/single/networking-guide/
(Optional) Install Browbeat instance workloads
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Browbeat instance workloads are orchestrated Rally plugins that ship with Browbeat.
We currently support a handful of workloads
- Pbench-Uperf - Networking throughput / RR test
- Linpack - Microbenchmark for CPU load
To enable installation of the Browbeat workloads set install_browbeat_workloads: true in
ansible/install/group_vars/all.yml.
It is also required to provide the neutron network id of a private network which
has external access. To set this, edit ansible/install/group_vars/all.yml and
provide the network id for the browbeat_network:
This work can either be done prior to installation of Browbeat, or after Browbeat
has been installed. To skip directly to this task execute:
::
$ ansible-playbook -i hosts install/browbeat.yml --start-at-task "Check browbeat_network"
...
(Optional) Install Collectd
~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
[stack@ospd ansible]$ ansible-playbook -i hosts install/collectd-openstack.yml
(Optional) Install Rsyslogd logging with aggregation
~~~~~~~~~~~~~~~~~~~~~~~~~~~
First configure the values rsyslog values and elasticsearch parameters in
`ansible/install/group_vars/all.yml`. If you have a large number of hosts
deploying an aggregator using `ansible/install/rsyslog-aggregator.yml`
is strongly suggested. If you have a small scale, change the value
rsyslog_forwarding in `all.yml` to `false`. Once things are configured
to your liking deploy logging on the cloud using the `rsyslog-logging.yml`
playbook.
Firewall configuration for the aggregator is left up to the user. The logging
install playbook will check that the aggregator is up and the port is open if
you deploy with aggregation.
::
[stack@ospd ansible]$ vim install/group_vars/all.yml
[stack@ospd ansible]$ ansible-playbook -i hosts install/rsyslog-aggregator.yml
[stack@ospd ansible]$ ansible-playbook -i hosts install/rsyslog-logging.yml
(Optional) Install Browbeat Grafana dashboards
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Browbeat uses Grafyaml to upload dashboards to Grafana. Grafyaml is installed by browbeat
at the location pointed to by the variable `browbeat_venv` in `ansible/install/group_vars/all.yml`.
So, you need to first run the browbeat install playbook `ansible/install/browbeat.yml` before running
the below playbook.
::
[stack@ospd ansible]$ ansible-playbook -i hosts install/grafana-dashboards.yml
Not mantained (Pre-Pike): Run Overcloud checks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
[stack@ospd ansible]$ ansible-playbook -i hosts check/site.yml
Your Overcloud check output is located in results/bug_report.log
Install Browbeat from your local machine
----------------------------------------
This installs Browbeat onto your Undercloud but the playbook is run from your
local machine rather than directly on the Undercloud machine.
From your local machine
~~~~~~~~~~~~~~~~~~~~~~~
::
$ ssh-copy-id stack@<undercloud-ip>
$ git clone https://github.com/openstack/browbeat.git
$ cd browbeat/ansible
$ ./generate_tripleo_hostfile.sh -t <undercloud-ip>
$ vi install/group_vars/all.yml # Review and edit configuration items
$ ansible-playbook -i hosts install/browbeat.yml
$ ansible-playbook -i hosts install/shaker_build.yml
.. note:: Your default network might not work for you depending on your
underlay/overlay network setup. In such cases, user needs to create
appropriate networks for instances to allow them to reach the
internet. Some useful documentation can be found at:
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/networking_guide/
(Optional) Install collectd
~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
$ ansible-playbook -i hosts install/collectd-openstack.yml
(Optional) Install Browbeat Grafana dashboards
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Browbeat uses Grafyaml to upload dashboards to Grafana. Grafyaml is installed by browbeat
at the location pointed to by the variable `browbeat_venv` in `ansible/install/group_vars/all.yml`.
So, you need to first run the browbeat install playbook `ansible/install/browbeat.yml` before running
the below playbook.
::
$ ansible-playbook -i hosts install/grafana-dashboards.yml
Install/Setup Browbeat Machine
------------------------------
This setup is used when running Browbeat on a separate machine than the
Undercloud. Using this method, you can create multiple users on the machine
and each user can be pointed at a different cloud or the same cloud.
Requirements
~~~~~~~~~~~~
Hardware
* Baremetal or Virtual Machine
Networking
* Access to Public API endpoints
* Access to Keystone Admin Endpoint
RPM
* epel-release
* ansible
* git
OpenStack
* overcloudrc file placed in browbeat user home directory
.. note:: For tripleo, public API endpoints are located on the External
Network by default. The Keystone Admin Endpoint is deployed on the ctlplane
network by default. These networking requirements should be validated before
attempting an installation.
Preparing the Machine (CentOS 7)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. Install Machine either from Image, ISO, or PXE
2. Check for Required Network Connectivity
Determine Overcloud Keystone endpoints
::
[stack@undercloud-1 ~]$ . overcloudrc
[stack@undercloud-1 ~]$ openstack catalog show identity
+-----------+----------------------------------------+
| Field | Value |
+-----------+----------------------------------------+
| endpoints | regionOne |
| | publicURL: http://172.21.0.10:5000 |
| | internalURL: http://172.16.0.16:5000 |
| | adminURL: http://192.168.24.61:35357 |
| | |
| name | keystone |
| type | identity |
+-----------+----------------------------------------+
Check network connectivity
::
$ ssh root@browbeatvm
[root@browbeatvm ~]$ # Ping Keystone Admin API IP Address
[root@browbeatvm ~]# ping -c 2 192.168.24.61
PING 192.168.24.61 (192.168.24.61) 56(84) bytes of data.
64 bytes from 192.168.24.61: icmp_seq=1 ttl=64 time=1.60 ms
64 bytes from 192.168.24.61: icmp_seq=2 ttl=64 time=0.312 ms
--- 192.168.24.61 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.312/0.957/1.603/0.646 ms
[root@browbeatvm ~]$ # Ping Keystone Public API IP Address
[root@browbeatvm ~]# ping -c 2 172.21.0.10
PING 172.21.0.10 (172.21.0.10) 56(84) bytes of data.
64 bytes from 172.21.0.10: icmp_seq=1 ttl=64 time=0.947 ms
64 bytes from 172.21.0.10: icmp_seq=2 ttl=64 time=0.304 ms
--- 172.21.0.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.304/0.625/0.947/0.322 ms
3. Create user for Browbeat and generate SSH key
::
[root@browbeatvm ~]# useradd browbeat1
[root@browbeatvm ~]# passwd browbeat1
Changing password for user browbeat1.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
[root@browbeatvm ~]# echo "browbeat1 ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/browbeat1; chmod 0440 /etc/sudoers.d/browbeat1
browbeat1 ALL=(root) NOPASSWD:ALL
[root@browbeatvm ~]# su - browbeat1
[browbeat1@browbeatvm ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/browbeat1/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/browbeat1/.ssh/id_rsa.
Your public key has been saved in /home/browbeat1/.ssh/id_rsa.pub.
The key fingerprint is:
c2:b2:f0:cd:ef:d2:2b:a8:9a:5a:bb:ca:ce:c1:8c:3b browbeat1@browbeatvm
The key's randomart image is:
+--[ RSA 2048]----+
| |
| |
| |
| . |
| . . o S |
|+ o = . |
|.+. o.o. |
|E+... o.. |
|OB+o ++. |
+-----------------+
4. Enable passwordless SSH into localhost and Undercloud then copy overcloudrc over to Browbeat VM
::
[browbeat1@browbeatvm ansible]$ ssh-copy-id browbeat1@localhost
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
browbeat1@localhost's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'browbeat1@localhost'"
and check to make sure that only the key(s) you wanted were added.
[browbeat1@browbeatvm ~]$ ssh-copy-id stack@undercloud-1
The authenticity of host 'undercloud-1 (undercloud-1)' can't be established.
ECDSA key fingerprint is fa:3a:02:e8:8e:92:4d:a7:9c:90:68:6a:c2:eb:fe:e1.
Are you sure you want to continue connecting (yes/no)? yes
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
stack@undercloud-1's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'stack@undercloud-1'"
and check to make sure that only the key(s) you wanted were added.
[browbeat1@browbeatvm ~]$ scp stack@undercloud-1:/home/stack/overcloudrc .
overcloudrc 100% 553 0.5KB/s 00:00
.. note:: In SSL environments, you must copy the certificate over and
check that the "OS_CA_CERT" variable is set correctly to the copied
certificate location
5. Install RPM requirements
::
[browbeat1@browbeatvm ~]$ sudo yum install -y epel-release
[browbeat1@browbeatvm ~]$ sudo yum install -y ansible git
6. Clone Browbeat
::
[browbeatuser1@browbeat-vm ~]$ git clone https://github.com/openstack/browbeat.git
Cloning into 'browbeat'...
remote: Counting objects: 7425, done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 7425 (delta 14), reused 12 (delta 12), pack-reused 7398
Receiving objects: 100% (7425/7425), 5.23 MiB | 0 bytes/s, done.
Resolving deltas: 100% (4280/4280), done.
7. Generate hosts, ssh-config, and retrieve heat-admin-id_rsa.
::
[browbeat1@browbeatvm ~]$ cd browbeat/ansible/
[browbeat1@browbeatvm ansible]$ ./generate_tripleo_hostfile.sh -t undercloud-1 --localhost
...
[browbeat1@browbeatvm ansible]$ ls ssh-config hosts heat-admin-id_rsa
heat-admin-id_rsa hosts ssh-config
Note use of "--localhost" to indicate the desire to install browbeat on the
localhost rather than the undercloud.
8. Edit installation variables
::
[browbeat1@browbeatvm ansible]$ vi install/group_vars/all.yml
In this case, adjust browbeat_user, iptables_file and dns_server. Each
environment is different and thus your configuration options will vary.
.. note:: If you require a proxy to get outside your network, you must
configure http_proxy, https_proxy, no_proxy variables in the proxy_env
dictionary in install/group_vars/all.yml
9. Run Browbeat install playbook
::
[browbeat1@browbeatvm ansible]$ ansible-playbook -i hosts install/browbeat.yml
10. Setup browbeat-config.yaml and test run Rally against cloud
::
[browbeat1@browbeatvm ansible]$ cd ..
[browbeat1@browbeatvm browbeat]$ vi browbeat-config.yaml
[browbeat1@browbeatvm browbeat]$ . .browbeat-venv/bin/activate
(browbeat-venv) [browbeat1@browbeatvm browbeat]$ python browbeat.py rally
11. Build Shaker image
::
[browbeatuser1@browbeat-vm ~]$ ansible-playbook -i hosts install/shaker_build.yml
.. note:: Your default network might not work for you depending on your
underlay/overlay network setup. In such cases, user needs to create
appropriate networks for instances to allow them to reach the
internet. Some useful documentation can be found at:
https://access.redhat.com/documentation/en/red-hat-openstack-platform/11/single/networking-guide/
(Optional) Install collectd
~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
[browbeatuser1@browbeat-vm ~]$ ansible-playbook -i hosts install/collectd-openstack.yml
(Optional) Install Browbeat Grafana dashboards
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Browbeat uses Grafyaml to upload dashboards to Grafana. Grafyaml is installed by browbeat
at the location pointed to by the variable `browbeat_venv` in `ansible/install/group_vars/all.yml`.
So, you need to first run the browbeat install playbook `ansible/install/browbeat.yml` before running
the below playbook.
::
[browbeatuser1@browbeat-vm ~]$ ansible-playbook -i hosts install/grafana-dashboards.yml
Considerations for additional Browbeat Installs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If it is desired to run Browbeat against multiple clouds from the same machine.
It is recommended to create a second user (Ex. browbeat2) and repeat above
instructions. In order to expose the second user's Browbeat results via httpd,
change the port (Variable browbeat_results_port) and thus each user's results
will be available via http on different ports.
.. note:: Keep in mind that running multiple sets of control plane workloads
from multiple Browbeat users at the same time will introduce variation into
resulting performance data if the machine on which Browbeat is installed is
resource constrained.
Using Keystone Public Endpoint
------------------------------
If your Browbeat installation can not reach the Keystone Admin API endpoint due
to the networking, you can use Keystone V3 options. In your overcloudrc or rc
file you can add the following environment variables.
::
export OS_IDENTITY_API_VERSION=3
export OS_INTERFACE=public
Uploading Images to the overcloud
---------------------------------
Browbeat by default uploads CentOS and CirrOS images to the cloud for use in
Rally and other workloads. It is recommended to upload RAW images if using ceph
and hence the convert_to_raw variable must be set to true as shown below in
ansible/install/group_vars/all.yml. The default is false.
::
images:
centos7:
name: centos7
url: http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2
type: qcow2
convert_to_raw: true
==================================
Additional Components Installation
==================================
Install Monitoring Host (Carbon/Graphite/Grafana)
-------------------------------------------------
A monitoring host exposes System and Application performance metrics to the
Browbeat user via Grafana. It helps expose what may be causing your bottleneck
when you encounter a performance issue.
Prerequisites
~~~~~~~~~~~~~
Hardware
* Baremetal or Virtual Machine
* SSD storage
Operating System
* RHEL 7
* CentOS 7
Repos
* Red Hat Enterprise Linux 7Server - x86_64 - Server
* Red Hat Enterprise Linux 7Server - x86_64 - Server Optional
RPM
* epel-release
* ansible
* git
Installation
~~~~~~~~~~~~
1. Deploy machine (RHEL7 is used in this example)
2. Install RPMS
::
[root@dhcp23-93 ~]# yum install -y https://download.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
...
[root@dhcp23-93 ~]# yum install -y ansible git
3. Clone Browbeat
::
[root@dhcp23-93 ~]# git clone https://github.com/openstack/browbeat.git
Cloning into 'browbeat'...
remote: Counting objects: 7533, done.
remote: Compressing objects: 100% (38/38), done.
remote: Total 7533 (delta 30), reused 36 (delta 23), pack-reused 7469
Receiving objects: 100% (7533/7533), 5.26 MiB | 5.79 MiB/s, done.
Resolving deltas: 100% (4330/4330), done.
4. Add a hosts file into ansible directory
::
[root@dhcp23-93 ~]# cd browbeat/ansible/
[root@dhcp23-93 ansible]# vi hosts
Content of hosts file should be following
::
[graphite]
localhost
[grafana]
localhost
5. Setup SSH config, SSH key and exchange for Ansible
::
[root@dhcp23-93 ansible]# touch ssh-config
[root@dhcp23-93 ansible]# ssh-keygen
Generating public/private rsa key pair.
...
[root@dhcp23-93 ansible]# ssh-copy-id root@localhost
...
6. Edit install variables
::
[root@dhcp23-93 ansible]# vi install/group_vars/all.yml
Depending on the environment you may need to edit more than just the following
variables - graphite_host and grafana_host
.. note:: If you require a proxy to get outside your network, you must
configure http_proxy, https_proxy, no_proxy variables in the proxy_env
dictionary in install/group_vars/all.yml
7. Install Carbon and Graphite via Ansible playbook
::
[root@dhcp23-93 ansible]# ansible-playbook -i hosts install/graphite.yml
...
8. Install Grafana via Ansible playbook
::
[root@dhcp23-93 ansible]# ansible-playbook -i hosts install/grafana.yml
...
9. Install Grafana dashboards via Ansible playbook
::
[root@dhcp23-93 ansible]# ansible-playbook -i hosts install/grafana-dashboards.yml -e 'cloud_dashboards=false'
...
10. (Optional) Monitor the Monitor Host
::
[root@dhcp23-93 ansible]# ansible-playbook -i hosts install/collectd-generic.yml --tags graphite
...
Now navigate to http://monitoring-host-address:3000 to verify Grafana is
installed, the Graphite data source exists and custom dashboards are uploaded.
You can now point other clouds at this host in order to view System and
Application performance metrics. Depending on the number of clouds and
machines pointed at your monitoring server, you may need to add more disk IO
capacity, disk storage or carbon-cache+carbon-relay processes depending
entirely on the number of metrics and your environments capacity. There is a
Graphite dashboard included and it is recommended to install collectd on your
monitoring host such that you can see if you hit resource issues with your
monitoring host.
Install ELK Host (ElasticSearch/LogStash/Kibana)
-------------------------------------------------
An ELK server allows you to publish resulting benchmark data into ElasticSearch
which allows you to build querys and dashboards to examine your benchmarking
result data over various metadata points.
Prerequisites
~~~~~~~~~~~~~
Hardware
* Baremetal or Virtual Machine
Operating System
* RHEL 7
* CentOS 7
Repos
* Red Hat Enterprise Linux 7Server - x86_64 - Server
* Red Hat Enterprise Linux 7Server - x86_64 - Server Optional
RPM
* epel-release
* ansible
* git
Installation
~~~~~~~~~~~~
1. Deploy machine (RHEL7 is used in this example)
2. Install RPMS
::
[root@dhcp23-93 ~]# yum install -y https://download.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
...
[root@dhcp23-93 ~]# yum install -y ansible git
3. Clone Browbeat
::
[root@dhcp23-93 ~]# git clone https://github.com/openstack/browbeat.git
Cloning into 'browbeat'...
remote: Counting objects: 7533, done.
remote: Compressing objects: 100% (38/38), done.
remote: Total 7533 (delta 30), reused 36 (delta 23), pack-reused 7469
Receiving objects: 100% (7533/7533), 5.26 MiB | 5.79 MiB/s, done.
Resolving deltas: 100% (4330/4330), done.
4. Add a hosts file into ansible directory
::
[root@dhcp23-93 ~]# cd browbeat/ansible/
[root@dhcp23-93 ansible]# vi hosts
Content of hosts file should be following
::
[elk]
localhost
5. Setup SSH config, SSH key and exchange for Ansible
::
[root@dhcp23-93 ansible]# touch ssh-config
[root@dhcp23-93 ansible]# ssh-keygen
Generating public/private rsa key pair.
...
[root@dhcp23-93 ansible]# ssh-copy-id root@localhost
...
6. Edit install variables
::
[root@dhcp23-93 ansible]# vi install/group_vars/all.yml
Depending on the environment you may need to edit more than just the following
variables - es_ip
If you are deploying using a machine that is not an OSP undercloud, be sure to edit
the home_dir/browbeat_path to match its actual path.
.. note:: If you require a proxy to get outside your network, you must
configure http_proxy, https_proxy, no_proxy variables in the proxy_env
dictionary in install/group_vars/all.yml
7. Install ELK via Ansible playbook
::
[root@dhcp23-93 ansible]# ansible-playbook -i hosts install/elk.yml
...
8. Install Kibana Visualizations via Ansible playbook
::
[root@dhcp23-93 ansible]# ansible-playbook -i hosts install/kibana-visuals.yml
...
Now navigate to http://elk-host-address to verify Kibana is
installed and custom visualizations are uploaded.
View Git Blame Copy Permalink