Initial commit to openstack/kloudbuster
Change-Id: Id7e009e3a9ed61e86c45c8e4839208ecfa30bd77
This commit is contained in:
Yichen Wang
parent
bfdeefc89c
commit
90fbf3012b
@ -1,7 +1,7 @@
|
||||
[run]
|
||||
branch = True
|
||||
source = vmtp
|
||||
omit = vmtp/tests/*,vmtp/openstack/*
|
||||
source = kloudbuster
|
||||
omit = kloudbuster/openstack/*
|
||||
|
||||
[report]
|
||||
ignore-errors = True
|
||||
|
||||
@ -1,8 +0,0 @@
|
||||
ansible
|
||||
installer
|
||||
requirements-dev.txt
|
||||
cloud_init*
|
||||
.git
|
||||
.gitignore
|
||||
.gitreview
|
||||
.pylintrc
|
||||
5
.gitignore
vendored
5
.gitignore
vendored
@ -54,11 +54,8 @@ ChangeLog
|
||||
*cscope*
|
||||
.ropeproject/
|
||||
|
||||
# vmtp
|
||||
*.local*
|
||||
*.json
|
||||
|
||||
# KloudBuster
|
||||
*.json
|
||||
*.html
|
||||
*.qcow2
|
||||
scale/dib/kloudbuster.d/
|
||||
|
||||
2
.mailmap
2
.mailmap
@ -2,4 +2,4 @@
|
||||
# <preferred e-mail> <other e-mail 1>
|
||||
# <preferred e-mail> <other e-mail 2>
|
||||
|
||||
vmtp-core@cisco.com
|
||||
kloudbuster-core@lists.launchpad.net
|
||||
|
||||
17
CONTRIBUTING.rst
Normal file
17
CONTRIBUTING.rst
Normal file
@ -0,0 +1,17 @@
|
||||
If you would like to contribute to the development of OpenStack, you must
|
||||
follow the steps in this page:
|
||||
|
||||
http://docs.openstack.org/infra/manual/developers.html
|
||||
|
||||
If you already have a good understanding of how the system works and your
|
||||
OpenStack accounts are set up, you can skip to the development workflow
|
||||
section of this documentation to learn how changes to OpenStack should be
|
||||
submitted for review via the Gerrit tool:
|
||||
|
||||
http://docs.openstack.org/infra/manual/developers.html#development-workflow
|
||||
|
||||
Pull requests submitted through GitHub will be ignored.
|
||||
|
||||
Bugs should be filed on Launchpad, not GitHub:
|
||||
|
||||
https://bugs.launchpad.net/kloudbuster
|
||||
23
Dockerfile
23
Dockerfile
@ -1,23 +0,0 @@
|
||||
# docker file for creating a container that has vmtp installed and ready to use
|
||||
FROM ubuntu:14.04
|
||||
MAINTAINER vmtp-core <vmtp-core@lists.launchpad.net>
|
||||
|
||||
# Install VMTP script and dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
lib32z1-dev \
|
||||
libffi-dev \
|
||||
libssl-dev \
|
||||
libxml2-dev \
|
||||
libxslt1-dev \
|
||||
libyaml-dev \
|
||||
openssh-client \
|
||||
python \
|
||||
python-dev \
|
||||
python-lxml \
|
||||
python-pip \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY . /vmtp/
|
||||
|
||||
RUN pip install -r /vmtp/requirements.txt
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
vmtp Style Commandments
|
||||
kloudbuster Style Commandments
|
||||
===============================================
|
||||
|
||||
Read the OpenStack Style Commandments http://docs.openstack.org/developer/hacking/
|
||||
|
||||
133
README.rst
133
README.rst
@ -1,126 +1,19 @@
|
||||
========
|
||||
Overview
|
||||
========
|
||||
===============================
|
||||
kloudbuster
|
||||
===============================
|
||||
|
||||
VMTP is a data path performance measurement tool for OpenStack clouds.
|
||||
KloudBuster is a open source tool that allows anybody to load any Neutron OpenStack cloud at massive data plane scale swiftly and effortlessly.
|
||||
|
||||
Please feel here a long description which must be at least 3 lines wrapped on
|
||||
80 cols, so that distribution package maintainers can use it in their packages.
|
||||
Note that this is a hard requirement.
|
||||
|
||||
* Free software: Apache license
|
||||
* Documentation: http://docs.openstack.org/developer/kloudbuster
|
||||
* Source: http://git.openstack.org/cgit/openstack/kloudbuster
|
||||
* Bugs: http://bugs.launchpad.net/kloudbuster
|
||||
|
||||
Features
|
||||
--------
|
||||
|
||||
Have you ever had the need for a quick, simple and automatable way to get VM-level or host-level single-flow throughput and latency numbers from any OpenStack cloud, and take into account various Neutron topologies? Or check whether some OpenStack configuration option, Neutron plug-in performs to expectation or if there is any data path impact for upgrading to a different OpenStack release?
|
||||
|
||||
VMTP is a small python application that will automatically perform ping connectivity, round trip time measurement (latency) and TCP/UDP throughput measurement for the following East/West flows on any OpenStack deployment:
|
||||
|
||||
* VM to VM same network (private fixed IP, flow #1)
|
||||
* VM to VM different network using fixed IP (same as intra-tenant L3 fixed IP, flow #2)
|
||||
* VM to VM different network using floating IP and NAT (same as floating IP inter-tenant L3, flow #3)
|
||||
|
||||
Optionally, when an external Linux host is available for testing North/South flows:
|
||||
|
||||
* External host/VM download and upload throughput/latency (L3/floating IP, flow #4 and #5)
|
||||
|
||||
.. image:: images/flows.png
|
||||
|
||||
Optionally, when SSH login to any Linux host (native or virtual) is available:
|
||||
|
||||
* Host to host process-level throughput/latency (intra-node and inter-node)
|
||||
|
||||
Optionally, VMTP can extract automatically CPU usage from all native hosts in the cloud during the throughput tests, provided the Ganglia monitoring service (gmond) is installed and enabled on those hosts.
|
||||
|
||||
For VM-related flows, VMTP will automatically create the necessary OpenStack resources (router, networks, subnets, key pairs, security groups, test VMs) using the public OpenStack API, install the test tools then orchestrate them to gather the throughput measurements then cleanup all related resources before exiting.
|
||||
|
||||
See the usage page for the description of all the command line arguments supported by VMTP.
|
||||
|
||||
|
||||
Pre-requisite
|
||||
-------------
|
||||
|
||||
VMTP runs on any Python 2.X envirnment (validated on Linux and MacOSX).
|
||||
|
||||
For VM related performance measurements
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
* Access to the cloud Horizon Dashboard (to retrieve the openrc file)
|
||||
* 1 working external network pre-configured on the cloud (VMTP will pick the first one found)
|
||||
* At least 2 floating IP if an external router is configured or 3 floating IP if there is no external router configured
|
||||
* 1 Linux image available in OpenStack (any distribution)
|
||||
* A configuration file that is properly set for the cloud to test (see "Configuration File" section below)
|
||||
|
||||
For native/external host throughputs
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
* A public key must be installed on the target hosts (see ssh password-less access below)
|
||||
|
||||
For pre-existing native host throughputs
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
* Firewalls must be configured to allow TCP/UDP ports 5001 and TCP port 5002
|
||||
|
||||
For running VMTP Docker Image
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
* Docker is installed. See `here <https://docs.docker.com/installation/#installation/>`_ for instructions.
|
||||
|
||||
Sample Results Output
|
||||
---------------------
|
||||
|
||||
VMTP will display the results to stdout with the following data:
|
||||
|
||||
.. code::
|
||||
|
||||
- Session general information (date, auth_url, OpenStack encaps, VMTP version, OpenStack release, Agent type, CPU...)
|
||||
- List of results per flow, for each flow:
|
||||
| flow name
|
||||
| to and from IP addresses
|
||||
| to and from availability zones (if VM)
|
||||
| - results:
|
||||
| | -TCP
|
||||
| | | packet size
|
||||
| | | throughput value
|
||||
| | | number of retransmissions
|
||||
| | | round trip time in ms
|
||||
| | | - CPU usage (if enabled), for each host in the openstack cluster
|
||||
| | | | baseline (before test starts)
|
||||
| | | | 1 or more readings during test
|
||||
| | -UDP
|
||||
| | | - for each packet size
|
||||
| | | | throughput value
|
||||
| | | | loss rate
|
||||
| | | | CPU usage (if enabled)
|
||||
| | - ICMP
|
||||
| | | average, min, max and stddev round trip time in ms
|
||||
|
||||
Detailed results can also be stored in a file in JSON format using the *--json* command line argument and/or stored directly into a MongoDB server. See :download:`here <_static/example.json>` for an example JSON file that is generated by VMTP.
|
||||
The packaged python tool genchart.py can be used to generate from the JSON result files column charts in HTML format visible from any browser.
|
||||
|
||||
Example of column chart generated by genchart.py:
|
||||
|
||||
.. image:: images/genchart-sample.png
|
||||
|
||||
Limitations and Caveats
|
||||
-----------------------
|
||||
|
||||
VMTP only measures performance for single-flows at the socket/TCP/UDP level (in a VM or natively). Measured numbers therefore reflect what most applications will see.
|
||||
|
||||
It is not designed to measure driver level data path performance from inside a VM (such as bypassing the kernel TCP stack and write directly to virtio), there are better tools that can address this type of mesurement.
|
||||
|
||||
|
||||
Licensing
|
||||
---------
|
||||
|
||||
VMTP is licensed under Apache License 2.0 and comes packaged with the following tools for convenience:
|
||||
|
||||
* iperf: BSD License (https://iperf.fr/license.html, source code: https://iperf.fr)
|
||||
* nuttcp: GPL v2 License (http://nuttcp.net/nuttcp/beta/LICENSE, source code: http://nuttcp.net/nuttcp/beta/nuttcp-7.3.2.c)
|
||||
|
||||
Redistribution of nuttcp and iperf is governed by their respective licenses. Please make sure you read and understand each one before further redistributing VMTP downstream.
|
||||
|
||||
Links
|
||||
-----
|
||||
|
||||
* Documentation: http://vmtp.readthedocs.org/en/latest
|
||||
* Source: http://git.openstack.org/cgit/stackforge/vmtp
|
||||
* Supports/Bugs: https://launchpad.net/vmtp
|
||||
* Mailing List: vmtp-core@lists.launchpad.net
|
||||
|
||||
* TODO
|
||||
|
||||
185
cfg.default.yaml
185
cfg.default.yaml
@ -1,185 +0,0 @@
|
||||
#
|
||||
# VMTP default configuration file
|
||||
#
|
||||
# This configuration file is ALWAYS loaded by VMTP and should never be modified by users.
|
||||
# To specify your own property values, always define them in a separate config file
|
||||
# and pass that file to the script using -c or --config <file>
|
||||
# Property values in that config file will override the default values in the current file
|
||||
#
|
||||
---
|
||||
|
||||
# Name of the image to use for launching the test VMs. This name must be
|
||||
# the exact same name used in OpenStack (as shown from 'nova image-list')
|
||||
# Any image running Linux should work (Fedora, Ubuntu, CentOS...)
|
||||
image_name: 'Ubuntu Server 14.04'
|
||||
#image_name: 'Fedora 21'
|
||||
|
||||
# User name to use to ssh to the test VMs
|
||||
# This is specific to the image being used
|
||||
ssh_vm_username: 'ubuntu'
|
||||
#ssh_vm_username: fedora
|
||||
|
||||
# Name of the flavor to use for the test VMs
|
||||
# This name must be an exact match to a flavor name known by the target
|
||||
# OpenStack deployment (as shown from 'nova flavor-list')
|
||||
flavor_type: 'm1.small'
|
||||
|
||||
# Name of the availability zone to use for the test VMs
|
||||
# Must be one of the zones listed by 'nova availability-zone-list'
|
||||
# If the zone selected contains more than 1 compute node, the script
|
||||
# will determine inter-node and intra-node throughput. If it contains only
|
||||
# 1 compute node, only intra-node troughput will be measured.
|
||||
# If empty (default), VMTP will automatically pick the first 2 hosts
|
||||
# that are compute nodes regardless of the availability zone
|
||||
#availability_zone: 'nova'
|
||||
availability_zone:
|
||||
|
||||
# DNS server IP addresses to use for the VM (list of 1 or more DNS servers)
|
||||
# This default DNS server is available on the Internet,
|
||||
# Change this to use a different DNS server if necessary,
|
||||
dns_nameservers: [ '8.8.8.8' ]
|
||||
|
||||
# VMTP can automatically download a VM image if the image named by
|
||||
# image_name is missing, for that you need to specify a URL where
|
||||
# the image can be retrieved
|
||||
#
|
||||
# A link to a Ubuntu Server 14.04 qcow2 image can be used here:
|
||||
# https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img
|
||||
vm_image_url: ''
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# These variables are not likely to be changed
|
||||
|
||||
# Set this variable to a network name if you want the script to reuse
|
||||
# a specific existing external network. If empty, the script will reuse the
|
||||
# first external network it can find (the cloud must have at least 1
|
||||
# external network defined and available for use)
|
||||
# When set, ignore floating ip creation and reuse existing management network for tests
|
||||
reuse_network_name :
|
||||
|
||||
# Use of the script for special deployments
|
||||
floating_ip: True
|
||||
|
||||
# Set this to an existing VM name if the script should not create new VM
|
||||
# and reuse existing VM
|
||||
reuse_existing_vm :
|
||||
|
||||
# Set config drive to true to bypass metadata service and use config drive
|
||||
# An option of config_drive to True is provided to nova boot to enable this
|
||||
config_drive:
|
||||
|
||||
# ipv6 mode. Set this to one of the following 3 modes
|
||||
# slaac : VM obtains IPV6 address from Openstack radvd using SLAAC
|
||||
# dhcpv6-stateful : VM obtains ipv6 address from dnsmasq using DHCPv6 stateful
|
||||
# dhcpv6-stateless : VM obtains ipv6 address from Openstack radvd using SLAAC and options from dnsmasq
|
||||
# If left blank use ipv4
|
||||
ipv6_mode:
|
||||
|
||||
|
||||
# Default name for the router to use to connect the internal mgmt network
|
||||
# with the external network. If a router exists with this name it will be
|
||||
# reused, otherwise a new router will be created
|
||||
router_name: 'pns-router'
|
||||
|
||||
# Defaul names for the internal networks used by the
|
||||
# script. If an existing network with this name exists it will be reused.
|
||||
# Otherwise a new internal network will be created with that name.
|
||||
# 2 networks are needed to test the case of network to network communication
|
||||
internal_network_name: ['pns-internal-net', 'pns-internal-net2']
|
||||
|
||||
# Name of the subnets associated to the internal mgmt network
|
||||
internal_subnet_name: ['pns-internal-subnet', 'pns-internal-subnet2']
|
||||
|
||||
# Name of the subnets for ipv6
|
||||
internal_subnet_name_ipv6: ['pns-internal-v6-subnet','pns-internal-v6-subnet2']
|
||||
|
||||
# Default CIDRs to use for the internal mgmt subnet
|
||||
internal_cidr: ['192.168.1.0/24' , '192.168.2.0/24']
|
||||
|
||||
# Default CIDRs to use for data network for ipv6
|
||||
internal_cidr_v6: ['2001:45::/64','2001:46::/64']
|
||||
|
||||
# The public and private keys to use to ssh to all targets (VMs, containers, hosts)
|
||||
# By default the SSH library will try several methods to authenticate:
|
||||
# - password if provided on the command line
|
||||
# - user's own key pair (under the home directory $HOME) if already setup
|
||||
# - the below key pair if not empty
|
||||
# If you want to use a specific key pair, specify the key pair files here.
|
||||
# This can be a pathname that is absolute or relative to the current directory
|
||||
public_key_file:
|
||||
private_key_file:
|
||||
|
||||
# Name of the P&S public key in OpenStack to create for all test VMs
|
||||
public_key_name: 'pns_public_key'
|
||||
|
||||
# name of the server VM
|
||||
vm_name_server: 'TestServer'
|
||||
|
||||
# name of the client VM
|
||||
vm_name_client: 'TestClient'
|
||||
|
||||
# name of the security group to create and use
|
||||
security_group_name: 'pns-security'
|
||||
|
||||
# Location to the performance test tools.
|
||||
# If relative, is relative to the vmtp directory
|
||||
perf_tool_path: './tools'
|
||||
|
||||
# ping variables
|
||||
ping_count: 2
|
||||
ping_pass_threshold: 80
|
||||
|
||||
# Max retry count for ssh to a VM (5 seconds between retries)
|
||||
ssh_retry_count: 50
|
||||
|
||||
# General retry count
|
||||
generic_retry_count: 50
|
||||
|
||||
# Times to run when measuring TCP Throughput
|
||||
tcp_tp_loop_count: 3
|
||||
|
||||
# TCP throughput list of packet sizes to measure
|
||||
# Can be overridden at the command line using --tcpbuf
|
||||
tcp_pkt_sizes: [65536]
|
||||
|
||||
# UDP throughput list of packet sizes to measure
|
||||
# By default we measure for small, medium and large packets
|
||||
# Can be overridden at the command line using --udpbuf
|
||||
udp_pkt_sizes: [128, 1024, 8192]
|
||||
|
||||
# UDP packet loss rate threshold in percentage beyond which bandwidth
|
||||
# iterations stop and below which iteration with a higher
|
||||
# bandwidth continues
|
||||
# The first number is the minimal loss rate (inclusive)
|
||||
# The second number is the maximum loss rate (inclusive)
|
||||
# Iteration to find the "optimal" bandwidth will stop as soon as the loss rate
|
||||
# falls within that range: min <= loss_rate <= max
|
||||
# The final throughput measurement may return a loss rate out of this range
|
||||
# as that measurement is taken on a longer time than when iterating to find
|
||||
# the optimal throughput
|
||||
#
|
||||
udp_loss_rate_range: [2, 5]
|
||||
|
||||
# The default bandwidth limit (in Kbps) for TCP/UDP flow measurement
|
||||
# 0 means unlimited, which can be overridden at the command line using --bandwidth
|
||||
vm_bandwidth: 0
|
||||
|
||||
#######################################
|
||||
# VMTP MongoDB Connection information
|
||||
#######################################
|
||||
|
||||
########################################
|
||||
# Default MongoDB port is 27017, to override
|
||||
#vmtp_mongod_port: <port no>
|
||||
|
||||
########################################
|
||||
# MongoDB pns database.
|
||||
# use "official_db" for offical runs only.
|
||||
########################################
|
||||
vmtp_db: "client_db"
|
||||
|
||||
########################################
|
||||
# MongoDB collection name.
|
||||
########################################
|
||||
vmtp_collection: "pns_web_entry"
|
||||
|
||||
@ -1,10 +0,0 @@
|
||||
#
|
||||
# Example of configuration where we froce the use of a specific external network and
|
||||
# use provider network (no floating IP)
|
||||
reuse_network_name : 'prov1'
|
||||
|
||||
# Floating ip false is a provider network where we simply attach to it
|
||||
floating_ip : False
|
||||
|
||||
# Floating ip is true by default:
|
||||
# attach to existing network, create a floating ip and attach instance to it
|
||||
463
compute.py
463
compute.py
@ -1,463 +0,0 @@
|
||||
# Copyright 2014 Cisco Systems, Inc. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
'''Module for Openstack compute operations'''
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import time
|
||||
|
||||
import novaclient
|
||||
import novaclient.exceptions as exceptions
|
||||
|
||||
class Compute(object):
|
||||
|
||||
def __init__(self, nova_client, config):
|
||||
self.novaclient = nova_client
|
||||
self.config = config
|
||||
|
||||
def find_image(self, image_name):
|
||||
try:
|
||||
image = self.novaclient.images.find(name=image_name)
|
||||
return image
|
||||
except novaclient.exceptions.NotFound:
|
||||
return None
|
||||
|
||||
def upload_image_via_url(self, glance_client, final_image_name, image_url, retry_count=60):
|
||||
'''
|
||||
Directly uploads image to Nova via URL if image is not present
|
||||
'''
|
||||
|
||||
# Here is the deal:
|
||||
# Idealy, we should better to use the python library glanceclient to perform the
|
||||
# image uploades. However, due to a limitation of the v2.0 API right now, it is
|
||||
# impossible to tell Glance to download the image from a URL directly.
|
||||
#
|
||||
# There are two steps to create the image:
|
||||
# (1) Store the binary image data into Glance;
|
||||
# (2) Store the metadata about the image into Glance;
|
||||
# PS: The order does not matter.
|
||||
#
|
||||
# The REST API allows to do two steps in one if a Location header is provided with
|
||||
# the POST request. (REF: http://developer.openstack.org/api-ref-image-v2.html)
|
||||
#
|
||||
# However the python API doesn't support a customized header in POST request.
|
||||
# So we have to do two steps in two calls.
|
||||
#
|
||||
# The good thing is: the API does support (2) perfectly, but for (1) it is only
|
||||
# accepting the data from local, not remote URL. So... Ur... Let's keep the CLI
|
||||
# version as the workaround for now.
|
||||
|
||||
# # upload in glance
|
||||
# image = glance_client.images.create(
|
||||
# name=str(final_image_name), disk_format="qcow2", container_format="bare",
|
||||
# Location=image_url)
|
||||
# glance_client.images.add_location(image.id, image_url, image)
|
||||
|
||||
# sys.exit(0)
|
||||
# for retry_attempt in range(retry_count):
|
||||
# if image.status == "active":
|
||||
# print 'Image: %s successfully uploaded to Nova' % (final_image_name)
|
||||
# return 1
|
||||
# # Sleep between retries
|
||||
# if self.config.debug:
|
||||
# print "Image is not yet active, retrying %s of %s... [%s]" \
|
||||
# % ((retry_attempt + 1), retry_count, image.status)
|
||||
# time.sleep(5)
|
||||
|
||||
# upload in glance
|
||||
glance_cmd = "glance image-create --name=\"" + str(final_image_name) + \
|
||||
"\" --disk-format=qcow2" + " --container-format=bare " + \
|
||||
" --is-public True --copy-from " + image_url
|
||||
if self.config.debug:
|
||||
print "Will update image to glance via CLI: %s" % (glance_cmd)
|
||||
subprocess.check_output(glance_cmd, shell=True)
|
||||
|
||||
# check for the image in glance
|
||||
glance_check_cmd = "glance image-list --name \"" + str(final_image_name) + "\""
|
||||
for retry_attempt in range(retry_count):
|
||||
result = subprocess.check_output(glance_check_cmd, shell=True)
|
||||
if "active" in result:
|
||||
print 'Image: %s successfully uploaded to Nova' % (final_image_name)
|
||||
return 1
|
||||
# Sleep between retries
|
||||
if self.config.debug:
|
||||
print "Image not yet active, retrying %s of %s..." \
|
||||
% ((retry_attempt + 1), retry_count)
|
||||
time.sleep(2)
|
||||
|
||||
print 'ERROR: Cannot upload image %s from URL: %s' % (final_image_name, image_url)
|
||||
return 0
|
||||
|
||||
# Remove keypair name from openstack if exists
|
||||
def remove_public_key(self, name):
|
||||
keypair_list = self.novaclient.keypairs.list()
|
||||
for key in keypair_list:
|
||||
if key.name == name:
|
||||
self.novaclient.keypairs.delete(name)
|
||||
print 'Removed public key %s' % (name)
|
||||
break
|
||||
|
||||
# Test if keypair file is present if not create it
|
||||
def create_keypair(self, name, private_key_pair_file):
|
||||
self.remove_public_key(name)
|
||||
keypair = self.novaclient.keypairs.create(name)
|
||||
# Now write the keypair to the file if requested
|
||||
if private_key_pair_file:
|
||||
kpf = os.open(private_key_pair_file,
|
||||
os.O_WRONLY | os.O_CREAT, 0o600)
|
||||
with os.fdopen(kpf, 'w') as kpf:
|
||||
kpf.write(keypair.private_key)
|
||||
return keypair
|
||||
|
||||
# Add an existing public key to openstack
|
||||
def add_public_key(self, name, public_key_file):
|
||||
self.remove_public_key(name)
|
||||
# extract the public key from the file
|
||||
public_key = None
|
||||
try:
|
||||
with open(os.path.expanduser(public_key_file)) as pkf:
|
||||
public_key = pkf.read()
|
||||
except IOError as exc:
|
||||
print 'ERROR: Cannot open public key file %s: %s' % \
|
||||
(public_key_file, exc)
|
||||
return None
|
||||
keypair = self.novaclient.keypairs.create(name, public_key)
|
||||
return keypair
|
||||
|
||||
def init_key_pair(self, kp_name, ssh_access):
|
||||
'''Initialize the key pair for all test VMs
|
||||
if a key pair is specified in access, use that key pair else
|
||||
create a temporary key pair
|
||||
'''
|
||||
if ssh_access.public_key_file:
|
||||
return self.add_public_key(kp_name, ssh_access.public_key_file)
|
||||
else:
|
||||
keypair = self.create_keypair(kp_name, None)
|
||||
ssh_access.private_key = keypair.private_key
|
||||
return keypair
|
||||
|
||||
def find_network(self, label):
|
||||
net = self.novaclient.networks.find(label=label)
|
||||
return net
|
||||
|
||||
# Create a server instance with name vmname
|
||||
# and check that it gets into the ACTIVE state
|
||||
def create_server(self, vmname, image, flavor, key_name,
|
||||
nic, sec_group, avail_zone=None, user_data=None,
|
||||
config_drive=None,
|
||||
retry_count=10):
|
||||
|
||||
if sec_group:
|
||||
security_groups = [sec_group.id]
|
||||
else:
|
||||
security_groups = None
|
||||
# Also attach the created security group for the test
|
||||
instance = self.novaclient.servers.create(name=vmname,
|
||||
image=image,
|
||||
flavor=flavor,
|
||||
key_name=key_name,
|
||||
nics=nic,
|
||||
availability_zone=avail_zone,
|
||||
userdata=user_data,
|
||||
config_drive=config_drive,
|
||||
security_groups=security_groups)
|
||||
if not instance:
|
||||
return None
|
||||
# Verify that the instance gets into the ACTIVE state
|
||||
for retry_attempt in range(retry_count):
|
||||
instance = self.novaclient.servers.get(instance.id)
|
||||
if instance.status == 'ACTIVE':
|
||||
return instance
|
||||
if instance.status == 'ERROR':
|
||||
print 'Instance creation error:' + instance.fault['message']
|
||||
break
|
||||
if self.config.debug:
|
||||
print "[%s] VM status=%s, retrying %s of %s..." \
|
||||
% (vmname, instance.status, (retry_attempt + 1), retry_count)
|
||||
time.sleep(2)
|
||||
|
||||
# instance not in ACTIVE state
|
||||
print('Instance failed status=' + instance.status)
|
||||
self.delete_server(instance)
|
||||
return None
|
||||
|
||||
def get_server_list(self):
|
||||
servers_list = self.novaclient.servers.list()
|
||||
return servers_list
|
||||
|
||||
def find_floating_ips(self):
|
||||
floating_ip = self.novaclient.floating_ips.list()
|
||||
return floating_ip
|
||||
|
||||
# Return the server network for a server
|
||||
def find_server_network(self, vmname):
|
||||
servers_list = self.get_server_list()
|
||||
for server in servers_list:
|
||||
if server.name == vmname and server.status == "ACTIVE":
|
||||
return server.networks
|
||||
return None
|
||||
|
||||
# Returns True if server is present false if not.
|
||||
# Retry for a few seconds since after VM creation sometimes
|
||||
# it takes a while to show up
|
||||
def find_server(self, vmname, retry_count):
|
||||
for retry_attempt in range(retry_count):
|
||||
servers_list = self.get_server_list()
|
||||
for server in servers_list:
|
||||
if server.name == vmname and server.status == "ACTIVE":
|
||||
return True
|
||||
# Sleep between retries
|
||||
if self.config.debug:
|
||||
print "[%s] VM not yet found, retrying %s of %s..." \
|
||||
% (vmname, (retry_attempt + 1), retry_count)
|
||||
time.sleep(2)
|
||||
print "[%s] VM not found, after %s attempts" % (vmname, retry_count)
|
||||
return False
|
||||
|
||||
# Returns True if server is found and deleted/False if not,
|
||||
# retry the delete if there is a delay
|
||||
def delete_server_by_name(self, vmname):
|
||||
servers_list = self.get_server_list()
|
||||
for server in servers_list:
|
||||
if server.name == vmname:
|
||||
print 'deleting server %s' % (server)
|
||||
self.novaclient.servers.delete(server)
|
||||
return True
|
||||
return False
|
||||
|
||||
def delete_server(self, server):
|
||||
self.novaclient.servers.delete(server)
|
||||
|
||||
def find_flavor(self, flavor_type):
|
||||
flavor = self.novaclient.flavors.find(name=flavor_type)
|
||||
return flavor
|
||||
|
||||
def normalize_az_host(self, az, host):
|
||||
if not az:
|
||||
az = self.config.availability_zone
|
||||
return az + ':' + host
|
||||
|
||||
def auto_fill_az(self, host_list, host):
|
||||
'''
|
||||
no az provided, if there is a host list we can auto-fill the az
|
||||
else we use the configured az if available
|
||||
else we return an error
|
||||
'''
|
||||
if host_list:
|
||||
for hyp in host_list:
|
||||
if hyp.host_name == host:
|
||||
return self.normalize_az_host(hyp.zone, host)
|
||||
# no match on host
|
||||
print('Error: passed host name does not exist: ' + host)
|
||||
return None
|
||||
if self.config.availability_zone:
|
||||
return self.normalize_az_host(None, host)
|
||||
print('Error: --hypervisor passed without an az and no az configured')
|
||||
return None
|
||||
|
||||
def sanitize_az_host(self, host_list, az_host):
|
||||
'''
|
||||
host_list: list of hosts as retrieved from openstack (can be empty)
|
||||
az_host: either a host or a az:host string
|
||||
if a host, will check host is in the list, find the corresponding az and
|
||||
return az:host
|
||||
if az:host is passed will check the host is in the list and az matches
|
||||
if host_list is empty, will return the configured az if there is no
|
||||
az passed
|
||||
'''
|
||||
if ':' in az_host:
|
||||
# no host_list, return as is (no check)
|
||||
if not host_list:
|
||||
return az_host
|
||||
# if there is a host_list, extract and verify the az and host
|
||||
az_host_list = az_host.split(':')
|
||||
zone = az_host_list[0]
|
||||
host = az_host_list[1]
|
||||
for hyp in host_list:
|
||||
if hyp.host_name == host:
|
||||
if hyp.zone == zone:
|
||||
# matches
|
||||
return az_host
|
||||
# else continue - another zone with same host name?
|
||||
# no match
|
||||
print('Error: no match for availability zone and host ' + az_host)
|
||||
return None
|
||||
else:
|
||||
return self.auto_fill_az(host_list, az_host)
|
||||
|
||||
#
|
||||
# Return a list of 0, 1 or 2 az:host
|
||||
#
|
||||
# The list is computed as follows:
|
||||
# The list of all hosts is retrieved first from openstack
|
||||
# if this fails, checks and az auto-fill are disabled
|
||||
#
|
||||
# If the user provides a list of hypervisors (--hypervisor)
|
||||
# that list is checked and returned
|
||||
#
|
||||
# If the user provides a configured az name (config.availability_zone)
|
||||
# up to the first 2 hosts from the list that match the az are returned
|
||||
#
|
||||
# If the user did not configure an az name
|
||||
# up to the first 2 hosts from the list are returned
|
||||
# Possible return values:
|
||||
# [ az ]
|
||||
# [ az:hyp ]
|
||||
# [ az1:hyp1, az2:hyp2 ]
|
||||
# [] if an error occurred (error message printed to console)
|
||||
#
|
||||
def get_az_host_list(self):
|
||||
avail_list = []
|
||||
host_list = []
|
||||
|
||||
try:
|
||||
host_list = self.novaclient.services.list()
|
||||
except novaclient.exceptions.Forbidden:
|
||||
print ('Warning: Operation Forbidden: could not retrieve list of hosts'
|
||||
' (likely no permission)')
|
||||
|
||||
# the user has specified a list of 1 or 2 hypervisors to use
|
||||
if self.config.hypervisors:
|
||||
for hyp in self.config.hypervisors:
|
||||
hyp = self.sanitize_az_host(host_list, hyp)
|
||||
if hyp:
|
||||
avail_list.append(hyp)
|
||||
else:
|
||||
return []
|
||||
# if the user did not specify an az, insert the configured az
|
||||
if ':' not in hyp:
|
||||
if self.config.availability_zone:
|
||||
hyp = self.normalize_az_host(None, hyp)
|
||||
else:
|
||||
return []
|
||||
# pick first 2 matches at most
|
||||
if len(avail_list) == 2:
|
||||
break
|
||||
print 'Using hypervisors:' + ', '.join(avail_list)
|
||||
else:
|
||||
for host in host_list:
|
||||
# this host must be a compute node
|
||||
if host.binary != 'nova-compute' or host.state != 'up':
|
||||
continue
|
||||
candidate = None
|
||||
if self.config.availability_zone:
|
||||
if host.zone == self.config.availability_zone:
|
||||
candidate = self.normalize_az_host(None, host.host)
|
||||
else:
|
||||
candidate = self.normalize_az_host(host.zone, host.host)
|
||||
if candidate:
|
||||
avail_list.append(candidate)
|
||||
# pick first 2 matches at most
|
||||
if len(avail_list) == 2:
|
||||
break
|
||||
|
||||
# if empty we insert the configured az
|
||||
if not avail_list:
|
||||
|
||||
if not self.config.availability_zone:
|
||||
print('Error: availability_zone must be configured')
|
||||
elif host_list:
|
||||
print('Error: no host matching the selection for availability zone: '
|
||||
+ self.config.availability_zone)
|
||||
avail_list = []
|
||||
else:
|
||||
avail_list = [self.config.availability_zone]
|
||||
return avail_list
|
||||
|
||||
# Given 2 VMs test if they are running on same Host or not
|
||||
def check_vm_placement(self, vm_instance1, vm_instance2):
|
||||
try:
|
||||
server_instance_1 = self.novaclient.servers.get(vm_instance1)
|
||||
server_instance_2 = self.novaclient.servers.get(vm_instance2)
|
||||
if server_instance_1.hostId == server_instance_2.hostId:
|
||||
return True
|
||||
else:
|
||||
return False
|
||||
except novaclient.exceptions:
|
||||
print "Exception in retrieving the hostId of servers"
|
||||
|
||||
# Create a new security group with appropriate rules
|
||||
def security_group_create(self):
|
||||
# check first the security group exists
|
||||
# May throw exceptions.NoUniqueMatch or NotFound
|
||||
try:
|
||||
group = self.novaclient.security_groups.find(name=self.config.security_group_name)
|
||||
return group
|
||||
except exceptions.NotFound:
|
||||
group = self.novaclient.security_groups.create(name=self.config.security_group_name,
|
||||
description="PNS Security group")
|
||||
# Once security group try to find it iteratively
|
||||
# (this check may no longer be necessary)
|
||||
for _ in range(self.config.generic_retry_count):
|
||||
group = self.novaclient.security_groups.get(group)
|
||||
if group:
|
||||
self.security_group_add_rules(group)
|
||||
return group
|
||||
else:
|
||||
time.sleep(1)
|
||||
return None
|
||||
# except exceptions.NoUniqueMatch as exc:
|
||||
# raise exc
|
||||
|
||||
# Delete a security group
|
||||
def security_group_delete(self, group):
|
||||
if group:
|
||||
print "Deleting security group"
|
||||
self.novaclient.security_groups.delete(group)
|
||||
|
||||
# Add rules to the security group
|
||||
def security_group_add_rules(self, group):
|
||||
# Allow ping traffic
|
||||
self.novaclient.security_group_rules.create(group.id,
|
||||
ip_protocol="icmp",
|
||||
from_port=-1,
|
||||
to_port=-1)
|
||||
if self.config.ipv6_mode:
|
||||
self.novaclient.security_group_rules.create(group.id,
|
||||
ip_protocol="icmp",
|
||||
from_port=-1,
|
||||
to_port=-1,
|
||||
cidr="::/0")
|
||||
# Allow SSH traffic
|
||||
self.novaclient.security_group_rules.create(group.id,
|
||||
ip_protocol="tcp",
|
||||
from_port=22,
|
||||
to_port=22)
|
||||
# Allow TCP/UDP traffic for perf tools like iperf/nuttcp
|
||||
# 5001: Data traffic (standard iperf data port)
|
||||
# 5002: Control traffic (non standard)
|
||||
# note that 5000/tcp is already picked by openstack keystone
|
||||
if not self.config.ipv6_mode:
|
||||
self.novaclient.security_group_rules.create(group.id,
|
||||
ip_protocol="tcp",
|
||||
from_port=5001,
|
||||
to_port=5002)
|
||||
self.novaclient.security_group_rules.create(group.id,
|
||||
ip_protocol="udp",
|
||||
from_port=5001,
|
||||
to_port=5001)
|
||||
else:
|
||||
# IPV6 rules addition
|
||||
self.novaclient.security_group_rules.create(group.id,
|
||||
ip_protocol="tcp",
|
||||
from_port=5001,
|
||||
to_port=5002,
|
||||
cidr="::/0")
|
||||
self.novaclient.security_group_rules.create(group.id,
|
||||
ip_protocol="udp",
|
||||
from_port=5001,
|
||||
to_port=5001,
|
||||
cidr="::/0")
|
||||
110
credentials.py
110
credentials.py
@ -1,110 +0,0 @@
|
||||
# Copyright 2014 Cisco Systems, Inc. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
|
||||
# Module for credentials in Openstack
|
||||
import getpass
|
||||
import os
|
||||
import re
|
||||
|
||||
|
||||
class Credentials(object):
|
||||
|
||||
def get_credentials(self):
|
||||
dct = {}
|
||||
dct['username'] = self.rc_username
|
||||
dct['password'] = self.rc_password
|
||||
dct['auth_url'] = self.rc_auth_url
|
||||
dct['tenant_name'] = self.rc_tenant_name
|
||||
return dct
|
||||
|
||||
def get_nova_credentials(self):
|
||||
dct = {}
|
||||
dct['username'] = self.rc_username
|
||||
dct['api_key'] = self.rc_password
|
||||
dct['auth_url'] = self.rc_auth_url
|
||||
dct['project_id'] = self.rc_tenant_name
|
||||
return dct
|
||||
|
||||
def get_nova_credentials_v2(self):
|
||||
dct = self.get_nova_credentials()
|
||||
dct['version'] = 2
|
||||
return dct
|
||||
|
||||
#
|
||||
# Read a openrc file and take care of the password
|
||||
# The 2 args are passed from the command line and can be None
|
||||
#
|
||||
def __init__(self, openrc_file, pwd, no_env):
|
||||
self.rc_password = None
|
||||
self.rc_username = None
|
||||
self.rc_tenant_name = None
|
||||
self.rc_auth_url = None
|
||||
success = True
|
||||
|
||||
if openrc_file:
|
||||
if os.path.exists(openrc_file):
|
||||
export_re = re.compile('export OS_([A-Z_]*)="?(.*)')
|
||||
for line in open(openrc_file):
|
||||
line = line.strip()
|
||||
mstr = export_re.match(line)
|
||||
if mstr:
|
||||
# get rif of posible trailing double quote
|
||||
# the first one was removed by the re
|
||||
name = mstr.group(1)
|
||||
value = mstr.group(2)
|
||||
if value.endswith('"'):
|
||||
value = value[:-1]
|
||||
# get rid of password assignment
|
||||
# echo "Please enter your OpenStack Password: "
|
||||
# read -sr OS_PASSWORD_INPUT
|
||||
# export OS_PASSWORD=$OS_PASSWORD_INPUT
|
||||
if value.startswith('$'):
|
||||
continue
|
||||
# now match against wanted variable names
|
||||
if name == 'USERNAME':
|
||||
self.rc_username = value
|
||||
elif name == 'AUTH_URL':
|
||||
self.rc_auth_url = value
|
||||
elif name == 'TENANT_NAME':
|
||||
self.rc_tenant_name = value
|
||||
else:
|
||||
print 'Error: rc file does not exist %s' % (openrc_file)
|
||||
success = False
|
||||
elif not no_env:
|
||||
# no openrc file passed - we assume the variables have been
|
||||
# sourced by the calling shell
|
||||
# just check that they are present
|
||||
for varname in ['OS_USERNAME', 'OS_AUTH_URL', 'OS_TENANT_NAME']:
|
||||
if varname not in os.environ:
|
||||
# print 'Warning: %s is missing' % (varname)
|
||||
success = False
|
||||
if success:
|
||||
self.rc_username = os.environ['OS_USERNAME']
|
||||
self.rc_auth_url = os.environ['OS_AUTH_URL']
|
||||
self.rc_tenant_name = os.environ['OS_TENANT_NAME']
|
||||
|
||||
# always override with CLI argument if provided
|
||||
if pwd:
|
||||
self.rc_password = pwd
|
||||
# if password not know, check from env variable
|
||||
elif self.rc_auth_url and not self.rc_password and success:
|
||||
if 'OS_PASSWORD' in os.environ and not no_env:
|
||||
self.rc_password = os.environ['OS_PASSWORD']
|
||||
else:
|
||||
# interactively ask for password
|
||||
self.rc_password = getpass.getpass(
|
||||
'Please enter your OpenStack Password: ')
|
||||
if not self.rc_password:
|
||||
self.rc_password = ""
|
||||
@ -85,17 +85,17 @@ qthelp:
|
||||
@echo
|
||||
@echo "Build finished; now you can run "qcollectiongenerator" with the" \
|
||||
".qhcp project file in $(BUILDDIR)/qthelp, like this:"
|
||||
@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/vmtp.qhcp"
|
||||
@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/kloudbuster.qhcp"
|
||||
@echo "To view the help file:"
|
||||
@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/vmtp.qhc"
|
||||
@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/kloudbuster.qhc"
|
||||
|
||||
devhelp:
|
||||
$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
|
||||
@echo
|
||||
@echo "Build finished."
|
||||
@echo "To view the help file:"
|
||||
@echo "# mkdir -p $$HOME/.local/share/devhelp/vmtp"
|
||||
@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/vmtp"
|
||||
@echo "# mkdir -p $$HOME/.local/share/devhelp/kloudbuster"
|
||||
@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/kloudbuster"
|
||||
@echo "# devhelp"
|
||||
|
||||
epub:
|
||||
|
||||
@ -1,409 +0,0 @@
|
||||
{
|
||||
"args": "vmtp.py -c cfg.default.yaml -r ../admin-openrc.sh -p <MASKED> --json juno_ovs_vxlan_2.json --mongod_server 172.29.87.29 --controller-node <MASKED> -d --test_description Yichen's testbed",
|
||||
"auth_url": "http://172.29.87.180:5000/v2.0",
|
||||
"cpu_info": "40 * Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz",
|
||||
"date": "2015-03-04 22:33:40",
|
||||
"distro": "CentOS Linux 7",
|
||||
"encapsulation": "vxlan",
|
||||
"flows": [
|
||||
{
|
||||
"az_from": "nova:hh23-6",
|
||||
"az_to": "nova:hh23-6",
|
||||
"desc": "VM to VM same network fixed IP (intra-node)",
|
||||
"distro_id": "Ubuntu",
|
||||
"distro_version": "14.04",
|
||||
"ip_from": "192.168.1.4",
|
||||
"ip_to": "192.168.1.2",
|
||||
"results": [
|
||||
{
|
||||
"pkt_size": 65536,
|
||||
"protocol": "TCP",
|
||||
"rtt_ms": 0.28,
|
||||
"throughput_kbps": 14318464,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"pkt_size": 65536,
|
||||
"protocol": "TCP",
|
||||
"rtt_ms": 0.12,
|
||||
"throughput_kbps": 14426352,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"pkt_size": 65536,
|
||||
"protocol": "TCP",
|
||||
"rtt_ms": 0.13,
|
||||
"throughput_kbps": 14247563,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"loss_rate": 0.11,
|
||||
"pkt_size": 128,
|
||||
"protocol": "UDP",
|
||||
"throughput_kbps": 127744,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"loss_rate": 0.12,
|
||||
"pkt_size": 1024,
|
||||
"protocol": "UDP",
|
||||
"throughput_kbps": 1021703,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"loss_rate": 0.17,
|
||||
"pkt_size": 8192,
|
||||
"protocol": "UDP",
|
||||
"throughput_kbps": 2496542,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"protocol": "ICMP",
|
||||
"rtt_avg_ms": "0.321",
|
||||
"rtt_max_ms": "0.741",
|
||||
"rtt_min_ms": "0.187",
|
||||
"rtt_stddev": "0.212",
|
||||
"rx_packets": "5",
|
||||
"tool": "ping",
|
||||
"tx_packets": "5"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"az_from": "nova:hh23-6",
|
||||
"az_to": "nova:hh23-6",
|
||||
"desc": "VM to VM different network fixed IP (intra-node)",
|
||||
"distro_id": "Ubuntu",
|
||||
"distro_version": "14.04",
|
||||
"ip_from": "192.168.2.2",
|
||||
"ip_to": "192.168.1.2",
|
||||
"results": [
|
||||
{
|
||||
"pkt_size": 65536,
|
||||
"protocol": "TCP",
|
||||
"retrans": 116,
|
||||
"rtt_ms": 0.67,
|
||||
"throughput_kbps": 1292957,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"pkt_size": 65536,
|
||||
"protocol": "TCP",
|
||||
"retrans": 218,
|
||||
"rtt_ms": 0.58,
|
||||
"throughput_kbps": 1602299,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"pkt_size": 65536,
|
||||
"protocol": "TCP",
|
||||
"retrans": 606,
|
||||
"rtt_ms": 0.59,
|
||||
"throughput_kbps": 1583186,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"loss_rate": 0.94,
|
||||
"pkt_size": 128,
|
||||
"protocol": "UDP",
|
||||
"throughput_kbps": 152745,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"loss_rate": 0.39,
|
||||
"pkt_size": 1024,
|
||||
"protocol": "UDP",
|
||||
"throughput_kbps": 1222784,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"loss_rate": 2.52,
|
||||
"pkt_size": 8192,
|
||||
"protocol": "UDP",
|
||||
"throughput_kbps": 1342442,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"protocol": "ICMP",
|
||||
"rtt_avg_ms": "0.771",
|
||||
"rtt_max_ms": "1.126",
|
||||
"rtt_min_ms": "0.677",
|
||||
"rtt_stddev": "0.180",
|
||||
"rx_packets": "5",
|
||||
"tool": "ping",
|
||||
"tx_packets": "5"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"az_from": "nova:hh23-6",
|
||||
"az_to": "nova:hh23-6",
|
||||
"desc": "VM to VM different network floating IP (intra-node)",
|
||||
"distro_id": "Ubuntu",
|
||||
"distro_version": "14.04",
|
||||
"ip_from": "192.168.2.2",
|
||||
"ip_to": "172.29.87.183",
|
||||
"results": [
|
||||
{
|
||||
"pkt_size": 65536,
|
||||
"protocol": "TCP",
|
||||
"retrans": 560,
|
||||
"rtt_ms": 0.69,
|
||||
"throughput_kbps": 1407148,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"pkt_size": 65536,
|
||||
"protocol": "TCP",
|
||||
"retrans": 184,
|
||||
"rtt_ms": 0.62,
|
||||
"throughput_kbps": 1475068,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"pkt_size": 65536,
|
||||
"protocol": "TCP",
|
||||
"retrans": 310,
|
||||
"rtt_ms": 0.59,
|
||||
"throughput_kbps": 1529674,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"loss_rate": 3.62,
|
||||
"pkt_size": 128,
|
||||
"protocol": "UDP",
|
||||
"throughput_kbps": 153493,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"loss_rate": 4.14,
|
||||
"pkt_size": 1024,
|
||||
"protocol": "UDP",
|
||||
"throughput_kbps": 1241424,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"loss_rate": 4.37,
|
||||
"pkt_size": 8192,
|
||||
"protocol": "UDP",
|
||||
"throughput_kbps": 1311624,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"protocol": "ICMP",
|
||||
"rtt_avg_ms": "0.646",
|
||||
"rtt_max_ms": "0.693",
|
||||
"rtt_min_ms": "0.613",
|
||||
"rtt_stddev": "0.043",
|
||||
"rx_packets": "5",
|
||||
"tool": "ping",
|
||||
"tx_packets": "5"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"az_from": "nova:hh23-5",
|
||||
"az_to": "nova:hh23-6",
|
||||
"desc": "VM to VM same network fixed IP (inter-node)",
|
||||
"distro_id": "Ubuntu",
|
||||
"distro_version": "14.04",
|
||||
"ip_from": "192.168.1.5",
|
||||
"ip_to": "192.168.1.2",
|
||||
"results": [
|
||||
{
|
||||
"pkt_size": 65536,
|
||||
"protocol": "TCP",
|
||||
"retrans": 99,
|
||||
"rtt_ms": 0.34,
|
||||
"throughput_kbps": 2340466,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"pkt_size": 65536,
|
||||
"protocol": "TCP",
|
||||
"retrans": 67,
|
||||
"rtt_ms": 0.43,
|
||||
"throughput_kbps": 2313315,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"pkt_size": 65536,
|
||||
"protocol": "TCP",
|
||||
"retrans": 63,
|
||||
"rtt_ms": 0.32,
|
||||
"throughput_kbps": 2020005,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"loss_rate": 50.66,
|
||||
"pkt_size": 128,
|
||||
"protocol": "UDP",
|
||||
"throughput_kbps": 76095,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"loss_rate": 24.04,
|
||||
"pkt_size": 1024,
|
||||
"protocol": "UDP",
|
||||
"throughput_kbps": 920877,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"loss_rate": 28.84,
|
||||
"pkt_size": 8192,
|
||||
"protocol": "UDP",
|
||||
"throughput_kbps": 1901142,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"protocol": "ICMP",
|
||||
"rtt_avg_ms": "0.657",
|
||||
"rtt_max_ms": "1.555",
|
||||
"rtt_min_ms": "0.331",
|
||||
"rtt_stddev": "0.453",
|
||||
"rx_packets": "5",
|
||||
"tool": "ping",
|
||||
"tx_packets": "5"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"az_from": "nova:hh23-5",
|
||||
"az_to": "nova:hh23-6",
|
||||
"desc": "VM to VM different network fixed IP (inter-node)",
|
||||
"distro_id": "Ubuntu",
|
||||
"distro_version": "14.04",
|
||||
"ip_from": "192.168.2.4",
|
||||
"ip_to": "192.168.1.2",
|
||||
"results": [
|
||||
{
|
||||
"pkt_size": 65536,
|
||||
"protocol": "TCP",
|
||||
"retrans": 121,
|
||||
"rtt_ms": 0.68,
|
||||
"throughput_kbps": 1344370,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"pkt_size": 65536,
|
||||
"protocol": "TCP",
|
||||
"retrans": 224,
|
||||
"rtt_ms": 0.61,
|
||||
"throughput_kbps": 1448398,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"pkt_size": 65536,
|
||||
"protocol": "TCP",
|
||||
"retrans": 75,
|
||||
"rtt_ms": 0.5,
|
||||
"throughput_kbps": 1301634,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"loss_rate": 1.04,
|
||||
"pkt_size": 128,
|
||||
"protocol": "UDP",
|
||||
"throughput_kbps": 161581,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"loss_rate": 0.98,
|
||||
"pkt_size": 1024,
|
||||
"protocol": "UDP",
|
||||
"throughput_kbps": 1207335,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"loss_rate": 3.82,
|
||||
"pkt_size": 8192,
|
||||
"protocol": "UDP",
|
||||
"throughput_kbps": 1330237,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"protocol": "ICMP",
|
||||
"rtt_avg_ms": "0.648",
|
||||
"rtt_max_ms": "0.984",
|
||||
"rtt_min_ms": "0.489",
|
||||
"rtt_stddev": "0.175",
|
||||
"rx_packets": "5",
|
||||
"tool": "ping",
|
||||
"tx_packets": "5"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"az_from": "nova:hh23-5",
|
||||
"az_to": "nova:hh23-6",
|
||||
"desc": "VM to VM different network floating IP (inter-node)",
|
||||
"distro_id": "Ubuntu",
|
||||
"distro_version": "14.04",
|
||||
"ip_from": "192.168.2.4",
|
||||
"ip_to": "172.29.87.183",
|
||||
"results": [
|
||||
{
|
||||
"pkt_size": 65536,
|
||||
"protocol": "TCP",
|
||||
"retrans": 201,
|
||||
"rtt_ms": 0.65,
|
||||
"throughput_kbps": 1371518,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"pkt_size": 65536,
|
||||
"protocol": "TCP",
|
||||
"retrans": 133,
|
||||
"rtt_ms": 0.57,
|
||||
"throughput_kbps": 1388169,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"pkt_size": 65536,
|
||||
"protocol": "TCP",
|
||||
"retrans": 68,
|
||||
"rtt_ms": 0.56,
|
||||
"throughput_kbps": 1250003,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"loss_rate": 2.66,
|
||||
"pkt_size": 128,
|
||||
"protocol": "UDP",
|
||||
"throughput_kbps": 148525,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"loss_rate": 2.02,
|
||||
"pkt_size": 1024,
|
||||
"protocol": "UDP",
|
||||
"throughput_kbps": 1174606,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"loss_rate": 1.12,
|
||||
"pkt_size": 8192,
|
||||
"protocol": "UDP",
|
||||
"throughput_kbps": 1310265,
|
||||
"tool": "nuttcp-7.3.2"
|
||||
},
|
||||
{
|
||||
"protocol": "ICMP",
|
||||
"rtt_avg_ms": "0.606",
|
||||
"rtt_max_ms": "0.698",
|
||||
"rtt_min_ms": "0.462",
|
||||
"rtt_stddev": "0.086",
|
||||
"rx_packets": "5",
|
||||
"tool": "ping",
|
||||
"tx_packets": "5"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"l2agent_type": "Open vSwitch agent",
|
||||
"l2agent_version": "OVS 2.3.1",
|
||||
"nic_name": "Cisco Systems Inc VIC Ethernet NIC (rev a2)",
|
||||
"openstack_version": "Juno (2014.2.1)",
|
||||
"test_description": "Yichen's testbed",
|
||||
"version": "2.0.1"
|
||||
}
|
||||
23
doc/source/conf.py
Normal file → Executable file
23
doc/source/conf.py
Normal file → Executable file
@ -1,6 +1,6 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
#
|
||||
# vmtp documentation build configuration file, created by
|
||||
# kloudbuster documentation build configuration file, created by
|
||||
# sphinx-quickstart on Fri Feb 13 14:43:59 2015.
|
||||
#
|
||||
# This file is execfile()d with the current directory set to its
|
||||
@ -14,9 +14,10 @@
|
||||
|
||||
import datetime
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
from pbr import version as pbr_ver
|
||||
|
||||
# If extensions (or modules to document with autodoc) are in another directory,
|
||||
# add these directories to sys.path here. If the directory is relative to the
|
||||
# documentation root, use os.path.abspath to make it absolute, like shown here.
|
||||
@ -50,7 +51,7 @@ source_suffix = '.rst'
|
||||
master_doc = 'index'
|
||||
|
||||
# General information about the project.
|
||||
project = u'vmtp'
|
||||
project = u'kloudbuster'
|
||||
copyright = u"%d, OpenStack Foundation" % datetime.datetime.now().year
|
||||
|
||||
# The version info for the project you're documenting, acts as replacement for
|
||||
@ -58,11 +59,9 @@ copyright = u"%d, OpenStack Foundation" % datetime.datetime.now().year
|
||||
# built documents.
|
||||
#
|
||||
# The short X.Y version.
|
||||
vmtp_file = open("../../vmtp.py")
|
||||
raw_text = vmtp_file.read()
|
||||
version = re.search(r"__version__\s=\s'(\d+\.\d+\.\d+)'", raw_text).group(1)
|
||||
version = pbr_ver.VersionInfo(project).version_string()
|
||||
# The full version, including alpha/beta/rc tags.
|
||||
release = version
|
||||
release = pbr_ver.VersionInfo(project).version_string_with_vcs()
|
||||
|
||||
# The language for content autogenerated by Sphinx. Refer to documentation
|
||||
# for a list of supported languages.
|
||||
@ -185,7 +184,7 @@ html_static_path = ['_static']
|
||||
#html_file_suffix = None
|
||||
|
||||
# Output file base name for HTML help builder.
|
||||
htmlhelp_basename = 'vmtpdoc'
|
||||
htmlhelp_basename = 'kloudbusterdoc'
|
||||
|
||||
|
||||
# -- Options for LaTeX output ---------------------------------------------
|
||||
@ -205,7 +204,7 @@ latex_elements = {
|
||||
# (source start file, target name, title,
|
||||
# author, documentclass [howto, manual, or own class]).
|
||||
latex_documents = [
|
||||
('index', 'vmtp.tex', u'vmtp Documentation',
|
||||
('index', 'kloudbuster.tex', u'KloudBuster Documentation',
|
||||
u'OpenStack Foundation', 'manual'),
|
||||
]
|
||||
|
||||
@ -235,7 +234,7 @@ latex_documents = [
|
||||
# One entry per manual page. List of tuples
|
||||
# (source start file, name, description, authors, manual section).
|
||||
man_pages = [
|
||||
('index', 'vmtp', u'vmtp Documentation',
|
||||
('index', 'kloudbuster', u'KloudBuster Documentation',
|
||||
[u'OpenStack Foundation'], 1)
|
||||
]
|
||||
|
||||
@ -249,8 +248,8 @@ man_pages = [
|
||||
# (source start file, target name, title, author,
|
||||
# dir menu entry, description, category)
|
||||
texinfo_documents = [
|
||||
('index', 'vmtp', u'vmtp Documentation',
|
||||
u'OpenStack Foundation', 'vmtp', 'One line description of project.',
|
||||
('index', 'kloudbuster', u'KloudBuster Documentation',
|
||||
u'OpenStack Foundation', 'kloudbuster', 'One line description of project.',
|
||||
'Miscellaneous'),
|
||||
]
|
||||
|
||||
|
||||
@ -1,115 +1,4 @@
|
||||
============
|
||||
Contributing
|
||||
============
|
||||
|
||||
Contribute to VMTP
|
||||
------------------
|
||||
|
||||
Below are a simplified version of the workflow to work on VMTP. For complete instructions, you have to follow the Developer's Guide in OpenStack official documents. Refer to :ref:`below section <developer_guide_of_openstack>` for links.
|
||||
|
||||
|
||||
Start working
|
||||
^^^^^^^^^^^^^
|
||||
|
||||
Before starting, a GitHub/StackForge respository based installation must be done. Refer :ref:`here <git_installation>` for detailed documentation.
|
||||
|
||||
1. From the root of your workspace, check out a new branch to work on::
|
||||
|
||||
$ git checkout -b <TOPIC-BRANCH>
|
||||
|
||||
2. Happy working on your code for features or bugfixes;
|
||||
|
||||
|
||||
Before Commit
|
||||
^^^^^^^^^^^^^
|
||||
|
||||
There are some criteria that are enforced to commit to VMTP. Below commands will perform the check and make sure your code complys with it.
|
||||
|
||||
3. PEP 8::
|
||||
|
||||
$ tox -epep8
|
||||
|
||||
**Note:** The first run usually takes longer, as tox will create a new virtual environment and download all dependencies. Once that is the done, further run will be very fast.
|
||||
|
||||
4. Run the test suite::
|
||||
|
||||
$ tox -epython27
|
||||
|
||||
5. If you made a documentation change (i.e. changes to .rst files), make sure the documentation is built as you expected::
|
||||
|
||||
$ cd <vmtp-ws-root>/doc
|
||||
$ make html
|
||||
|
||||
Once finished, the documentation in HTML format will be ready at <vmtp-ws-root>/doc/build/html.
|
||||
|
||||
|
||||
Submit Review
|
||||
^^^^^^^^^^^^^
|
||||
|
||||
6. Commit the code::
|
||||
|
||||
$ git commit -a
|
||||
|
||||
**Note:** For a feature commit, please supply a clear commit message indicating what the feature is; for a bugfix commit, please also containing a launchpad link to the bug you are working on.
|
||||
|
||||
7. Submit the review::
|
||||
|
||||
$ git review <TOPIC-BRANCH>
|
||||
|
||||
The members in the VMTP team will get notified once the Jenkin verification is passed. So watch your email from the review site, as it will contain the updates for your submission.
|
||||
|
||||
8. If the code is approved with a +2 review, Gerrit will automatically merge your code.
|
||||
|
||||
|
||||
File Bugs
|
||||
---------
|
||||
|
||||
Bugs should be filed on Launchpad, not GitHub:
|
||||
|
||||
https://bugs.launchpad.net/vmtp
|
||||
|
||||
|
||||
Build VMTP Docker Image
|
||||
-----------------------
|
||||
|
||||
Two files are used to build the Docker image: *Dockerfile* and *.dockerignore*. The former provides all the build instructions while the latter provides the list of files/directories that should not be copied to the Docker image.
|
||||
|
||||
In order to make the Docker image clean, remove all auto generated files from the root of your workspace first. It is strongly recommeneded to simply pull a new one from GitHub/StackForge. Specify the image name and the tag, and feed them to docker build. Examples to build the image with name "$USER/vmtp", tag "2.0.0" and "latest"::
|
||||
|
||||
$ cd <vmtp-ws-root>
|
||||
$ sudo docker build --tag=$USER/vmtp:2.0.0 .
|
||||
$ sudo docker build --tag=$USER/vmtp:latest .
|
||||
|
||||
The images should be available for use::
|
||||
|
||||
$ sudo docker images
|
||||
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
|
||||
ahothan/vmtp 2.0.0 9f08056496d7 27 hours ago 494.6 MB
|
||||
ahothan/vmtp latest 9f08056496d7 27 hours ago 494.6 MB
|
||||
|
||||
For exchanging purposes, the image could be saved to a tar archive. You can distribute the VMTP Docker image among your servers easily with this feature::
|
||||
|
||||
$ sudo docker save -o <IMAGE_FILE> <IMAGE_ID>
|
||||
|
||||
To publish the image to Docker Hub::
|
||||
|
||||
$ sudo docker login
|
||||
$ sudo docker push $USER/vmtp:2.0.0
|
||||
$ sudo docker push $USER/vmtp:latest
|
||||
|
||||
|
||||
.. _developer_guide_of_openstack:
|
||||
|
||||
Developer's Guide of OpenStack
|
||||
------------------------------
|
||||
|
||||
If you would like to contribute to the development of OpenStack, you must follow the steps in this page:
|
||||
|
||||
http://docs.openstack.org/infra/manual/developers.html
|
||||
|
||||
Once those steps have been completed, changes to OpenStack should be submitted for review via the Gerrit tool, following the workflow documented at:
|
||||
|
||||
http://docs.openstack.org/infra/manual/developers.html#development-workflow
|
||||
|
||||
Pull requests submitted through GitHub will be ignored.
|
||||
|
||||
.. include:: ../../CONTRIBUTING.rst
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 46 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 79 KiB |
@ -1,19 +0,0 @@
|
||||
==============
|
||||
Implementation
|
||||
==============
|
||||
|
||||
TCP Throughput Measurement
|
||||
--------------------------
|
||||
|
||||
The TCP throughput reported is measured using the default message size of the test tool (64KB with nuttcp). The TCP MSS (maximum segment size) used is the one suggested by the TCP-IP stack (which is dependent on the MTU).
|
||||
|
||||
|
||||
UDP Throughput Measurement
|
||||
--------------------------
|
||||
UDP throughput is tricky because of limitations of the performance tools used, limitations of the Linux kernel used and criteria for finding the throughput to report.
|
||||
|
||||
The default setting is to find the "optimal" throughput with packet loss rate within the 2%~5% range. This is achieved by successive iterations at different throughput values.
|
||||
|
||||
In some cases, it is not possible to converge with a loss rate within that range and trying to do so may require too many iterations. The algorithm used is empiric and tries to achieve a result within a reasonable and bounded number of iterations. In most cases the optimal throughput is found in less than 30 seconds for any given flow.
|
||||
|
||||
**Note:** UDP measurements are only available with nuttcp (not available with iperf).
|
||||
@ -1,10 +1,10 @@
|
||||
.. vmtp documentation master file, created by
|
||||
sphinx-quickstart on Fri Feb 13 14:43:59 2015.
|
||||
.. kloudbuster documentation master file, created by
|
||||
sphinx-quickstart on Tue Jul 9 22:26:36 2013.
|
||||
You can adapt this file completely to your liking, but it should at least
|
||||
contain the root `toctree` directive.
|
||||
|
||||
Welcome to vmtp's documentation!
|
||||
================================
|
||||
Welcome to kloudbuster's documentation!
|
||||
========================================================
|
||||
|
||||
Contents:
|
||||
|
||||
@ -14,15 +14,12 @@ Contents:
|
||||
readme
|
||||
installation
|
||||
usage
|
||||
setup
|
||||
implementation
|
||||
issue
|
||||
contributing
|
||||
|
||||
.. Indices and tables
|
||||
.. ==================
|
||||
Indices and tables
|
||||
==================
|
||||
|
||||
.. * :ref:`genindex`
|
||||
.. * :ref:`modindex`
|
||||
.. * :ref:`search`
|
||||
* :ref:`genindex`
|
||||
* :ref:`modindex`
|
||||
* :ref:`search`
|
||||
|
||||
|
||||
@ -2,118 +2,11 @@
|
||||
Installation
|
||||
============
|
||||
|
||||
There are two ways to install and run VMTP tool, Docker based, and GitHub/StackForge Repository based. Normally, a VMTP Docker image will satisfy most of use cases, and it is easy to start and use. Docker image is recommended if running under a production environment, or running through an automated or scheduled job. A git repository based installation gives more flexibility, and it is recommended for developing purposes.
|
||||
At the command line::
|
||||
|
||||
$ pip install kloudbuster
|
||||
|
||||
Docker based Installation
|
||||
-------------------------
|
||||
Or, if you have virtualenvwrapper installed::
|
||||
|
||||
Docker provides an easy and convenient way to run VMTP on Linux. The docker image pre-builds all the dependencies needed to run VMTP, including all the OpenStack python client libraries needed to access any OpenStack cloud, all the dependent python libraries and all the dependent distribution packages needed by these python libraries.
|
||||
|
||||
To run the container image all you need is docker.io installed on your Linux host. Refer `here <https://docs.docker.com/installation/#installation>`_ for details about how to install docker.
|
||||
|
||||
**Note:** An official image from Docker Hub is coming soon and this is a temporary private image.
|
||||
|
||||
Once the docker.io is installed, download the latest VMTP image from Docker Hub::
|
||||
|
||||
$ sudo docker pull ahothan/vmtp
|
||||
|
||||
The new image will be shown in the list::
|
||||
|
||||
$ sudo docker images
|
||||
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
|
||||
ahothan/vmtp 2.0.0 9f08056496d7 27 hours ago 494.6 MB
|
||||
ahothan/vmtp latest 9f08056496d7 27 hours ago 494.6 MB
|
||||
|
||||
Alternatively, for development or test purpose, a binary image could be loaded from a filesystem as well::
|
||||
|
||||
$ sudo docker load -i vmtp_image
|
||||
|
||||
Note that the image loaded from archive doesn't have a TAG, so the exact image ID must be specified to all docker commands mentioned below.
|
||||
|
||||
In its Docker image form, VMTP is located under the /vmtp directory in the container and can either take arguments from the host shell, or can be executed from inside the Docker image shell.
|
||||
|
||||
To run VMTP directly from the host shell::
|
||||
|
||||
$ sudo docker run <vmtp-docker-image-name> python /vmtp/vmtp.py <args>
|
||||
|
||||
To run VMTP from the Docker image shell::
|
||||
|
||||
$ sudo docker run <vmtp-docker-image-name>
|
||||
$ cd /vmtp.py
|
||||
$ python vmtp.py <args>
|
||||
|
||||
(then type exit to exit and terminate the container instance)
|
||||
|
||||
|
||||
Docker Shared Volume to Share Files with the Container
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
VMTP can accept files as input (e.g. configuration and openrc file) and can generate json results into a file. It is possible to use the VMTP Docker image with files persisted on the host by using Docker shared volumes.
|
||||
|
||||
For example, to get a copy of the VMTP default configuration file from the container::
|
||||
|
||||
$ sudo docker run -v $PWD:/vmtp/shared:rw -t <docker-vmtp-image-name> cp /vmtp/cfg.default.yaml /vmtp/shared/mycfg.yaml
|
||||
|
||||
The local directory to share ($PWD) is to be mapped to /vmtp/shared in the container in read/write mode. That way, mycfg.yaml will be copied to the local directory on the host.
|
||||
|
||||
Assume you have edited the configuration file "mycfg.yaml", retrieved an openrc file "admin-openrc.sh" from Horizon on the local directory, and would like to get results back in the "res.json" file. What you can do is to map the current directory ($PWD) to /vmtp/shared inside the container in read/write mode, then run the script inside the container and use use files from the shared directory.
|
||||
|
||||
E.g. From the host shell, you could do that in one-shot::
|
||||
|
||||
$ sudo docker run -v $PWD:/vmtp/shared:rw -t <docker-vmtp-image-name> python /vmtp/vmtp.py -c shared/mycfg.yaml -r shared/admin-openrc.sh -p admin --json shared/res.json
|
||||
$ cat res.json
|
||||
|
||||
Or from the Docker image shell::
|
||||
|
||||
$ sudo docker run -v $PWD:/vmtp/shared:rw -t <docker-vmtp-image-name>
|
||||
$ python /vmtp/vmtp.py -c shared/mycfg.yaml -r shared/admin-openrc.sh -p admin --json shared/res.json
|
||||
$ cat shared/res.json
|
||||
|
||||
|
||||
.. _git_installation:
|
||||
|
||||
GitHub/StackForge Repository based Installation
|
||||
-----------------------------------------------
|
||||
|
||||
It is recommended to run VMTP inside a virtual environment. However, it can be skipped if installed in a dedicated VM.
|
||||
|
||||
|
||||
Super quick installation on Ubuntu/Debian
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code::
|
||||
|
||||
$ sudo apt-get install python-dev python-virtualenv git git-review
|
||||
$ sudo apt-get install libxml2-dev libxslt-dev libffi-dev libz-dev libyaml-dev libssl-dev
|
||||
$ # create a virtual environment
|
||||
$ virtualenv ./vmtpenv
|
||||
$ source ./vmtpenv/bin/activate
|
||||
$ git clone git://git.openstack.org/stackforge/vmtp
|
||||
$ cd vmtp
|
||||
$ pip install -r requirements-dev.txt
|
||||
$ python vmtp.py -h
|
||||
|
||||
|
||||
Super quick installation on MacOSX
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
VMTP can run natively on MacOSX. These instructions have been verified to work on MacOSX 10.10 (Yosemite).
|
||||
|
||||
First, download XCode from App Store, then execute below commands:
|
||||
|
||||
.. code::
|
||||
|
||||
$ # Download the XCode command line tools
|
||||
$ code-select --install
|
||||
$ # Install pip
|
||||
$ sudo easy_install pip
|
||||
$ # Install python virtualenv
|
||||
$ sudo pip install virtualenv
|
||||
$ # create a virtual environment
|
||||
$ virtualenv ./vmtpenv
|
||||
$ source ./vmtpenv/bin/activate
|
||||
$ git clone git://git.openstack.org/stackforge/vmtp
|
||||
$ cd vmtp
|
||||
$ pip install -r requirements-dev.txt
|
||||
$ python vmtp.py -h
|
||||
$ mkvirtualenv kloudbuster
|
||||
$ pip install kloudbuster
|
||||
|
||||
@ -1,9 +0,0 @@
|
||||
========================
|
||||
Caveats and Known Issues
|
||||
========================
|
||||
|
||||
* UDP throughput is not available if iperf is selected (the iperf UDP reported results are not reliable enough for iterating)
|
||||
|
||||
* If VMTP hangs for native hosts throughputs, check firewall rules on the hosts to allow TCP/UDP ports 5001 and TCP port 5002
|
||||
|
||||
* When storing the results to JSON or MongoDB, the quotes in the command-line will not be saved. In a unix-like environment, the magic happened even before Python can see them. e.g. quotes get consumed, variables get interpolated, etc. Keep this in mind when you want to execute the command stored in "*args*", and pay more attention in any parameter that may have quotes inside like *test_description*.
|
||||
@ -1,33 +0,0 @@
|
||||
=====
|
||||
Setup
|
||||
=====
|
||||
|
||||
|
||||
SSH Authentication
|
||||
------------------
|
||||
|
||||
VMTP can optionally SSH to the following hosts:
|
||||
- OpenStack controller node (if the --controller-node option is used)
|
||||
- External host for cloud upload/download performance test (if the --external-host option is used)
|
||||
- Native host throughput (if the --host option is used)
|
||||
|
||||
To connect to these hosts, the SSH library used by VMTP will try a number of authentication methods:
|
||||
- if provided at the command line, try the provided password (e.g. --controller-node localadmin@10.1.1.78:secret)
|
||||
- user's personal private key (~/.ssh/id_rsa)
|
||||
- if provided in the configuration file, a specific private key file (private_key_file variable)
|
||||
|
||||
SSH to the test VMs is always based on key pairs with the following precedence:
|
||||
- if provided in the passed configuration file, use the configured key pair (private_key_file and public_key_file variables),
|
||||
- otherwise use the user's personal key pair (~/.ssh/id_rsa and ~/.ssh/id_rsa.pub)
|
||||
- otherwise if there is no personal key pair configured, create a temporary key pair to access all test VMs
|
||||
|
||||
To summarize:
|
||||
- if you have a personal key pair configured in your home directory, VMTP will use that key pair for all SSH connections (including to the test VMs)
|
||||
- if you want to use your personal key pair, there is nothing to do other than making sure that the targeted hosts have been configured with the associated public key
|
||||
|
||||
In any case make sure you specify the correct username.
|
||||
If there is a problem, you should see an error message and stack trace after the SSH library times out.
|
||||
|
||||
|
||||
|
||||
|
||||
@ -1,321 +1,7 @@
|
||||
=====
|
||||
========
|
||||
Usage
|
||||
=====
|
||||
========
|
||||
|
||||
VMTP Usage
|
||||
----------
|
||||
|
||||
.. code::
|
||||
|
||||
usage: vmtp.py [-h] [-c <config_file>] [-r <openrc_file>]
|
||||
[-m <gmond_ip>[:<port>]] [-p <password>] [-t <time>]
|
||||
[--host <user>@<host_ssh_ip>[:<password>:<server-listen-if-name>]]
|
||||
[--external-host <user>@<host_ssh_ip>[:password>]]
|
||||
[--controller-node <user>@<host_ssh_ip>[:<password>]]
|
||||
[--mongod-server <server ip>] [--json <file>]
|
||||
[--tp-tool <nuttcp|iperf>] [--hypervisor [<az>:] <hostname>]
|
||||
[--inter-node-only] [--protocols <T|U|I>]
|
||||
[--bandwidth <bandwidth>] [--tcpbuf <tcp_pkt_size1,...>]
|
||||
[--udpbuf <udp_pkt_size1,...>] [--no-env]
|
||||
[--vnic-type <direct|macvtap|normal>] [-d] [-v]
|
||||
[--stop-on-error] [--vm-image-url <url_to_image>]
|
||||
[--test-description <test_description>]
|
||||
|
||||
OpenStack VM Throughput V2.1.0
|
||||
|
||||
optional arguments:
|
||||
-h, --help show this help message and exit
|
||||
-c <config_file>, --config <config_file>
|
||||
override default values with a config file
|
||||
-r <openrc_file>, --rc <openrc_file>
|
||||
source OpenStack credentials from rc file
|
||||
-m <gmond_ip>[:<port>], --monitor <gmond_ip>[:<port>]
|
||||
Enable CPU monitoring (requires Ganglia)
|
||||
-p <password>, --password <password>
|
||||
OpenStack password
|
||||
-t <time>, --time <time>
|
||||
throughput test duration in seconds (default 10 sec)
|
||||
--host <user>@<host_ssh_ip>[:<password>:<server-listen-if-name>]
|
||||
native host throughput (password or public key
|
||||
required)
|
||||
--external-host <user>@<host_ssh_ip>[:password>]
|
||||
external-VM throughput (password or public key
|
||||
required)
|
||||
--controller-node <user>@<host_ssh_ip>[:<password>]
|
||||
controller node ssh (password or public key required)
|
||||
--mongod-server <server ip>
|
||||
provide mongoDB server IP to store results
|
||||
--json <file> store results in json format file
|
||||
--tp-tool <nuttcp|iperf>
|
||||
transport perf tool to use (default=nuttcp)
|
||||
--hypervisor [<az>:] <hostname>
|
||||
hypervisor to use (1 per arg, up to 2 args)
|
||||
--inter-node-only only measure inter-node
|
||||
--protocols <T|U|I> protocols T(TCP), U(UDP), I(ICMP) - default=TUI (all)
|
||||
--bandwidth <bandwidth>
|
||||
the bandwidth limit for TCP/UDP flows in K/M/Gbps,
|
||||
e.g. 128K/32M/5G. (default=no limit)
|
||||
--tcpbuf <tcp_pkt_size1,...>
|
||||
list of buffer length when transmitting over TCP in
|
||||
Bytes, e.g. --tcpbuf 8192,65536. (default=65536)
|
||||
--udpbuf <udp_pkt_size1,...>
|
||||
list of buffer length when transmitting over UDP in
|
||||
Bytes, e.g. --udpbuf 128,2048. (default=128,1024,8192)
|
||||
--no-env do not read env variables
|
||||
--vnic-type <direct|macvtap|normal>
|
||||
binding vnic type for test VMs
|
||||
-d, --debug debug flag (very verbose)
|
||||
-v, --version print version of this script and exit
|
||||
--stop-on-error Stop and keep everything as-is on error (must cleanup
|
||||
manually)
|
||||
--vm-image-url <url_to_image>
|
||||
URL to a Linux image in qcow2 format that can be
|
||||
downloaded from
|
||||
--test-description <test_description>
|
||||
The test description to be stored in JSON or MongoDB
|
||||
|
||||
Configuration File
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
|
||||
VMTP configuration files follow the yaml syntax and contain variables used by VMTP to run and collect performance data.
|
||||
The default configuration is stored in the cfg.default.yaml file.
|
||||
|
||||
Default values should be overwritten for any cloud under test by defining new variable values in a new configuration file that follows the same format.
|
||||
Variables that are not defined in the new configuration file will retain their default values.
|
||||
|
||||
The precedence order for configuration files is as follows:
|
||||
- the command line argument "-c <file>" has highest precedence
|
||||
- $HOME/.vmtp.yaml if the file exists in the user home directory
|
||||
- cfg.default.yaml has the lowest precedence (always exists in the VMTP package root directory)
|
||||
|
||||
To override a default value set in cfg.default.yaml, simply redefine that value in the configuration file passed in -c or in the $HOME/.vmtp.yaml file.
|
||||
Check the content of cfg.default.yaml file as it contains the list of configuration variables and instructions on how to set them.
|
||||
|
||||
**Note:** the configuration file is not needed if VMTP only runs the native host throughput option (*--host*)
|
||||
|
||||
|
||||
OpenStack openrc File
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
VMTP requires downloading an "openrc" file from the OpenStack Dashboard (Project|Acces&Security!Api Access|Download OpenStack RC File)
|
||||
|
||||
This file should then be passed to VMTP using the *-r* option or should be sourced prior to invoking VMTP.
|
||||
|
||||
**Note:** the openrc file is not needed if VMTP only runs the native host throughput option (*--host*)
|
||||
|
||||
|
||||
Access Info for Controller Node
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
By default, VMTP is not able to get the Linux distro nor the OpenStack version of the cloud deployment under test.
|
||||
However, by providing the credentials of the controller node under test, VMTP will try to fetch these information, and output them along in the JSON file or to the MongoDB server.
|
||||
For example to retrieve the OpenStack distribution information on a given controller node:
|
||||
|
||||
.. code:
|
||||
python vmtp.py --json tb172.json --test-description 'Testbed 172' --controller-node root@172.22.191.172
|
||||
|
||||
Bandwidth Limit for TCP/UDP Flow Measurements
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Specify a value in *--bandwidth* will limit the bandwidth when performing throughput tests.
|
||||
|
||||
The default behavior for both TCP/UDP are unlimited. For TCP, we are leveraging on the protocol itself to get the best performance; while for UDP, we are doing a binary search to find the optimal bandwidth.
|
||||
|
||||
This is useful when running vmtp on production clouds. The test tool will use up all the bandwidth that may be needed by any other live VMs if we don't set any bandwidth limit. This feature will help to prevent impacting other VMs while running the test tool.
|
||||
|
||||
|
||||
Host Selection and Availability Zone
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
VMTP requires 1 physical host to perform intra-node tests and 2 hosts to perform inter-node tests.
|
||||
There are multiple ways to specify the placement of test VMs to VMTP. By default, VMTP will pick the first 2 compute hosts it can find, regardless of the availability zone.
|
||||
|
||||
It is possible to limit the host selection to a specific availability zone by specifying its name in the yaml configuration file ('availability_name' parameter).
|
||||
|
||||
The *--hypervisor* argument can also be used to specify explicitly on which hosts to run the test VMs. The first *--hypervisor* argument specifies on which host to run the test server VM. The second *--hypervisor* argument (in the command line) specifies on which host to run the test client VMs.
|
||||
|
||||
The syntax to use for the argument value is either availability_zone and host name separated by a column (e.g. "--hypervisor nova:host26") or host name (e.g. "--hypervisor host12"). In the latter case, VMTP will automaticaly pick the availability zone of the host.
|
||||
|
||||
Picking a particular host can be handy for example when exact VM placement can impact the data path performance (for example rack based placement).
|
||||
|
||||
The value of the argument must match the hypervisor host name as known by OpenStack (or as displayed using "nova hypervisor-list").
|
||||
|
||||
If an availability zone is provided, VMTP will check that the host name exists in that availability zone.
|
||||
|
||||
|
||||
Upload Images to Glance
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
VMTP requires a Linux image available in Glance to spawn VMs. It could be uploaded manually through Horizon or CLI, or VMTP will try to upload the image defined in the configuration file automatically.
|
||||
|
||||
There is a candidate image defined in the default config already. It has been verified working, but of course it is OK to try other Linux distro as well.
|
||||
|
||||
**Note:** Due to the limitation of the Python glanceclient API (v2.0), it is not able to create the image directly from a remote URL. So the implementation of this feature used a glance CLI command instead. Be sure to source the OpenStack rc file first before running VMTP with this feature.
|
||||
|
||||
VNIC Type
|
||||
^^^^^^^^^
|
||||
|
||||
By default test VMs will be created with ports that have a "normal" VNIC type.
|
||||
To create test VMs with ports that use PCI passthrough SRIOV, specify "--vnic_type direct". This will assume that the host where the VM are instantiated have SRIOV capable NIC.
|
||||
An exception will be thrown if a test VM is lauched on a host that does not have SRIOV capable NIC or has not been configured to use such feature.
|
||||
|
||||
Quick guide to run VMTP on an OpenStack Cloud
|
||||
----------------------------------------------
|
||||
|
||||
Preparation
|
||||
^^^^^^^^^^^
|
||||
|
||||
* Step 1)
|
||||
|
||||
Download the openrc file from OpenStack Dashboard, and saved it to your local file system. (In Horizon dashboard: Project|Acces&Security!Api Access|Download OpenStack RC File)
|
||||
|
||||
* Step 2)
|
||||
|
||||
Create one configuration file for your specific cloud and use the *-c* option to pass that file name to VMTP. Parameters that you are most certainly required to change are:
|
||||
|
||||
**image_name**: The name of the Linux image that will run the test VMs created by vmtp. It must be set to an existing image available in openstack (check the name with Horizon or using "nova image-list" from the shell). Any recent Ubuntu or CentOS/Fedora image should work -- if needed you will need to upload an image to OpenStack manually prior to running VMTP.
|
||||
|
||||
**ssh_vm_username**: VM SSH username to use (specific to the image)
|
||||
|
||||
**flavor_type**: The flavor name to use (often specific to each cloud)
|
||||
|
||||
* Step 3)
|
||||
|
||||
Upload the Linux image to the OpenStack controller node, so that OpenStack is able to spawning VMs. You will be prompted an error if the image defined in the config file is not available to use when running the tool. The image can be uploaded using either Horizon dashboard, or the command below::
|
||||
|
||||
.. code::
|
||||
python vmtp.py -r admin-openrc.sh -p admin --vm_image_url http://<url_to_the_image>
|
||||
|
||||
**Note:** Currently, VMTP only supports the qcow2 format.
|
||||
|
||||
|
||||
Examples of running VMTP on an OpenStack Cloud
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Example 1: Typical Run
|
||||
""""""""""""""""""""""
|
||||
|
||||
Run VMTP on an OpenStack cloud with the default configuration file, use "admin-openrc.sh" as the rc file, and "admin" as the password::
|
||||
|
||||
.. code::
|
||||
python vmtp.py -r admin-openrc.sh -p admin
|
||||
|
||||
This will generate 6 standard sets of performance data:
|
||||
(1) VM to VM same network (intra-node, private fixed IP)
|
||||
(2) VM to VM different network (intra-node, L3 fixed IP)
|
||||
(3) VM to VM different network and tenant (intra-node, floating IP)
|
||||
(4) VM to VM same network (inter-node, private fixed IP)
|
||||
(5) VM to VM different network (inter-node, L3 fixed IP)
|
||||
(6) VM to VM different network and tenant (inter-node, floating IP)
|
||||
|
||||
By default, the performance data of all three protocols (TCP/UDP/ICMP) will be measured for each scenario mentioned above. However, it can be overridden by providing *--protocols*::
|
||||
|
||||
python vmtp.py -r admin-openrc.sh -p admin --protocols IT
|
||||
|
||||
This will tell VMTP to only collect ICMP and TCP measurements.
|
||||
|
||||
|
||||
Example 2: Cloud upload/download performance measurement
|
||||
""""""""""""""""""""""""""""""""""""""""""""""""""""""""
|
||||
|
||||
Run VMTP on an OpenStack cloud with a specified configuration file (mycfg.yaml), and saved the result to a JSON file::
|
||||
|
||||
.. code::
|
||||
python vmtp.py -c mycfg.yaml -r admin-openrc.sh -p admin --external_host localadmin@172.29.87.29 --json res.json
|
||||
|
||||
This run will generate 8 sets of performance data, the standard 6 sets mentioned above, plus two sets of upload/download performance data for both TCP and UDP.
|
||||
If you do not have ssh password-less access to the external host (public key) you must specify a password:
|
||||
|
||||
.. code::
|
||||
python vmtp.py -c mycfg.yaml -r admin-openrc.sh -p admin --external_host localadmin@172.29.87.29:secret --json res.json
|
||||
|
||||
Example 3: Store the OpenStack deployment details
|
||||
"""""""""""""""""""""""""""""""""""""""""""""""""
|
||||
|
||||
Run VMTP on an OpenStack cloud, fetch the defails of the deployment and store it to JSON file. Assume the controlloer node is on 192.168.12.34 with admin/admin::
|
||||
|
||||
.. code::
|
||||
python vmtp.py -r admin-openrc.sh -p admin --json res.json --controller-node root@192.168.12.34:admin
|
||||
|
||||
In addition, VMTP also supports to store the results to a MongoDB server::
|
||||
|
||||
.. code::
|
||||
python vmtp.py -r admin-openrc.sh -p admin --json res.json --mongod_server 172.29.87.29 --controller-node root@192.168.12.34:admin
|
||||
|
||||
Before storing info into MongoDB, some configurations are needed to change to fit in your environment. By default, VMTP will store to database "client_db" with collection name "pns_web_entry", and of course these can be changed in the configuration file. Below are the fields which are related to accessing MongoDB::
|
||||
|
||||
vmtp_mongod_port
|
||||
vmtp_db
|
||||
vmtp_collection
|
||||
|
||||
|
||||
Example 4: Specify which compute nodes to spawn VMs
|
||||
"""""""""""""""""""""""""""""""""""""""""""""""""""
|
||||
|
||||
Run VMTP on an OpenStack cloud, spawn the test server VM on tme212, and the test client VM on tme210. Save the result, and perform the inter-node measurement only::
|
||||
|
||||
.. code::
|
||||
python vmtp.py -r admin-openrc.sh -p admin --inter-node-only --json res.json --hypervisor tme212 --hypervisor tme210
|
||||
|
||||
|
||||
Example 5: Collect native host performance data
|
||||
"""""""""""""""""""""""""""""""""""""""""""""""
|
||||
|
||||
Run VMTP to get native host throughput between 172.29.87.29 and 172.29.87.30 using the localadmin ssh username and run each tcp/udp test session for 120 seconds (instead of the default 10 seconds)::
|
||||
|
||||
.. code::
|
||||
python vmtp.py --host localadmin@172.29.87.29 --host localadmin@172.29.87.30 --time 120
|
||||
|
||||
The first IP passed (*--host*) is always the one running the server side.
|
||||
If you do not have public keys setup on these targets, you must provide a password:
|
||||
|
||||
.. code::
|
||||
python vmtp.py --host localadmin@172.29.87.29:secret --host localadmin@172.29.87.30:secret --time 120
|
||||
|
||||
It is also possible to run VMTP between pre-existing VMs that are accessible through SSH (using floating IP) if you have the corresponding private key to access them.
|
||||
|
||||
In the case of servers that have multiple NIC and IP addresses, it is possible to specify the server side listening interface name to use (if you want the client side to connect using the associated IP address)
|
||||
For example, to measure throughput between 2 hosts using the network attached to the server interface "eth5"::
|
||||
|
||||
.. code::
|
||||
python vmtp.py --host localadmin@172.29.87.29::eth5 --host localadmin@172.29.87.30
|
||||
|
||||
|
||||
Example 6: IPV6 throughput measurement
|
||||
""""""""""""""""""""""""""""""""""""""
|
||||
|
||||
It is possible to use VMTP to measure throughput for IPv6
|
||||
|
||||
Set ipv6_mode to slaac, dhcpv6-stateful or dhcpv6-stateless. If SLAAC or DHCPv6 stateless is enabled make sure to have radvd packaged in as part of openstack install. For DHCPv6 stateful you need dnsmasq version >= 2.68. The test creates 2 networks and creates 1 IPv4 and 1 IPv6 subnet inside each of these networks. The subnets are created based on the IPv6 mode that you set in the configuration file. The Floating IP result case is skipped for IPv6 since there is no concept of a floating ip with IPv6.
|
||||
|
||||
Generating charts from JSON results
|
||||
-----------------------------------
|
||||
|
||||
.. code::
|
||||
usage: genchart.py [-h] [-c <file>] [-b] [-p <all|tcp|udp>] [-v]
|
||||
<file> [<file> ...]
|
||||
|
||||
VMTP Chart Generator V0.0.1
|
||||
|
||||
positional arguments:
|
||||
<file> vmtp json result file
|
||||
|
||||
optional arguments:
|
||||
-h, --help show this help message and exit
|
||||
-c <file>, --chart <file>
|
||||
create and save chart in html file
|
||||
-b, --browser display (-c) chart in the browser
|
||||
-p <all|tcp|udp>, --protocol <all|tcp|udp>
|
||||
select protocols:all, tcp, udp
|
||||
-v, --version print version of this script and exit
|
||||
|
||||
Examples of use:
|
||||
|
||||
Generate charts from the JSON results file "tb172.json", store resulting html to "tb172.html" and open that file in the browser:
|
||||
.. code::
|
||||
python genchart.py --chart tb172.html --browser tb172.json
|
||||
|
||||
Same but only show UDP numbers:
|
||||
.. code::
|
||||
python genchart.py --chart tb172.html --browser --protocol udp tb172.json
|
||||
To use kloudbuster in a project::
|
||||
|
||||
import kloudbuster
|
||||
|
||||
273
genchart.py
273
genchart.py
@ -1,273 +0,0 @@
|
||||
# Copyright 2014 Cisco Systems, Inc. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
|
||||
# This is an example of tool that can represent VMTP json results in
|
||||
# a nicer form using HTML and the Google Charts Javascript library
|
||||
#
|
||||
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import os.path
|
||||
import sys
|
||||
import webbrowser
|
||||
|
||||
__version__ = '0.0.1'
|
||||
|
||||
html_main_template = '''
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<script type="text/javascript" src="https://www.google.com/jsapi"></script>
|
||||
<script type="text/javascript">
|
||||
google.load("visualization", "1.1", {packages:["bar", "table"]});
|
||||
google.setOnLoadCallback(drawChart);
|
||||
function drawChart() {
|
||||
var options;
|
||||
var data;
|
||||
var chart;
|
||||
%s
|
||||
}
|
||||
</script>
|
||||
</head>
|
||||
<body>
|
||||
<table align="center">
|
||||
%s
|
||||
</table>
|
||||
</body>
|
||||
</html>
|
||||
'''
|
||||
|
||||
js_options_tpl = '''
|
||||
var options = {
|
||||
chart: {
|
||||
title: 'OpenStack Data Plane Performance (Gbps)',
|
||||
subtitle: '%s',
|
||||
}
|
||||
};
|
||||
'''
|
||||
|
||||
js_data_tpl = '''
|
||||
data = google.visualization.arrayToDataTable([
|
||||
%s
|
||||
]);
|
||||
chart = new %s(document.getElementById('vmtp-%s%d'));
|
||||
chart.draw(data, options);
|
||||
'''
|
||||
|
||||
div_tpl = ' ' * 6 + \
|
||||
'<tr><td><div id="vmtp-%s%d" style="width:900px;height:500px;">' + \
|
||||
'</div></td></tr>\n'
|
||||
|
||||
# must be exact match
|
||||
label_match = {
|
||||
'VM to VM same network fixed IP (inter-node)': 'L2',
|
||||
'VM to VM different network fixed IP (inter-node)': 'L3 fixed',
|
||||
'VM to VM different network floating IP (inter-node)': 'L3 floating',
|
||||
# This one is special because it is bidirectional
|
||||
'External-VM': ''
|
||||
}
|
||||
|
||||
prop_match = {
|
||||
'cpu_info': 'CPU Info',
|
||||
'distro': 'Host Linux distribution',
|
||||
'date': 'Date',
|
||||
'nic_name': 'NIC name',
|
||||
'openstack_version': 'OpenStack release',
|
||||
'test_description': 'Description',
|
||||
'version': 'VMTP version',
|
||||
'encapsulation': 'Encapsulation',
|
||||
'l2agent_type': 'L2 agent type'
|
||||
}
|
||||
|
||||
# what goes in the subtitle
|
||||
subtitle_match = ['test_description', 'openstack_version', 'distro',
|
||||
'encapsulation', 'l2agent_type']
|
||||
|
||||
class GoogleChartsBarChart:
|
||||
def __init__(self, results, protocols):
|
||||
self.results = results
|
||||
if protocols not in ['udp', 'tcp']:
|
||||
protocols = 'all'
|
||||
self.show_udp = protocols in ['all', 'udp']
|
||||
self.show_tcp = protocols in ['all', 'tcp']
|
||||
|
||||
def _get_subtitle(self, res):
|
||||
sub = 'inter-node'
|
||||
for key in subtitle_match:
|
||||
if key in res:
|
||||
sub += ' ' + res[key].encode('ascii')
|
||||
return sub
|
||||
|
||||
def _get_categories(self, flow):
|
||||
categories = ['Flow']
|
||||
# start with UDP first
|
||||
if self.show_udp:
|
||||
# iterate through all results in this flow to pick the sizes
|
||||
for flow_res in flow['results']:
|
||||
if flow_res['protocol'] == 'UDP':
|
||||
categories.append('UDP ' + str(flow_res['pkt_size']))
|
||||
if self.show_tcp:
|
||||
categories.append('TCP')
|
||||
return categories
|
||||
|
||||
def _get_flow_data(self, label, flow, reverse=False):
|
||||
data = [label]
|
||||
# start with UDP first
|
||||
if self.show_udp:
|
||||
# iterate through all results in this flow to pick the sizes
|
||||
for flow_res in flow['results']:
|
||||
reverse_flow = 'direction' in flow_res
|
||||
if reverse_flow != reverse:
|
||||
continue
|
||||
if flow_res['protocol'] == 'UDP':
|
||||
data.append(float(flow_res['throughput_kbps']) / (1024 * 1024))
|
||||
if self.show_tcp:
|
||||
# TCP may have multiple samples - pick the average for now
|
||||
res = []
|
||||
for flow_res in flow['results']:
|
||||
if reverse and 'direction' not in flow_res:
|
||||
continue
|
||||
if flow_res['protocol'] == 'TCP':
|
||||
res.append(float(flow_res['throughput_kbps']) / (1024 * 1024))
|
||||
break
|
||||
if res:
|
||||
total_tp = 0
|
||||
for tp in res:
|
||||
total_tp += tp
|
||||
data.append(total_tp / len(res))
|
||||
return data
|
||||
|
||||
def _get_flows(self, flows):
|
||||
res = []
|
||||
for flow in flows:
|
||||
desc = flow['desc']
|
||||
if desc in label_match:
|
||||
if label_match[desc]:
|
||||
res.append(self._get_flow_data(label_match[desc], flow))
|
||||
else:
|
||||
# upload/download
|
||||
res.append(self._get_flow_data('Upload', flow, reverse=False))
|
||||
res.append(self._get_flow_data('Download', flow, reverse=True))
|
||||
return res
|
||||
|
||||
def _get_js_options(self, res):
|
||||
subtitle = self._get_subtitle(res)
|
||||
return js_options_tpl % (subtitle)
|
||||
|
||||
def _get_js_chart(self, chart_class, rows, chart_name, id):
|
||||
data = ''
|
||||
for row in rows:
|
||||
data += ' ' * 12 + str(row) + ',\n'
|
||||
return js_data_tpl % (data, chart_class, chart_name, id)
|
||||
|
||||
def _get_js_data(self, flows, id):
|
||||
rows = [self._get_categories(flows[0])]
|
||||
rows.extend(self._get_flows(flows))
|
||||
return self._get_js_chart('google.charts.Bar', rows, 'chart', id)
|
||||
|
||||
def _get_js_props(self, res, id):
|
||||
rows = [['Property', 'Value']]
|
||||
for key in prop_match:
|
||||
if key in res:
|
||||
rows.append([prop_match[key], res[key].encode('ascii', 'ignore')])
|
||||
return self._get_js_chart('google.visualization.Table', rows, 'table', id)
|
||||
|
||||
def _get_js(self, res, id):
|
||||
js = ''
|
||||
js += self._get_js_options(res)
|
||||
js += self._get_js_data(res['flows'], id)
|
||||
# Add property table
|
||||
js += self._get_js_props(res, id)
|
||||
return js
|
||||
|
||||
def _get_jss(self):
|
||||
js = ''
|
||||
id = 0
|
||||
for res in self.results:
|
||||
js += self._get_js(res, id)
|
||||
id += 1
|
||||
return js
|
||||
|
||||
def _get_divs(self):
|
||||
divs = ''
|
||||
id = 0
|
||||
for _ in self.results:
|
||||
divs += div_tpl % ('chart', id)
|
||||
divs += div_tpl % ('table', id)
|
||||
id += 1
|
||||
return divs
|
||||
|
||||
def _plot(self, dest):
|
||||
dest.write(html_main_template % (self._get_jss(), self._get_divs()))
|
||||
|
||||
def plot(self, dest_file):
|
||||
with open(dest_file, 'w') as dest:
|
||||
print('Generating chart drawing code to ' + dest_file + '...')
|
||||
self._plot(dest)
|
||||
|
||||
def gen_chart(files, chart_dest, browser, protocols=''):
|
||||
results = []
|
||||
for ff in files:
|
||||
if not os.path.isfile(ff):
|
||||
print('Error: No such file %s: ' + ff)
|
||||
sys.exit(1)
|
||||
with open(ff) as data_file:
|
||||
res = json.load(data_file)
|
||||
results.append(res)
|
||||
|
||||
chart = GoogleChartsBarChart(results, protocols.lower())
|
||||
chart.plot(chart_dest)
|
||||
if browser:
|
||||
url = 'file://' + os.path.abspath(opts.chart)
|
||||
webbrowser.open(url, new=2)
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
parser = argparse.ArgumentParser(description='VMTP Chart Generator V' + __version__)
|
||||
|
||||
parser.add_argument('-c', '--chart', dest='chart',
|
||||
action='store',
|
||||
help='create and save chart in html file',
|
||||
metavar='<file>')
|
||||
|
||||
parser.add_argument('-b', '--browser', dest='browser',
|
||||
action='store_true',
|
||||
default=False,
|
||||
help='display (-c) chart in the browser')
|
||||
|
||||
parser.add_argument('-p', '--protocol', dest='protocols',
|
||||
action='store',
|
||||
default='all',
|
||||
help='select protocols:all, tcp, udp',
|
||||
metavar='<all|tcp|udp>')
|
||||
|
||||
parser.add_argument('-v', '--version', dest='version',
|
||||
default=False,
|
||||
action='store_true',
|
||||
help='print version of this script and exit')
|
||||
|
||||
parser.add_argument(dest='files',
|
||||
help='vmtp json result file', nargs='+',
|
||||
metavar='<file>')
|
||||
|
||||
opts = parser.parse_args()
|
||||
|
||||
if opts.version:
|
||||
print('Version ' + __version__)
|
||||
sys.exit(0)
|
||||
|
||||
gen_chart(opts.files, opts.chart, opts.browser, opts.protocols)
|
||||
320
instance.py
320
instance.py
@ -1,320 +0,0 @@
|
||||
# Copyright 2014 Cisco Systems, Inc. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
|
||||
import os
|
||||
import re
|
||||
import stat
|
||||
import subprocess
|
||||
|
||||
|
||||
import monitor
|
||||
import sshutils
|
||||
|
||||
|
||||
from netaddr import IPAddress
|
||||
# a dictionary of sequence number indexed by a name prefix
|
||||
prefix_seq = {}
|
||||
|
||||
#
|
||||
# An openstack instance (can be a VM or a LXC)
|
||||
#
|
||||
class Instance(object):
|
||||
|
||||
def __init__(self, name, config, comp=None, net=None):
|
||||
if name not in prefix_seq:
|
||||
prefix_seq[name] = 1
|
||||
seq = prefix_seq[name]
|
||||
prefix_seq[name] = seq + 1
|
||||
self.name = name + str(seq)
|
||||
self.comp = comp
|
||||
self.net = net
|
||||
self.az = None
|
||||
self.config = config
|
||||
# internal network IP
|
||||
self.internal_ip = None
|
||||
self.ssh_access = sshutils.SSHAccess()
|
||||
self.ssh_ip_id = None
|
||||
self.instance = None
|
||||
self.ssh = None
|
||||
self.port = None
|
||||
if config.gmond_svr_ip:
|
||||
self.gmond_svr = config.gmond_svr_ip
|
||||
else:
|
||||
self.gmond_svr = None
|
||||
if config.gmond_svr_port:
|
||||
self.gmond_port = int(config.gmond_svr_port)
|
||||
else:
|
||||
self.gmond_port = 0
|
||||
self.config_drive = None
|
||||
|
||||
# Setup the ssh connectivity
|
||||
# this function is only used for native hosts
|
||||
# Returns True if success
|
||||
def setup_ssh(self, host_access):
|
||||
# used for displaying the source IP in json results
|
||||
if not self.internal_ip:
|
||||
self.internal_ip = host_access.host
|
||||
self.ssh_access = host_access
|
||||
self.buginf('Setup SSH for %s@%s' % (host_access.username, host_access.host))
|
||||
self.ssh = sshutils.SSH(self.ssh_access,
|
||||
connect_retry_count=self.config.ssh_retry_count)
|
||||
return True
|
||||
|
||||
# Create a new VM instance, associate a floating IP for ssh access
|
||||
# and extract internal network IP
|
||||
# Retruns True if success, False otherwise
|
||||
def create(self, image, flavor_type,
|
||||
ssh_access, int_net,
|
||||
az,
|
||||
internal_network_name,
|
||||
sec_group,
|
||||
init_file_name=None):
|
||||
# if ssh is created it means this is a native host not a vm
|
||||
if self.ssh:
|
||||
return True
|
||||
self.buginf('Starting on zone %s', az)
|
||||
self.az = az
|
||||
|
||||
if init_file_name:
|
||||
user_data = open(init_file_name)
|
||||
else:
|
||||
user_data = None
|
||||
|
||||
if self.config.vnic_type:
|
||||
# create the VM by passing a port ID instead of a net ID
|
||||
self.port = self.net.create_port(int_net['id'],
|
||||
[sec_group.id],
|
||||
self.config.vnic_type)
|
||||
nics = [{'port-id': self.port['id']}]
|
||||
# no need to create server with a security group since
|
||||
# we already have the port created with it
|
||||
sec_group = None
|
||||
else:
|
||||
# create the VM by passing a net ID
|
||||
nics = [{'net-id': int_net['id']}]
|
||||
|
||||
self.instance = self.comp.create_server(self.name,
|
||||
image,
|
||||
flavor_type,
|
||||
self.config.public_key_name,
|
||||
nics,
|
||||
sec_group,
|
||||
az,
|
||||
user_data,
|
||||
self.config_drive,
|
||||
self.config.generic_retry_count)
|
||||
if user_data:
|
||||
user_data.close()
|
||||
if not self.instance:
|
||||
self.display('Server creation failed')
|
||||
return False
|
||||
|
||||
# clone the provided ssh access to pick up user name and key pair
|
||||
self.ssh_access.copy_from(ssh_access)
|
||||
|
||||
# If reusing existing management network skip the floating ip creation and association to VM
|
||||
# Assume management network has direct access
|
||||
if self.config.reuse_network_name:
|
||||
self.ssh_access.host = self.instance.networks[internal_network_name][0]
|
||||
self.internal_ip = self.ssh_access.host
|
||||
else:
|
||||
# Set the internal ip to the correct ip for v4 and v6
|
||||
for ip_address in self.instance.networks[internal_network_name]:
|
||||
ip = IPAddress(ip_address)
|
||||
if self.config.ipv6_mode:
|
||||
if ip.version == 6:
|
||||
self.internal_ip = ip_address
|
||||
else:
|
||||
ipv4_fixed_address = ip_address
|
||||
else:
|
||||
if ip.version == 4:
|
||||
self.internal_ip = ip_address
|
||||
ipv4_fixed_address = ip_address
|
||||
fip = self.net.create_floating_ip()
|
||||
if not fip:
|
||||
self.display('Floating ip creation failed')
|
||||
return False
|
||||
self.ssh_access.host = fip['floatingip']['floating_ip_address']
|
||||
self.ssh_ip_id = fip['floatingip']['id']
|
||||
self.buginf('Floating IP %s created', self.ssh_access.host)
|
||||
self.buginf('Started - associating floating IP %s', self.ssh_access.host)
|
||||
self.instance.add_floating_ip(self.ssh_access.host, ipv4_fixed_address)
|
||||
|
||||
# extract the IP for the data network
|
||||
self.buginf('Internal network IP: %s', self.internal_ip)
|
||||
self.buginf('SSH IP: %s', self.ssh_access.host)
|
||||
|
||||
# create ssh session
|
||||
if not self.setup_ssh(self.ssh_access):
|
||||
return False
|
||||
return True
|
||||
|
||||
# Send a command on the ssh session
|
||||
# returns stdout
|
||||
def exec_command(self, cmd, timeout=30):
|
||||
(status, cmd_output, err) = self.ssh.execute(cmd, timeout=timeout)
|
||||
if status:
|
||||
self.display('ERROR cmd=%s' % (cmd))
|
||||
if cmd_output:
|
||||
self.display("%s", cmd_output)
|
||||
if err:
|
||||
self.display('error=%s' % (err))
|
||||
return None
|
||||
self.buginf('%s', cmd_output)
|
||||
return cmd_output
|
||||
|
||||
# Display a status message with the standard header that has the instance
|
||||
# name (e.g. [foo] some text)
|
||||
def display(self, fmt, *args):
|
||||
print ('[%s] ' + fmt) % ((self.name,) + args)
|
||||
|
||||
# Debugging message, to be printed only in debug mode
|
||||
def buginf(self, fmt, *args):
|
||||
if self.config.debug:
|
||||
self.display(fmt, *args)
|
||||
|
||||
# Ping an IP from this instance
|
||||
def ping_check(self, target_ip, ping_count, pass_threshold):
|
||||
return self.ssh.ping_check(target_ip, ping_count, pass_threshold)
|
||||
|
||||
# Given a message size verify if ping without fragmentation works or fails
|
||||
# Returns True if success
|
||||
def ping_do_not_fragment(self, msg_size, ip_address):
|
||||
cmd = "ping -M do -c 1 -s " + str(msg_size) + " " + ip_address
|
||||
cmd_output = self.exec_command(cmd)
|
||||
match = re.search('100% packet loss', cmd_output)
|
||||
if match:
|
||||
return False
|
||||
else:
|
||||
return True
|
||||
|
||||
# Set the interface IP address and mask
|
||||
def set_interface_ip(self, if_name, ip, mask):
|
||||
self.buginf('Setting interface %s to %s mask %s', if_name, ip, mask)
|
||||
cmd2apply = "sudo ifconfig %s %s netmask %s" % (if_name, ip, mask)
|
||||
(rc, _, _) = self.ssh.execute(cmd2apply)
|
||||
return rc
|
||||
|
||||
# Get an interface IP address (returns None if error)
|
||||
def get_interface_ip(self, if_name):
|
||||
self.buginf('Getting interface %s IP and mask', if_name)
|
||||
cmd2apply = "ifconfig %s" % (if_name)
|
||||
(rc, res, _) = self.ssh.execute(cmd2apply)
|
||||
if rc:
|
||||
return None
|
||||
# eth5 Link encap:Ethernet HWaddr 90:e2:ba:40:74:05
|
||||
# inet addr:172.29.87.29 Bcast:172.29.87.31 Mask:255.255.255.240
|
||||
# inet6 addr: fe80::92e2:baff:fe40:7405/64 Scope:Link
|
||||
match = re.search(r'inet addr:([\d\.]*) ', res)
|
||||
if not match:
|
||||
return None
|
||||
return match.group(1)
|
||||
|
||||
# Set an interface MTU to passed in value
|
||||
def set_interface_mtu(self, if_name, mtu):
|
||||
self.buginf('Setting interface %s mtu to %d', if_name, mtu)
|
||||
cmd2apply = "sudo ifconfig %s mtu %d" % (if_name, mtu)
|
||||
(rc, _, _) = self.ssh.execute(cmd2apply)
|
||||
return rc
|
||||
|
||||
# Get the MTU of an interface
|
||||
def get_interface_mtu(self, if_name):
|
||||
cmd = "cat /sys/class/net/%s/mtu" % (if_name)
|
||||
cmd_output = self.exec_command(cmd)
|
||||
return int(cmd_output)
|
||||
|
||||
# scp a file from the local host to the instance
|
||||
# Returns True if dest file already exists or scp succeeded
|
||||
# False in case of scp error
|
||||
def scp(self, tool_name, source, dest):
|
||||
|
||||
# check if the dest file is already present
|
||||
if self.ssh.stat(dest):
|
||||
self.buginf('tool %s already present - skipping install',
|
||||
tool_name)
|
||||
return True
|
||||
# scp over the tool binary
|
||||
# first chmod the local copy since git does not keep the permission
|
||||
os.chmod(source, stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO)
|
||||
|
||||
# scp to the target
|
||||
scp_opts = '-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no'
|
||||
scp_cmd = 'scp -i %s %s %s %s@%s:%s' % (self.config.private_key_file,
|
||||
scp_opts,
|
||||
source,
|
||||
self.ssh_access.username,
|
||||
self.ssh_access.host,
|
||||
dest)
|
||||
self.buginf('Copying %s to target...', tool_name)
|
||||
self.buginf(scp_cmd)
|
||||
devnull = open(os.devnull, 'wb')
|
||||
rc = subprocess.call(scp_cmd, shell=True,
|
||||
stdout=devnull, stderr=devnull)
|
||||
if rc:
|
||||
self.display('Copy to target failed rc=%d', rc)
|
||||
self.display(scp_cmd)
|
||||
return False
|
||||
return True
|
||||
|
||||
def get_cmd_duration(self):
|
||||
'''Get the duration of the client run
|
||||
Will normally return the time configured in config.time
|
||||
If cpu monitoring is enabled will make sure that this time is at least
|
||||
30 seconds (to be adjusted based on metric collection frequency)
|
||||
'''
|
||||
if self.gmond_svr:
|
||||
return max(30, self.config.time)
|
||||
return self.config.time
|
||||
|
||||
def exec_with_cpu(self, cmd):
|
||||
'''If cpu monitoring is enabled (--monitor) collect CPU in the background
|
||||
while the test is running
|
||||
:param duration: how long the command will run in seconds
|
||||
:return: a tuple (cmd_output, cpu_load)
|
||||
'''
|
||||
# ssh timeout should be at least set to the command duration
|
||||
# we add 20 seconds to it as a safety
|
||||
timeout = self.get_cmd_duration() + 20
|
||||
if self.gmond_svr:
|
||||
gmon = monitor.Monitor(self.gmond_svr, self.gmond_port)
|
||||
# Adjust this frequency based on the collectors update frequency
|
||||
# Here we assume 10 second and a max of 20 samples
|
||||
gmon.start_monitoring_thread(freq=10, count=20)
|
||||
cmd_output = self.exec_command(cmd, timeout)
|
||||
gmon.stop_monitoring_thread()
|
||||
# insert the cpu results into the results
|
||||
cpu_load = gmon.build_cpu_metrics()
|
||||
else:
|
||||
cmd_output = self.exec_command(cmd, timeout)
|
||||
cpu_load = None
|
||||
return (cmd_output, cpu_load)
|
||||
|
||||
# Delete the floating IP
|
||||
# Delete the server instance
|
||||
# Dispose the ssh session
|
||||
def dispose(self):
|
||||
if self.ssh_ip_id:
|
||||
self.net.delete_floating_ip(self.ssh_ip_id)
|
||||
self.buginf('Floating IP %s deleted', self.ssh_access.host)
|
||||
self.ssh_ip_id = None
|
||||
if self.instance:
|
||||
self.comp.delete_server(self.instance)
|
||||
self.buginf('Instance deleted')
|
||||
self.instance = None
|
||||
if self.port:
|
||||
self.net.delete_port(self.port)
|
||||
if self.ssh:
|
||||
self.ssh.close()
|
||||
self.ssh = None
|
||||
206
iperf_tool.py
206
iperf_tool.py
@ -1,206 +0,0 @@
|
||||
# Copyright 2014 Cisco Systems, Inc. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
|
||||
import re
|
||||
|
||||
from perf_tool import PerfTool
|
||||
|
||||
# The resulting unit should be in K
|
||||
MULTIPLIERS = {'K': 1,
|
||||
'M': 1.0e3,
|
||||
'G': 1.0e6}
|
||||
|
||||
def get_bdw_kbps(bdw, bdw_unit):
|
||||
if not bdw_unit:
|
||||
# bits/sec
|
||||
return bdw / 1000
|
||||
if bdw_unit in MULTIPLIERS:
|
||||
return int(bdw * MULTIPLIERS[bdw_unit])
|
||||
print('Error: unknown multiplier: ' + bdw_unit)
|
||||
return bdw
|
||||
|
||||
class IperfTool(PerfTool):
|
||||
|
||||
def __init__(self, instance, perf_tool_path):
|
||||
PerfTool.__init__(self, 'iperf', perf_tool_path, instance)
|
||||
|
||||
def get_server_launch_cmd(self):
|
||||
'''Return the command to launch the server side.'''
|
||||
# Need 1 server for tcp (port 5001) and 1 for udp (port 5001)
|
||||
return [self.dest_path + ' -s >/dev/null &',
|
||||
self.dest_path + ' -s -u >/dev/null &']
|
||||
|
||||
def run_client(self, target_ip, target_instance,
|
||||
mss=None, bandwidth=0, bidirectional=False):
|
||||
'''Run the test
|
||||
:return: list containing one or more dictionary results
|
||||
'''
|
||||
res_list = []
|
||||
|
||||
# Get list of protocols and packet sizes to measure
|
||||
(proto_list, proto_pkt_sizes) = self.get_proto_profile()
|
||||
|
||||
for udp, pkt_size_list in zip(proto_list, proto_pkt_sizes):
|
||||
# bidirectional is not supported for udp
|
||||
# (need to find the right iperf options to make it work as there are
|
||||
# issues for the server to send back results to the client in reverse
|
||||
# direction
|
||||
if udp:
|
||||
bidir = False
|
||||
loop_count = 1
|
||||
else:
|
||||
# For accuracy purpose, TCP throughput will be measured 3 times
|
||||
bidir = bidirectional
|
||||
loop_count = self.instance.config.tcp_tp_loop_count
|
||||
for pkt_size in pkt_size_list:
|
||||
for _ in xrange(loop_count):
|
||||
res = self.run_client_dir(target_ip, mss,
|
||||
bandwidth_kbps=bandwidth,
|
||||
bidirectional=bidir,
|
||||
udp=udp,
|
||||
length=pkt_size)
|
||||
# for bidirectional the function returns a list of 2 results
|
||||
res_list.extend(res)
|
||||
return res_list
|
||||
|
||||
def run_client_dir(self, target_ip,
|
||||
mss,
|
||||
bidirectional=False,
|
||||
bandwidth_kbps=0,
|
||||
udp=False,
|
||||
length=0,
|
||||
no_cpu_timed=0):
|
||||
'''Run client for given protocol and packet size
|
||||
:param bandwidth_kbps: transmit rate limit in Kbps
|
||||
:param udp: if true get UDP throughput, else get TCP throughput
|
||||
:param length: length of network write|read buf (default 1K|8K/udp, 64K/tcp)
|
||||
for udp is the packet size
|
||||
:param no_cpu_timed: if non zero will disable cpu collection and override
|
||||
the time with the provided value - used mainly for udp
|
||||
to find quickly the optimal throughput using short
|
||||
tests at various throughput values
|
||||
:return: a list of dictionary with the 1 or 2 results (see parse_results())
|
||||
'''
|
||||
# run client using the default TCP window size (tcp window
|
||||
# scaling is normally enabled by default so setting explicit window
|
||||
# size is not going to help achieve better results)
|
||||
opts = ''
|
||||
|
||||
# run iperf client using the default TCP window size (tcp window
|
||||
# scaling is normally enabled by default so setting explicit window
|
||||
# size is not going to help achieve better results)
|
||||
if mss:
|
||||
opts += " -M " + str(mss)
|
||||
|
||||
if bidirectional:
|
||||
opts += " -r"
|
||||
|
||||
if length:
|
||||
opts += " -l" + str(length)
|
||||
|
||||
if udp:
|
||||
opts += " -u"
|
||||
# for UDP if the bandwidth is not provided we need to calculate
|
||||
# the optimal bandwidth
|
||||
if not bandwidth_kbps:
|
||||
udp_res = self.find_udp_bdw(length, target_ip)
|
||||
if 'error' in udp_res:
|
||||
return [udp_res]
|
||||
if not self.instance.gmond_svr:
|
||||
# if we do not collect CPU we might as well return
|
||||
# the results found through iteration
|
||||
return [udp_res]
|
||||
bandwidth_kbps = udp_res['throughput_kbps']
|
||||
|
||||
if bandwidth_kbps:
|
||||
opts += " -b%dK" % (bandwidth_kbps)
|
||||
|
||||
if no_cpu_timed:
|
||||
duration_sec = no_cpu_timed
|
||||
else:
|
||||
duration_sec = self.instance.get_cmd_duration()
|
||||
|
||||
cmd = "%s -c %s -t %d %s" % (self.dest_path,
|
||||
target_ip,
|
||||
duration_sec,
|
||||
opts)
|
||||
self.instance.buginf(cmd)
|
||||
if no_cpu_timed:
|
||||
# force the timeout value with 20 second extra for the command to
|
||||
# complete and do not collect CPU
|
||||
cpu_load = None
|
||||
cmd_out = self.instance.exec_command(cmd, duration_sec + 20)
|
||||
else:
|
||||
(cmd_out, cpu_load) = self.instance.exec_with_cpu(cmd)
|
||||
|
||||
if udp:
|
||||
# Decode UDP output (unicast and multicast):
|
||||
#
|
||||
# [ 3] local 127.0.0.1 port 54244 connected with 127.0.0.1 port 5001
|
||||
# [ ID] Interval Transfer Bandwidth
|
||||
# [ 3] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec
|
||||
# [ 3] Sent 893 datagrams
|
||||
# [ 3] Server Report:
|
||||
# [ ID] Interval Transfer Bandwidth Jitter Lost/Total Da
|
||||
# [ 3] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec 0.032 ms 1/894 (0.11%)
|
||||
# [ 3] 0.0-15.0 sec 14060 datagrams received out-of-order
|
||||
re_udp = r'([\d\.]*)\s*([KMG]?)bits/sec\s*[\d\.]*\s*ms\s*(\d*)/\s*(\d*) '
|
||||
match = re.search(re_udp, cmd_out)
|
||||
if match:
|
||||
bdw = float(match.group(1))
|
||||
bdw_unit = match.group(2)
|
||||
drop = float(match.group(3))
|
||||
pkt = int(match.group(4))
|
||||
# iperf uses multiple of 1000 for K - not 1024
|
||||
return [self.parse_results('UDP',
|
||||
get_bdw_kbps(bdw, bdw_unit),
|
||||
lossrate=round(drop * 100 / pkt, 2),
|
||||
msg_size=length,
|
||||
cpu_load=cpu_load)]
|
||||
else:
|
||||
# TCP output:
|
||||
# [ 3] local 127.0.0.1 port 57936 connected with 127.0.0.1 port 5001
|
||||
# [ ID] Interval Transfer Bandwidth
|
||||
# [ 3] 0.0-10.0 sec 2.09 GBytes 1.79 Gbits/sec
|
||||
#
|
||||
# For bi-directional option (-r), last 3 lines:
|
||||
# [ 5] 0.0-10.0 sec 36.0 GBytes 31.0 Gbits/sec
|
||||
# [ 4] local 127.0.0.1 port 5002 connected with 127.0.0.1 port 39118
|
||||
# [ 4] 0.0-10.0 sec 36.0 GBytes 30.9 Gbits/sec
|
||||
re_tcp = r'Bytes\s*([\d\.]*)\s*([KMG])bits/sec'
|
||||
match = re.search(re_tcp, cmd_out)
|
||||
if match:
|
||||
bdw = float(match.group(1))
|
||||
bdw_unit = match.group(2)
|
||||
res = [self.parse_results('TCP',
|
||||
get_bdw_kbps(bdw, bdw_unit),
|
||||
msg_size=length,
|
||||
cpu_load=cpu_load)]
|
||||
if bidirectional:
|
||||
# decode the last row results
|
||||
re_tcp = r'Bytes\s*([\d\.]*)\s*([KMG])bits/sec$'
|
||||
match = re.search(re_tcp, cmd_out)
|
||||
if match:
|
||||
bdw = float(match.group(1))
|
||||
bdw_unit = match.group(2)
|
||||
# use the same cpu load since the same run
|
||||
# does both directions
|
||||
res.append(self.parse_results('TCP',
|
||||
get_bdw_kbps(bdw, bdw_unit),
|
||||
reverse_dir=True,
|
||||
msg_size=length,
|
||||
cpu_load=cpu_load))
|
||||
return res
|
||||
return [self.parse_error('Could not parse: %s' % (cmd_out))]
|
||||
@ -16,4 +16,4 @@ import pbr.version
|
||||
|
||||
|
||||
__version__ = pbr.version.VersionInfo(
|
||||
'vmtp').version_string()
|
||||
'kloudbuster').version_string_with_vcs()
|
||||
6
scale/kloudbuster.py → kloudbuster/kloudbuster.py
Normal file → Executable file
6
scale/kloudbuster.py → kloudbuster/kloudbuster.py
Normal file → Executable file
@ -1,3 +1,4 @@
|
||||
#!/usr/bin/env python
|
||||
# Copyright 2015 Cisco Systems, Inc. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
@ -33,12 +34,9 @@ from oslo_config import cfg
|
||||
from tabulate import tabulate
|
||||
import tenant
|
||||
|
||||
import sshutils
|
||||
|
||||
CONF = cfg.CONF
|
||||
LOG = logging.getLogger(__name__)
|
||||
|
||||
__version__ = '1.0.0'
|
||||
KB_IMAGE_MAJOR_VERSION = 1
|
||||
|
||||
class KBVMCreationException(Exception):
|
||||
@ -392,7 +390,7 @@ class KloudBuster(object):
|
||||
LOG.info(self.final_result)
|
||||
except KeyboardInterrupt:
|
||||
traceback.format_exc()
|
||||
except (sshutils.SSHError, ClientException, Exception):
|
||||
except (ClientException, Exception):
|
||||
traceback.print_exc()
|
||||
|
||||
# Cleanup: start with tested side first
|
||||
@ -13,16 +13,16 @@
|
||||
# under the License.
|
||||
|
||||
"""
|
||||
test_vmtp
|
||||
test_kloudbuster
|
||||
----------------------------------
|
||||
|
||||
Tests for `vmtp` module.
|
||||
Tests for `kloudbuster` module.
|
||||
"""
|
||||
|
||||
from vmtp.tests import base
|
||||
from kloudbuster.tests import base
|
||||
|
||||
|
||||
class TestVmtp(base.TestCase):
|
||||
class TestKloudbuster(base.TestCase):
|
||||
|
||||
def test_something(self):
|
||||
pass
|
||||
443
monitor.py
443
monitor.py
@ -1,443 +0,0 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2014 Cisco Systems, Inc. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
|
||||
|
||||
'''
|
||||
Module for parsing statistical output from Ganglia (gmond) server
|
||||
The module opens a socket connection to collect statistical data.
|
||||
It parses the raw data in xml format.
|
||||
|
||||
The data from ganglia/gmond is in a heirarchical xml format as below:
|
||||
<CLUSTER>
|
||||
<HOST..>
|
||||
<METRIC ../>
|
||||
<METRIC ../>
|
||||
:
|
||||
</HOST>
|
||||
:
|
||||
<HOST..>
|
||||
<METRIC ../>
|
||||
<METRIC ../>
|
||||
</HOST>
|
||||
</CLUSTER>
|
||||
|
||||
## Usage:
|
||||
Using the module is simple.
|
||||
|
||||
1. instantiate the Monitor with the gmond server ip and port to poll.
|
||||
|
||||
gmon = Monitor("172.22.191.151", 8649)
|
||||
|
||||
2. Start the monitoring thread
|
||||
gmon.start_monitoring_thread(frequency, count)
|
||||
|
||||
< run tests/tasks>
|
||||
|
||||
gmon.stop_monitoring_thread()
|
||||
|
||||
3. Collecting stats:
|
||||
cpu_metric = gmon.build_cpu_metric()
|
||||
|
||||
Returns a dictionary object with all the cpu stats for each
|
||||
node
|
||||
|
||||
|
||||
'''
|
||||
|
||||
import datetime
|
||||
import re
|
||||
import socket
|
||||
import subprocess
|
||||
from threading import Thread
|
||||
import time
|
||||
|
||||
from lxml import etree
|
||||
|
||||
class MonitorExecutor(Thread):
|
||||
'''
|
||||
Thread handler class to asynchronously collect stats
|
||||
'''
|
||||
THREAD_STOPPED = 0
|
||||
THREAD_RUNNING = 1
|
||||
|
||||
def __init__(self, gmond_svr, gmond_port, freq=5, count=5):
|
||||
super(MonitorExecutor, self).__init__()
|
||||
self.gmond_svr_ip = gmond_svr
|
||||
self.gmond_port = gmond_port
|
||||
|
||||
self.freq = freq
|
||||
self.count = count
|
||||
|
||||
self.force_stop = False
|
||||
self.thread_status = MonitorExecutor.THREAD_STOPPED
|
||||
|
||||
# This dictionary always holds the latest metric.
|
||||
self.gmond_parsed_tree_list = []
|
||||
|
||||
|
||||
def run(self):
|
||||
'''
|
||||
The thread runnable method.
|
||||
The function will periodically poll the gmond server and
|
||||
collect the metrics.
|
||||
'''
|
||||
self.thread_status = MonitorExecutor.THREAD_RUNNING
|
||||
|
||||
count = self.count
|
||||
while count > 0:
|
||||
if self.force_stop:
|
||||
self.thread_status = MonitorExecutor.THREAD_STOPPED
|
||||
return
|
||||
|
||||
self.parse_gmond_xml_data()
|
||||
count -= 1
|
||||
time.sleep(self.freq)
|
||||
self.thread_status = MonitorExecutor.THREAD_STOPPED
|
||||
|
||||
|
||||
def set_force_stop(self):
|
||||
'''
|
||||
Setting the force stop flag to stop the thread. By default
|
||||
the thread stops after the specific count/iterations is reached
|
||||
'''
|
||||
self.force_stop = True
|
||||
|
||||
def parse_gmond_xml_data(self):
|
||||
'''
|
||||
Parse gmond data (V2)
|
||||
Retrieve the ganglia stats from the aggregation node
|
||||
:return: None in case of error or a dictionary containing the stats
|
||||
'''
|
||||
gmond_parsed_tree = {}
|
||||
raw_data = self.retrieve_stats_raw()
|
||||
|
||||
if raw_data is None or len(raw_data) == 0:
|
||||
print "Failed to retrieve stats from server"
|
||||
return
|
||||
|
||||
xtree = etree.XML(raw_data)
|
||||
############################################
|
||||
# Populate cluster information.
|
||||
############################################
|
||||
for elem in xtree.iter('CLUSTER'):
|
||||
gmond_parsed_tree['CLUSTER-NAME'] = str(elem.get('NAME'))
|
||||
gmond_parsed_tree['LOCALTIME'] = str(elem.get('LOCALTIME'))
|
||||
gmond_parsed_tree['URL'] = str(elem.get('URL'))
|
||||
|
||||
host_list = []
|
||||
for helem in elem.iterchildren():
|
||||
host = {}
|
||||
host['NAME'] = str(helem.get('NAME'))
|
||||
host['IP'] = str(helem.get('IP'))
|
||||
host['REPORTED'] = str(helem.get('REPORTED'))
|
||||
host['TN'] = str(helem.get('TN'))
|
||||
host['TMAX'] = str(helem.get('TMAX'))
|
||||
host['DMAX'] = str(helem.get('DMAX'))
|
||||
host['LOCATION'] = str(helem.get('LOCATION'))
|
||||
host['GMOND_STARTED'] = str(helem.get('GMOND_STARTED'))
|
||||
|
||||
mlist = []
|
||||
for metric in helem.iterchildren():
|
||||
mdic = {}
|
||||
mdic['NAME'] = str(metric.get('NAME'))
|
||||
mdic['VAL'] = str(metric.get('VAL'))
|
||||
mlist.append(mdic)
|
||||
|
||||
host['metrics'] = mlist
|
||||
host_list.append(host)
|
||||
|
||||
gmond_parsed_tree['hosts'] = host_list
|
||||
stat_dt = datetime.datetime.now()
|
||||
gmond_parsed_tree['dt'] = stat_dt
|
||||
self.gmond_parsed_tree_list.append(gmond_parsed_tree)
|
||||
|
||||
|
||||
def retrieve_stats_raw(self):
|
||||
'''
|
||||
Retrieve stats from the gmond process.
|
||||
'''
|
||||
soc = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
soc.settimeout(10)
|
||||
try:
|
||||
soc.connect((self.gmond_svr_ip, self.gmond_port))
|
||||
except socket.error as exp:
|
||||
print "Connection failure host: %s [%s]" % (self.gmond_svr_ip, exp)
|
||||
return None
|
||||
|
||||
data = ""
|
||||
while True:
|
||||
try:
|
||||
rbytes = soc.recv(4096)
|
||||
except socket.error as exp:
|
||||
print "Read failed for host: ", str(exp)
|
||||
return None
|
||||
|
||||
if len(rbytes) == 0:
|
||||
break
|
||||
data += rbytes
|
||||
|
||||
soc.close()
|
||||
return data
|
||||
|
||||
|
||||
class Monitor(object):
|
||||
gmond_svr_ip = None
|
||||
gmond_port = None
|
||||
gmond_parsed_tree = {}
|
||||
|
||||
def __init__(self, gmond_svr, gmond_port=8649):
|
||||
'''
|
||||
The constructor simply sets the values of the gmond server and port.
|
||||
'''
|
||||
self.gmond_svr_ip = gmond_svr
|
||||
self.gmond_port = gmond_port
|
||||
# List of all stats.
|
||||
self.gmond_parsed_tree_list = []
|
||||
# series for all cpu loads
|
||||
self.cpu_res = {}
|
||||
|
||||
self.mon_thread = None
|
||||
|
||||
def start_monitoring_thread(self, freq=10, count=10):
|
||||
'''
|
||||
Start the monitoring thread.
|
||||
'''
|
||||
self.mon_thread = MonitorExecutor(self.gmond_svr_ip,
|
||||
self.gmond_port, freq, count)
|
||||
self.mon_thread.start()
|
||||
|
||||
|
||||
def stop_monitoring_thread(self):
|
||||
self.mon_thread.set_force_stop()
|
||||
self.gmond_parsed_tree_list = self.mon_thread.gmond_parsed_tree_list
|
||||
|
||||
|
||||
def strip_raw_telnet_output(self, raw_data):
|
||||
'''
|
||||
When using the retrieve_stats_raw_telent api, the raw data
|
||||
has some additional text along with the xml data. We need to
|
||||
strip that before we can invoke pass it through the lxml parser.
|
||||
'''
|
||||
data = ""
|
||||
xml_flag = False
|
||||
for line in raw_data.splitlines():
|
||||
if re.match(r".*<?xml version.*", line):
|
||||
xml_flag = True
|
||||
if xml_flag:
|
||||
data += line + "\n"
|
||||
|
||||
return data
|
||||
|
||||
|
||||
def retrieve_stats_raw_telnet(self):
|
||||
'''
|
||||
This way of retrieval is to create a subprocess and execute
|
||||
the telnet command on the port to retrieve the xml raw data.
|
||||
'''
|
||||
cmd = "telnet " + self.gmond_svr_ip + " " + str(self.gmond_port)
|
||||
print "cmd: ", cmd
|
||||
port = str(self.gmond_port)
|
||||
|
||||
proc = subprocess.Popen(["telnet", self.gmond_svr_ip, port],
|
||||
stdout=subprocess.PIPE)
|
||||
(output, _) = proc.communicate()
|
||||
|
||||
newout = self.strip_raw_telnet_output(output)
|
||||
return newout
|
||||
|
||||
|
||||
def get_host_list(self, gmond_parsed_tree):
|
||||
'''
|
||||
Function returns all the hosts {} as a list.
|
||||
'''
|
||||
return gmond_parsed_tree['hosts']
|
||||
|
||||
|
||||
def get_metric_value(self, parsed_node, host_name, name):
|
||||
'''
|
||||
The function returns the value of a specific metric, given
|
||||
the host name and the metric name to collect.
|
||||
'''
|
||||
for host in parsed_node['hosts']:
|
||||
if host['NAME'] == host_name:
|
||||
for metric in host['metrics']:
|
||||
if metric['NAME'] == name:
|
||||
return metric['VAL']
|
||||
|
||||
return 0
|
||||
|
||||
def get_aggregate_cpu_usage(self, parsed_node, host_name):
|
||||
'''
|
||||
The function returns the aggregate CPU usage for a specific host.
|
||||
eqation: [user cpu + system cpu * no of cpu /100]
|
||||
'''
|
||||
cpu_user = float(self.get_metric_value(parsed_node, host_name, "cpu_user"))
|
||||
cpu_system = float(self.get_metric_value(parsed_node, host_name, "cpu_system"))
|
||||
cpu_num = int(self.get_metric_value(parsed_node, host_name, "cpu_num"))
|
||||
|
||||
return (cpu_user + cpu_system) * cpu_num / 100
|
||||
|
||||
|
||||
def build_cpu_metrics(self):
|
||||
'''Add a new set of cpu metrics to the results dictionary self.cpu_res
|
||||
The result dest dictionary should look like this:
|
||||
key = host IP, value = list of cpu load where the
|
||||
the first value is the baseline value followed by 1 or more
|
||||
values collected during the test
|
||||
{
|
||||
'10.0.0.1': [ 0.03, 1.23, 1.20 ],
|
||||
'10.0.0.2': [ 0.10, 1.98, 2.72 ]
|
||||
}
|
||||
After another xml is decoded:
|
||||
{
|
||||
'10.0.0.1': [ 0.03, 1.23, 1.20, 1.41 ],
|
||||
'10.0.0.2': [ 0.10, 1.98, 2.72, 2.04 ]
|
||||
}
|
||||
Each value in the list is the cpu load calculated as
|
||||
(cpu_user + cpu_system) * num_cpu / 100
|
||||
The load_five metric cannot be used as it is the average for last 5'
|
||||
'''
|
||||
cpu_res = {}
|
||||
for parsed_node in self.gmond_parsed_tree_list:
|
||||
for host in self.get_host_list(parsed_node):
|
||||
host_ip = host['IP']
|
||||
cpu_num = 0
|
||||
cpu_user = 0.0
|
||||
cpu_system = 0.0
|
||||
|
||||
cpu_user = float(self.get_metric_value(parsed_node, host['NAME'], "cpu_user"))
|
||||
cpu_system = float(self.get_metric_value(parsed_node, host['NAME'], "cpu_system"))
|
||||
cpu_num = int(self.get_metric_value(parsed_node, host['NAME'], "cpu_num"))
|
||||
cpu_load = round(((cpu_user + cpu_system) * cpu_num) / 100, 2)
|
||||
try:
|
||||
cpu_res[host_ip].append(cpu_load)
|
||||
except KeyError:
|
||||
cpu_res[host_ip] = [cpu_load]
|
||||
|
||||
return cpu_res
|
||||
|
||||
def get_formatted_datetime(self, parsed_node):
|
||||
'''
|
||||
Returns the data in formated string. This is the
|
||||
time when the last stat was collected.
|
||||
'''
|
||||
now = parsed_node['dt']
|
||||
fmt_dt = "[" + str(now.hour) + ":" + str(now.minute) + \
|
||||
":" + str(now.second) + "]"
|
||||
return fmt_dt
|
||||
|
||||
|
||||
def get_formatted_host_row(self, host_list):
|
||||
'''
|
||||
Returns the hosts in formated order (for printing purposes)
|
||||
'''
|
||||
row_str = "".ljust(10)
|
||||
for host in host_list:
|
||||
row_str += host['NAME'].ljust(15)
|
||||
return row_str
|
||||
|
||||
def get_formatted_metric_row(self, parsed_node, metric, justval):
|
||||
'''
|
||||
Returns a specific metric for all hosts in the same row
|
||||
in formated string (for printing)
|
||||
'''
|
||||
host_list = self.get_host_list(parsed_node)
|
||||
|
||||
row_str = metric.ljust(len(metric) + 2)
|
||||
for host in host_list:
|
||||
val = self.get_metric_value(parsed_node, host['NAME'], metric)
|
||||
row_str += str(val).ljust(justval)
|
||||
return row_str
|
||||
|
||||
|
||||
def dump_cpu_stats(self):
|
||||
'''
|
||||
Print the CPU stats
|
||||
'''
|
||||
hl_len = 80
|
||||
print "-" * hl_len
|
||||
print "CPU Statistics: ",
|
||||
|
||||
for parsed_node in self.gmond_parsed_tree_list:
|
||||
hosts = self.get_host_list(parsed_node)
|
||||
|
||||
print self.get_formatted_datetime(parsed_node)
|
||||
print self.get_formatted_host_row(hosts)
|
||||
print "-" * hl_len
|
||||
print self.get_formatted_metric_row(parsed_node, "cpu_user", 18)
|
||||
print self.get_formatted_metric_row(parsed_node, "cpu_system", 18)
|
||||
|
||||
print "Aggregate ",
|
||||
for host in hosts:
|
||||
print str(self.get_aggregate_cpu_usage(parsed_node,
|
||||
host['NAME'])).ljust(16),
|
||||
print "\n"
|
||||
|
||||
def dump_gmond_parsed_tree(self):
|
||||
'''
|
||||
Display the full tree parsed from the gmond server stats.
|
||||
'''
|
||||
hl_len = 60
|
||||
|
||||
for parsed_node in self.gmond_parsed_tree_list:
|
||||
print "%-20s (%s) URL: %s " % \
|
||||
(parsed_node['CLUSTER-NAME'],
|
||||
parsed_node['LOCALTIME'],
|
||||
parsed_node['URL'])
|
||||
print "-" * hl_len
|
||||
|
||||
row_str = " ".ljust(9)
|
||||
for host in parsed_node['hosts']:
|
||||
row_str += host['NAME'].ljust(15)
|
||||
row_str += "\n"
|
||||
print row_str
|
||||
print "-" * hl_len
|
||||
metric_count = len(parsed_node['hosts'][0]['metrics'])
|
||||
for count in range(0, metric_count):
|
||||
row_str = ""
|
||||
host = parsed_node['hosts'][0]
|
||||
row_str += parsed_node['hosts'][0]['metrics'][count]['NAME'].ljust(18)
|
||||
for host in parsed_node['hosts']:
|
||||
val = str(self.get_metric_value(parsed_node, host['NAME'],
|
||||
host['metrics'][count]['NAME']))
|
||||
row_str += val.ljust(12)
|
||||
|
||||
row_str += str(parsed_node['hosts'][0]).ljust(5)
|
||||
|
||||
print row_str
|
||||
|
||||
|
||||
##################################################
|
||||
# Only invoke the module directly for test purposes. Should be
|
||||
# invoked from pns script.
|
||||
##################################################
|
||||
def main():
|
||||
print "main: monitor"
|
||||
gmon = Monitor("172.22.191.151", 8649)
|
||||
gmon.start_monitoring_thread(freq=5, count=20)
|
||||
print "wait for 15 seconds"
|
||||
time.sleep(20)
|
||||
print "Now force the thread to stop"
|
||||
gmon.stop_monitoring_thread()
|
||||
gmon.dump_cpu_stats()
|
||||
|
||||
cpu_metric = gmon.build_cpu_metrics()
|
||||
print "cpu_metric: ", cpu_metric
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
390
network.py
390
network.py
@ -1,390 +0,0 @@
|
||||
# Copyright 2014 Cisco Systems, Inc. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
|
||||
import time
|
||||
|
||||
# Module containing a helper class for operating on OpenStack networks
|
||||
from neutronclient.common.exceptions import NetworkInUseClient
|
||||
from neutronclient.common.exceptions import NeutronException
|
||||
|
||||
class Network(object):
|
||||
|
||||
#
|
||||
# This constructor will try to find an external network (will use the
|
||||
# first network that is tagged as external - irrespective of its name)
|
||||
# and a router attached to it (irrespective of the router name).
|
||||
# ext_router_name is the name of the external router to create if not None
|
||||
# and if no external router is found
|
||||
#
|
||||
def __init__(self, neutron_client, config):
|
||||
self.neutron_client = neutron_client
|
||||
self.networks = neutron_client.list_networks()['networks']
|
||||
self.ext_net = None
|
||||
self.ext_router = None
|
||||
self.ext_router_created = False
|
||||
self.config = config
|
||||
# mgmt/data network:
|
||||
# - first for same network
|
||||
# - second for network to network communication
|
||||
self.vm_int_net = []
|
||||
self.ext_router_name = None
|
||||
# Store state if the network is ipv4/ipv6 dual stack
|
||||
self.ipv6_enabled = False
|
||||
|
||||
# If reusing existing management network just find this network
|
||||
if self.config.reuse_network_name:
|
||||
# An existing management network must be reused
|
||||
int_net = self.lookup_network(self.config.reuse_network_name)
|
||||
self.vm_int_net.append(int_net)
|
||||
return
|
||||
|
||||
##############################################
|
||||
# If a user provided ext_net_name is not available,
|
||||
# then find the first network that is external
|
||||
##############################################
|
||||
for network in self.networks:
|
||||
if network['router:external']:
|
||||
try:
|
||||
if network['name'] == config.ext_net_name:
|
||||
self.ext_net = network
|
||||
break
|
||||
if not self.ext_net:
|
||||
self.ext_net = network
|
||||
except KeyError:
|
||||
###############################################
|
||||
# A key error indicates, no user defined
|
||||
# external network defined, so use the first one
|
||||
###############################################
|
||||
self.ext_net = network
|
||||
break
|
||||
|
||||
if not self.ext_net:
|
||||
print "No external network found."
|
||||
return
|
||||
|
||||
print "Using external network: " + self.ext_net['name']
|
||||
|
||||
# Find or create the router to the external network
|
||||
ext_net_id = self.ext_net['id']
|
||||
routers = neutron_client.list_routers()['routers']
|
||||
for router in routers:
|
||||
external_gw_info = router['external_gateway_info']
|
||||
if external_gw_info:
|
||||
if external_gw_info['network_id'] == ext_net_id:
|
||||
self.ext_router = router
|
||||
print 'Found external router: %s' % \
|
||||
(self.ext_router['name'])
|
||||
break
|
||||
|
||||
# create a new external router if none found and a name was given
|
||||
self.ext_router_name = config.router_name
|
||||
if (not self.ext_router) and self.ext_router_name:
|
||||
self.ext_router = self.create_router(self.ext_router_name,
|
||||
self.ext_net['id'])
|
||||
print '[%s] Created ext router' % (self.ext_router_name)
|
||||
self.ext_router_created = True
|
||||
|
||||
if config.ipv6_mode:
|
||||
self.ipv6_enabled = True
|
||||
|
||||
# Create the networks and subnets depending on v4 or v6
|
||||
if config.ipv6_mode:
|
||||
for (net, subnet, cidr, subnet_ipv6, cidr_ipv6) in zip(config.internal_network_name,
|
||||
config.internal_subnet_name,
|
||||
config.internal_cidr,
|
||||
config.internal_subnet_name_ipv6,
|
||||
config.internal_cidr_v6):
|
||||
int_net = self.create_net(net, subnet, cidr,
|
||||
config.dns_nameservers,
|
||||
subnet_ipv6, cidr_ipv6, config.ipv6_mode)
|
||||
self.vm_int_net.append(int_net)
|
||||
else:
|
||||
for (net, subnet, cidr) in zip(config.internal_network_name,
|
||||
config.internal_subnet_name,
|
||||
config.internal_cidr):
|
||||
int_net = self.create_net(net, subnet, cidr,
|
||||
config.dns_nameservers)
|
||||
self.vm_int_net.append(int_net)
|
||||
|
||||
self.l2agent_type = self._get_l2agent_type()
|
||||
self.internal_iface_dict = self._get_internal_iface_dict()
|
||||
|
||||
# Add both internal networks to router interface to enable network to network connectivity
|
||||
self.__add_router_interface()
|
||||
|
||||
# Create a network with associated subnet
|
||||
# Check first if a network with the same name exists, if it exists
|
||||
# return that network.
|
||||
# dns_nameservers: a list of name servers e.g. ['8.8.8.8']
|
||||
def create_net(self, network_name, subnet_name, cidr, dns_nameservers,
|
||||
subnet_name_ipv6=None, cidr_ipv6=None, ipv6_mode=None):
|
||||
|
||||
for network in self.networks:
|
||||
if network['name'] == network_name:
|
||||
print ('Found existing internal network: %s'
|
||||
% (network_name))
|
||||
return network
|
||||
|
||||
body = {
|
||||
'network': {
|
||||
'name': network_name,
|
||||
'admin_state_up': True
|
||||
}
|
||||
}
|
||||
network = self.neutron_client.create_network(body)['network']
|
||||
body = {
|
||||
'subnet': {
|
||||
'name': subnet_name,
|
||||
'cidr': cidr,
|
||||
'network_id': network['id'],
|
||||
'enable_dhcp': True,
|
||||
'ip_version': 4,
|
||||
'dns_nameservers': dns_nameservers
|
||||
}
|
||||
}
|
||||
subnet = self.neutron_client.create_subnet(body)['subnet']
|
||||
# add subnet id to the network dict since it has just been added
|
||||
network['subnets'] = [subnet['id']]
|
||||
# If ipv6 is enabled than create and add ipv6 network
|
||||
if ipv6_mode:
|
||||
body = {
|
||||
'subnet': {
|
||||
'name': subnet_name_ipv6,
|
||||
'cidr': cidr_ipv6,
|
||||
'network_id': network['id'],
|
||||
'enable_dhcp': True,
|
||||
'ip_version': 6,
|
||||
'ipv6_ra_mode': ipv6_mode,
|
||||
'ipv6_address_mode': ipv6_mode
|
||||
}
|
||||
}
|
||||
subnet = self.neutron_client.create_subnet(body)['subnet']
|
||||
# add the subnet id to the network dict
|
||||
network['subnets'].append(subnet['id'])
|
||||
print 'Created internal network: %s' % (network_name)
|
||||
return network
|
||||
|
||||
# Delete a network and associated subnet
|
||||
def delete_net(self, network):
|
||||
if network:
|
||||
name = network['name']
|
||||
# it may take some time for ports to be cleared so we need to retry
|
||||
for _ in range(1, 5):
|
||||
try:
|
||||
self.neutron_client.delete_network(network['id'])
|
||||
print 'Network %s deleted' % (name)
|
||||
break
|
||||
except NetworkInUseClient:
|
||||
time.sleep(1)
|
||||
|
||||
# Add a network/subnet to a logical router
|
||||
# Check that it is not already attached to the network/subnet
|
||||
def __add_router_interface(self):
|
||||
|
||||
# and pick the first in the list - the list should be non empty and
|
||||
# contain only 1 subnet since it is supposed to be a private network
|
||||
|
||||
# But first check that the router does not already have this subnet
|
||||
# so retrieve the list of all ports, then check if there is one port
|
||||
# - matches the subnet
|
||||
# - and is attached to the router
|
||||
# Assumed that both management networks are created together so checking for one of them
|
||||
ports = self.neutron_client.list_ports()['ports']
|
||||
for port in ports:
|
||||
port_ip = port['fixed_ips'][0]
|
||||
if (port['device_id'] == self.ext_router['id']) and \
|
||||
(port_ip['subnet_id'] == self.vm_int_net[0]['subnets'][0]):
|
||||
print 'Ext router already associated to the internal network'
|
||||
return
|
||||
|
||||
for int_net in self.vm_int_net:
|
||||
body = {
|
||||
'subnet_id': int_net['subnets'][0]
|
||||
}
|
||||
self.neutron_client.add_interface_router(self.ext_router['id'], body)
|
||||
if self.config.debug:
|
||||
print 'Ext router associated to ' + int_net['name']
|
||||
# If ipv6 is enabled than add second subnet
|
||||
if self.ipv6_enabled:
|
||||
body = {
|
||||
'subnet_id': int_net['subnets'][1]
|
||||
}
|
||||
self.neutron_client.add_interface_router(self.ext_router['id'], body)
|
||||
|
||||
# Detach the ext router from the mgmt network
|
||||
def __remove_router_interface(self):
|
||||
for int_net in self.vm_int_net:
|
||||
if int_net:
|
||||
# If ipv6 is enabled remove that subnet too
|
||||
if self.ipv6_enabled:
|
||||
body = {
|
||||
'subnet_id': int_net['subnets'][1]
|
||||
}
|
||||
self.neutron_client.remove_interface_router(self.ext_router['id'],
|
||||
body)
|
||||
body = {
|
||||
'subnet_id': int_net['subnets'][0]
|
||||
}
|
||||
try:
|
||||
self.neutron_client.remove_interface_router(self.ext_router['id'],
|
||||
body)
|
||||
except NeutronException:
|
||||
# May fail with neutronclient.common.exceptions.Conflict
|
||||
# if there are floating IP in use - just ignore
|
||||
print('Router interface may have floating IP in use: not deleted')
|
||||
|
||||
# Lookup network given network name
|
||||
def lookup_network(self, network_name):
|
||||
networks = self.neutron_client.list_networks(name=network_name)
|
||||
return networks['networks'][0]
|
||||
|
||||
# Create a router and up-date external gateway on router
|
||||
# to external network
|
||||
def create_router(self, router_name, net_id):
|
||||
body = {
|
||||
"router": {
|
||||
"name": router_name,
|
||||
"admin_state_up": True,
|
||||
"external_gateway_info": {
|
||||
"network_id": net_id
|
||||
}
|
||||
}
|
||||
}
|
||||
router = self.neutron_client.create_router(body)
|
||||
return router['router']
|
||||
|
||||
# Show a router based on name
|
||||
def show_router(self, router_name):
|
||||
router = self.neutron_client.show_router(router_name)
|
||||
return router
|
||||
|
||||
# Update a router given router and network id
|
||||
def update_router(self, router_id, net_id):
|
||||
print net_id
|
||||
body = {
|
||||
"router": {
|
||||
"name": "pns-router",
|
||||
"external_gateway_info": {
|
||||
"network_id": net_id
|
||||
}
|
||||
}
|
||||
}
|
||||
router = self.neutron_client.update_router(router_id, body)
|
||||
return router['router']
|
||||
|
||||
# Create a port
|
||||
def create_port(self, net_id, sec_group_list, vnic_type):
|
||||
body = {
|
||||
"port": {
|
||||
"network_id": net_id,
|
||||
"security_groups": sec_group_list
|
||||
}
|
||||
}
|
||||
if vnic_type:
|
||||
body['port']['binding:vnic_type'] = vnic_type
|
||||
port = self.neutron_client.create_port(body)
|
||||
if self.config.debug:
|
||||
print 'Created port ' + port['port']['id']
|
||||
return port['port']
|
||||
|
||||
def delete_port(self, port):
|
||||
if self.config.debug:
|
||||
print 'Deleting port ' + port['id']
|
||||
self.neutron_client.delete_port(port['id'])
|
||||
|
||||
# Create a floating ip on the external network and return it
|
||||
def create_floating_ip(self):
|
||||
body = {
|
||||
"floatingip": {
|
||||
"floating_network_id": self.ext_net['id']
|
||||
}
|
||||
}
|
||||
fip = self.neutron_client.create_floatingip(body)
|
||||
return fip
|
||||
|
||||
# Delete floating ip given a floating ip ad
|
||||
def delete_floating_ip(self, floatingip):
|
||||
self.neutron_client.delete_floatingip(floatingip)
|
||||
|
||||
# Dispose all network resources, call after all VM have been deleted
|
||||
def dispose(self):
|
||||
# Delete the internal networks only of we did not reuse an existing
|
||||
# network
|
||||
if not self.config.reuse_network_name:
|
||||
self.__remove_router_interface()
|
||||
for int_net in self.vm_int_net:
|
||||
self.delete_net(int_net)
|
||||
# delete the router only if its name matches the pns router name
|
||||
if self.ext_router_created:
|
||||
try:
|
||||
if self.ext_router['name'] == self.ext_router_name:
|
||||
self.neutron_client.delete_router(self.ext_router['id'])
|
||||
print 'External router %s deleted' % \
|
||||
(self.ext_router['name'])
|
||||
except TypeError:
|
||||
print "No external router set"
|
||||
|
||||
def _get_l2agent_type(self):
|
||||
'''
|
||||
Retrieve the list of agents
|
||||
return 'Linux bridge agent' or 'Open vSwitch agent' or 'Unknown agent'
|
||||
'''
|
||||
agents = self.neutron_client.list_agents(fields='agent_type')['agents']
|
||||
for agent in agents:
|
||||
agent_type = agent['agent_type']
|
||||
if 'Linux bridge' in agent_type or 'Open vSwitch' in agent_type:
|
||||
return agent_type
|
||||
|
||||
return 'Unknown agent'
|
||||
|
||||
def _get_internal_iface_dict(self):
|
||||
'''
|
||||
return a dictionary which contains the information needed to determine
|
||||
which pysical interface(s) are holding the internal traffic
|
||||
|
||||
For Linux Bridge, the Neutron L2 Agent will automatically put the
|
||||
configurations from Linux Bridge into Neutron config. So just use
|
||||
the Neutron API to fetch it.
|
||||
|
||||
For OVS, the Neutron L2 Agent is not pushing all information to Neutron
|
||||
config, so we need a second step look-up which will happen in
|
||||
sshutils.get_nic_name(). Here we just maintain:
|
||||
|
||||
In the case of VLAN:
|
||||
{ '<HOSTNAME>' : '<The bridge which has the interface for internal traffic>' }
|
||||
In the case of GRE/VxLAN:
|
||||
{ '<HOSTNAME>' : '<IP Address of local interface>
|
||||
'''
|
||||
|
||||
agents = self.neutron_client.list_agents()['agents']
|
||||
internal_iface_dict = {}
|
||||
for agent in agents:
|
||||
agent_type = agent['agent_type']
|
||||
hostname = agent['host']
|
||||
if 'Linux bridge' in agent_type:
|
||||
agent_detail = self.neutron_client.show_agent(agent['id'])['agent']
|
||||
ifname = agent_detail['configurations']['interface_mappings']['physnet1']
|
||||
internal_iface_dict[hostname] = ifname
|
||||
elif 'Open vSwitch' in agent_type:
|
||||
network_type = self.vm_int_net[0]['provider:network_type']
|
||||
agent_detail = self.neutron_client.show_agent(agent['id'])['agent']
|
||||
if network_type == "vlan":
|
||||
brname = agent_detail['configurations']['bridge_mappings']['physnet1']
|
||||
internal_iface_dict[hostname] = brname
|
||||
elif network_type == "vxlan" or network_type == 'gre':
|
||||
ipaddr = agent_detail['configurations']['tunneling_ip']
|
||||
internal_iface_dict[hostname] = ipaddr
|
||||
|
||||
return internal_iface_dict
|
||||
203
nuttcp_tool.py
203
nuttcp_tool.py
@ -1,203 +0,0 @@
|
||||
# Copyright 2014 Cisco Systems, Inc. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
|
||||
import re
|
||||
|
||||
from perf_tool import PerfTool
|
||||
import sshutils
|
||||
|
||||
class NuttcpTool(PerfTool):
|
||||
|
||||
def __init__(self, instance, perf_tool_path):
|
||||
PerfTool.__init__(self, 'nuttcp-7.3.2', perf_tool_path, instance)
|
||||
|
||||
def get_server_launch_cmd(self):
|
||||
'''Return the commands to launch the server side.'''
|
||||
if self.instance.config.ipv6_mode:
|
||||
return [self.dest_path + ' -P5002 -S --single-threaded -6 &']
|
||||
else:
|
||||
return [self.dest_path + ' -P5002 -S --single-threaded &']
|
||||
|
||||
def run_client(self, target_ip, target_instance,
|
||||
mss=None, bandwidth=0, bidirectional=False):
|
||||
'''Run the test
|
||||
:return: list containing one or more dictionary results
|
||||
'''
|
||||
res_list = []
|
||||
if bidirectional:
|
||||
reverse_dir_list = [False, True]
|
||||
else:
|
||||
reverse_dir_list = [False]
|
||||
|
||||
# Get list of protocols and packet sizes to measure
|
||||
(proto_list, proto_pkt_sizes) = self.get_proto_profile()
|
||||
|
||||
for udp, pkt_size_list in zip(proto_list, proto_pkt_sizes):
|
||||
for pkt_size in pkt_size_list:
|
||||
for reverse_dir in reverse_dir_list:
|
||||
# nuttcp does not support reverse dir for UDP...
|
||||
if reverse_dir and udp:
|
||||
continue
|
||||
if udp:
|
||||
self.instance.display('Measuring UDP Throughput (packet size=%d)...',
|
||||
pkt_size)
|
||||
loop_count = 1
|
||||
else:
|
||||
# For accuracy purpose, TCP throughput will be measured 3 times
|
||||
self.instance.display('Measuring TCP Throughput (packet size=%d)...',
|
||||
pkt_size)
|
||||
loop_count = self.instance.config.tcp_tp_loop_count
|
||||
for _ in xrange(loop_count):
|
||||
res = self.run_client_dir(target_ip, mss,
|
||||
reverse_dir=reverse_dir,
|
||||
bandwidth_kbps=bandwidth,
|
||||
udp=udp,
|
||||
length=pkt_size)
|
||||
res_list.extend(res)
|
||||
|
||||
# For UDP reverse direction we need to start the server on self.instance
|
||||
# and run the client on target_instance
|
||||
if bidirectional and 'U' in self.instance.config.protocols:
|
||||
# Start the server on the client (this tool instance)
|
||||
self.instance.display('Start UDP server for reverse dir')
|
||||
if self.start_server():
|
||||
# Start the client on the target instance
|
||||
target_instance.display('Starting UDP client for reverse dir')
|
||||
|
||||
for pkt_size in self.instance.config.udp_pkt_sizes:
|
||||
self.instance.display('Measuring UDP Throughput packet size=%d'
|
||||
' (reverse direction)...',
|
||||
pkt_size)
|
||||
res = target_instance.tp_tool.run_client_dir(self.instance.internal_ip,
|
||||
mss,
|
||||
bandwidth_kbps=bandwidth,
|
||||
udp=True,
|
||||
length=pkt_size)
|
||||
res[0]['direction'] = 'reverse'
|
||||
res_list.extend(res)
|
||||
else:
|
||||
self.instance.display('Failed to start UDP server for reverse dir')
|
||||
return res_list
|
||||
|
||||
def run_client_dir(self, target_ip,
|
||||
mss,
|
||||
reverse_dir=False,
|
||||
bandwidth_kbps=0,
|
||||
udp=False,
|
||||
length=0,
|
||||
no_cpu_timed=0):
|
||||
'''Run client in one direction
|
||||
:param reverse_dir: True if reverse the direction (tcp only for now)
|
||||
:param bandwidth_kbps: transmit rate limit in Kbps
|
||||
:param udp: if true get UDP throughput, else get TCP throughput
|
||||
:param length: length of network write|read buf (default 1K|8K/udp, 64K/tcp)
|
||||
for udp is the packet size
|
||||
:param no_cpu_timed: if non zero will disable cpu collection and override
|
||||
the time with the provided value - used mainly for udp
|
||||
to find quickly the optimal throughput using short
|
||||
tests at various throughput values
|
||||
:return: a list of 1 dictionary with the results (see parse_results())
|
||||
'''
|
||||
# run client using the default TCP window size (tcp window
|
||||
# scaling is normally enabled by default so setting explicit window
|
||||
# size is not going to help achieve better results)
|
||||
opts = ''
|
||||
if mss:
|
||||
opts += "-M" + str(mss)
|
||||
if reverse_dir:
|
||||
opts += " -F -r"
|
||||
if length:
|
||||
opts += " -l" + str(length)
|
||||
if self.instance.config.ipv6_mode:
|
||||
opts += " -6 "
|
||||
if udp:
|
||||
opts += " -u"
|
||||
# for UDP if the bandwidth is not provided we need to calculate
|
||||
# the optimal bandwidth
|
||||
if not bandwidth_kbps:
|
||||
udp_res = self.find_udp_bdw(length, target_ip)
|
||||
if 'error' in udp_res:
|
||||
return [udp_res]
|
||||
if not self.instance.gmond_svr:
|
||||
# if we do not collect CPU we miught as well return
|
||||
# the results found through iteration
|
||||
return [udp_res]
|
||||
bandwidth_kbps = udp_res['throughput_kbps']
|
||||
if bandwidth_kbps:
|
||||
opts += " -R%sK" % (bandwidth_kbps)
|
||||
|
||||
if no_cpu_timed:
|
||||
duration_sec = no_cpu_timed
|
||||
else:
|
||||
duration_sec = self.instance.get_cmd_duration()
|
||||
# use data port 5001 and control port 5002
|
||||
# must be enabled in the VM security group
|
||||
cmd = "%s -T%d %s -p5001 -P5002 -fparse %s" % (self.dest_path,
|
||||
duration_sec,
|
||||
opts,
|
||||
target_ip)
|
||||
self.instance.buginf(cmd)
|
||||
try:
|
||||
if no_cpu_timed:
|
||||
# force the timeout value with 20 second extra for the command to
|
||||
# complete and do not collect CPU
|
||||
cpu_load = None
|
||||
cmd_out = self.instance.exec_command(cmd, duration_sec + 20)
|
||||
else:
|
||||
(cmd_out, cpu_load) = self.instance.exec_with_cpu(cmd)
|
||||
except sshutils.SSHError as exc:
|
||||
# Timout or any SSH error
|
||||
self.instance.display('SSH Error:' + str(exc))
|
||||
return [self.parse_error(str(exc))]
|
||||
|
||||
if udp:
|
||||
# UDP output (unicast and multicast):
|
||||
# megabytes=1.1924 real_seconds=10.01 rate_Mbps=0.9997 tx_cpu=99 rx_cpu=0
|
||||
# drop=0 pkt=1221 data_loss=0.00000
|
||||
re_udp = r'rate_Mbps=([\d\.]*) tx_cpu=\d* rx_cpu=\d* drop=(\-*\d*) pkt=(\d*)'
|
||||
match = re.search(re_udp, cmd_out)
|
||||
if match:
|
||||
rate_mbps = float(match.group(1))
|
||||
drop = float(match.group(2))
|
||||
pkt = int(match.group(3))
|
||||
# Workaround for a bug of nuttcp that sometimes it will return a
|
||||
# negative number for drop.
|
||||
if drop < 0:
|
||||
drop = 0
|
||||
|
||||
return [self.parse_results('UDP',
|
||||
int(rate_mbps * 1024),
|
||||
lossrate=round(drop * 100 / pkt, 2),
|
||||
reverse_dir=reverse_dir,
|
||||
msg_size=length,
|
||||
cpu_load=cpu_load)]
|
||||
else:
|
||||
# TCP output:
|
||||
# megabytes=1083.4252 real_seconds=10.04 rate_Mbps=905.5953 tx_cpu=3 rx_cpu=19
|
||||
# retrans=0 rtt_ms=0.55
|
||||
re_tcp = r'rate_Mbps=([\d\.]*) tx_cpu=\d* rx_cpu=\d* retrans=(\d*) rtt_ms=([\d\.]*)'
|
||||
match = re.search(re_tcp, cmd_out)
|
||||
if match:
|
||||
rate_mbps = float(match.group(1))
|
||||
retrans = int(match.group(2))
|
||||
rtt_ms = float(match.group(3))
|
||||
return [self.parse_results('TCP',
|
||||
int(rate_mbps * 1024),
|
||||
retrans=retrans,
|
||||
rtt_ms=rtt_ms,
|
||||
reverse_dir=reverse_dir,
|
||||
msg_size=length,
|
||||
cpu_load=cpu_load)]
|
||||
return [self.parse_error('Could not parse: %s' % (cmd_out))]
|
||||
@ -3,4 +3,4 @@
|
||||
# The list of modules to copy from oslo-incubator.git
|
||||
|
||||
# The base module to hold the copy of openstack.common
|
||||
base=vmtp
|
||||
base=kloudbuster
|
||||
|
||||
110
perf_instance.py
110
perf_instance.py
@ -1,110 +0,0 @@
|
||||
# Copyright 2014 Cisco Systems, Inc. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
|
||||
from instance import Instance as Instance
|
||||
from perf_tool import PingTool
|
||||
|
||||
class PerfInstance(Instance):
|
||||
'''An openstack instance to run performance tools
|
||||
'''
|
||||
def __init__(self, name, config, comp=None, net=None, server=False):
|
||||
Instance.__init__(self, name, config, comp, net)
|
||||
self.is_server = server
|
||||
if 'I' in config.protocols:
|
||||
self.ping = PingTool(self)
|
||||
else:
|
||||
self.ping = None
|
||||
if config.tp_tool:
|
||||
self.tp_tool = config.tp_tool(self, config.perf_tool_path)
|
||||
else:
|
||||
self.tp_tool = None
|
||||
# Override the config drive option to save in instance
|
||||
if config.config_drive:
|
||||
self.config_drive = True
|
||||
else:
|
||||
self.config_drive = None
|
||||
|
||||
# No args is reserved for native host server
|
||||
def create(self, image=None, flavor_type=None,
|
||||
ssh_access=None, nics=None, az=None,
|
||||
management_network_name=None,
|
||||
sec_group=None,
|
||||
init_file_name=None):
|
||||
'''Create an instance
|
||||
:return: True on success, False on error
|
||||
'''
|
||||
rc = Instance.create(self, image, flavor_type, ssh_access,
|
||||
nics, az,
|
||||
management_network_name,
|
||||
sec_group,
|
||||
init_file_name)
|
||||
if not rc:
|
||||
return False
|
||||
if self.tp_tool and not self.tp_tool.install():
|
||||
return False
|
||||
if not self.is_server:
|
||||
return True
|
||||
if self.tp_tool and not self.tp_tool.start_server():
|
||||
return False
|
||||
return True
|
||||
|
||||
def run_client(self, label, dest_ip, target_instance, mss=None,
|
||||
bandwidth=0,
|
||||
bidirectional=False,
|
||||
az_to=None):
|
||||
'''test iperf client using the default TCP window size
|
||||
(tcp window scaling is normally enabled by default so setting explicit window
|
||||
size is not going to help achieve better results)
|
||||
:return: a dictionary containing the results of the run
|
||||
'''
|
||||
# Latency (ping rtt)
|
||||
if 'I' in self.config.protocols:
|
||||
ping_res = self.ping.run_client(dest_ip)
|
||||
else:
|
||||
ping_res = None
|
||||
|
||||
# TCP/UDP throughput with tp_tool, returns a list of dict
|
||||
if self.tp_tool and (not ping_res or 'error' not in ping_res):
|
||||
tp_tool_res = self.tp_tool.run_client(dest_ip,
|
||||
target_instance,
|
||||
mss=mss,
|
||||
bandwidth=bandwidth,
|
||||
bidirectional=bidirectional)
|
||||
else:
|
||||
tp_tool_res = []
|
||||
|
||||
res = {'ip_to': dest_ip}
|
||||
if self.internal_ip:
|
||||
res['ip_from'] = self.internal_ip
|
||||
if label:
|
||||
res['desc'] = label
|
||||
if self.az:
|
||||
res['az_from'] = self.az
|
||||
if az_to:
|
||||
res['az_to'] = az_to
|
||||
res['distro_id'] = self.ssh.distro_id
|
||||
res['distro_version'] = self.ssh.distro_version
|
||||
|
||||
# consolidate results for all tools
|
||||
if ping_res:
|
||||
tp_tool_res.append(ping_res)
|
||||
res['results'] = tp_tool_res
|
||||
return res
|
||||
|
||||
# Override in order to terminate the perf server
|
||||
def dispose(self):
|
||||
if self.tp_tool:
|
||||
self.tp_tool.dispose()
|
||||
Instance.dispose(self)
|
||||
293
perf_tool.py
293
perf_tool.py
@ -1,293 +0,0 @@
|
||||
# Copyright 2014 Cisco Systems, Inc. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
|
||||
import abc
|
||||
import os
|
||||
import re
|
||||
|
||||
# where to copy the tool on the target, must end with slash
|
||||
SCP_DEST_DIR = '/tmp/'
|
||||
|
||||
|
||||
|
||||
#
|
||||
# A base class for all tools that can be associated to an instance
|
||||
#
|
||||
class PerfTool(object):
|
||||
__metaclass__ = abc.ABCMeta
|
||||
|
||||
def __init__(self, name, perf_tool_path, instance):
|
||||
self.name = name
|
||||
self.instance = instance
|
||||
self.dest_path = SCP_DEST_DIR + name
|
||||
self.pid = None
|
||||
self.perf_tool_path = perf_tool_path
|
||||
|
||||
# install the tool to the instance
|
||||
# returns False if fail, True if success
|
||||
def install(self):
|
||||
if self.perf_tool_path:
|
||||
local_path = os.path.join(self.perf_tool_path, self.name)
|
||||
return self.instance.scp(self.name, local_path, self.dest_path)
|
||||
# no install needed
|
||||
return True
|
||||
|
||||
@abc.abstractmethod
|
||||
def get_server_launch_cmd(self):
|
||||
'''To be implemented by sub-classes.'''
|
||||
return None
|
||||
|
||||
def start_server(self):
|
||||
'''Launch the server side of this tool
|
||||
:return: True if success, False if error
|
||||
'''
|
||||
# check if server is already started
|
||||
if not self.pid:
|
||||
self.pid = self.instance.ssh.pidof(self.name)
|
||||
if not self.pid:
|
||||
cmd_list = self.get_server_launch_cmd()
|
||||
# Start the tool server
|
||||
self.instance.buginf('Starting %s server...' % (self.name))
|
||||
for launch_cmd in cmd_list:
|
||||
launch_out = self.instance.exec_command(launch_cmd)
|
||||
self.pid = self.instance.ssh.pidof(self.name)
|
||||
else:
|
||||
self.instance.buginf('%s server already started pid=%s' % (self.name, self.pid))
|
||||
if self.pid:
|
||||
return True
|
||||
else:
|
||||
self.instance.display('Cannot launch server %s: %s' % (self.name, launch_out))
|
||||
return False
|
||||
|
||||
# Terminate pid if started
|
||||
def dispose(self):
|
||||
if self.pid:
|
||||
# Terminate the iperf server
|
||||
self.instance.buginf('Terminating %s', self.name)
|
||||
self.instance.ssh.kill_proc(self.pid)
|
||||
self.pid = None
|
||||
|
||||
def parse_error(self, msg):
|
||||
return {'error': msg, 'tool': self.name}
|
||||
|
||||
def parse_results(self, protocol, throughput, lossrate=None, retrans=None,
|
||||
rtt_ms=None, reverse_dir=False,
|
||||
msg_size=None,
|
||||
cpu_load=None):
|
||||
res = {'throughput_kbps': throughput,
|
||||
'protocol': protocol,
|
||||
'tool': self.name}
|
||||
if self.instance.config.vm_bandwidth:
|
||||
res['bandwidth_limit_kbps'] = self.instance.config.vm_bandwidth
|
||||
if lossrate is not None:
|
||||
res['loss_rate'] = lossrate
|
||||
if retrans:
|
||||
res['retrans'] = retrans
|
||||
if rtt_ms:
|
||||
res['rtt_ms'] = rtt_ms
|
||||
if reverse_dir:
|
||||
res['direction'] = 'reverse'
|
||||
if msg_size:
|
||||
res['pkt_size'] = msg_size
|
||||
if cpu_load:
|
||||
res['cpu_load'] = cpu_load
|
||||
return res
|
||||
|
||||
@abc.abstractmethod
|
||||
def run_client_dir(self, target_ip,
|
||||
mss,
|
||||
reverse_dir=False,
|
||||
bandwidth_kbps=0,
|
||||
udp=False,
|
||||
length=0,
|
||||
no_cpu_timed=0):
|
||||
# must be implemented by sub classes
|
||||
return None
|
||||
|
||||
def find_udp_bdw(self, pkt_size, target_ip):
|
||||
'''Find highest UDP bandwidth within max loss rate for given packet size
|
||||
:return: a dictionary describing the optimal bandwidth (see parse_results())
|
||||
'''
|
||||
# we use a binary search to converge to the optimal throughput
|
||||
# start with 5Gbps - mid-range between 1 and 10Gbps
|
||||
# Convergence can be *very* tricky because UDP throughput behavior
|
||||
# can vary dramatically between host runs and guest runs.
|
||||
# The packet rate limitation is going to dictate the effective
|
||||
# send rate, meaning that small packet sizes will yield the worst
|
||||
# throughput.
|
||||
# The measured throughput can be vastly smaller than the requested
|
||||
# throughput even when the loss rate is zero when the sender cannot
|
||||
# send fast enough to fill the network, in that case increasing the
|
||||
# requested rate will not make it any better
|
||||
# Examples:
|
||||
# 1. too much difference between requested/measured bw - regardless of loss rate
|
||||
# => retry with bw mid-way between the requested bw and the measured bw
|
||||
# /tmp/nuttcp-7.3.2 -T2 -u -l128 -R5000000K -p5001 -P5002 -fparse 192.168.1.2
|
||||
# megabytes=36.9785 real_seconds=2.00 rate_Mbps=154.8474 tx_cpu=23 rx_cpu=32
|
||||
# drop=78149 pkt=381077 data_loss=20.50746
|
||||
# /tmp/nuttcp-7.3.2 -T2 -u -l128 -R2500001K -p5001 -P5002 -fparse 192.168.1.2
|
||||
# megabytes=47.8063 real_seconds=2.00 rate_Mbps=200.2801 tx_cpu=24 rx_cpu=34
|
||||
# drop=0 pkt=391629 data_loss=0.00000
|
||||
# 2. measured and requested bw are very close :
|
||||
# if loss_rate is too low
|
||||
# increase bw mid-way between requested and last max bw
|
||||
# if loss rate is too high
|
||||
# decrease bw mid-way between the measured bw and the last min bw
|
||||
# else stop iteration (converged)
|
||||
# /tmp/nuttcp-7.3.2 -T2 -u -l8192 -R859376K -p5001 -P5002 -fparse 192.168.1.2
|
||||
# megabytes=204.8906 real_seconds=2.00 rate_Mbps=859.2992 tx_cpu=99 rx_cpu=10
|
||||
# drop=0 pkt=26226 data_loss=0.00000
|
||||
|
||||
min_kbps = 1
|
||||
max_kbps = 10000000
|
||||
kbps = 5000000
|
||||
min_loss_rate = self.instance.config.udp_loss_rate_range[0]
|
||||
max_loss_rate = self.instance.config.udp_loss_rate_range[1]
|
||||
# stop if the remaining range to cover is less than 5%
|
||||
while (min_kbps * 100 / max_kbps) < 95:
|
||||
res_list = self.run_client_dir(target_ip, 0, bandwidth_kbps=kbps,
|
||||
udp=True, length=pkt_size,
|
||||
no_cpu_timed=1)
|
||||
# always pick the first element in the returned list of dict(s)
|
||||
# should normally only have 1 element
|
||||
res = res_list[0]
|
||||
if 'error' in res:
|
||||
return res
|
||||
loss_rate = res['loss_rate']
|
||||
measured_kbps = res['throughput_kbps']
|
||||
self.instance.buginf('pkt-size=%d throughput=%d<%d/%d<%d Kbps loss-rate=%d' %
|
||||
(pkt_size, min_kbps, measured_kbps, kbps, max_kbps, loss_rate))
|
||||
# expected rate must be at least 80% of the requested rate
|
||||
if (measured_kbps * 100 / kbps) < 80:
|
||||
# the measured bw is too far away from the requested bw
|
||||
# take half the distance or 3x the measured bw whichever is lowest
|
||||
kbps = measured_kbps + (kbps - measured_kbps) / 2
|
||||
if measured_kbps:
|
||||
kbps = min(kbps, measured_kbps * 3)
|
||||
max_kbps = kbps
|
||||
continue
|
||||
# The measured bw is within striking distance from the requested bw
|
||||
# increase bw if loss rate is too small
|
||||
if loss_rate < min_loss_rate:
|
||||
# undershot
|
||||
if measured_kbps > min_kbps:
|
||||
min_kbps = measured_kbps
|
||||
else:
|
||||
# to make forward progress we need to increase min_kbps
|
||||
# and try a higher bw since the loss rate is too low
|
||||
min_kbps = int((max_kbps + min_kbps) / 2)
|
||||
|
||||
kbps = int((max_kbps + min_kbps) / 2)
|
||||
# print ' undershot, min=%d kbps=%d max=%d' % (min_kbps, kbps, max_kbps)
|
||||
elif loss_rate > max_loss_rate:
|
||||
# overshot
|
||||
max_kbps = kbps
|
||||
if measured_kbps < kbps:
|
||||
kbps = measured_kbps
|
||||
else:
|
||||
kbps = int((max_kbps + min_kbps) / 2)
|
||||
# print ' overshot, min=%d kbps=%d max=%d' % (min_kbps, kbps, max_kbps)
|
||||
else:
|
||||
# converged within loss rate bracket
|
||||
break
|
||||
return res
|
||||
|
||||
def get_proto_profile(self):
|
||||
'''Return a tuple containing the list of protocols (tcp/udp) and
|
||||
list of packet sizes (udp only)
|
||||
'''
|
||||
# start with TCP (udp=False) then UDP
|
||||
proto_list = []
|
||||
proto_pkt_sizes = []
|
||||
if 'T' in self.instance.config.protocols:
|
||||
proto_list.append(False)
|
||||
proto_pkt_sizes.append(self.instance.config.tcp_pkt_sizes)
|
||||
if 'U' in self.instance.config.protocols:
|
||||
proto_list.append(True)
|
||||
proto_pkt_sizes.append(self.instance.config.udp_pkt_sizes)
|
||||
return (proto_list, proto_pkt_sizes)
|
||||
|
||||
class PingTool(PerfTool):
|
||||
'''
|
||||
A class to run ping and get loss rate and round trip time
|
||||
'''
|
||||
|
||||
def __init__(self, instance):
|
||||
PerfTool.__init__(self, 'ping', None, instance)
|
||||
|
||||
def run_client(self, target_ip, ping_count=5):
|
||||
'''Perform the ping operation
|
||||
:return: a dict containing the results stats
|
||||
|
||||
Example of output:
|
||||
10 packets transmitted, 10 packets received, 0.0% packet loss
|
||||
round-trip min/avg/max/stddev = 55.855/66.074/103.915/13.407 ms
|
||||
or
|
||||
5 packets transmitted, 5 received, 0% packet loss, time 3998ms
|
||||
rtt min/avg/max/mdev = 0.455/0.528/0.596/0.057 ms
|
||||
'''
|
||||
if self.instance.config.ipv6_mode:
|
||||
cmd = "ping6 -c " + str(ping_count) + " " + str(target_ip)
|
||||
else:
|
||||
cmd = "ping -c " + str(ping_count) + " " + str(target_ip)
|
||||
cmd_out = self.instance.exec_command(cmd)
|
||||
if not cmd_out:
|
||||
res = {'protocol': 'ICMP',
|
||||
'tool': 'ping',
|
||||
'error': 'failed'}
|
||||
return res
|
||||
match = re.search(r'(\d*) packets transmitted, (\d*) ',
|
||||
cmd_out)
|
||||
if match:
|
||||
tx_packets = match.group(1)
|
||||
rx_packets = match.group(2)
|
||||
else:
|
||||
tx_packets = 0
|
||||
rx_packets = 0
|
||||
match = re.search(r'min/avg/max/[a-z]* = ([\d\.]*)/([\d\.]*)/([\d\.]*)/([\d\.]*)',
|
||||
cmd_out)
|
||||
if match:
|
||||
rtt_min = match.group(1)
|
||||
rtt_avg = match.group(2)
|
||||
rtt_max = match.group(3)
|
||||
rtt_stddev = match.group(4)
|
||||
else:
|
||||
rtt_min = 0
|
||||
rtt_max = 0
|
||||
rtt_avg = 0
|
||||
rtt_stddev = 0
|
||||
res = {'protocol': 'ICMP',
|
||||
'tool': 'ping',
|
||||
'tx_packets': tx_packets,
|
||||
'rx_packets': rx_packets,
|
||||
'rtt_min_ms': rtt_min,
|
||||
'rtt_max_ms': rtt_max,
|
||||
'rtt_avg_ms': rtt_avg,
|
||||
'rtt_stddev': rtt_stddev}
|
||||
return res
|
||||
|
||||
def get_server_launch_cmd(self):
|
||||
# not applicable
|
||||
return None
|
||||
|
||||
def run_client_dir(self, target_ip,
|
||||
mss,
|
||||
reverse_dir=False,
|
||||
bandwidth_kbps=0,
|
||||
udp=False,
|
||||
length=0,
|
||||
no_cpu_timed=0):
|
||||
# not applicable
|
||||
return None
|
||||
142
pns_mongo.py
142
pns_mongo.py
@ -1,142 +0,0 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2014 Cisco Systems, Inc. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
|
||||
import pymongo
|
||||
|
||||
def connect_to_mongod(mongod_ip, mongod_port):
|
||||
'''
|
||||
Create a connection to the mongo deamon.
|
||||
'''
|
||||
if mongod_ip is None:
|
||||
mongod_ip = "localhost"
|
||||
|
||||
if mongod_port is None:
|
||||
mongod_port = 27017
|
||||
|
||||
client = None
|
||||
|
||||
try:
|
||||
client = pymongo.MongoClient(mongod_ip, mongod_port)
|
||||
except pymongo.errors.ConnectionFailure:
|
||||
print "ERROR: pymongo. Connection Failure (%s) (%d)" % \
|
||||
(mongod_ip, mongod_port)
|
||||
return None
|
||||
|
||||
return client
|
||||
|
||||
|
||||
def get_mongod_collection(db_client, database_name, collection_name):
|
||||
'''
|
||||
Given db name and collection name, get the collection object.
|
||||
'''
|
||||
mongo_db = db_client[database_name]
|
||||
if mongo_db is None:
|
||||
print "Invalid database name"
|
||||
return None
|
||||
|
||||
collection = mongo_db[collection_name]
|
||||
if collection is None:
|
||||
return None
|
||||
|
||||
return collection
|
||||
|
||||
|
||||
def is_type_dict(var):
|
||||
if isinstance(var, dict):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def add_new_document_to_collection(collection, document):
|
||||
if collection is None:
|
||||
print "collection cannot be none"
|
||||
return None
|
||||
|
||||
if not is_type_dict(document):
|
||||
print "Document type should be a dictionary"
|
||||
return None
|
||||
|
||||
post_id = collection.insert(document)
|
||||
|
||||
return post_id
|
||||
|
||||
|
||||
def search_documents_in_collection(collection, pattern):
|
||||
if collection is None:
|
||||
print "collection cannot be None"
|
||||
return None
|
||||
|
||||
if pattern is None:
|
||||
pattern = {}
|
||||
|
||||
if not is_type_dict(pattern):
|
||||
print "pattern type should be a dictionary"
|
||||
return None
|
||||
|
||||
try:
|
||||
output = collection.find(pattern)
|
||||
except TypeError:
|
||||
print "A TypeError occured. Invalid pattern: ", pattern
|
||||
return None
|
||||
|
||||
return output
|
||||
|
||||
|
||||
def pns_add_test_result_to_mongod(mongod_ip,
|
||||
mongod_port, pns_database,
|
||||
pns_collection, document):
|
||||
'''
|
||||
Invoked from vmtp to add a new result to the mongod database.
|
||||
'''
|
||||
client = connect_to_mongod(mongod_ip, mongod_port)
|
||||
if client is None:
|
||||
print "ERROR: Failed to connect to mongod (%s) (%d)" % \
|
||||
(mongod_ip, mongod_port)
|
||||
return None
|
||||
|
||||
collection = get_mongod_collection(client, pns_database, pns_collection)
|
||||
if collection is None:
|
||||
print "ERROR: Failed to get collection DB: %s, %s" % \
|
||||
(pns_database, pns_collection)
|
||||
return None
|
||||
|
||||
post_id = add_new_document_to_collection(collection, document)
|
||||
|
||||
return post_id
|
||||
|
||||
|
||||
def pns_search_results_from_mongod(mongod_ip, mongod_port,
|
||||
pns_database, pns_collection,
|
||||
pattern):
|
||||
'''
|
||||
Can be invoked from a helper script to query the mongod database
|
||||
'''
|
||||
client = connect_to_mongod(mongod_ip, mongod_port)
|
||||
if client is None:
|
||||
print "ERROR: Failed to connect to mongod (%s) (%d)" % \
|
||||
(mongod_ip, mongod_port)
|
||||
return
|
||||
|
||||
collection = get_mongod_collection(client, pns_database, pns_collection)
|
||||
if collection is None:
|
||||
print "ERROR: Failed to get collection DB: %s, %s" % \
|
||||
(pns_database, pns_collection)
|
||||
return
|
||||
|
||||
docs = search_documents_in_collection(collection, pattern)
|
||||
|
||||
return docs
|
||||
328
pnsdb_summary.py
328
pnsdb_summary.py
@ -1,328 +0,0 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2014 Cisco Systems, Inc. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import sys
|
||||
|
||||
import pns_mongo
|
||||
|
||||
import tabulate
|
||||
|
||||
###########################################
|
||||
# Global list of all result functions
|
||||
# that are displayed as a menu/list.
|
||||
###########################################
|
||||
pnsdb_results_list = [
|
||||
("Summary of all results", "show_summary_all"),
|
||||
("Show TCP results for vlan encap", "show_tcp_summary_encap_vlan"),
|
||||
("Show UDP results for vlan encap", "show_udp_summary_encap_vlan"),
|
||||
]
|
||||
|
||||
network_type = [
|
||||
(0, "L2 Network"),
|
||||
(1, "L3 Network"),
|
||||
(100, "Unknown"),
|
||||
]
|
||||
|
||||
vm_loc = [
|
||||
(0, "Intra-node"),
|
||||
(1, "Inter-node"),
|
||||
]
|
||||
|
||||
flow_re = re.compile(r".*(same|different) network.*(fixed|floating).*"
|
||||
"IP.*(inter|intra).*",
|
||||
re.IGNORECASE)
|
||||
|
||||
|
||||
def get_flow_type(flow_desc):
|
||||
vm_location = None
|
||||
nw_type = None
|
||||
fixed_ip = None
|
||||
|
||||
mobj = flow_re.match(flow_desc)
|
||||
if mobj:
|
||||
if mobj.group(1) == "same":
|
||||
nw_type = network_type[0][0]
|
||||
elif mobj.group(1) == "different":
|
||||
nw_type = network_type[1][0]
|
||||
else:
|
||||
nw_type = network_type[2][0]
|
||||
|
||||
if mobj.group(2) == "fixed":
|
||||
fixed_ip = True
|
||||
else:
|
||||
fixed_ip = False
|
||||
|
||||
if mobj.group(3) == "inter":
|
||||
vm_location = vm_loc[1][0]
|
||||
else:
|
||||
vm_location = vm_loc[0][0]
|
||||
|
||||
return(vm_location, nw_type, fixed_ip)
|
||||
|
||||
|
||||
def get_tcp_flow_data(data):
|
||||
record_list = []
|
||||
for record in data:
|
||||
for flow in record['flows']:
|
||||
results = flow['results']
|
||||
get_flow_type(flow['desc'])
|
||||
for result in results:
|
||||
show_record = {}
|
||||
if result['protocol'] == "TCP" or result['protocol'] == "tcp":
|
||||
show_record['throughput_kbps'] = result['throughput_kbps']
|
||||
show_record['rtt_ms'] = result['rtt_ms']
|
||||
show_record['pkt_size'] = result['pkt_size']
|
||||
show_record['openstack_version'] = record['openstack_version']
|
||||
show_record['date'] = record['date']
|
||||
show_record['distro'] = record['distro']
|
||||
# show_record['desc'] = flow['desc']
|
||||
record_list.append(show_record)
|
||||
|
||||
return record_list
|
||||
|
||||
|
||||
def get_udp_flow_data(data):
|
||||
record_list = []
|
||||
for record in data:
|
||||
for flow in record['flows']:
|
||||
results = flow['results']
|
||||
get_flow_type(flow['desc'])
|
||||
for result in results:
|
||||
show_record = {}
|
||||
if result['protocol'] == "UDP" or result['protocol'] == "udp":
|
||||
show_record['throughput_kbps'] = result['throughput_kbps']
|
||||
show_record['loss_rate'] = result['loss_rate']
|
||||
show_record['openstack_version'] = record['openstack_version']
|
||||
show_record['date'] = record['date']
|
||||
show_record['distro'] = record['distro']
|
||||
# show_record['desc'] = flow['desc']
|
||||
record_list.append(show_record)
|
||||
return record_list
|
||||
|
||||
|
||||
def show_pnsdb_summary(db_server, db_port, db_name, db_collection):
|
||||
'''
|
||||
Show a summary of results.
|
||||
'''
|
||||
pattern = {}
|
||||
data = pns_mongo.pns_search_results_from_mongod(db_server,
|
||||
db_port,
|
||||
db_name,
|
||||
db_collection,
|
||||
pattern)
|
||||
record_list = get_tcp_flow_data(data)
|
||||
print tabulate.tabulate(record_list, headers="keys", tablefmt="grid")
|
||||
print data.count()
|
||||
|
||||
data = pns_mongo.pns_search_results_from_mongod(db_server,
|
||||
db_port,
|
||||
db_name,
|
||||
db_collection,
|
||||
pattern)
|
||||
record_list = get_udp_flow_data(data)
|
||||
print "UDP:"
|
||||
print tabulate.tabulate(record_list, headers="keys", tablefmt="grid")
|
||||
|
||||
|
||||
def get_results_info(results, cols, protocol=None):
|
||||
result_list = []
|
||||
|
||||
for result in results:
|
||||
show_result = {}
|
||||
if protocol is not None:
|
||||
if result['protocol'] != protocol:
|
||||
continue
|
||||
for col in cols:
|
||||
if col in result.keys():
|
||||
show_result[col] = result[col]
|
||||
|
||||
result_list.append(show_result)
|
||||
|
||||
return result_list
|
||||
|
||||
|
||||
def get_flow_info(flow, cols):
|
||||
flow_list = []
|
||||
show_flow = {}
|
||||
for col in cols:
|
||||
show_flow[col] = flow[col]
|
||||
(vmloc, nw_type, fixed_ip) = get_flow_type(flow['desc'])
|
||||
show_flow['nw_type'] = network_type[nw_type][1]
|
||||
show_flow['vm_loc'] = vm_loc[vmloc][1]
|
||||
if fixed_ip:
|
||||
show_flow['fixed_float'] = "Fixed IP"
|
||||
else:
|
||||
show_flow['fixed_float'] = "Floating IP"
|
||||
flow_list.append(show_flow)
|
||||
|
||||
return flow_list
|
||||
|
||||
|
||||
def get_record_info(record, cols):
|
||||
record_list = []
|
||||
show_record = {}
|
||||
for col in cols:
|
||||
show_record[col] = record[col]
|
||||
|
||||
record_list.append(show_record)
|
||||
|
||||
return record_list
|
||||
|
||||
|
||||
def print_record_header(record):
|
||||
print "#" * 60
|
||||
print "RUN: %s" % (record['date'])
|
||||
cols = ['date', 'distro', 'openstack_version', 'encapsulation']
|
||||
record_list = get_record_info(record, cols)
|
||||
print tabulate.tabulate(record_list)
|
||||
|
||||
|
||||
def print_flow_header(flow):
|
||||
cols = ['desc']
|
||||
flow_list = get_flow_info(flow, cols)
|
||||
print tabulate.tabulate(flow_list, tablefmt="simple")
|
||||
|
||||
|
||||
def show_tcp_summary_encap_vlan(db_server, db_port, db_name, db_collection):
|
||||
pattern = {"encapsulation": "vlan"}
|
||||
|
||||
data = pns_mongo.pns_search_results_from_mongod(db_server,
|
||||
db_port,
|
||||
db_name,
|
||||
db_collection,
|
||||
pattern)
|
||||
for record in data:
|
||||
print_record_header(record)
|
||||
for flow in record['flows']:
|
||||
print_flow_header(flow)
|
||||
cols = ['throughput_kbps', 'protocol', 'tool', 'rtt_ms']
|
||||
result_list = get_results_info(flow['results'], cols,
|
||||
protocol="TCP")
|
||||
print tabulate.tabulate(result_list,
|
||||
headers="keys", tablefmt="grid")
|
||||
|
||||
print "\n"
|
||||
|
||||
|
||||
def show_udp_summary_encap_vlan(db_server, db_port, db_name, db_collection):
|
||||
pattern = {"encapsulation": "vlan"}
|
||||
|
||||
data = pns_mongo.pns_search_results_from_mongod(db_server,
|
||||
db_port,
|
||||
db_name,
|
||||
db_collection,
|
||||
pattern)
|
||||
for record in data:
|
||||
print_record_header(record)
|
||||
for flow in record['flows']:
|
||||
print_flow_header(flow)
|
||||
cols = ['throughput_kbps', 'protocol', 'loss_rate', 'pkt_size']
|
||||
result_list = get_results_info(flow['results'], cols,
|
||||
protocol="UDP")
|
||||
print tabulate.tabulate(result_list,
|
||||
headers="keys", tablefmt="grid")
|
||||
|
||||
|
||||
def show_summary_all(db_server, db_port, db_name, db_collection):
|
||||
pattern = {}
|
||||
|
||||
print "-" * 60
|
||||
print "Summary Data: "
|
||||
print "-" * 60
|
||||
|
||||
data = pns_mongo.pns_search_results_from_mongod(db_server,
|
||||
db_port,
|
||||
db_name,
|
||||
db_collection,
|
||||
pattern)
|
||||
for record in data:
|
||||
print_record_header(record)
|
||||
for flow in record['flows']:
|
||||
print_flow_header(flow)
|
||||
|
||||
# Display the results for each flow.
|
||||
cols = ['throughput_kbps', 'protocol', 'tool',
|
||||
'rtt_ms', 'loss_rate', 'pkt_size',
|
||||
'rtt_avg_ms']
|
||||
result_list = get_results_info(flow['results'], cols)
|
||||
print tabulate.tabulate(result_list,
|
||||
headers="keys", tablefmt="grid")
|
||||
|
||||
print "\n"
|
||||
|
||||
|
||||
def main():
|
||||
####################################################################
|
||||
# parse arguments.
|
||||
# --server-ip [required]
|
||||
# --server-port [optional] [default: 27017]
|
||||
# --official [optional]
|
||||
####################################################################
|
||||
parser = argparse.ArgumentParser(description="VMTP Results formatter")
|
||||
parser.add_argument('-s', "--server-ip", dest="server_ip",
|
||||
action="store",
|
||||
help="MongoDB Server IP address")
|
||||
parser.add_argument('-p', "--server-port", dest="server_port",
|
||||
action="store",
|
||||
help="MongoDB Server port (default 27017)")
|
||||
parser.add_argument("-o", "--official", default=False,
|
||||
action="store_true",
|
||||
help="Access offcial results collection")
|
||||
|
||||
(opts, _) = parser.parse_known_args()
|
||||
|
||||
if not opts.server_ip:
|
||||
print "Provide the pns db server ip address"
|
||||
sys.exit()
|
||||
|
||||
db_server = opts.server_ip
|
||||
|
||||
if not opts.server_port:
|
||||
db_port = 27017
|
||||
else:
|
||||
db_port = opts.server_port
|
||||
|
||||
db_name = "pnsdb"
|
||||
|
||||
if opts.official:
|
||||
print "Use db collection officialdata"
|
||||
db_collection = "officialdata"
|
||||
else:
|
||||
db_collection = "testdata"
|
||||
|
||||
print "-" * 40
|
||||
print "Reports Menu:"
|
||||
print "-" * 40
|
||||
count = 0
|
||||
for option in pnsdb_results_list:
|
||||
print "%d: %s" % (count, option[0])
|
||||
count += 1
|
||||
print "\n"
|
||||
|
||||
try:
|
||||
user_opt = int(raw_input("Choose a report [no] : "))
|
||||
except ValueError:
|
||||
print "Invalid option"
|
||||
sys.exit()
|
||||
|
||||
globals()[pnsdb_results_list[user_opt][1]](db_server,
|
||||
db_port, db_name, db_collection)
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
17
pylintrc
17
pylintrc
@ -1,17 +0,0 @@
|
||||
[BASIC]
|
||||
# Allow constant names to be lower case
|
||||
const-rgx=[a-zA-Z_][a-zA-Z0-9_]{2,30}$
|
||||
module-rgx=[a-zA-Z_][a-zA-Z0-9_]{2,30}$
|
||||
max-line-length=100
|
||||
max-args=10
|
||||
max-branches=20
|
||||
max-locals=20
|
||||
good-names=az,ip,_,rc
|
||||
max-statements=100
|
||||
|
||||
|
||||
[MESSAGE CONTROL]
|
||||
disable=missing-docstring,too-many-public-methods,too-many-instance-attributes,star-args,pointless-string-statement,no-self-use,too-many-locals,superfluous-parens,too-few-public-methods,unused-argument
|
||||
|
||||
[SIMILARITIES]
|
||||
min-similarity-lines=10
|
||||
@ -6,22 +6,14 @@ pbr>=0.6,!=0.7,<1.0
|
||||
Babel>=1.3
|
||||
|
||||
configure>=0.5
|
||||
ecdsa>=0.11
|
||||
hdrhistogram>=0.0.4
|
||||
jsonpatch>=1.9
|
||||
jsonschema>=2.4.0
|
||||
lxml>=3.4.0
|
||||
oslo.log>=1.0.0
|
||||
oslo.utils>=1.2.0
|
||||
paramiko>=1.14.0
|
||||
pecan>=0.9.0
|
||||
pycrypto>=2.6.1
|
||||
pymongo>=2.7.2
|
||||
python-glanceclient>=0.15.0
|
||||
python-neutronclient<3,>=2.3.6
|
||||
python-novaclient>=2.18.1
|
||||
python-openstackclient>=0.4.1
|
||||
python-keystoneclient>=1.0.0
|
||||
redis>=2.10.3
|
||||
scp>=0.8.0
|
||||
tabulate>=0.7.3
|
||||
|
||||
@ -1,2 +0,0 @@
|
||||
#! /bin/bash
|
||||
vmtp.py -h
|
||||
@ -1,8 +0,0 @@
|
||||
This is for VMTP scale testing
|
||||
This is work in progress for now
|
||||
The idea is to be able to
|
||||
1. Create tenant and users to load the cloud
|
||||
2. Create routers within a User in a tenant
|
||||
3. Create N networks per router
|
||||
4. Create N VMs per network
|
||||
5. Clean up all resources by default (Provide ability to avoid cleanup)
|
||||
@ -1,668 +0,0 @@
|
||||
# Copyright 2013: Mirantis Inc.
|
||||
# All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
"""High level ssh library.
|
||||
|
||||
Usage examples:
|
||||
|
||||
Execute command and get output:
|
||||
|
||||
ssh = sshclient.SSH('root', 'example.com', port=33)
|
||||
status, stdout, stderr = ssh.execute('ps ax')
|
||||
if status:
|
||||
raise Exception('Command failed with non-zero status.')
|
||||
print stdout.splitlines()
|
||||
|
||||
Execute command with huge output:
|
||||
|
||||
class PseudoFile(object):
|
||||
def write(chunk):
|
||||
if 'error' in chunk:
|
||||
email_admin(chunk)
|
||||
|
||||
ssh = sshclient.SSH('root', 'example.com')
|
||||
ssh.run('tail -f /var/log/syslog', stdout=PseudoFile(), timeout=False)
|
||||
|
||||
Execute local script on remote side:
|
||||
|
||||
ssh = sshclient.SSH('user', 'example.com')
|
||||
status, out, err = ssh.execute('/bin/sh -s arg1 arg2',
|
||||
stdin=open('~/myscript.sh', 'r'))
|
||||
|
||||
Upload file:
|
||||
|
||||
ssh = sshclient.SSH('user', 'example.com')
|
||||
ssh.run('cat > ~/upload/file.gz', stdin=open('/store/file.gz', 'rb'))
|
||||
|
||||
Eventlet:
|
||||
|
||||
eventlet.monkey_patch(select=True, time=True)
|
||||
or
|
||||
eventlet.monkey_patch()
|
||||
or
|
||||
sshclient = eventlet.import_patched("opentstack.common.sshclient")
|
||||
|
||||
"""
|
||||
|
||||
import re
|
||||
import select
|
||||
import socket
|
||||
import StringIO
|
||||
import sys
|
||||
import time
|
||||
|
||||
import log as logging
|
||||
import paramiko
|
||||
import scp
|
||||
|
||||
LOG = logging.getLogger(__name__)
|
||||
|
||||
class SSHError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class SSHTimeout(SSHError):
|
||||
pass
|
||||
|
||||
# Check IPv4 address syntax - not completely fool proof but will catch
|
||||
# some invalid formats
|
||||
def is_ipv4(address):
|
||||
try:
|
||||
socket.inet_aton(address)
|
||||
except socket.error:
|
||||
return False
|
||||
return True
|
||||
|
||||
class SSHAccess(object):
|
||||
'''
|
||||
A class to contain all the information needed to access a host
|
||||
(native or virtual) using SSH
|
||||
'''
|
||||
def __init__(self, arg_value=None):
|
||||
'''
|
||||
decode user@host[:pwd]
|
||||
'hugo@1.1.1.1:secret' -> ('hugo', '1.1.1.1', 'secret', None)
|
||||
'huggy@2.2.2.2' -> ('huggy', '2.2.2.2', None, None)
|
||||
None ->(None, None, None, None)
|
||||
Examples of fatal errors (will call exit):
|
||||
'hutch@q.1.1.1' (invalid IP)
|
||||
'@3.3.3.3' (missing username)
|
||||
'hiro@' or 'buggy' (missing host IP)
|
||||
The error field will be None in case of success or will
|
||||
contain a string describing the error
|
||||
'''
|
||||
self.username = None
|
||||
self.host = None
|
||||
self.password = None
|
||||
# name of the file that contains the private key
|
||||
self.private_key_file = None
|
||||
# this is the private key itself (a long string starting with
|
||||
# -----BEGIN RSA PRIVATE KEY-----
|
||||
# used when the private key is not saved in any file
|
||||
self.private_key = None
|
||||
self.public_key_file = None
|
||||
self.port = 22
|
||||
self.error = None
|
||||
|
||||
if not arg_value:
|
||||
return
|
||||
match = re.search(r'^([^@]+)@([0-9\.]+):?(.*)$', arg_value)
|
||||
if not match:
|
||||
self.error = 'Invalid argument: ' + arg_value
|
||||
return
|
||||
if not is_ipv4(match.group(2)):
|
||||
self.error = 'Invalid IPv4 address ' + match.group(2)
|
||||
return
|
||||
(self.username, self.host, self.password) = match.groups()
|
||||
|
||||
def copy_from(self, ssh_access):
|
||||
self.username = ssh_access.username
|
||||
self.host = ssh_access.host
|
||||
self.port = ssh_access.port
|
||||
self.password = ssh_access.password
|
||||
self.private_key = ssh_access.private_key
|
||||
self.public_key_file = ssh_access.public_key_file
|
||||
self.private_key_file = ssh_access.private_key_file
|
||||
|
||||
class SSH(object):
|
||||
"""Represent ssh connection."""
|
||||
|
||||
def __init__(self, ssh_access,
|
||||
connect_timeout=60,
|
||||
connect_retry_count=30,
|
||||
connect_retry_wait_sec=2):
|
||||
"""Initialize SSH client.
|
||||
|
||||
:param user: ssh username
|
||||
:param host: hostname or ip address of remote ssh server
|
||||
:param port: remote ssh port
|
||||
:param pkey: RSA or DSS private key string or file object
|
||||
:param key_filename: private key filename
|
||||
:param password: password
|
||||
:param connect_timeout: timeout when connecting ssh
|
||||
:param connect_retry_count: how many times to retry connecting
|
||||
:param connect_retry_wait_sec: seconds to wait between retries
|
||||
"""
|
||||
|
||||
self.ssh_access = ssh_access
|
||||
if ssh_access.private_key:
|
||||
self.pkey = self._get_pkey(ssh_access.private_key)
|
||||
else:
|
||||
self.pkey = None
|
||||
self._client = False
|
||||
self.connect_timeout = connect_timeout
|
||||
self.connect_retry_count = connect_retry_count
|
||||
self.connect_retry_wait_sec = connect_retry_wait_sec
|
||||
self.distro_id = None
|
||||
self.distro_id_like = None
|
||||
self.distro_version = None
|
||||
self.__get_distro()
|
||||
|
||||
def _get_pkey(self, key):
|
||||
'''Get the binary form of the private key
|
||||
from the text form
|
||||
'''
|
||||
if isinstance(key, basestring):
|
||||
key = StringIO.StringIO(key)
|
||||
errors = []
|
||||
for key_class in (paramiko.rsakey.RSAKey, paramiko.dsskey.DSSKey):
|
||||
try:
|
||||
return key_class.from_private_key(key)
|
||||
except paramiko.SSHException as exc:
|
||||
errors.append(exc)
|
||||
raise SSHError('Invalid pkey: %s' % (errors))
|
||||
|
||||
def _get_client(self):
|
||||
if self._client:
|
||||
return self._client
|
||||
self._client = paramiko.SSHClient()
|
||||
self._client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
|
||||
for _ in range(self.connect_retry_count):
|
||||
try:
|
||||
self._client.connect(self.ssh_access.host,
|
||||
username=self.ssh_access.username,
|
||||
port=self.ssh_access.port,
|
||||
pkey=self.pkey,
|
||||
key_filename=self.ssh_access.private_key_file,
|
||||
password=self.ssh_access.password,
|
||||
timeout=self.connect_timeout)
|
||||
return self._client
|
||||
except (paramiko.AuthenticationException,
|
||||
paramiko.BadHostKeyException,
|
||||
paramiko.SSHException,
|
||||
socket.error):
|
||||
time.sleep(self.connect_retry_wait_sec)
|
||||
|
||||
self._client = None
|
||||
msg = '[%s] SSH Connection failed after %s attempts' % (self.ssh_access.host,
|
||||
self.connect_retry_count)
|
||||
raise SSHError(msg)
|
||||
|
||||
def close(self):
|
||||
self._client.close()
|
||||
self._client = False
|
||||
|
||||
def run(self, cmd, stdin=None, stdout=None, stderr=None,
|
||||
raise_on_error=True, timeout=3600):
|
||||
"""Execute specified command on the server.
|
||||
|
||||
:param cmd: Command to be executed.
|
||||
:param stdin: Open file or string to pass to stdin.
|
||||
:param stdout: Open file to connect to stdout.
|
||||
:param stderr: Open file to connect to stderr.
|
||||
:param raise_on_error: If False then exit code will be return. If True
|
||||
then exception will be raized if non-zero code.
|
||||
:param timeout: Timeout in seconds for command execution.
|
||||
Default 1 hour. No timeout if set to 0.
|
||||
"""
|
||||
|
||||
client = self._get_client()
|
||||
|
||||
if isinstance(stdin, basestring):
|
||||
stdin = StringIO.StringIO(stdin)
|
||||
|
||||
return self._run(client, cmd, stdin=stdin, stdout=stdout,
|
||||
stderr=stderr, raise_on_error=raise_on_error,
|
||||
timeout=timeout)
|
||||
|
||||
def _run(self, client, cmd, stdin=None, stdout=None, stderr=None,
|
||||
raise_on_error=True, timeout=3600):
|
||||
|
||||
transport = client.get_transport()
|
||||
session = transport.open_session()
|
||||
session.exec_command(cmd)
|
||||
start_time = time.time()
|
||||
|
||||
data_to_send = ''
|
||||
stderr_data = None
|
||||
|
||||
# If we have data to be sent to stdin then `select' should also
|
||||
# check for stdin availability.
|
||||
if stdin and not stdin.closed:
|
||||
writes = [session]
|
||||
else:
|
||||
writes = []
|
||||
|
||||
while True:
|
||||
# Block until data can be read/write.
|
||||
select.select([session], writes, [session], 1)
|
||||
|
||||
if session.recv_ready():
|
||||
data = session.recv(4096)
|
||||
if stdout is not None:
|
||||
stdout.write(data)
|
||||
continue
|
||||
|
||||
if session.recv_stderr_ready():
|
||||
stderr_data = session.recv_stderr(4096)
|
||||
if stderr is not None:
|
||||
stderr.write(stderr_data)
|
||||
continue
|
||||
|
||||
if session.send_ready():
|
||||
if stdin is not None and not stdin.closed:
|
||||
if not data_to_send:
|
||||
data_to_send = stdin.read(4096)
|
||||
if not data_to_send:
|
||||
stdin.close()
|
||||
session.shutdown_write()
|
||||
writes = []
|
||||
continue
|
||||
sent_bytes = session.send(data_to_send)
|
||||
data_to_send = data_to_send[sent_bytes:]
|
||||
|
||||
if session.exit_status_ready():
|
||||
break
|
||||
|
||||
if timeout and (time.time() - timeout) > start_time:
|
||||
args = {'cmd': cmd, 'host': self.ssh_access.host}
|
||||
raise SSHTimeout(('Timeout executing command '
|
||||
'"%(cmd)s" on host %(host)s') % args)
|
||||
# if e:
|
||||
# raise SSHError('Socket error.')
|
||||
|
||||
exit_status = session.recv_exit_status()
|
||||
if 0 != exit_status and raise_on_error:
|
||||
fmt = ('Command "%(cmd)s" failed with exit_status %(status)d.')
|
||||
details = fmt % {'cmd': cmd, 'status': exit_status}
|
||||
if stderr_data:
|
||||
details += (' Last stderr data: "%s".') % stderr_data
|
||||
raise SSHError(details)
|
||||
return exit_status
|
||||
|
||||
def execute(self, cmd, stdin=None, timeout=3600):
|
||||
"""Execute the specified command on the server.
|
||||
|
||||
:param cmd: Command to be executed.
|
||||
:param stdin: Open file to be sent on process stdin.
|
||||
:param timeout: Timeout for execution of the command.
|
||||
|
||||
Return tuple (exit_status, stdout, stderr)
|
||||
|
||||
"""
|
||||
stdout = StringIO.StringIO()
|
||||
stderr = StringIO.StringIO()
|
||||
|
||||
exit_status = self.run(cmd, stderr=stderr,
|
||||
stdout=stdout, stdin=stdin,
|
||||
timeout=timeout, raise_on_error=False)
|
||||
stdout.seek(0)
|
||||
stderr.seek(0)
|
||||
return (exit_status, stdout.read(), stderr.read())
|
||||
|
||||
def wait(self, timeout=120, interval=1):
|
||||
"""Wait for the host will be available via ssh."""
|
||||
start_time = time.time()
|
||||
while True:
|
||||
try:
|
||||
return self.execute('uname')
|
||||
except (socket.error, SSHError):
|
||||
time.sleep(interval)
|
||||
if time.time() > (start_time + timeout):
|
||||
raise SSHTimeout(('Timeout waiting for "%s"') % self.ssh_access.host)
|
||||
|
||||
def __extract_property(self, name, input_str):
|
||||
expr = name + r'="?([\w\.]*)"?'
|
||||
match = re.search(expr, input_str)
|
||||
if match:
|
||||
return match.group(1)
|
||||
return 'Unknown'
|
||||
|
||||
# Get the linux distro
|
||||
def __get_distro(self):
|
||||
'''cat /etc/*-release | grep ID
|
||||
Ubuntu:
|
||||
DISTRIB_ID=Ubuntu
|
||||
ID=ubuntu
|
||||
ID_LIKE=debian
|
||||
VERSION_ID="14.04"
|
||||
RHEL:
|
||||
ID="rhel"
|
||||
ID_LIKE="fedora"
|
||||
VERSION_ID="7.0"
|
||||
'''
|
||||
distro_cmd = "grep ID /etc/*-release"
|
||||
(status, distro_out, _) = self.execute(distro_cmd)
|
||||
if status:
|
||||
distro_out = ''
|
||||
self.distro_id = self.__extract_property('ID', distro_out)
|
||||
self.distro_id_like = self.__extract_property('ID_LIKE', distro_out)
|
||||
self.distro_version = self.__extract_property('VERSION_ID', distro_out)
|
||||
|
||||
def pidof(self, proc_name):
|
||||
'''
|
||||
Return a list containing the pids of all processes of a given name
|
||||
the list is empty if there is no pid
|
||||
'''
|
||||
# the path update is necessary for RHEL
|
||||
cmd = "PATH=$PATH:/usr/sbin pidof " + proc_name
|
||||
(status, cmd_output, _) = self.execute(cmd)
|
||||
if status:
|
||||
return []
|
||||
cmd_output = cmd_output.strip()
|
||||
result = cmd_output.split()
|
||||
return result
|
||||
|
||||
# kill pids in the given list of pids
|
||||
def kill_proc(self, pid_list):
|
||||
cmd = "kill -9 " + ' '.join(pid_list)
|
||||
self.execute(cmd)
|
||||
|
||||
# check stats for a given path
|
||||
def stat(self, path):
|
||||
(status, cmd_output, _) = self.execute('stat ' + path)
|
||||
if status:
|
||||
return None
|
||||
return cmd_output
|
||||
|
||||
def ping_check(self, target_ip, ping_count=2, pass_threshold=80):
|
||||
'''helper function to ping from one host to an IP address,
|
||||
for a given count and pass_threshold;
|
||||
Steps:
|
||||
ssh to the host and then ping to the target IP
|
||||
then match the output and verify that the loss% is
|
||||
less than the pass_threshold%
|
||||
Return 1 if the criteria passes
|
||||
Return 0, if it fails
|
||||
'''
|
||||
cmd = "ping -c " + str(ping_count) + " " + str(target_ip)
|
||||
(_, cmd_output, _) = self.execute(cmd)
|
||||
|
||||
match = re.search(r'(\d*)% packet loss', cmd_output)
|
||||
pkt_loss = match.group(1)
|
||||
if int(pkt_loss) < int(pass_threshold):
|
||||
return 1
|
||||
else:
|
||||
LOG.error('Ping to %s failed: %s' % (target_ip, cmd_output))
|
||||
return 0
|
||||
|
||||
def get_file_from_host(self, from_path, to_path):
|
||||
'''
|
||||
A wrapper api on top of paramiko scp module, to scp
|
||||
a local file to the host.
|
||||
'''
|
||||
sshcon = self._get_client()
|
||||
scpcon = scp.SCPClient(sshcon.get_transport())
|
||||
try:
|
||||
scpcon.get(from_path, to_path)
|
||||
except scp.SCPException as exp:
|
||||
LOG.error("Send failed: [%s]", exp)
|
||||
return 0
|
||||
return 1
|
||||
|
||||
def read_remote_file(self, from_path):
|
||||
'''
|
||||
Read a remote file and save it to a buffer.
|
||||
'''
|
||||
cmd = "cat " + from_path
|
||||
(status, cmd_output, _) = self.execute(cmd)
|
||||
if status:
|
||||
return None
|
||||
return cmd_output
|
||||
|
||||
def get_host_os_version(self):
|
||||
'''
|
||||
Identify the host distribution/relase.
|
||||
'''
|
||||
os_release_file = "/etc/os-release"
|
||||
sys_release_file = "/etc/system-release"
|
||||
name = ""
|
||||
version = ""
|
||||
|
||||
if self.stat(os_release_file):
|
||||
data = self.read_remote_file(os_release_file)
|
||||
if data is None:
|
||||
LOG.error("ERROR:Failed to read file %s" % os_release_file)
|
||||
return None
|
||||
|
||||
for line in data.splitlines():
|
||||
mobj = re.match(r'NAME=(.*)', line)
|
||||
if mobj:
|
||||
name = mobj.group(1).strip("\"")
|
||||
|
||||
mobj = re.match(r'VERSION_ID=(.*)', line)
|
||||
if mobj:
|
||||
version = mobj.group(1).strip("\"")
|
||||
|
||||
os_name = name + " " + version
|
||||
return os_name
|
||||
|
||||
if self.stat(sys_release_file):
|
||||
data = self.read_remote_file(sys_release_file)
|
||||
if data is None:
|
||||
LOG.error("ERROR:Failed to read file %s" % sys_release_file)
|
||||
return None
|
||||
|
||||
for line in data.splitlines():
|
||||
mobj = re.match(r'Red Hat.*', line)
|
||||
if mobj:
|
||||
return mobj.group(0)
|
||||
|
||||
return None
|
||||
|
||||
def check_rpm_package_installed(self, rpm_pkg):
|
||||
'''
|
||||
Given a host and a package name, check if it is installed on the
|
||||
system.
|
||||
'''
|
||||
check_pkg_cmd = "rpm -qa | grep " + rpm_pkg
|
||||
|
||||
(status, cmd_output, _) = self.execute(check_pkg_cmd)
|
||||
if status:
|
||||
return None
|
||||
|
||||
pkg_pattern = ".*" + rpm_pkg + ".*"
|
||||
rpm_pattern = re.compile(pkg_pattern, re.IGNORECASE)
|
||||
|
||||
for line in cmd_output.splitlines():
|
||||
mobj = rpm_pattern.match(line)
|
||||
if mobj:
|
||||
return mobj.group(0)
|
||||
|
||||
print "%s pkg installed " % rpm_pkg
|
||||
|
||||
return None
|
||||
|
||||
def get_openstack_release(self, ver_str):
|
||||
'''
|
||||
Get the release series name from the package version
|
||||
Refer to here for release tables:
|
||||
https://wiki.openstack.org/wiki/Releases
|
||||
'''
|
||||
ver_table = {"2015.1": "Kilo",
|
||||
"2014.2": "Juno",
|
||||
"2014.1": "Icehouse",
|
||||
"2013.2": "Havana",
|
||||
"2013.1": "Grizzly",
|
||||
"2012.2": "Folsom",
|
||||
"2012.1": "Essex",
|
||||
"2011.3": "Diablo",
|
||||
"2011.2": "Cactus",
|
||||
"2011.1": "Bexar",
|
||||
"2010.1": "Austin"}
|
||||
|
||||
ver_prefix = re.search(r"20\d\d\.\d", ver_str).group(0)
|
||||
if ver_prefix in ver_table:
|
||||
return ver_table[ver_prefix]
|
||||
else:
|
||||
return "Unknown"
|
||||
|
||||
def check_openstack_version(self):
|
||||
'''
|
||||
Identify the openstack version running on the controller.
|
||||
'''
|
||||
nova_cmd = "nova-manage --version"
|
||||
(status, _, err_output) = self.execute(nova_cmd)
|
||||
|
||||
if status:
|
||||
return "Unknown"
|
||||
|
||||
ver_str = err_output.strip()
|
||||
release_str = self.get_openstack_release(err_output)
|
||||
return release_str + " (" + ver_str + ")"
|
||||
|
||||
def get_cpu_info(self):
|
||||
'''
|
||||
Get the CPU info of the controller.
|
||||
|
||||
Note: Here we are assuming the controller node has the exact
|
||||
hardware as the compute nodes.
|
||||
'''
|
||||
|
||||
cmd = 'cat /proc/cpuinfo | grep -m1 "model name"'
|
||||
(status, std_output, _) = self.execute(cmd)
|
||||
if status:
|
||||
return "Unknown"
|
||||
model_name = re.search(r":\s(.*)", std_output).group(1)
|
||||
|
||||
cmd = 'cat /proc/cpuinfo | grep "model name" | wc -l'
|
||||
(status, std_output, _) = self.execute(cmd)
|
||||
if status:
|
||||
return "Unknown"
|
||||
cores = std_output.strip()
|
||||
|
||||
return (cores + " * " + model_name)
|
||||
|
||||
def get_nic_name(self, agent_type, encap, internal_iface_dict):
|
||||
'''
|
||||
Get the NIC info of the controller.
|
||||
|
||||
Note: Here we are assuming the controller node has the exact
|
||||
hardware as the compute nodes.
|
||||
'''
|
||||
|
||||
# The internal_ifac_dict is a dictionary contains the mapping between
|
||||
# hostname and the internal interface name like below:
|
||||
# {u'hh23-4': u'eth1', u'hh23-5': u'eth1', u'hh23-6': u'eth1'}
|
||||
|
||||
cmd = "hostname"
|
||||
(status, std_output, _) = self.execute(cmd)
|
||||
if status:
|
||||
return "Unknown"
|
||||
hostname = std_output.strip()
|
||||
|
||||
if hostname in internal_iface_dict:
|
||||
iface = internal_iface_dict[hostname]
|
||||
else:
|
||||
return "Unknown"
|
||||
|
||||
# Figure out which interface is for internal traffic
|
||||
if 'Linux bridge' in agent_type:
|
||||
ifname = iface
|
||||
elif 'Open vSwitch' in agent_type:
|
||||
if encap == 'vlan':
|
||||
# [root@hh23-10 ~]# ovs-vsctl list-ports br-inst
|
||||
# eth1
|
||||
# phy-br-inst
|
||||
cmd = 'ovs-vsctl list-ports ' + iface + ' | grep -E "^[^phy].*"'
|
||||
(status, std_output, _) = self.execute(cmd)
|
||||
if status:
|
||||
return "Unknown"
|
||||
ifname = std_output.strip()
|
||||
elif encap == 'vxlan' or encap == 'gre':
|
||||
# This is complicated. We need to first get the local IP address on
|
||||
# br-tun, then do a reverse lookup to get the physical interface.
|
||||
#
|
||||
# [root@hh23-4 ~]# ip addr show to "23.23.2.14"
|
||||
# 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
|
||||
# inet 23.23.2.14/24 brd 23.23.2.255 scope global eth1
|
||||
# valid_lft forever preferred_lft forever
|
||||
cmd = "ip addr show to " + iface + " | awk -F: '{print $2}'"
|
||||
(status, std_output, _) = self.execute(cmd)
|
||||
if status:
|
||||
return "Unknown"
|
||||
ifname = std_output.strip()
|
||||
else:
|
||||
return "Unknown"
|
||||
|
||||
cmd = 'ethtool -i ' + ifname + ' | grep bus-info'
|
||||
(status, std_output, _) = self.execute(cmd)
|
||||
if status:
|
||||
return "Unknown"
|
||||
bus_info = re.search(r":\s(.*)", std_output).group(1)
|
||||
|
||||
cmd = 'lspci -s ' + bus_info
|
||||
(status, std_output, _) = self.execute(cmd)
|
||||
if status:
|
||||
return "Unknown"
|
||||
nic_name = re.search(r"Ethernet controller:\s(.*)", std_output).group(1)
|
||||
|
||||
return (nic_name)
|
||||
|
||||
def get_l2agent_version(self, agent_type):
|
||||
'''
|
||||
Get the L2 agent version of the controller.
|
||||
|
||||
Note: Here we are assuming the controller node has the exact
|
||||
hardware as the compute nodes.
|
||||
'''
|
||||
if 'Linux bridge' in agent_type:
|
||||
cmd = "brctl --version | awk -F',' '{print $2}'"
|
||||
ver_string = "Linux Bridge "
|
||||
elif 'Open vSwitch' in agent_type:
|
||||
cmd = "ovs-vsctl --version | awk -F')' '{print $2}'"
|
||||
ver_string = "OVS "
|
||||
else:
|
||||
return "Unknown"
|
||||
|
||||
(status, std_output, _) = self.execute(cmd)
|
||||
if status:
|
||||
return "Unknown"
|
||||
|
||||
return ver_string + std_output.strip()
|
||||
|
||||
|
||||
##################################################
|
||||
# Only invoke the module directly for test purposes. Should be
|
||||
# invoked from pns script.
|
||||
##################################################
|
||||
def main():
|
||||
# As argument pass the SSH access string, e.g. "localadmin@1.1.1.1:secret"
|
||||
test_ssh = SSH(SSHAccess(sys.argv[1]))
|
||||
|
||||
print 'ID=' + test_ssh.distro_id
|
||||
print 'ID_LIKE=' + test_ssh.distro_id_like
|
||||
print 'VERSION_ID=' + test_ssh.distro_version
|
||||
|
||||
# ssh.wait()
|
||||
# print ssh.pidof('bash')
|
||||
# print ssh.stat('/tmp')
|
||||
print test_ssh.check_openstack_version()
|
||||
print test_ssh.get_cpu_info()
|
||||
print test_ssh.get_l2agent_version("Open vSwitch agent")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
23
setup.cfg
23
setup.cfg
@ -1,6 +1,6 @@
|
||||
[metadata]
|
||||
name = vmtp
|
||||
summary = A data path performance tool for OpenStack clouds.
|
||||
name = kloudbuster
|
||||
summary = KloudBuster is a open source tool that allows anybody to load any Neutron OpenStack cloud at massive data plane scale swiftly and effortlessly.
|
||||
description-file =
|
||||
README.rst
|
||||
author = OpenStack
|
||||
@ -8,21 +8,20 @@ author-email = openstack-dev@lists.openstack.org
|
||||
home-page = http://www.openstack.org/
|
||||
classifier =
|
||||
Environment :: OpenStack
|
||||
Intended Audience :: Developers
|
||||
Intended Audience :: Information Technology
|
||||
Intended Audience :: System Administrators
|
||||
License :: OSI Approved :: Apache Software License
|
||||
Operating System :: POSIX :: Linux
|
||||
Operating System :: MacOS
|
||||
Programming Language :: Python
|
||||
Programming Language :: Python :: 2
|
||||
Programming Language :: Python :: 2.7
|
||||
Programming Language :: Python :: 2.6
|
||||
Programming Language :: Python :: 3
|
||||
Programming Language :: Python :: 3.3
|
||||
Programming Language :: Python :: 3.4
|
||||
|
||||
[files]
|
||||
packages =
|
||||
vmtp
|
||||
kloudbuster
|
||||
|
||||
[build_sphinx]
|
||||
source-dir = doc/source
|
||||
@ -33,15 +32,15 @@ all_files = 1
|
||||
upload-dir = doc/build/html
|
||||
|
||||
[compile_catalog]
|
||||
directory = vmtp/locale
|
||||
domain = vmtp
|
||||
directory = kloudbuster/locale
|
||||
domain = kloudbuster
|
||||
|
||||
[update_catalog]
|
||||
domain = vmtp
|
||||
output_dir = vmtp/locale
|
||||
input_file = vmtp/locale/vmtp.pot
|
||||
domain = kloudbuster
|
||||
output_dir = kloudbuster/locale
|
||||
input_file = kloudbuster/locale/kloudbuster.pot
|
||||
|
||||
[extract_messages]
|
||||
keywords = _ gettext ngettext l_ lazy_gettext
|
||||
mapping_file = babel.cfg
|
||||
output_file = vmtp/locale/vmtp.pot
|
||||
output_file = kloudbuster/locale/kloudbuster.pot
|
||||
|
||||
9
setup.py
Executable file → Normal file
9
setup.py
Executable file → Normal file
@ -1,4 +1,3 @@
|
||||
#!/usr/bin/env python
|
||||
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
@ -17,6 +16,14 @@
|
||||
# THIS FILE IS MANAGED BY THE GLOBAL REQUIREMENTS REPO - DO NOT EDIT
|
||||
import setuptools
|
||||
|
||||
# In python < 2.7.4, a lazy loading of package `pbr` will break
|
||||
# setuptools if some other modules registered functions in `atexit`.
|
||||
# solution from: http://bugs.python.org/issue15881#msg170215
|
||||
try:
|
||||
import multiprocessing # noqa
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
setuptools.setup(
|
||||
setup_requires=['pbr'],
|
||||
pbr=True)
|
||||
|
||||
668
sshutils.py
668
sshutils.py
@ -1,668 +0,0 @@
|
||||
# Copyright 2013: Mirantis Inc.
|
||||
# All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
"""High level ssh library.
|
||||
|
||||
Usage examples:
|
||||
|
||||
Execute command and get output:
|
||||
|
||||
ssh = sshclient.SSH('root', 'example.com', port=33)
|
||||
status, stdout, stderr = ssh.execute('ps ax')
|
||||
if status:
|
||||
raise Exception('Command failed with non-zero status.')
|
||||
print stdout.splitlines()
|
||||
|
||||
Execute command with huge output:
|
||||
|
||||
class PseudoFile(object):
|
||||
def write(chunk):
|
||||
if 'error' in chunk:
|
||||
email_admin(chunk)
|
||||
|
||||
ssh = sshclient.SSH('root', 'example.com')
|
||||
ssh.run('tail -f /var/log/syslog', stdout=PseudoFile(), timeout=False)
|
||||
|
||||
Execute local script on remote side:
|
||||
|
||||
ssh = sshclient.SSH('user', 'example.com')
|
||||
status, out, err = ssh.execute('/bin/sh -s arg1 arg2',
|
||||
stdin=open('~/myscript.sh', 'r'))
|
||||
|
||||
Upload file:
|
||||
|
||||
ssh = sshclient.SSH('user', 'example.com')
|
||||
ssh.run('cat > ~/upload/file.gz', stdin=open('/store/file.gz', 'rb'))
|
||||
|
||||
Eventlet:
|
||||
|
||||
eventlet.monkey_patch(select=True, time=True)
|
||||
or
|
||||
eventlet.monkey_patch()
|
||||
or
|
||||
sshclient = eventlet.import_patched("opentstack.common.sshclient")
|
||||
|
||||
"""
|
||||
|
||||
import re
|
||||
import select
|
||||
import socket
|
||||
import StringIO
|
||||
import sys
|
||||
import time
|
||||
|
||||
import paramiko
|
||||
import scp
|
||||
|
||||
# from rally.openstack.common.gettextutils import _
|
||||
|
||||
|
||||
class SSHError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class SSHTimeout(SSHError):
|
||||
pass
|
||||
|
||||
# Check IPv4 address syntax - not completely fool proof but will catch
|
||||
# some invalid formats
|
||||
def is_ipv4(address):
|
||||
try:
|
||||
socket.inet_aton(address)
|
||||
except socket.error:
|
||||
return False
|
||||
return True
|
||||
|
||||
class SSHAccess(object):
|
||||
'''
|
||||
A class to contain all the information needed to access a host
|
||||
(native or virtual) using SSH
|
||||
'''
|
||||
def __init__(self, arg_value=None):
|
||||
'''
|
||||
decode user@host[:pwd]
|
||||
'hugo@1.1.1.1:secret' -> ('hugo', '1.1.1.1', 'secret', None)
|
||||
'huggy@2.2.2.2' -> ('huggy', '2.2.2.2', None, None)
|
||||
None ->(None, None, None, None)
|
||||
Examples of fatal errors (will call exit):
|
||||
'hutch@q.1.1.1' (invalid IP)
|
||||
'@3.3.3.3' (missing username)
|
||||
'hiro@' or 'buggy' (missing host IP)
|
||||
The error field will be None in case of success or will
|
||||
contain a string describing the error
|
||||
'''
|
||||
self.username = None
|
||||
self.host = None
|
||||
self.password = None
|
||||
# name of the file that contains the private key
|
||||
self.private_key_file = None
|
||||
# this is the private key itself (a long string starting with
|
||||
# -----BEGIN RSA PRIVATE KEY-----
|
||||
# used when the private key is not saved in any file
|
||||
self.private_key = None
|
||||
self.public_key_file = None
|
||||
self.port = 22
|
||||
self.error = None
|
||||
|
||||
if not arg_value:
|
||||
return
|
||||
match = re.search(r'^([^@]+)@([0-9\.]+):?(.*)$', arg_value)
|
||||
if not match:
|
||||
self.error = 'Invalid argument: ' + arg_value
|
||||
return
|
||||
if not is_ipv4(match.group(2)):
|
||||
self.error = 'Invalid IPv4 address ' + match.group(2)
|
||||
return
|
||||
(self.username, self.host, self.password) = match.groups()
|
||||
|
||||
def copy_from(self, ssh_access):
|
||||
self.username = ssh_access.username
|
||||
self.host = ssh_access.host
|
||||
self.port = ssh_access.port
|
||||
self.password = ssh_access.password
|
||||
self.private_key = ssh_access.private_key
|
||||
self.public_key_file = ssh_access.public_key_file
|
||||
self.private_key_file = ssh_access.private_key_file
|
||||
|
||||
class SSH(object):
|
||||
"""Represent ssh connection."""
|
||||
|
||||
def __init__(self, ssh_access,
|
||||
connect_timeout=60,
|
||||
connect_retry_count=30,
|
||||
connect_retry_wait_sec=2):
|
||||
"""Initialize SSH client.
|
||||
|
||||
:param user: ssh username
|
||||
:param host: hostname or ip address of remote ssh server
|
||||
:param port: remote ssh port
|
||||
:param pkey: RSA or DSS private key string or file object
|
||||
:param key_filename: private key filename
|
||||
:param password: password
|
||||
:param connect_timeout: timeout when connecting ssh
|
||||
:param connect_retry_count: how many times to retry connecting
|
||||
:param connect_retry_wait_sec: seconds to wait between retries
|
||||
"""
|
||||
|
||||
self.ssh_access = ssh_access
|
||||
if ssh_access.private_key:
|
||||
self.pkey = self._get_pkey(ssh_access.private_key)
|
||||
else:
|
||||
self.pkey = None
|
||||
self._client = False
|
||||
self.connect_timeout = connect_timeout
|
||||
self.connect_retry_count = connect_retry_count
|
||||
self.connect_retry_wait_sec = connect_retry_wait_sec
|
||||
self.distro_id = None
|
||||
self.distro_id_like = None
|
||||
self.distro_version = None
|
||||
self.__get_distro()
|
||||
|
||||
def _get_pkey(self, key):
|
||||
'''Get the binary form of the private key
|
||||
from the text form
|
||||
'''
|
||||
if isinstance(key, basestring):
|
||||
key = StringIO.StringIO(key)
|
||||
errors = []
|
||||
for key_class in (paramiko.rsakey.RSAKey, paramiko.dsskey.DSSKey):
|
||||
try:
|
||||
return key_class.from_private_key(key)
|
||||
except paramiko.SSHException as exc:
|
||||
errors.append(exc)
|
||||
raise SSHError('Invalid pkey: %s' % (errors))
|
||||
|
||||
def _get_client(self):
|
||||
if self._client:
|
||||
return self._client
|
||||
self._client = paramiko.SSHClient()
|
||||
self._client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
|
||||
for _ in range(self.connect_retry_count):
|
||||
try:
|
||||
self._client.connect(self.ssh_access.host,
|
||||
username=self.ssh_access.username,
|
||||
port=self.ssh_access.port,
|
||||
pkey=self.pkey,
|
||||
key_filename=self.ssh_access.private_key_file,
|
||||
password=self.ssh_access.password,
|
||||
timeout=self.connect_timeout)
|
||||
return self._client
|
||||
except (paramiko.AuthenticationException,
|
||||
paramiko.BadHostKeyException,
|
||||
paramiko.SSHException,
|
||||
socket.error):
|
||||
time.sleep(self.connect_retry_wait_sec)
|
||||
|
||||
self._client = None
|
||||
msg = '[%s] SSH Connection failed after %s attempts' % (self.ssh_access.host,
|
||||
self.connect_retry_count)
|
||||
raise SSHError(msg)
|
||||
|
||||
def close(self):
|
||||
self._client.close()
|
||||
self._client = False
|
||||
|
||||
def run(self, cmd, stdin=None, stdout=None, stderr=None,
|
||||
raise_on_error=True, timeout=3600):
|
||||
"""Execute specified command on the server.
|
||||
|
||||
:param cmd: Command to be executed.
|
||||
:param stdin: Open file or string to pass to stdin.
|
||||
:param stdout: Open file to connect to stdout.
|
||||
:param stderr: Open file to connect to stderr.
|
||||
:param raise_on_error: If False then exit code will be return. If True
|
||||
then exception will be raized if non-zero code.
|
||||
:param timeout: Timeout in seconds for command execution.
|
||||
Default 1 hour. No timeout if set to 0.
|
||||
"""
|
||||
|
||||
client = self._get_client()
|
||||
|
||||
if isinstance(stdin, basestring):
|
||||
stdin = StringIO.StringIO(stdin)
|
||||
|
||||
return self._run(client, cmd, stdin=stdin, stdout=stdout,
|
||||
stderr=stderr, raise_on_error=raise_on_error,
|
||||
timeout=timeout)
|
||||
|
||||
def _run(self, client, cmd, stdin=None, stdout=None, stderr=None,
|
||||
raise_on_error=True, timeout=3600):
|
||||
|
||||
transport = client.get_transport()
|
||||
session = transport.open_session()
|
||||
session.exec_command(cmd)
|
||||
start_time = time.time()
|
||||
|
||||
data_to_send = ''
|
||||
stderr_data = None
|
||||
|
||||
# If we have data to be sent to stdin then `select' should also
|
||||
# check for stdin availability.
|
||||
if stdin and not stdin.closed:
|
||||
writes = [session]
|
||||
else:
|
||||
writes = []
|
||||
|
||||
while True:
|
||||
# Block until data can be read/write.
|
||||
select.select([session], writes, [session], 1)
|
||||
|
||||
if session.recv_ready():
|
||||
data = session.recv(4096)
|
||||
if stdout is not None:
|
||||
stdout.write(data)
|
||||
continue
|
||||
|
||||
if session.recv_stderr_ready():
|
||||
stderr_data = session.recv_stderr(4096)
|
||||
if stderr is not None:
|
||||
stderr.write(stderr_data)
|
||||
continue
|
||||
|
||||
if session.send_ready():
|
||||
if stdin is not None and not stdin.closed:
|
||||
if not data_to_send:
|
||||
data_to_send = stdin.read(4096)
|
||||
if not data_to_send:
|
||||
stdin.close()
|
||||
session.shutdown_write()
|
||||
writes = []
|
||||
continue
|
||||
sent_bytes = session.send(data_to_send)
|
||||
data_to_send = data_to_send[sent_bytes:]
|
||||
|
||||
if session.exit_status_ready():
|
||||
break
|
||||
|
||||
if timeout and (time.time() - timeout) > start_time:
|
||||
args = {'cmd': cmd, 'host': self.ssh_access.host}
|
||||
raise SSHTimeout(('Timeout executing command '
|
||||
'"%(cmd)s" on host %(host)s') % args)
|
||||
# if e:
|
||||
# raise SSHError('Socket error.')
|
||||
|
||||
exit_status = session.recv_exit_status()
|
||||
if 0 != exit_status and raise_on_error:
|
||||
fmt = ('Command "%(cmd)s" failed with exit_status %(status)d.')
|
||||
details = fmt % {'cmd': cmd, 'status': exit_status}
|
||||
if stderr_data:
|
||||
details += (' Last stderr data: "%s".') % stderr_data
|
||||
raise SSHError(details)
|
||||
return exit_status
|
||||
|
||||
def execute(self, cmd, stdin=None, timeout=3600):
|
||||
"""Execute the specified command on the server.
|
||||
|
||||
:param cmd: Command to be executed.
|
||||
:param stdin: Open file to be sent on process stdin.
|
||||
:param timeout: Timeout for execution of the command.
|
||||
|
||||
Return tuple (exit_status, stdout, stderr)
|
||||
|
||||
"""
|
||||
stdout = StringIO.StringIO()
|
||||
stderr = StringIO.StringIO()
|
||||
|
||||
exit_status = self.run(cmd, stderr=stderr,
|
||||
stdout=stdout, stdin=stdin,
|
||||
timeout=timeout, raise_on_error=False)
|
||||
stdout.seek(0)
|
||||
stderr.seek(0)
|
||||
return (exit_status, stdout.read(), stderr.read())
|
||||
|
||||
def wait(self, timeout=120, interval=1):
|
||||
"""Wait for the host will be available via ssh."""
|
||||
start_time = time.time()
|
||||
while True:
|
||||
try:
|
||||
return self.execute('uname')
|
||||
except (socket.error, SSHError):
|
||||
time.sleep(interval)
|
||||
if time.time() > (start_time + timeout):
|
||||
raise SSHTimeout(('Timeout waiting for "%s"') % self.ssh_access.host)
|
||||
|
||||
def __extract_property(self, name, input_str):
|
||||
expr = name + r'="?([\w\.]*)"?'
|
||||
match = re.search(expr, input_str)
|
||||
if match:
|
||||
return match.group(1)
|
||||
return 'Unknown'
|
||||
|
||||
# Get the linux distro
|
||||
def __get_distro(self):
|
||||
'''cat /etc/*-release | grep ID
|
||||
Ubuntu:
|
||||
DISTRIB_ID=Ubuntu
|
||||
ID=ubuntu
|
||||
ID_LIKE=debian
|
||||
VERSION_ID="14.04"
|
||||
RHEL:
|
||||
ID="rhel"
|
||||
ID_LIKE="fedora"
|
||||
VERSION_ID="7.0"
|
||||
'''
|
||||
distro_cmd = "grep ID /etc/*-release"
|
||||
(status, distro_out, _) = self.execute(distro_cmd)
|
||||
if status:
|
||||
distro_out = ''
|
||||
self.distro_id = self.__extract_property('ID', distro_out)
|
||||
self.distro_id_like = self.__extract_property('ID_LIKE', distro_out)
|
||||
self.distro_version = self.__extract_property('VERSION_ID', distro_out)
|
||||
|
||||
def pidof(self, proc_name):
|
||||
'''
|
||||
Return a list containing the pids of all processes of a given name
|
||||
the list is empty if there is no pid
|
||||
'''
|
||||
# the path update is necessary for RHEL
|
||||
cmd = "PATH=$PATH:/usr/sbin pidof " + proc_name
|
||||
(status, cmd_output, _) = self.execute(cmd)
|
||||
if status:
|
||||
return []
|
||||
cmd_output = cmd_output.strip()
|
||||
result = cmd_output.split()
|
||||
return result
|
||||
|
||||
# kill pids in the given list of pids
|
||||
def kill_proc(self, pid_list):
|
||||
cmd = "kill -9 " + ' '.join(pid_list)
|
||||
self.execute(cmd)
|
||||
|
||||
# check stats for a given path
|
||||
def stat(self, path):
|
||||
(status, cmd_output, _) = self.execute('stat ' + path)
|
||||
if status:
|
||||
return None
|
||||
return cmd_output
|
||||
|
||||
def ping_check(self, target_ip, ping_count=2, pass_threshold=80):
|
||||
'''helper function to ping from one host to an IP address,
|
||||
for a given count and pass_threshold;
|
||||
Steps:
|
||||
ssh to the host and then ping to the target IP
|
||||
then match the output and verify that the loss% is
|
||||
less than the pass_threshold%
|
||||
Return 1 if the criteria passes
|
||||
Return 0, if it fails
|
||||
'''
|
||||
cmd = "ping -c " + str(ping_count) + " " + str(target_ip)
|
||||
(_, cmd_output, _) = self.execute(cmd)
|
||||
|
||||
match = re.search(r'(\d*)% packet loss', cmd_output)
|
||||
pkt_loss = match.group(1)
|
||||
if int(pkt_loss) < int(pass_threshold):
|
||||
return 1
|
||||
else:
|
||||
print 'Ping to %s failed: %s' % (target_ip, cmd_output)
|
||||
return 0
|
||||
|
||||
def get_file_from_host(self, from_path, to_path):
|
||||
'''
|
||||
A wrapper api on top of paramiko scp module, to scp
|
||||
a local file to the host.
|
||||
'''
|
||||
sshcon = self._get_client()
|
||||
scpcon = scp.SCPClient(sshcon.get_transport())
|
||||
try:
|
||||
scpcon.get(from_path, to_path)
|
||||
except scp.SCPException as exp:
|
||||
print ("Send failed: [%s]", exp)
|
||||
return 0
|
||||
return 1
|
||||
|
||||
def read_remote_file(self, from_path):
|
||||
'''
|
||||
Read a remote file and save it to a buffer.
|
||||
'''
|
||||
cmd = "cat " + from_path
|
||||
(status, cmd_output, _) = self.execute(cmd)
|
||||
if status:
|
||||
return None
|
||||
return cmd_output
|
||||
|
||||
def get_host_os_version(self):
|
||||
'''
|
||||
Identify the host distribution/relase.
|
||||
'''
|
||||
os_release_file = "/etc/os-release"
|
||||
sys_release_file = "/etc/system-release"
|
||||
name = ""
|
||||
version = ""
|
||||
|
||||
if self.stat(os_release_file):
|
||||
data = self.read_remote_file(os_release_file)
|
||||
if data is None:
|
||||
print "ERROR:Failed to read file %s" % os_release_file
|
||||
return None
|
||||
|
||||
for line in data.splitlines():
|
||||
mobj = re.match(r'NAME=(.*)', line)
|
||||
if mobj:
|
||||
name = mobj.group(1).strip("\"")
|
||||
|
||||
mobj = re.match(r'VERSION_ID=(.*)', line)
|
||||
if mobj:
|
||||
version = mobj.group(1).strip("\"")
|
||||
|
||||
os_name = name + " " + version
|
||||
return os_name
|
||||
|
||||
if self.stat(sys_release_file):
|
||||
data = self.read_remote_file(sys_release_file)
|
||||
if data is None:
|
||||
print "ERROR:Failed to read file %s" % sys_release_file
|
||||
return None
|
||||
|
||||
for line in data.splitlines():
|
||||
mobj = re.match(r'Red Hat.*', line)
|
||||
if mobj:
|
||||
return mobj.group(0)
|
||||
|
||||
return None
|
||||
|
||||
def check_rpm_package_installed(self, rpm_pkg):
|
||||
'''
|
||||
Given a host and a package name, check if it is installed on the
|
||||
system.
|
||||
'''
|
||||
check_pkg_cmd = "rpm -qa | grep " + rpm_pkg
|
||||
|
||||
(status, cmd_output, _) = self.execute(check_pkg_cmd)
|
||||
if status:
|
||||
return None
|
||||
|
||||
pkg_pattern = ".*" + rpm_pkg + ".*"
|
||||
rpm_pattern = re.compile(pkg_pattern, re.IGNORECASE)
|
||||
|
||||
for line in cmd_output.splitlines():
|
||||
mobj = rpm_pattern.match(line)
|
||||
if mobj:
|
||||
return mobj.group(0)
|
||||
|
||||
print "%s pkg installed " % rpm_pkg
|
||||
|
||||
return None
|
||||
|
||||
def get_openstack_release(self, ver_str):
|
||||
'''
|
||||
Get the release series name from the package version
|
||||
Refer to here for release tables:
|
||||
https://wiki.openstack.org/wiki/Releases
|
||||
'''
|
||||
ver_table = {"2015.1": "Kilo",
|
||||
"2014.2": "Juno",
|
||||
"2014.1": "Icehouse",
|
||||
"2013.2": "Havana",
|
||||
"2013.1": "Grizzly",
|
||||
"2012.2": "Folsom",
|
||||
"2012.1": "Essex",
|
||||
"2011.3": "Diablo",
|
||||
"2011.2": "Cactus",
|
||||
"2011.1": "Bexar",
|
||||
"2010.1": "Austin"}
|
||||
|
||||
ver_prefix = re.search(r"20\d\d\.\d", ver_str).group(0)
|
||||
if ver_prefix in ver_table:
|
||||
return ver_table[ver_prefix]
|
||||
else:
|
||||
return "Unknown"
|
||||
|
||||
def check_openstack_version(self):
|
||||
'''
|
||||
Identify the openstack version running on the controller.
|
||||
'''
|
||||
nova_cmd = "nova-manage --version"
|
||||
(status, _, err_output) = self.execute(nova_cmd)
|
||||
|
||||
if status:
|
||||
return "Unknown"
|
||||
|
||||
ver_str = err_output.strip()
|
||||
release_str = self.get_openstack_release(err_output)
|
||||
return release_str + " (" + ver_str + ")"
|
||||
|
||||
def get_cpu_info(self):
|
||||
'''
|
||||
Get the CPU info of the controller.
|
||||
|
||||
Note: Here we are assuming the controller node has the exact
|
||||
hardware as the compute nodes.
|
||||
'''
|
||||
|
||||
cmd = 'cat /proc/cpuinfo | grep -m1 "model name"'
|
||||
(status, std_output, _) = self.execute(cmd)
|
||||
if status:
|
||||
return "Unknown"
|
||||
model_name = re.search(r":\s(.*)", std_output).group(1)
|
||||
|
||||
cmd = 'cat /proc/cpuinfo | grep "model name" | wc -l'
|
||||
(status, std_output, _) = self.execute(cmd)
|
||||
if status:
|
||||
return "Unknown"
|
||||
cores = std_output.strip()
|
||||
|
||||
return (cores + " * " + model_name)
|
||||
|
||||
def get_nic_name(self, agent_type, encap, internal_iface_dict):
|
||||
'''
|
||||
Get the NIC info of the controller.
|
||||
|
||||
Note: Here we are assuming the controller node has the exact
|
||||
hardware as the compute nodes.
|
||||
'''
|
||||
|
||||
# The internal_ifac_dict is a dictionary contains the mapping between
|
||||
# hostname and the internal interface name like below:
|
||||
# {u'hh23-4': u'eth1', u'hh23-5': u'eth1', u'hh23-6': u'eth1'}
|
||||
|
||||
cmd = "hostname"
|
||||
(status, std_output, _) = self.execute(cmd)
|
||||
if status:
|
||||
return "Unknown"
|
||||
hostname = std_output.strip()
|
||||
|
||||
if hostname in internal_iface_dict:
|
||||
iface = internal_iface_dict[hostname]
|
||||
else:
|
||||
return "Unknown"
|
||||
|
||||
# Figure out which interface is for internal traffic
|
||||
if 'Linux bridge' in agent_type:
|
||||
ifname = iface
|
||||
elif 'Open vSwitch' in agent_type:
|
||||
if encap == 'vlan':
|
||||
# [root@hh23-10 ~]# ovs-vsctl list-ports br-inst
|
||||
# eth1
|
||||
# phy-br-inst
|
||||
cmd = 'ovs-vsctl list-ports ' + iface + ' | grep -E "^[^phy].*"'
|
||||
(status, std_output, _) = self.execute(cmd)
|
||||
if status:
|
||||
return "Unknown"
|
||||
ifname = std_output.strip()
|
||||
elif encap == 'vxlan' or encap == 'gre':
|
||||
# This is complicated. We need to first get the local IP address on
|
||||
# br-tun, then do a reverse lookup to get the physical interface.
|
||||
#
|
||||
# [root@hh23-4 ~]# ip addr show to "23.23.2.14"
|
||||
# 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
|
||||
# inet 23.23.2.14/24 brd 23.23.2.255 scope global eth1
|
||||
# valid_lft forever preferred_lft forever
|
||||
cmd = "ip addr show to " + iface + " | awk -F: '{print $2}'"
|
||||
(status, std_output, _) = self.execute(cmd)
|
||||
if status:
|
||||
return "Unknown"
|
||||
ifname = std_output.strip()
|
||||
else:
|
||||
return "Unknown"
|
||||
|
||||
cmd = 'ethtool -i ' + ifname + ' | grep bus-info'
|
||||
(status, std_output, _) = self.execute(cmd)
|
||||
if status:
|
||||
return "Unknown"
|
||||
bus_info = re.search(r":\s(.*)", std_output).group(1)
|
||||
|
||||
cmd = 'lspci -s ' + bus_info
|
||||
(status, std_output, _) = self.execute(cmd)
|
||||
if status:
|
||||
return "Unknown"
|
||||
nic_name = re.search(r"Ethernet controller:\s(.*)", std_output).group(1)
|
||||
|
||||
return (nic_name)
|
||||
|
||||
def get_l2agent_version(self, agent_type):
|
||||
'''
|
||||
Get the L2 agent version of the controller.
|
||||
|
||||
Note: Here we are assuming the controller node has the exact
|
||||
hardware as the compute nodes.
|
||||
'''
|
||||
if 'Linux bridge' in agent_type:
|
||||
cmd = "brctl --version | awk -F',' '{print $2}'"
|
||||
ver_string = "Linux Bridge "
|
||||
elif 'Open vSwitch' in agent_type:
|
||||
cmd = "ovs-vsctl --version | awk -F')' '{print $2}'"
|
||||
ver_string = "OVS "
|
||||
else:
|
||||
return "Unknown"
|
||||
|
||||
(status, std_output, _) = self.execute(cmd)
|
||||
if status:
|
||||
return "Unknown"
|
||||
|
||||
return ver_string + std_output.strip()
|
||||
|
||||
|
||||
##################################################
|
||||
# Only invoke the module directly for test purposes. Should be
|
||||
# invoked from pns script.
|
||||
##################################################
|
||||
def main():
|
||||
# As argument pass the SSH access string, e.g. "localadmin@1.1.1.1:secret"
|
||||
test_ssh = SSH(SSHAccess(sys.argv[1]))
|
||||
|
||||
print 'ID=' + test_ssh.distro_id
|
||||
print 'ID_LIKE=' + test_ssh.distro_id_like
|
||||
print 'VERSION_ID=' + test_ssh.distro_version
|
||||
|
||||
# ssh.wait()
|
||||
# print ssh.pidof('bash')
|
||||
# print ssh.stat('/tmp')
|
||||
print test_ssh.check_openstack_version()
|
||||
print test_ssh.get_cpu_info()
|
||||
print test_ssh.get_l2agent_version("Open vSwitch agent")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -2,15 +2,14 @@
|
||||
# of appearance. Changing the order has an impact on the overall integration
|
||||
# process, which may cause wedges in the gate later.
|
||||
|
||||
hacking>=0.9.2,<0.10
|
||||
hacking<0.11,>=0.10.0
|
||||
|
||||
coverage>=3.6
|
||||
discover
|
||||
python-subunit>=0.0.18
|
||||
sphinx>=1.1.2,!=1.2.0,!=1.3b1,<1.3
|
||||
oslosphinx>=2.2.0 # Apache-2.0
|
||||
oslotest>=1.2.0 # Apache-2.0
|
||||
testrepository>=0.0.18
|
||||
testscenarios>=0.4
|
||||
testtools>=0.9.36,!=1.2.0
|
||||
|
||||
oslosphinx>=2.2.0 # Apache-2.0
|
||||
oslotest>=1.2.0 # Apache-2.0
|
||||
|
||||
BIN
tools/iperf
BIN
tools/iperf
Binary file not shown.
Binary file not shown.
20
tox.ini
20
tox.ini
@ -1,6 +1,6 @@
|
||||
[tox]
|
||||
minversion = 1.6
|
||||
envlist = py33,py34,py26,py27,pypy,pep8
|
||||
envlist = py26,py27,pypy,pep8
|
||||
skipsdist = True
|
||||
|
||||
[testenv]
|
||||
@ -10,7 +10,7 @@ setenv =
|
||||
VIRTUAL_ENV={envdir}
|
||||
deps = -r{toxinidir}/requirements.txt
|
||||
-r{toxinidir}/test-requirements.txt
|
||||
commands = python setup.py testr --slowest --testr-args='{posargs}'
|
||||
commands = python setup.py test --slowest --testr-args='{posargs}'
|
||||
|
||||
[testenv:pep8]
|
||||
commands = flake8
|
||||
@ -19,24 +19,24 @@ commands = flake8
|
||||
commands = {posargs}
|
||||
|
||||
[testenv:cover]
|
||||
commands = python setup.py testr --coverage --testr-args='{posargs}'
|
||||
commands = python setup.py test --coverage --testr-args='{posargs}'
|
||||
|
||||
[testenv:docs]
|
||||
commands = python setup.py build_sphinx
|
||||
|
||||
[testenv:debug]
|
||||
commands = oslo_debug_helper {posargs}
|
||||
|
||||
[flake8]
|
||||
# H803 skipped on purpose per list discussion.
|
||||
# E123, E125 skipped as they are invalid PEP-8.
|
||||
max-line-length = 100
|
||||
show-source = True
|
||||
#E302: expected 2 blank linee
|
||||
#E303: too many blank lines (2)
|
||||
# E123, E125 skipped as they are invalid PEP-8.
|
||||
# H233: Python 3.x incompatible use of print operator
|
||||
# H236: Python 3.x incompatible __metaclass__, use six.add_metaclass()
|
||||
#H302: import only modules.
|
||||
# E302: expected 2 blank linee
|
||||
# E303: too many blank lines (2)
|
||||
# H404: multi line docstring should start without a leading new line
|
||||
# H405: multi line docstring summary not separated with an empty line
|
||||
#H904: Wrap long lines in parentheses instead of a backslash
|
||||
ignore = E123,E125,H803,E302,E303,H233,H236,H302,H404,H405,H904
|
||||
ignore = E123,E125,H233,H236,E302,E303,H404,H405
|
||||
builtins = _
|
||||
exclude=.venv,.git,.tox,dist,doc,*openstack/common*,*lib/python*,*egg,build
|
||||
|
||||
897
vmtp.py
897
vmtp.py
@ -1,897 +0,0 @@
|
||||
# Copyright 2014 Cisco Systems, Inc. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
|
||||
import argparse
|
||||
import datetime
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import pprint
|
||||
import re
|
||||
import sys
|
||||
import traceback
|
||||
|
||||
import compute
|
||||
import credentials
|
||||
import iperf_tool
|
||||
import network
|
||||
import nuttcp_tool
|
||||
import pns_mongo
|
||||
import sshutils
|
||||
|
||||
import configure
|
||||
from glanceclient.v2 import client as glanceclient
|
||||
from keystoneclient.v2_0 import client as keystoneclient
|
||||
from neutronclient.v2_0 import client as neutronclient
|
||||
from novaclient.client import Client
|
||||
from novaclient.exceptions import ClientException
|
||||
|
||||
__version__ = '2.1.0'
|
||||
|
||||
from perf_instance import PerfInstance as PerfInstance
|
||||
|
||||
def get_vmtp_absolute_path_for_file(file_name):
|
||||
'''
|
||||
Return the filename in absolute path for any file
|
||||
passed as relative path to the vmtp directory
|
||||
'''
|
||||
if os.path.isabs(__file__):
|
||||
abs_file_path = os.path.join(__file__.split("vmtp.py")[0],
|
||||
file_name)
|
||||
else:
|
||||
abs_file = os.path.abspath(__file__)
|
||||
abs_file_path = os.path.join(abs_file.split("vmtp.py")[0],
|
||||
file_name)
|
||||
|
||||
return abs_file_path
|
||||
|
||||
|
||||
def normalize_paths(cfg):
|
||||
'''
|
||||
Normalize the various paths to config files, tools, ssh priv and pub key
|
||||
files.
|
||||
If a relative path is entered:
|
||||
- the key pair file names are relative to the current directory
|
||||
- the perftool path is relative to vmtp itself
|
||||
'''
|
||||
if cfg.public_key_file:
|
||||
cfg.public_key_file = os.path.abspath(os.path.expanduser(cfg.public_key_file))
|
||||
if cfg.private_key_file:
|
||||
cfg.private_key_file = os.path.expanduser(os.path.expanduser(cfg.private_key_file))
|
||||
if cfg.perf_tool_path:
|
||||
cfg.perf_tool_path = get_vmtp_absolute_path_for_file(cfg.perf_tool_path)
|
||||
|
||||
class FlowPrinter(object):
|
||||
|
||||
def __init__(self):
|
||||
self.flow_num = 0
|
||||
|
||||
def print_desc(self, desc):
|
||||
self.flow_num += 1
|
||||
print "=" * 60
|
||||
print('Flow %d: %s' % (self.flow_num, desc))
|
||||
|
||||
class ResultsCollector(object):
|
||||
|
||||
def __init__(self):
|
||||
self.results = {'flows': []}
|
||||
self.results['date'] = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
|
||||
self.results['args'] = ' '.join(sys.argv)
|
||||
self.results['version'] = __version__
|
||||
self.ppr = pprint.PrettyPrinter(indent=4, width=100)
|
||||
|
||||
def add_property(self, name, value):
|
||||
self.results[name] = value
|
||||
|
||||
def add_properties(self, properties):
|
||||
self.results.update(properties)
|
||||
|
||||
def add_flow_result(self, flow_res):
|
||||
self.results['flows'].append(flow_res)
|
||||
self.ppr.pprint(flow_res)
|
||||
|
||||
def display(self):
|
||||
self.ppr.pprint(self.results)
|
||||
|
||||
def pprint(self, res):
|
||||
self.ppr.pprint(res)
|
||||
|
||||
def get_result(self, key):
|
||||
if keystoneclient in self.results:
|
||||
return self.results[key]
|
||||
return None
|
||||
|
||||
def mask_credentials(self):
|
||||
arguments = self.results['args']
|
||||
if not arguments:
|
||||
return
|
||||
|
||||
arg_list = ['-p', '--host', '--external-host', '--controller-node']
|
||||
for keyword in arg_list:
|
||||
pattern = keyword + r'\s+[^\s]+'
|
||||
string = keyword + ' <MASKED>'
|
||||
arguments = re.sub(pattern, string, arguments)
|
||||
|
||||
self.results['args'] = arguments
|
||||
|
||||
def generate_runid(self):
|
||||
key = self.results['args'] + self.results['date'] + self.results['version']
|
||||
self.results['run_id'] = hashlib.md5(key).hexdigest()[:7]
|
||||
|
||||
def save(self, cfg):
|
||||
'''Save results in json format file.'''
|
||||
print('Saving results in json file: ' + cfg.json_file + "...")
|
||||
with open(cfg.json_file, 'w') as jfp:
|
||||
json.dump(self.results, jfp, indent=4, sort_keys=True)
|
||||
|
||||
def save_to_db(self, cfg):
|
||||
'''Save results to MongoDB database.'''
|
||||
print "Saving results to MongoDB database..."
|
||||
post_id = pns_mongo.\
|
||||
pns_add_test_result_to_mongod(cfg.vmtp_mongod_ip,
|
||||
cfg.vmtp_mongod_port,
|
||||
cfg.vmtp_db,
|
||||
cfg.vmtp_collection,
|
||||
self.results)
|
||||
if post_id is None:
|
||||
print "ERROR: Failed to add result to DB"
|
||||
|
||||
class VmtpException(Exception):
|
||||
pass
|
||||
|
||||
class VmtpTest(object):
|
||||
def __init__(self):
|
||||
'''
|
||||
1. Authenticate nova and neutron with keystone
|
||||
2. Create new client objects for neutron and nova
|
||||
3. Find external network
|
||||
4. Find or create router for external network
|
||||
5. Find or create internal mgmt and data networks
|
||||
6. Add internal mgmt network to router
|
||||
7. Import public key for ssh
|
||||
8. Create 2 VM instances on internal networks
|
||||
9. Create floating ips for VMs
|
||||
10. Associate floating ip with VMs
|
||||
'''
|
||||
self.server = None
|
||||
self.client = None
|
||||
self.net = None
|
||||
self.comp = None
|
||||
self.ping_status = None
|
||||
self.client_az_list = None
|
||||
self.sec_group = None
|
||||
self.image_instance = None
|
||||
self.flavor_type = None
|
||||
|
||||
# Create an instance on a particular availability zone
|
||||
def create_instance(self, inst, az, int_net):
|
||||
self.assert_true(inst.create(self.image_instance,
|
||||
self.flavor_type,
|
||||
instance_access,
|
||||
int_net,
|
||||
az,
|
||||
int_net['name'],
|
||||
self.sec_group))
|
||||
|
||||
def assert_true(self, cond):
|
||||
if not cond:
|
||||
raise VmtpException('Assert failure')
|
||||
|
||||
def setup(self):
|
||||
# If we need to reuse existing vms just return without setup
|
||||
if not config.reuse_existing_vm:
|
||||
creds = cred.get_credentials()
|
||||
creds_nova = cred.get_nova_credentials_v2()
|
||||
# Create the nova and neutron instances
|
||||
nova_client = Client(**creds_nova)
|
||||
neutron = neutronclient.Client(**creds)
|
||||
|
||||
self.comp = compute.Compute(nova_client, config)
|
||||
|
||||
# Add the appropriate public key to openstack
|
||||
self.comp.init_key_pair(config.public_key_name, instance_access)
|
||||
|
||||
self.image_instance = self.comp.find_image(config.image_name)
|
||||
if self.image_instance is None:
|
||||
if config.vm_image_url is not None:
|
||||
print '%s: image for VM not found, uploading it ...' \
|
||||
% (config.image_name)
|
||||
keystone = keystoneclient.Client(**creds)
|
||||
glance_endpoint = keystone.service_catalog.url_for(
|
||||
service_type='image', endpoint_type='publicURL')
|
||||
glance_client = glanceclient.Client(
|
||||
glance_endpoint, token=keystone.auth_token)
|
||||
self.comp.upload_image_via_url(
|
||||
glance_client, config.image_name, config.vm_image_url)
|
||||
self.image_instance = self.comp.find_image(config.image_name)
|
||||
else:
|
||||
# Exit the pogram
|
||||
print '%s: image to launch VM not found. ABORTING.' \
|
||||
% (config.image_name)
|
||||
sys.exit(1)
|
||||
|
||||
self.assert_true(self.image_instance)
|
||||
print 'Found image %s to launch VM, will continue' % (config.image_name)
|
||||
self.flavor_type = self.comp.find_flavor(config.flavor_type)
|
||||
self.net = network.Network(neutron, config)
|
||||
|
||||
rescol.add_property('l2agent_type', self.net.l2agent_type)
|
||||
print "OpenStack agent: " + self.net.l2agent_type
|
||||
try:
|
||||
network_type = self.net.vm_int_net[0]['provider:network_type']
|
||||
print "OpenStack network type: " + network_type
|
||||
rescol.add_property('encapsulation', network_type)
|
||||
except KeyError as exp:
|
||||
network_type = 'Unknown'
|
||||
print "Provider network type not found: ", str(exp)
|
||||
|
||||
# Create a new security group for the test
|
||||
self.sec_group = self.comp.security_group_create()
|
||||
if not self.sec_group:
|
||||
raise VmtpException("Security group creation failed")
|
||||
if config.reuse_existing_vm:
|
||||
self.server.internal_ip = config.vm_server_internal_ip
|
||||
self.client.internal_ip = config.vm_client_internal_ip
|
||||
if config.vm_server_external_ip:
|
||||
self.server.ssh_access.host = config.vm_server_external_ip
|
||||
else:
|
||||
self.server.ssh_access.host = config.vm_server_internal_ip
|
||||
if config.vm_client_external_ip:
|
||||
self.client.ssh_access.host = config.vm_client_external_ip
|
||||
else:
|
||||
self.client.ssh_access.host = config.vm_client_internal_ip
|
||||
return
|
||||
|
||||
# this is the standard way of running the test
|
||||
# NICs to be used for the VM
|
||||
if config.reuse_network_name:
|
||||
# VM needs to connect to existing management and new data network
|
||||
# Reset the management network name
|
||||
config.internal_network_name[0] = config.reuse_network_name
|
||||
else:
|
||||
# Make sure we have an external network and an external router
|
||||
self.assert_true(self.net.ext_net)
|
||||
self.assert_true(self.net.ext_router)
|
||||
self.assert_true(self.net.vm_int_net)
|
||||
|
||||
# Get hosts for the availability zone to use
|
||||
# avail_list = self.comp.list_hypervisor(config.availability_zone)
|
||||
avail_list = self.comp.get_az_host_list()
|
||||
if not avail_list:
|
||||
sys.exit(5)
|
||||
|
||||
# compute the list of client vm placements to run
|
||||
# the first host is always where the server runs
|
||||
server_az = avail_list[0]
|
||||
if len(avail_list) > 1:
|
||||
# 2 hosts are known
|
||||
if config.inter_node_only:
|
||||
# in this case we do not want the client to run on the same host
|
||||
# as the server
|
||||
avail_list.pop(0)
|
||||
self.client_az_list = avail_list
|
||||
|
||||
self.server = PerfInstance(config.vm_name_server,
|
||||
config,
|
||||
self.comp,
|
||||
self.net,
|
||||
server=True)
|
||||
self.server.display('Creating server VM...')
|
||||
self.create_instance(self.server, server_az,
|
||||
self.net.vm_int_net[0])
|
||||
|
||||
# Test throughput for the case of the external host
|
||||
def ext_host_tp_test(self):
|
||||
client = PerfInstance('Host-' + config.ext_host.host + '-Client', config)
|
||||
if not client.setup_ssh(config.ext_host):
|
||||
client.display('SSH to ext host failed, check IP or make sure public key is configured')
|
||||
else:
|
||||
client.buginf('SSH connected')
|
||||
client.create()
|
||||
fpr.print_desc('External-VM (upload/download)')
|
||||
res = client.run_client('External-VM',
|
||||
self.server.ssh_access.host,
|
||||
self.server,
|
||||
bandwidth=config.vm_bandwidth,
|
||||
bidirectional=True)
|
||||
if res:
|
||||
rescol.add_flow_result(res)
|
||||
client.dispose()
|
||||
|
||||
def add_location(self, label):
|
||||
'''Add a note to a label to specify same node or differemt node.'''
|
||||
# We can only tell if there is a host part in the az
|
||||
# e.g. 'nova:GG34-7'
|
||||
if ':' in self.client.az:
|
||||
if self.client.az == self.server.az:
|
||||
return label + ' (intra-node)'
|
||||
else:
|
||||
return label + ' (inter-node)'
|
||||
return label
|
||||
|
||||
def create_flow_client(self, client_az, int_net):
|
||||
self.client = PerfInstance(config.vm_name_client, config,
|
||||
self.comp,
|
||||
self.net)
|
||||
self.create_instance(self.client, client_az, int_net)
|
||||
|
||||
def measure_flow(self, label, target_ip):
|
||||
label = self.add_location(label)
|
||||
fpr.print_desc(label)
|
||||
|
||||
# results for this flow as a dict
|
||||
perf_output = self.client.run_client(label, target_ip,
|
||||
self.server,
|
||||
bandwidth=config.vm_bandwidth,
|
||||
az_to=self.server.az)
|
||||
if opts.stop_on_error:
|
||||
# check if there is any error in the results
|
||||
results_list = perf_output['results']
|
||||
for res_dict in results_list:
|
||||
if 'error' in res_dict:
|
||||
print('Stopping execution on error, cleanup all VMs/networks manually')
|
||||
rescol.pprint(perf_output)
|
||||
sys.exit(2)
|
||||
|
||||
rescol.add_flow_result(perf_output)
|
||||
|
||||
def measure_vm_flows(self):
|
||||
# scenarios need to be tested for both inter and intra node
|
||||
# 1. VM to VM on same data network
|
||||
# 2. VM to VM on seperate networks fixed-fixed
|
||||
# 3. VM to VM on seperate networks floating-floating
|
||||
|
||||
# we should have 1 or 2 AZ to use (intra and inter-node)
|
||||
for client_az in self.client_az_list:
|
||||
self.create_flow_client(client_az, self.net.vm_int_net[0])
|
||||
self.measure_flow("VM to VM same network fixed IP",
|
||||
self.server.internal_ip)
|
||||
self.client.dispose()
|
||||
self.client = None
|
||||
if not config.reuse_network_name:
|
||||
# Different network
|
||||
self.create_flow_client(client_az, self.net.vm_int_net[1])
|
||||
|
||||
self.measure_flow("VM to VM different network fixed IP",
|
||||
self.server.internal_ip)
|
||||
if not config.ipv6_mode:
|
||||
self.measure_flow("VM to VM different network floating IP",
|
||||
self.server.ssh_access.host)
|
||||
|
||||
self.client.dispose()
|
||||
self.client = None
|
||||
|
||||
# If external network is specified run that case
|
||||
if config.ext_host:
|
||||
self.ext_host_tp_test()
|
||||
|
||||
def teardown(self):
|
||||
'''
|
||||
Clean up the floating ip and VMs
|
||||
'''
|
||||
print '---- Cleanup ----'
|
||||
if self.server:
|
||||
self.server.dispose()
|
||||
if self.client:
|
||||
self.client.dispose()
|
||||
if not config.reuse_existing_vm and self.net:
|
||||
self.net.dispose()
|
||||
# Remove the public key
|
||||
if self.comp:
|
||||
self.comp.remove_public_key(config.public_key_name)
|
||||
# Finally remove the security group
|
||||
try:
|
||||
if self.comp:
|
||||
self.comp.security_group_delete(self.sec_group)
|
||||
except ClientException:
|
||||
# May throw novaclient.exceptions.BadRequest if in use
|
||||
print('Security group in use: not deleted')
|
||||
|
||||
def run(self):
|
||||
error_flag = False
|
||||
|
||||
try:
|
||||
self.setup()
|
||||
self.measure_vm_flows()
|
||||
except KeyboardInterrupt:
|
||||
traceback.format_exc()
|
||||
except (VmtpException, sshutils.SSHError, ClientException, Exception):
|
||||
print 'print_exc:'
|
||||
traceback.print_exc()
|
||||
error_flag = True
|
||||
|
||||
if opts.stop_on_error and error_flag:
|
||||
print('Stopping execution on error, cleanup all VMs/networks manually')
|
||||
sys.exit(2)
|
||||
else:
|
||||
self.teardown()
|
||||
|
||||
def test_native_tp(nhosts, ifname):
|
||||
fpr.print_desc('Native Host to Host throughput')
|
||||
server_host = nhosts[0]
|
||||
server = PerfInstance('Host-' + server_host.host + '-Server', config, server=True)
|
||||
|
||||
if not server.setup_ssh(server_host):
|
||||
server.display('SSH failed, check IP or make sure public key is configured')
|
||||
else:
|
||||
server.display('SSH connected')
|
||||
server.create()
|
||||
# if inter-node-only requested we avoid running the client on the
|
||||
# same node as the server - but only if there is at least another
|
||||
# IP provided
|
||||
if config.inter_node_only and len(nhosts) > 1:
|
||||
# remove the first element of the list
|
||||
nhosts.pop(0)
|
||||
# IP address clients should connect to, check if the user
|
||||
# has passed a server listen interface name
|
||||
if ifname:
|
||||
# use the IP address configured on given interface
|
||||
server_ip = server.get_interface_ip(ifname)
|
||||
if not server_ip:
|
||||
print('Error: cannot get IP address for interface ' + ifname)
|
||||
else:
|
||||
server.display('Clients will use server IP address %s (%s)' %
|
||||
(server_ip, ifname))
|
||||
else:
|
||||
# use same as ssh IP
|
||||
server_ip = server_host.host
|
||||
|
||||
if server_ip:
|
||||
# start client side, 1 per host provided
|
||||
for client_host in nhosts:
|
||||
client = PerfInstance('Host-' + client_host.host + '-Client', config)
|
||||
if not client.setup_ssh(client_host):
|
||||
client.display('SSH failed, check IP or make sure public key is configured')
|
||||
else:
|
||||
client.buginf('SSH connected')
|
||||
client.create()
|
||||
res = client.run_client('Native host-host',
|
||||
server_ip,
|
||||
server,
|
||||
bandwidth=config.vm_bandwidth)
|
||||
rescol.add_flow_result(res)
|
||||
client.dispose()
|
||||
server.dispose()
|
||||
|
||||
def _get_ssh_access(opt_name, opt_value):
|
||||
'''Allocate a HostSshAccess instance to the option value
|
||||
Check that a password is provided or the key pair in the config file
|
||||
is valid.
|
||||
If invalid exit with proper error message
|
||||
'''
|
||||
if not opt_value:
|
||||
return None
|
||||
|
||||
host_access = sshutils.SSHAccess(opt_value)
|
||||
host_access.private_key_file = config.private_key_file
|
||||
host_access.public_key_file = config.public_key_file
|
||||
if host_access.error:
|
||||
print'Error for --' + (opt_name + ':' + host_access.error)
|
||||
sys.exit(2)
|
||||
return host_access
|
||||
|
||||
def _merge_config(cfg_file, source_config, required=False):
|
||||
'''
|
||||
returns the merged config or exits if the file does not exist and is required
|
||||
'''
|
||||
dest_config = source_config
|
||||
|
||||
fullname = os.path.expanduser(cfg_file)
|
||||
if os.path.isfile(fullname):
|
||||
print('Loading ' + fullname + '...')
|
||||
try:
|
||||
alt_config = configure.Configuration.from_file(fullname).configure()
|
||||
dest_config = source_config.merge(alt_config)
|
||||
|
||||
except configure.ConfigurationError:
|
||||
# this is in most cases when the config file passed is empty
|
||||
# configure.ConfigurationError: unconfigured
|
||||
# in case of syntax error, another exception is thrown:
|
||||
# TypeError: string indices must be integers, not str
|
||||
pass
|
||||
elif required:
|
||||
print('Error: configration file %s does not exist' % (fullname))
|
||||
sys.exit(1)
|
||||
return dest_config
|
||||
|
||||
def get_controller_info(ssh_access, net, res_col):
|
||||
if not ssh_access:
|
||||
return
|
||||
print 'Fetching OpenStack deployment details...'
|
||||
sshcon = sshutils.SSH(ssh_access,
|
||||
connect_retry_count=config.ssh_retry_count)
|
||||
if sshcon is None:
|
||||
print 'ERROR: Cannot connect to the controller node'
|
||||
return
|
||||
res = {}
|
||||
res['distro'] = sshcon.get_host_os_version()
|
||||
res['openstack_version'] = sshcon.check_openstack_version()
|
||||
res['cpu_info'] = sshcon.get_cpu_info()
|
||||
if net:
|
||||
l2type = res_col.get_result('l2agent_type')
|
||||
encap = res_col.get_result('encapsulation')
|
||||
if l2type:
|
||||
if encap:
|
||||
res['nic_name'] = sshcon.get_nic_name(l2type, encap,
|
||||
net.internal_iface_dict)
|
||||
res['l2agent_version'] = sshcon.get_l2agent_version(l2type)
|
||||
# print results
|
||||
res_col.pprint(res)
|
||||
res_col.add_properties(res)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
fpr = FlowPrinter()
|
||||
rescol = ResultsCollector()
|
||||
|
||||
parser = argparse.ArgumentParser(description='OpenStack VM Throughput V' + __version__)
|
||||
|
||||
parser.add_argument('-c', '--config', dest='config',
|
||||
action='store',
|
||||
help='override default values with a config file',
|
||||
metavar='<config_file>')
|
||||
|
||||
parser.add_argument('-r', '--rc', dest='rc',
|
||||
action='store',
|
||||
help='source OpenStack credentials from rc file',
|
||||
metavar='<openrc_file>')
|
||||
|
||||
parser.add_argument('-m', '--monitor', dest='monitor',
|
||||
action='store',
|
||||
help='Enable CPU monitoring (requires Ganglia)',
|
||||
metavar='<gmond_ip>[:<port>]')
|
||||
|
||||
parser.add_argument('-p', '--password', dest='pwd',
|
||||
action='store',
|
||||
help='OpenStack password',
|
||||
metavar='<password>')
|
||||
|
||||
parser.add_argument('-t', '--time', dest='time',
|
||||
action='store',
|
||||
help='throughput test duration in seconds (default 10 sec)',
|
||||
metavar='<time>')
|
||||
|
||||
parser.add_argument('--host', dest='hosts',
|
||||
action='append',
|
||||
help='native host throughput (password or public key required)',
|
||||
metavar='<user>@<host_ssh_ip>[:<password>:<server-listen-if-name>]')
|
||||
|
||||
parser.add_argument('--external-host', dest='ext_host',
|
||||
action='store',
|
||||
help='external-VM throughput (password or public key required)',
|
||||
metavar='<user>@<host_ssh_ip>[:password>]')
|
||||
|
||||
parser.add_argument('--controller-node', dest='controller_node',
|
||||
action='store',
|
||||
help='controller node ssh (password or public key required)',
|
||||
metavar='<user>@<host_ssh_ip>[:<password>]')
|
||||
|
||||
parser.add_argument('--mongod-server', dest='mongod_server',
|
||||
action='store',
|
||||
help='provide mongoDB server IP to store results',
|
||||
metavar='<server ip>')
|
||||
|
||||
parser.add_argument('--json', dest='json',
|
||||
action='store',
|
||||
help='store results in json format file',
|
||||
metavar='<file>')
|
||||
|
||||
parser.add_argument('--tp-tool', dest='tp_tool',
|
||||
action='store',
|
||||
default='nuttcp',
|
||||
help='transport perf tool to use (default=nuttcp)',
|
||||
metavar='<nuttcp|iperf>')
|
||||
|
||||
# note there is a bug in argparse that causes an AssertionError
|
||||
# when the metavar is set to '[<az>:]<hostname>', hence had to insert a space
|
||||
parser.add_argument('--hypervisor', dest='hypervisors',
|
||||
action='append',
|
||||
help='hypervisor to use (1 per arg, up to 2 args)',
|
||||
metavar='[<az>:] <hostname>')
|
||||
|
||||
parser.add_argument('--inter-node-only', dest='inter_node_only',
|
||||
default=False,
|
||||
action='store_true',
|
||||
help='only measure inter-node')
|
||||
|
||||
parser.add_argument('--protocols', dest='protocols',
|
||||
action='store',
|
||||
default='TUI',
|
||||
help='protocols T(TCP), U(UDP), I(ICMP) - default=TUI (all)',
|
||||
metavar='<T|U|I>')
|
||||
|
||||
parser.add_argument('--bandwidth', dest='vm_bandwidth',
|
||||
action='store',
|
||||
default=0,
|
||||
help='the bandwidth limit for TCP/UDP flows in K/M/Gbps, '
|
||||
'e.g. 128K/32M/5G. (default=no limit) ',
|
||||
metavar='<bandwidth>')
|
||||
|
||||
parser.add_argument('--tcpbuf', dest='tcp_pkt_sizes',
|
||||
action='store',
|
||||
default=0,
|
||||
help='list of buffer length when transmitting over TCP in Bytes, '
|
||||
'e.g. --tcpbuf 8192,65536. (default=65536)',
|
||||
metavar='<tcp_pkt_size1,...>')
|
||||
|
||||
parser.add_argument('--udpbuf', dest='udp_pkt_sizes',
|
||||
action='store',
|
||||
default=0,
|
||||
help='list of buffer length when transmitting over UDP in Bytes, '
|
||||
'e.g. --udpbuf 128,2048. (default=128,1024,8192)',
|
||||
metavar='<udp_pkt_size1,...>')
|
||||
|
||||
parser.add_argument('--no-env', dest='no_env',
|
||||
default=False,
|
||||
action='store_true',
|
||||
help='do not read env variables')
|
||||
|
||||
parser.add_argument('--vnic-type', dest='vnic_type',
|
||||
default=None,
|
||||
action='store',
|
||||
help='binding vnic type for test VMs',
|
||||
metavar='<direct|macvtap|normal>')
|
||||
|
||||
parser.add_argument('-d', '--debug', dest='debug',
|
||||
default=False,
|
||||
action='store_true',
|
||||
help='debug flag (very verbose)')
|
||||
|
||||
parser.add_argument('-v', '--version', dest='version',
|
||||
default=False,
|
||||
action='store_true',
|
||||
help='print version of this script and exit')
|
||||
|
||||
parser.add_argument('--stop-on-error', dest='stop_on_error',
|
||||
default=False,
|
||||
action='store_true',
|
||||
help='Stop and keep everything as-is on error (must cleanup manually)')
|
||||
|
||||
parser.add_argument('--vm-image-url', dest='vm_image_url',
|
||||
action='store',
|
||||
help='URL to a Linux image in qcow2 format that can be downloaded from',
|
||||
metavar='<url_to_image>')
|
||||
|
||||
parser.add_argument('--test-description', dest='test_description',
|
||||
action='store',
|
||||
help='The test description to be stored in JSON or MongoDB',
|
||||
metavar='<test_description>')
|
||||
|
||||
|
||||
(opts, args) = parser.parse_known_args()
|
||||
|
||||
default_cfg_file = get_vmtp_absolute_path_for_file("cfg.default.yaml")
|
||||
|
||||
# read the default configuration file and possibly an override config file
|
||||
# the precedence order is as follows:
|
||||
# $HOME/.vmtp.yaml if exists
|
||||
# -c <file> from command line if provided
|
||||
# cfg.default.yaml
|
||||
config = configure.Configuration.from_file(default_cfg_file).configure()
|
||||
config = _merge_config('~/.vmtp.yaml', config)
|
||||
|
||||
if opts.config:
|
||||
config = _merge_config(opts.config, config, required=True)
|
||||
|
||||
if opts.version:
|
||||
print('Version ' + __version__)
|
||||
sys.exit(0)
|
||||
|
||||
# debug flag
|
||||
config.debug = opts.debug
|
||||
config.inter_node_only = opts.inter_node_only
|
||||
|
||||
if config.public_key_file and not os.path.isfile(config.public_key_file):
|
||||
print('Warning: invalid public_key_file:' + config.public_key_file)
|
||||
config.public_key_file = None
|
||||
if config.private_key_file and not os.path.isfile(config.private_key_file):
|
||||
print('Warning: invalid private_key_file:' + config.private_key_file)
|
||||
config.private_key_file = None
|
||||
|
||||
# direct: use SR-IOV ports for all the test VMs
|
||||
if opts.vnic_type not in [None, 'direct', 'macvtap', 'normal']:
|
||||
print('Invalid vnic-type: ' + opts.vnic_type)
|
||||
sys.exit(1)
|
||||
config.vnic_type = opts.vnic_type
|
||||
|
||||
config.hypervisors = opts.hypervisors
|
||||
|
||||
# time to run each perf test in seconds
|
||||
if opts.time:
|
||||
config.time = int(opts.time)
|
||||
else:
|
||||
config.time = 10
|
||||
|
||||
if opts.json:
|
||||
config.json_file = opts.json
|
||||
else:
|
||||
config.json_file = None
|
||||
|
||||
# Initialize the external host access
|
||||
config.ext_host = _get_ssh_access('external-host', opts.ext_host)
|
||||
|
||||
# This is a template host access that will be used for all instances
|
||||
# (the only specific field specific to each instance is the host IP)
|
||||
# For test VM access, we never use password and always need a key pair
|
||||
instance_access = sshutils.SSHAccess()
|
||||
instance_access.username = config.ssh_vm_username
|
||||
# if the configuration does not have a
|
||||
# key pair specified, we check if the user has a personal key pair
|
||||
# if no key pair is configured or usable, a temporary key pair will be created
|
||||
if config.public_key_file and config.private_key_file:
|
||||
instance_access.public_key_file = config.public_key_file
|
||||
instance_access.private_key_file = config.private_key_file
|
||||
else:
|
||||
pub_key = os.path.expanduser('~/.ssh/id_rsa.pub')
|
||||
priv_key = os.path.expanduser('~/.ssh/id_rsa')
|
||||
if os.path.isfile(pub_key) and os.path.isfile(priv_key):
|
||||
instance_access.public_key_file = pub_key
|
||||
instance_access.private_key_file = priv_key
|
||||
|
||||
if opts.debug and instance_access.public_key_file:
|
||||
print('VM public key: ' + instance_access.public_key_file)
|
||||
print('VM private key: ' + instance_access.private_key_file)
|
||||
|
||||
|
||||
###################################################
|
||||
# VM Image URL
|
||||
###################################################
|
||||
if opts.vm_image_url:
|
||||
config.vm_image_url = opts.vm_image_url
|
||||
|
||||
###################################################
|
||||
# Test Description
|
||||
###################################################
|
||||
if opts.test_description:
|
||||
rescol.add_property('test_description', opts.test_description)
|
||||
|
||||
###################################################
|
||||
# MongoDB Server connection info.
|
||||
###################################################
|
||||
if opts.mongod_server:
|
||||
config.vmtp_mongod_ip = opts.mongod_server
|
||||
else:
|
||||
config.vmtp_mongod_ip = None
|
||||
|
||||
if 'vmtp_mongod_port' not in config:
|
||||
# Set MongoDB default port if not set.
|
||||
config.vmtp_mongod_port = 27017
|
||||
|
||||
# the bandwidth limit for VMs
|
||||
if opts.vm_bandwidth:
|
||||
opts.vm_bandwidth = opts.vm_bandwidth.upper().strip()
|
||||
ex_unit = 'KMG'.find(opts.vm_bandwidth[-1])
|
||||
try:
|
||||
if ex_unit == -1:
|
||||
raise ValueError
|
||||
val = int(opts.vm_bandwidth[0:-1])
|
||||
except ValueError:
|
||||
print 'Invalid --bandwidth parameter. A valid input must '\
|
||||
'specify only one unit (K|M|G).'
|
||||
sys.exit(1)
|
||||
config.vm_bandwidth = int(val * (10 ** (ex_unit * 3)))
|
||||
|
||||
# the pkt size for TCP and UDP
|
||||
if opts.tcp_pkt_sizes:
|
||||
try:
|
||||
config.tcp_pkt_sizes = opts.tcp_pkt_sizes.split(',')
|
||||
for i in xrange(len(config.tcp_pkt_sizes)):
|
||||
config.tcp_pkt_sizes[i] = int(config.tcp_pkt_sizes[i])
|
||||
except ValueError:
|
||||
print 'Invalid --tcpbuf parameter. A valid input must be '\
|
||||
'integers seperated by comma.'
|
||||
sys.exit(1)
|
||||
|
||||
if opts.udp_pkt_sizes:
|
||||
try:
|
||||
config.udp_pkt_sizes = opts.udp_pkt_sizes.split(',')
|
||||
for i in xrange(len(config.udp_pkt_sizes)):
|
||||
config.udp_pkt_sizes[i] = int(config.udp_pkt_sizes[i])
|
||||
except ValueError:
|
||||
print 'Invalid --udpbuf parameter. A valid input must be '\
|
||||
'integers seperated by comma.'
|
||||
sys.exit(1)
|
||||
|
||||
#####################################################
|
||||
# Set Ganglia server ip and port if the monitoring (-m)
|
||||
# option is enabled.
|
||||
#####################################################
|
||||
config.gmond_svr_ip = None
|
||||
config.gmond_svr_port = None
|
||||
if opts.monitor:
|
||||
# Add the default gmond port if not present
|
||||
if ':' not in opts.monitor:
|
||||
opts.monitor += ':8649'
|
||||
|
||||
mobj = re.match(r'(\d+\.\d+\.\d+\.\d+):(\d+)', opts.monitor)
|
||||
if mobj:
|
||||
config.gmond_svr_ip = mobj.group(1)
|
||||
config.gmond_svr_port = mobj.group(2)
|
||||
print "Ganglia monitoring enabled (%s:%s)" % \
|
||||
(config.gmond_svr_ip, config.gmond_svr_port)
|
||||
config.time = 30
|
||||
|
||||
else:
|
||||
print 'Invalid --monitor syntax: ' + opts.monitor
|
||||
|
||||
###################################################
|
||||
# Once we parse the config files, normalize
|
||||
# the paths so that all paths are absolute paths.
|
||||
###################################################
|
||||
normalize_paths(config)
|
||||
|
||||
# Check the tp-tool name
|
||||
config.protocols = opts.protocols.upper()
|
||||
if 'T' in config.protocols or 'U' in config.protocols:
|
||||
if opts.tp_tool.lower() == 'nuttcp':
|
||||
config.tp_tool = nuttcp_tool.NuttcpTool
|
||||
elif opts.tp_tool.lower() == 'iperf':
|
||||
config.tp_tool = iperf_tool.IperfTool
|
||||
else:
|
||||
print 'Invalid transport tool: ' + opts.tp_tool
|
||||
sys.exit(1)
|
||||
else:
|
||||
config.tp_tool = None
|
||||
|
||||
# 3 forms
|
||||
# A list of 0 to 2 HostSshAccess elements
|
||||
if opts.hosts:
|
||||
native_hosts = []
|
||||
if_name = None
|
||||
for host in opts.hosts:
|
||||
# decode and extract the trailing if name first
|
||||
# there is an if name if there are at least 2 ':' in the argument
|
||||
# e.g. "root@1.1.1.1:secret:eth0"
|
||||
if host.count(':') >= 2:
|
||||
last_column_index = host.rfind(':')
|
||||
# can be empty
|
||||
last_arg = host[last_column_index + 1:]
|
||||
if not if_name and last_arg:
|
||||
if_name = last_arg
|
||||
host = host[:last_column_index]
|
||||
native_hosts.append(_get_ssh_access('host', host))
|
||||
test_native_tp(native_hosts, if_name)
|
||||
|
||||
cred = credentials.Credentials(opts.rc, opts.pwd, opts.no_env)
|
||||
|
||||
# replace all command line arguments (after the prog name) with
|
||||
# those args that have not been parsed by this parser so that the
|
||||
# unit test parser is not bothered by local arguments
|
||||
sys.argv[1:] = args
|
||||
vmtp_net = None
|
||||
if cred.rc_auth_url:
|
||||
if opts.debug:
|
||||
print 'Using ' + cred.rc_auth_url
|
||||
rescol.add_property('auth_url', cred.rc_auth_url)
|
||||
vmtp = VmtpTest()
|
||||
vmtp.run()
|
||||
vmtp_net = vmtp.net
|
||||
|
||||
# Retrieve controller information if requested
|
||||
# controller node ssh access to collect metadata for the run.
|
||||
ctrl_host_access = _get_ssh_access('controller-node', opts.controller_node)
|
||||
get_controller_info(ctrl_host_access, vmtp_net, rescol)
|
||||
|
||||
# If saving the results to JSON or MongoDB, get additional details:
|
||||
if config.json_file or config.vmtp_mongod_ip:
|
||||
rescol.mask_credentials()
|
||||
rescol.generate_runid()
|
||||
|
||||
if config.json_file:
|
||||
rescol.save(config)
|
||||
|
||||
if config.vmtp_mongod_ip:
|
||||
rescol.save_to_db(config)
|
||||
Loading…
Reference in New Issue
Block a user