x/kloudbuster
Code Issues Proposed changes

doc update with gallery

Browse Source 
Change-Id: I25e13c27fcd4d88f14c04190050c16d654ead6bc
This commit is contained in:
ahothan 2016-04-03 22:24:18 -07:00
parent 879b366ac2
commit 6f1b2502d4
13 changed files with 1055 additions and 203 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

139
README.rst
View File

@ -1,126 +1,23 @@
========
Overview
========
=====================
KloudBuster version 6
=====================
How good is your OpenStack data plane or storage plane under real heavy load?
How good is your OpenStack *data plane* or *storage service* under real heavy load?
KloudBuster is a tool that can load the data or storage plane of any OpenStack
cloud at massive scale and can measure how well the cloud behaves under load.
KloudBuster is a tool that can load the data plane or storage infrastructure of any OpenStack
cloud at massive scale and measure how well the cloud behaves under load
where it matters: from the VMs standpoint, where cloud applications run.
Anybody with basic knowledge of OpenStack, data plane and storage
performance concepts can use the tool and get scale numbers for any OpenStack
cloud straight off the box with pre-defined default workloads.
Accessible to anybody with basic knowledge of OpenStack, installs in minutes and runs
off the box with sensible default workloads in a fully automated way.
CLI/REST or Web User Interface.. you pick what works best for you.
No need for complex installation as the KloudBuster installation takes care of
all the dependencies.
Read the full documentation with feature list, snapshots and diagrams,
scale test design, how-to and installation instructions:
Features
--------
`KloudBuster Documentation <http://kloudbuster.readthedocs.org>`_
* Neutron configuration agnostic (any encapsulation, any overlay, any plugin)
* OpenStack Storage backend agnostic
* User can specify any number of tenants, routers, networks, VM instances (only
limited by cloud capacity) and KloudBuster will stage all these resources in a
way that makes sense for operational usage
* Real VM-level performance and scale numbers (not bare metal)
* Punishing scale (thousands of VMs and enough load to fill even the fastest NIC
cards or load any storage cluster with ease - if your cloud can even support
that much)
* Data plane with HTTP traffic load:
* Can load the data plane with one OpenStack cloud (single-cloud operations
for L3 East-West scale) or 2 OpenStack clouds (dual-cloud operations with
one cloud hosting the HTTP servers and the other loading HTTP traffic for
L3 North-South scale testing)
* Real HTTP servers (Nginx) running in real Linux images (Ubuntu 14.04)
* Can specify any number of HTTP servers per tenant (as many as your cloud
can handle)
* High performance and highly scalable HTTP traffic generators to simulate
huge number of HTTP users and TCP connections (hundreds of thousands to
millions of concurrent and active connections)
* Overall throughput aggegation and loss-less latency aggregation for every
single HTTP request (typically millions per run) using the open source
HdrHistogram library
* Traffic shaping to specify on which links traffic should flow
* Can support periodic reporting and aggregation of results
* Storage load:
* VM-level Cinder volume (block storage) or Ephemeral disk file I/O performance measurement
using FIO running inside VMs (not bare metal)
* Supports random and sequential file access mode
* Supports read, write and read/write mix
* IOPs, bandwitdh and loss-less latency aggregation
* User configurable storage workload profiles
* Supports automated scale progressions (VM count series in any multiple
increment) to reduce dramatically scale testing time
* Highly efficient and scalable metric aggregation
* Automatic cleanup upon termination (can be disabled)
* Manual cleanup script
* KloudBuster Server to drive scale test:
* from any browser (KloudBuster Web UI)
* from any external programs (KloudBuster REST API)
* Aggregated results provide an easy to understand way to assess the scale of
the cloud under test
* KloudBuster VM image pre-built and available from the OpenStack Community App
Catalog (https://apps.openstack.org/)
Limitations and Non-Goals
-------------------------
* Requires Neutron networking (does not support Nova networking)
* Only supports HTTP and storage traffic in this version
Unlike some other scaling test frameworks, KloudBuster does *not* attempt to:
* provide a scale test framework that works across different cloud technologies
(OpenStack + AWS + Google Cloud + ...) - we are only focusing on OpenStack
* provide a scale test framework that is flexible and programmable to do everything -
we just focus on opinionated and well targeted performance and scale areas
with sensible use cases and available in a fully integrated and easy to consume
packaged format
* replace bare metal and domain specific native performance and scale frameworks
(line level traffic generators, ceph specific performance and scale tools...)
Contributions and Feedbacks
---------------------------
If you are interested in OpenStack Performance and Scale, contributions and
feedbacks are welcome!
If you have any feedbacks or would like to make small or large contributions,
simply send an email to openstack-dev@lists.openstack.org with a '[kloudbuster]'
tag in the subject.
Licensing
@ -150,13 +47,3 @@ the FIO binary copy was not modified and can be found directly at
`<https://github.com/axboe/fio>`_ or can be obtained by email request to the
maintainer of KloudBuster.
Links
-----
* Documentation: `<http://kloudbuster.readthedocs.org>`_
* `KloudBuster REST API documentation Preview <https://htmlpreview.github.io/?https://github.com/openstack/kloudbuster/blob/master/doc/source/_static/kloudbuster-swagger.html>`_
* Source: `<http://git.openstack.org/cgit/openstack/kloudbuster>`_
* Supports/Bugs: `<http://launchpad.net/kloudbuster>`_
* Mailing List: kloudbuster-core@lists.launchpad.net

29
doc/source/architecture.rst
View File

@ -1,4 +1,5 @@
========================
.. _arch:
KloudBuster Architecture
========================
@ -20,7 +21,7 @@ that run inside the same cloud. Such traffic can involve:
(that is the packets do not have to go through a router) - often called L2
East-West
- Packet routing if the 2 end points belong to differet Neutron networks
- Packet routing if the 2 end points belong to different Neutron networks
(packets have to go through router) - often called L3 East-West
The KloudBuster data plane scale test exercises L3 East-West traffic by running
@ -32,7 +33,7 @@ corresponding server router as illustrated in the following diagram:
.. image:: images/kb-http-east-west.png
The KloudBuster App typically runs outside the cloud under test on any server
that has a python interpreter (Macbook, Linux workstation...) with the
that has a python interpreter (MacBook, Linux workstation...) with the
requirement to have access to the OpenStack API of the cloud under test.
The KloudBuster app basically reads the requested scale config (which contains
@ -60,7 +61,7 @@ Rack to Rack Data Plane Scale
-----------------------------
By default KloudBuster will rely on the Nova scheduler to place the various
cient and server VMs. As a result these VMs will be load balanced across all
client and server VMs. As a result these VMs will be load balanced across all
servers and causing the data path of the HTTP traffic to be quite random. This
can be good to measure the scale on a random traffic pattern but sometimes it is
more interesting to shape the HTTP traffic can be shaped to follow certain
@ -83,12 +84,12 @@ North South Data Plane Scale Test
---------------------------------
The North South traffic refers to traffic flowing between external sources and
VMs runnin in the cloud. Such traffic follows a very different path than
VMs running in the cloud. Such traffic follows a very different path than
East-West traffic as it is generally always routed and requires the used of IP
address translation (SNAT and DNAT). One exception to this is the use of a
provider network which may avoid routing and NAT completely.
KloudBuster provides a option to test the North-South data plabe traffic by
KloudBuster provides a option to test the North-South data plane traffic by
separating the client VMs and server VMs into 2 different OpenStack clouds.
.. image:: images/kb-http-north-south.png
@ -101,12 +102,20 @@ Storage Scale Test
++++++++++++++++++
The storage scale test is a relatively simpler version of the data plane scale
test as it only involves 1 tenant 1 network and 1 router. Each test VM runs an
instance of the FIO test client (`FIO <https://github.com/axboe/fio>`_ is a
popular open source storage test client).
test as it only involves 1 tenant, 1 network and 1 router. Each test VM runs one
instance of the FIO test client (`FIO <https://github.com/axboe/fio>`_ which is a
widely adopted open source storage test client).
.. image:: images/kb-storage.png
VM staging, storage plumbing (using Cinder or Nova for ephemeral disks) is done
by the KloudBuster app using OpenStack APIs.
Because the Cinder API abstracts the storage back-end, it is agnostic of
the technology used (Ceph, EMC...).
After the test, all resources (volumes, VMs, network, router) are cleaned up in
the proper order.
Progression Runs
++++++++++++++++
@ -118,7 +127,7 @@ Progression runs are a very convenient feature as it allows to produce result
for series in a much shorter time by reusing the same set of staged VMs and
iterating the scale test to produce measurements at different scale level.
For example, to get storage performace measurement for 100 to 1000 VMs in
For example, to get storage performance measurement for 100 to 1000 VMs in
increments of 100, would require staging and unstaging 100+200+300+....+1,000 =
5,500 VM Instances without progression runs while it would only require staging
1,000 instances with VM reuse.

7
doc/source/cleanup.rst
View File

@ -1,4 +1,5 @@
===========================
.. _cleanup:
OpenStack Resources Cleanup
===========================
@ -85,8 +86,8 @@ Example of cleanup log file::
If such file is provided to the cleanup script using the *--file* option, only
the resources described in the file will be deleted.
Discoverty with Resource name filter (--filter <regex>)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Discovery with Resource name filter (--filter <regex>)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If no cleanup log file is provided, resources are discovered from OpenStack and
selected using a regular expression on the resource name (--filter <regex>). You

71
doc/source/gallery.rst Normal file
View File

@ -0,0 +1,71 @@
.. _gallery:
Gallery
=======
This page has links to examples of scale test reports in HTML format generated by KloudBuster.
These reports were generated within minutes after starting the scale test from a
bare bone OpenStack cloud (not running anything).
Click on the thumbnail images to view the result HTML file in your browser (you will need
access to the Internet to view these files as they reference multiple Java script libraries in CDN).
Sample HTTP Scale Report
------------------------
The following report shows an HTTP scale run with results for 1 to 20 HTTP servers
(running in as many server VMs) in increment of 5, where each HTTP server is receiving
HTTP requests from 1 HTTP traffic generator that runs in a separate VM and emulates 1,000
users sending 1 request per second each (for a total of 1000 requests per second per HTTP server).
The topology used for the test is 1 tenant, 1 router, 4 networks and 5 HTTP servers per network.
Each iteration is programmed to run for 30 seconds.
This scale settings can be viewed in the Configuration tab.
The table shows the results for each iteration step, with the requested and measured RPS
(HTTP requests per second) and the corresponding aggregated download throughput (the sum of all
downloads for all clients).
The latency distribution is shows in the chart, where each line corresponds to one load level (or
iteration in the progression). Lines can be individually shown/hidden by clicking on the
corresponding legend item.
For example, the largest scale involves 20,000 simultaneous users sending an aggregate of 18,621
HTTP requests per second and the latency chart tells us that 99.9% of these 18,621 requests are
replied within 34ms, which is actually excellent.
Note that this test is configured to reuse HTTP connections meaning that we do not have the
overhead of creating a new TCP connection for every HTTP request. This also means that this cloud
had 20,000 TCP active connections at all times during the scale test.
.. image:: images/kb-http-thumbnail.png
:target: https://htmlpreview.github.io/?https://github.com/openstack/kloudbuster/blob/master/doc/source/gallery/http.html
Sample Storage Scale Report
---------------------------
This is a report for a storage scale test using the default workload suite with a progression run
from 1 VM to 20 VMs in increment of 5, and 30 second run per iteration.
This results in 6 tabs of results (1 per workload). The mixed read/write tabs further split in
2 sub tabs (1 for read and 1 for write results).
The random read tab shows that each VM could achieve its requested 100 IOPs across the progression.
The lines represent the latency value at given percentile and can be individually shown/hidden by clicking
the corresponding legend item.
As an example, 20 VMs represents a combined 2,000 IOPs measured for a total of 60,000 random read operations.
The latency line tells us that 99.9% of these 60,000 read operations are completed within 1.576 msec.
.. image:: images/kb-storage-thumbnail.png
:target: https://htmlpreview.github.io/?https://github.com/openstack/kloudbuster/blob/master/doc/source/gallery/storage.html
The sequential write results are more challenging as they show that the VMs cannot achieve their requested
write bandwidth (60MB/s) and can only get 49MB/s each when there are 20 of such VMs.
The latency lines also reflect that stress by peaking at 500 ms for 99.99% of all write operations (although
latency is not nearly as critical for sequential access than for random access).

230
doc/source/gallery/http.html Normal file
View File

@ -0,0 +1,230 @@
<!--Copyright 2016 Cisco Systems, Inc. All rights reserved.-->
<!--Licensed under the Apache License, Version 2.0 (the "License"); you may-->
<!--not use this file except in compliance with the License. You may obtain-->
<!--a copy of the License at-->
<!--http://www.apache.org/licenses/LICENSE-2.0-->
<!--Unless required by applicable law or agreed to in writing, software-->
<!--distributed under the License is distributed on an "AS IS" BASIS, WITHOUT-->
<!--WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the-->
<!--License for the specific language governing permissions and limitations-->
<!--under the License.-->
<!DOCTYPE html>
<html lang="en-US" ng-app="app">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>KloudBuster Report</title>
<script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.4.2/angular.min.js"></script>
<script src="https://d3js.org/d3.v3.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/line-chart/2.0.3/LineChart.min.js"></script>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/line-chart/2.0.3/LineChart.min.css">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/ng-table/0.8.3/ng-table.min.css">
<script src="https://cdnjs.cloudflare.com/ajax/libs/ng-table/0.8.3/ng-table.min.js"></script>
<link rel="stylesheet" href="https://bootswatch.com/flatly/bootstrap.min.css">
<script src="https://code.jquery.com/jquery-2.2.0.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js"></script>
<style rel="stylesheet">
.hidden {
display: inline !important;
}
.label {
padding: 0;
font-size: 110%;
font-weight: normal;
line-height: 16;
color: #000000;
text-align: center;
}
</style>
</head>
<body ng-controller="MainCtrl">
<nav class="navbar navbar-default">
<div class="container-fluid">
<a class="navbar-brand" ng-href="#/" style="font-family: Arial">
<span style="color:#DF314D">K</span>loudBuster
</a>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav" ng-init="tab=0">
<li ng-class="{active:tab==0}" ng-click="tab=0">
<a><span class="glyphicon" aria-hidden="true" ></span>Report</a>
</li>
<li ng-class="{active:tab==1}" ng-click="tab=1" ng-show="config == true">
<a><span class="glyphicon" aria-hidden="true" ></span>Configuration</a>
</li>
</ul>
<!--<ul class="nav navbar-nav navbar-right">-->
<!--<li><a href="#"></a></li>-->
<!--</ul>-->
</div>
</div>
</nav>
<div class="container" ng-show="tab == 0">
<h3>KloudBuster HTTP Test Report</h3>
<h6 style="margin-bottom:0"><span style="float:left">Latency(ms)</span></h6>
<div class="my-chart" style="height: 400px;margin-bottom: 10%">
<linechart data="data" options="options"></linechart>
</div>
<table ng-table="tableParams" class="table table-responsive table-condensed table-bordered table-striped">
<tr ng-repeat="row in tableParams.data" style="text-align:center;">
<td title="cols[0].title" ng-if="cols[0].show" style="margin:0 auto;padding:0;">
<button class="btn btn-default btn-xs {{row.seq}}" ng-click=""
style="height: 22px;width: 24px;"></button>
</td>
<td title="cols[1].title" data-sortable="cols[1].field" ng-if="cols[1].show">{{row.connection}}</td>
<td title="cols[2].title" data-sortable="cols[2].field" ng-if="cols[2].show">{{row.server_vms}}</td>
<td title="cols[3].title" data-sortable="cols[3].field" ng-if="cols[3].show">{{row.requests}}</td>
<td title="cols[4].title" data-sortable="cols[4].field" ng-if="cols[4].show">{{row.sock_err}}</td>
<td title="cols[5].title" data-sortable="cols[5].field" ng-if="cols[5].show">{{row.rps}}</td>
<td title="cols[6].title" data-sortable="cols[6].field" ng-if="cols[6].show">{{row.rate_limit}}</td>
<td title="cols[7].title" data-sortable="cols[7].field" ng-if="cols[7].show">{{row.throughput}} Gbps</td>
</tr>
</table>
</div>
<div class="container" ng-show="tab == 1">
<h3>KloudBuster HTTP Test Configuration</h3>
<div class="panel panel-default">
<div class="panel-body" style="word-wrap:break-word">
<textarea style="font-family:Courier New, Monospace;width: 100%;height: 900px" disabled> {{from_outside_config}}</textarea>
</div>
</div>
</div>
<footer style="text-align: center;">
<hr style="margin:2px"/>
<h6 style="color:gray">{{from_outside["time"]}} - KloudBuster {{from_outside["version"]}}</h6>
</footer>
<script type="text/javascript">
angular.module("app", ["n3-line-chart", "ngTable"]).controller("MainCtrl", function ($scope, color, ngTableParams) {
if($scope.config = true) $scope.from_outside_config = JSON.stringify({"cleanup_resources": true, "client": {"availability_zone": null, "cleanup_resources": true, "flavor": {"disk": 0, "ram": 2048, "vcpus": 1}, "http_tool_configs": {"connection_type": "Keep-alive", "connections": 1000, "duration": 30, "rate_limit": 1000, "report_interval": 0, "threads": 1, "timeout": 5}, "image_name": "kloudbuster_v6", "keystone_admin_role": "admin", "networks_per_router": 1, "number_tenants": 1, "polling_interval": 5, "progression": {"enabled": true, "http_stop_limit": [50, 0], "storage_stop_limit": 20, "vm_multiple": 5, "vm_start": 1}, "prompt_before_run": false, "public_key_file": "/Users/xiyu3/.ssh/id_rsa.pub", "routers_per_tenant": 1, "secgroups_per_network": 1, "storage_stage_configs": {"disk_size": 10, "io_file_size": 1, "target": "volume", "vm_count": 20}, "storage_tool_configs": [{"block_size": "4k", "description": "Random Read", "iodepth": 4, "mode": "randread", "rate_iops": 100, "runtime": 30}, {"block_size": "4k", "description": "Random Write", "iodepth": 4, "mode": "randwrite", "rate_iops": 100, "runtime": 30}, {"block_size": "4k", "description": "Random Read/Write", "iodepth": 4, "mode": "randrw", "rate_iops": 100, "runtime": 30, "rwmixread": 70}, {"block_size": "64k", "description": "Sequential Read", "iodepth": 64, "mode": "read", "rate": "60M", "runtime": 30}, {"block_size": "64k", "description": "Sequential Write", "iodepth": 64, "mode": "write", "rate": "60M", "runtime": 30}, {"block_size": "64k", "description": "Sequential Read/Write", "iodepth": 64, "mode": "rw", "rate": "60M", "runtime": 30, "rwmixread": 70}], "use_floatingip": false, "vm_creation_concurrency": 5, "vms_per_network": 21}, "image_name": "kloudbuster_v6", "keystone_admin_role": "admin", "public_key_file": "/Users/xiyu3/.ssh/id_rsa.pub", "server": {"availability_zone": null, "cleanup_resources": true, "flavor": {"disk": 0, "ram": 2048, "vcpus": 1}, "http_server_configs": {"html_size": 32768}, "image_name": "kloudbuster_v6", "keystone_admin_role": "admin", "networks_per_router": 4, "number_tenants": 1, "public_key_file": "/Users/xiyu3/.ssh/id_rsa.pub", "routers_per_tenant": 1, "secgroups_per_network": 1, "use_floatingip": false, "vm_creation_concurrency": 5, "vms_per_network": 5}, "vm_creation_concurrency": 5}, null, "\t");
$scope.from_outside = {"kb_result": [{"description": "-- Stage 1: 1 VM(s), 1000 Connections, 1000 Expected RPS --", "http_rate_limit": 1000, "http_rps": 933, "http_sock_err": 0, "http_sock_timeout": 0, "http_throughput_kbytes": 30095, "http_total_req": 28001, "latency_stats": [[50, 1607], [75, 1919], [90, 2239], [99, 2815], [99.9, 3055], [99.99, 3279], [99.999, 3375]], "tool": "wrk2", "total_client_vms": 1, "total_connections": 1000, "total_server_vms": 1}, {"description": "-- Stage 2: 5 VM(s), 5000 Connections, 5000 Expected RPS --", "http_rate_limit": 5000, "http_rps": 4645, "http_sock_err": 0, "http_sock_timeout": 0, "http_throughput_kbytes": 149790, "http_total_req": 139408, "latency_stats": [[50, 1567], [75, 2191], [90, 2535], [99, 3735], [99.9, 5047], [99.99, 7591], [99.999, 8711]], "tool": "wrk2", "total_client_vms": 5, "total_connections": 5000, "total_server_vms": 5}, {"description": "-- Stage 3: 10 VM(s), 10000 Connections, 10000 Expected RPS --", "http_rate_limit": 10000, "http_rps": 9309, "http_sock_err": 0, "http_sock_timeout": 0, "http_throughput_kbytes": 300206, "http_total_req": 279355, "latency_stats": [[50, 1535], [75, 2023], [90, 2735], [99, 8991], [99.9, 17143], [99.99, 18967], [99.999, 20095]], "tool": "wrk2", "total_client_vms": 10, "total_connections": 10000, "total_server_vms": 10}, {"description": "-- Stage 4: 15 VM(s), 15000 Connections, 15000 Expected RPS --", "http_rate_limit": 15000, "http_rps": 13961, "http_sock_err": 0, "http_sock_timeout": 0, "http_throughput_kbytes": 450191, "http_total_req": 418951, "latency_stats": [[50, 1647], [75, 2159], [90, 4767], [99, 19359], [99.9, 27567], [99.99, 40551], [99.999, 47423]], "tool": "wrk2", "total_client_vms": 15, "total_connections": 15000, "total_server_vms": 15}, {"description": "-- Stage 5: 20 VM(s), 20000 Connections, 20000 Expected RPS --", "http_rate_limit": 20000, "http_rps": 18621, "http_sock_err": 0, "http_sock_timeout": 0, "http_throughput_kbytes": 600586, "http_total_req": 558895, "latency_stats": [[50, 1807], [75, 2335], [90, 3807], [99, 10703], [99.9, 34343], [99.99, 45439], [99.999, 55063]], "tool": "wrk2", "total_client_vms": 20, "total_connections": 20000, "total_server_vms": 20}], "test_mode": "http", "time": "2016-04-01 15:29:34", "version": "0.6.3"};
$scope.result = $scope.from_outside["kb_result"];
var countRep = $scope.result.length;
$scope.data = {dataset0: [{x: 0}, {x: 10}, {x: 20}, {x: 30}, {x: 40}, {x: 50}, {x: 60}]};
$scope.options = {
series: [], axes: {
x: {
key: "x", type: "linear", tickFormat: function (value) {
if (value === 0) {
return "50%"
} else if (value === 10) {
return "75%"
} else if (value === 20) {
return "90%"
} else if (value === 30) {
return "99%"
} else if (value === 40) {
return "99.9%"
} else if (value === 50) {
return "99.99%"
} else if (value === 60) {
return "99.999%"
}
}
}, y: {
type: "log", ticksFormat: "d", ticks: 10, tickFormat: function (value, index) {
return value
}
}
}, margin: {top: 20, right: 30, bottom: 20, left: 30}, grid: {x: false, y: true}
};
$scope.tabledata = [];
$scope.cols = [{field: "seq", title: "SEQ", sortable: "seq", show: true}, {
field: "connection",
title: "Connection",
sortable: "connection",
show: true
}, {field: "server_vms", title: "Server VMs", sortable: "server_vms", show: true}, {
field: "requests",
title: "Requests",
sortable: "requests",
show: true
}, {field: "sock_err", title: "Error", sortable: "sock_err", show: true}, {
field: "rps",
title: "RPS measured",
sortable: "rps",
show: true
}, {field: "rate_limit", title: "RPS requested", sortable: "rate_limit", show: true}, {
field: "throughput",
title: "Throughput",
sortable: "throughput",
show: true
},];
$scope.tableParams = new ngTableParams({sorting: {name: "asc"}, "count": 10}, {
counts: [],
data: $scope.tabledata
});
$scope.pushTableData = function (taName, taData, pickColor) {
var temThrou = (taData.http_throughput_kbytes * 8) / (1000 * 1000);
$scope.tabledata.push({
"seq": taName,
"connection": taData.total_connections,
"server_vms": taData.total_server_vms,
"requests": taData.http_total_req,
"sock_err": taData.http_sock_err + taData.http_sock_timeout,
"rps": taData.http_rps,
"rate_limit": taData.http_rate_limit,
"throughput": temThrou.toFixed(2),
"description": taData.description,
"color": pickColor
});
$("<style>button." + taName + " {background-color: " + pickColor + ";</style>").appendTo("head");
$scope.tableParams.reload()
};
for (var i = 0; i < countRep; i++) {
$scope.perrow = $scope.result[i];
var pickColor = color.getColor();
if (1) {
chName = "Connection-" + $scope.perrow.total_connections;
$scope.options.series.push({
label: chName,
color: pickColor,
dotSize: "3",
thickness: "2px",
axis: "y",
dataset: "dataset0",
key: chName,
type: ["line", "dot"],
id: chName,
interpolation: {mode: "cardinal", tension: 0.8}
});
for (var j = 0; j < 7; j++) {
$scope.data.dataset0[j][chName] = $scope.perrow.latency_stats[j][1] / 1000
}
$scope.pushTableData("Connection-" + $scope.perrow.total_connections, $scope.perrow, pickColor)
}
}
}).service("color", function () {
var self = this;
var num = -1;
var colorList = ["#F44336", "#673AB7", "#03A9F4", "#4CAF50", "#FFEB3B", "#BF360C", "#795548", "#E91E63", "#3F51B5", "#00BCD4", "#CDDC39", "#FF9800", "#9E9E9E", "#9C27B0", "#009688"];
var length = colorList.length;
this.getColor = function () {
num = (num + 1) % length;
return colorList[num]
};
this.reset = function () {
num = -1
}
});
</script>
</body>
</html>

450
doc/source/gallery/storage.html Normal file
View File

File diff suppressed because one or more lines are too long

BIN
doc/source/images/kb-http-thumbnail.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 66 KiB

BIN
doc/source/images/kb-storage-thumbnail.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 71 KiB

1
doc/source/index.rst
View File

@ -13,6 +13,7 @@ Contents:
readme
architecture
gallery
installation
usage
cleanup

70
doc/source/installation.rst
View File

@ -3,31 +3,41 @@ Installation
============
KloudBuster is already pre-installed as a web service in the KloudBuster VM image
available from the `OpenStack Community App Catalog <https://apps.openstack.org>`_
available from the `OpenStack Community App Catalog <https://apps.openstack.org>`_.
So if you just need to use the KloudBuster Web user interface you can instantiate
that VM image and point your browser to its public address as described in :ref:`run_server`.
There are two alternative ways to install and run KloudBuster tool.
Users of KloudBuster who prefers to use the CLI should use regular PyPI based installation.
Developers of KloudBuster should use the GitHub/OpenStack Repository based installation.
Users of KloudBuster who prefers to use the CLI or who prefer to run KloudBuster
locally on their workstation or laptop should use the PyPI based installation
(pip install).
Developers of KloudBuster should use the GitHub/OpenStack Repository based installation
(git clone).
Web Service and PyPI based installation will satisfy most use cases
and are the 2 recommended ways for running KloudBuster under production environments, or through an
automated or scheduled job.
The git repository based installation is targeted at developers of KloudBuster.
and are the 2 recommended ways for running KloudBuster under production environments,
or through an automated or scheduled job.
.. note:: Installation from PyPI will only have the latest stable version.
PyPI based Installation
-----------------------
This is the recommended way to install KloudBuster for non-development use if CLI is required.
KloudBuster is available in the Python Package Index (PyPI):
`KloudBuster PyPI <https://pypi.python.org/pypi/KloudBuster>`_
This is the recommended way to install KloudBuster for non-development use if CLI is preferred
or if you prefer to run KloudBuster locally.
Step 1
^^^^^^
KloudBuster is available in the Python Package Index (PyPI)
`KloudBuster PyPI <https://pypi.python.org/pypi/KloudBuster>`_
and can be installed on any system that has python 2.7.
Step 1: Install pip and the python virtualenv (if not installed already)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
You will need to have python 2.7, pip, and some dependencies installed
before installing KloudBuster, run the command based on your distro.
before installing KloudBuster depending on the operating system at the installation site.
These pre-requisites can be skipped if the corresponding dependencies are already installed.
Ubuntu/Debian based:
@ -50,8 +60,8 @@ MacOSX:
$ sudo easy_install pip
$ sudo pip install virtualenv
Step 2
^^^^^^
Step 2: Install KloudBuster in a virtual environment
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Create a virtual environment for Python, and install KloudBuster:
@ -92,8 +102,8 @@ It is recommended to run KloudBuster inside a virtual environment. However,
it can be skipped if installed in a dedicated VM.
Super quick installation on Ubuntu/Debian
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Quick installation on Ubuntu/Debian
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: bash
@ -105,8 +115,8 @@ Super quick installation on Ubuntu/Debian
$ cd kloudbuster
$ pip install -r requirements-dev.txt
Super quick installation on RHEL/Fedora/CentOS
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Quick installation on RHEL/Fedora/CentOS
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: bash
@ -118,11 +128,10 @@ Super quick installation on RHEL/Fedora/CentOS
$ cd kloudbuster
$ pip install -r requirements-dev.txt
Super quick installation on MacOSX
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Quick installation on MacOSX
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KloudBuster can run natively on MacOSX. These instructions have been verified
to work on MacOSX 10.10 (Yosemite).
KloudBuster can run natively on MacOSX.
First, download XCode from App Store, then execute below commands:
@ -157,9 +166,10 @@ Upload KloudBuster Image
.. note::
If your OpenStack Glance is able to access the Internet, and you don't
need to access the KloudBuster Web UI from the pre-built image, you can skip
this section and you are done with the installation.
If your OpenStack Glance is able to access the Internet and you only use
the CLI to launch KloudBuster, you can skip this section (KloudBuster CLI
will request Glance to download the image from the OpenStack App Catalog when
it is not present in Glance).
In the cloud under test, KloudBuster needs one "universal" test VM image
(referred to as "KloudBuster image") that contains the necessary test software.
@ -169,10 +179,7 @@ application using the appropriate role (HTTP server, HTTP traffic generator,
etc.).
Pre-built images are available for download from the
`OpenStack App Catalog <http://apps.openstack.org>`_ (preferred method). For
whatever reason the pre-built version doesn't work for you, the image can be
re-built from MacOSX using Vagrant or from any Linux server. See
:ref:`here <build_vm_image>` for more details.
`OpenStack App Catalog <http://apps.openstack.org>`_ (preferred method).
.. note::
@ -186,7 +193,8 @@ Manual upload of the KloudBuster VM image
In order to upload the KloudBuster Image to the cloud under test, the image
must be downloaded from the OpenStack App Catalog either directly from
the OpenStack App Catalog (if you have direct access to the Internet)
or through an intermediate location such as a jump host (a jump host has access
or through an intermediate location such as a laptop that has Internet access
or a jump host (a jump host has access
to both Internet and the cloud under
test and can be used to download the image from the App Catalog
and upload to Glance using either a Glance CLI command or via Horizon
@ -199,7 +207,7 @@ with the "kloudbuster_v" prefix and download the latest version from the list.
The name of the image in Glance must match exactly the image name in the App
Catalog (without the .qcow2 extension), for example to upload the image from
a local copy of that image:
a local copy of that image using the Glance CLI:
.. code-block:: bash

165
doc/source/readme.rst
View File

@ -1 +1,164 @@
.. include:: ../../README.rst
=====================
KloudBuster version 6
=====================
How good is your OpenStack *data plane* or *storage service* under real heavy load?
KloudBuster is a tool that can load the data plane or storage infrastructure of any OpenStack
cloud at massive scale and measure how well the cloud behaves under load
where it matters: from the VMs standpoint, where cloud applications run.
Accessible to anybody with basic knowledge of OpenStack, installs in minutes and runs
off the box with sensible default workloads in a fully automated way.
CLI/REST or Web User Interface.. you pick what works best for you.
Feature List
------------
* Neutron configuration agnostic (any encapsulation, any overlay, any plugin)
* OpenStack Storage backend agnostic
* Real VM-level performance and scale numbers (not bare metal)
* Punishing scale (thousands of VMs and enough load to fill even the fastest NIC
cards or load any storage cluster with ease)
* Data plane with HTTP traffic load:
* Can load the data plane with one OpenStack cloud (single-cloud operations
for L3 East-West scale) or 2 OpenStack clouds (dual-cloud operations with
one cloud hosting the HTTP servers and the other loading HTTP traffic for
L3 North-South scale testing)
* Real HTTP servers (Nginx) running in real Linux images (Ubuntu 14.04)
* Can specify any number of tenants, routers and networks
* Can specify any number of HTTP servers per tenant (as many as your cloud
can handle)
* High performance and highly scalable HTTP traffic generators to simulate
huge number of HTTP users and TCP connections (hundreds of thousands to
millions of concurrent and active connections)
* Overall throughput aggegation and loss-less millisecond-precision latency
aggregation for every single HTTP request (typically millions per run)
* Traffic shaping to specify on which links traffic should flow
* Can support periodic reporting and aggregation of results
* Storage load:
* VM-level Cinder volume (block storage) or Ephemeral disk file I/O performance measurement
using FIO running inside VMs (not bare metal)
* Supports random and sequential file access mode
* Supports any mix of read/write
* IOPs, bandwitdh and loss-less millisecond-precision latency aggregation for every
IO operation (typically millions per run)
* User configurable workload sequence
* Supports automated scale progressions (VM count series in any multiple
increment) to reduce dramatically scale testing time
* Highly efficient and scalable metric aggregation
* Automatic cleanup upon termination
* Regular expression based cleanup script (`cleanup`_)
* KloudBuster server mode to drive scale test:
* from any browser (KloudBuster Web UI)
* or from any external programs (KloudBuster REST API)
* Aggregated results provide an easy to understand way to assess the scale of
the cloud under test
* KloudBuster VM image pre-built and available from the OpenStack Community App
Catalog (https://apps.openstack.org/)
*Diagrams* describing how the scale test resources are staged and how the traffic flows are available
in :ref:`arch`.
Scale results are available in json form or in html form with javascript graphical charts generated straight off the tool.
*Examples of results* are available in :ref:`gallery`.
Limitations and Non-Goals
-------------------------
* Requires Neutron networking (does not support Nova networking)
* Only supports HTTP and storage traffic in this version
Unlike some other scaling test frameworks, KloudBuster does *not* attempt to:
* provide a scale test framework that works across different cloud technologies
(OpenStack + AWS + Google Cloud + ...) - we are only focusing on OpenStack
* provide a scale test framework that is flexible and programmable to do everything -
we just focus on opinionated and well targeted performance and scale areas
with sensible use cases and available in a fully integrated and easy to consume
packaged format
* replace bare metal and domain specific native performance and scale frameworks
(line level traffic generators, ceph specific performance and scale tools...)
Contributions and Feedbacks
---------------------------
If you are interested in OpenStack Performance and Scale, contributions and
feedbacks are welcome!
If you have any feedbacks or would like to make small or large contributions,
simply send an email to openstack-dev@lists.openstack.org with a '[kloudbuster]'
tag in the subject.
Licensing
---------
KloudBuster is licensed under the Apache License, Version 2.0 (the "License").
You may not use this tool except in compliance with the License. You may obtain
a copy of the License at `<http://www.apache.org/licenses/LICENSE-2.0>`_
Unless required by applicable law or agreed to in writing, software distributed
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
KloudBuster VM images contain multiple open source license components:
* nginx: BSD License (http://nginx.org/LICENSE)
* wrk2: Apache License 2.0
(https://raw.githubusercontent.com/giltene/wrk2/master/LICENSE)
* Redis: BSD License (http://redis.io/topics/license)
* FIO: GPL v2 (https://raw.githubusercontent.com/axboe/fio/master/MORAL-LICENSE)
Although the VM image includes a binary copy of the FIO code, it does not
include the source code used to build it. In accordance to the GPL V2 license
related to the inclusion of binary copies of FIO, the source code used to build
the FIO binary copy was not modified and can be found directly at
`<https://github.com/axboe/fio>`_ or can be obtained by email request to the
maintainer of KloudBuster.
Links
-----
* Complete documentation: `<http://kloudbuster.readthedocs.org>`_
* `KloudBuster REST API documentation Preview <https://htmlpreview.github.io/?https://github.com/openstack/kloudbuster/blob/master/doc/source/_static/kloudbuster-swagger.html>`_
* Source: `<https://github.com/openstack/kloudbuster>`_
* Supports/Bugs: `<http://launchpad.net/kloudbuster>`_
* Mailing List: kloudbuster-core@lists.launchpad.net

76
doc/source/usage.rst
View File

@ -16,14 +16,15 @@ run KloudBuster:
* Admin access to the cloud under test (non-admin might work with some
tweaks and limitations)
* 3 available floating IPs if running HTTP data plane testing, 2 available
floating IPs if running Storage testing
* 3 available floating IPs if running the HTTP data plane scale test,
* 2 available floating IPs if running the Storage scale test
Regardless of the way you launch KloudBuster, you will need the access info and
the credentials to the cloud under test. This information can be downloaded
from a Horizon dashboard (Project|Access&Security|Api Access|Download OpenStack
from the Horizon dashboard (Project|Access&Security|Api Access|Download OpenStack
RC File). Save it to your local file system for future use.
.. _run_server:
Running KloudBuster as a Web/REST Server
----------------------------------------
@ -37,7 +38,7 @@ and is ready to service HTTP and REST requests once up and running. To get the
KloudBuster Web server running in any OpenStack cloud:
1. Follow the steps :ref:`here <upload_kb_image>` to upload the KloudBuster
image to the openstack cloud that will host your KloudBuster web server
image to the OpenStack cloud that will host your KloudBuster web server
.. note::
This could be the same as the cloud under test or a different cloud.
@ -49,7 +50,7 @@ KloudBuster Web server running in any OpenStack cloud:
3. Create or reuse a security group which allows ingress TCP traffic on port
8080
4. Launch an instance using the KloudBuster image，with the proper security
4. Launch an instance using the KloudBuster image with the proper security
group, and connect to the appropriate network. Leave the Key Pair as blank,
as we don't need the SSH access to this VM
@ -82,13 +83,13 @@ is up running::
Starting the KloudBuster Server from a git clone
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you use git clone, you can bring up the KloudBuster Web/REST server fron the
If you use git clone, you can bring up the KloudBuster Web/REST server from the
CLI. KloudBuster uses the `Pecan <http://www.pecanpy.org/>`_ web server to host
both the KloudBuster REST server and the KloudBuster front-end website (which
listens to port 8080 by default).
From the root of the KloudBuster repository, go to the kb_server directory. If
this is the first time to start the server, run below command once to setup the
this is the first time you start the server, run the below command once to setup the
environment::
$ python setup.py develop
@ -111,8 +112,8 @@ Using any browser, point to the provided URL at port 8080. You will get a Login
page where you will need to enter:
* The type of scale test (HTTP data plane or storage)
* The location of openrc file of the cloud under test
* The credentials for the cloud under test
* The location of the openrc file of the cloud under test
* The password for the cloud under test
Interacting with the KloudBuster Server REST Interface
@ -133,8 +134,9 @@ If you do not really need a Web UI or REST interface, you can simply run
KloudBuster scale test straight from CLI.
KloudBuster is ready to run with the default configuration, which can be
displayed from the command line using *--show-config* option. By default,
KloudBuster will run on a single cloud and run the default HTTP data plane scale
displayed from the command line using *--show-config* option.
By default, KloudBuster will run on a single cloud and run the default HTTP data plane scale
test:
* Create 2 tenants, 2 users, and 2 routers;
@ -143,19 +145,41 @@ test:
* Create 1 VM running the Redis server (for orchestration)
* Create 1 VM running the HTTP traffic generator (default to 1000 connections,
1000 requests per second, and 30 seconds duration
* Measure/aggegate throughput and latency
* Measure/aggregate throughput and latency
* Bring down and cleanup
Run KloudBuster with the following options::
kloudbuster --tested-rc <path_to_the_admin_rc_file> --tested-passwd <admin_password>
.. note::
If you installed KloudBuster using git clone you need to explicitly call the python interpreter
since the "kloudbuster" wrapper is only installed with pip install)::
Simply adding *--storage* to the above command will run KloudBuster with
storage testing.
python kloudbuster.py --tested-rc <path_to_the_admin_rc_file> --tested-passwd <admin_password>
The run should take couple of minutes (depending on how fast the cloud can
If you want to run the default storage scale test, simply adding *--storage*::
kloudbuster --storage --tested-rc <path_to_the_admin_rc_file> --tested-passwd <admin_password>
The default storage scale test will use the following settings:
* Create 1 tenant
* Create 1 router
* Create 1 private network
* Create 1 VM and attach a 10 GB Cinder volume to it
* Perform the default storage workload sequence
* random access 4KB block size, IO depth 4, 100 IOPs for 30 seconds each
* 100% read
* 100% write
* 70% read, 30% write
* sequential access 64KB block size, IO depth 64, 60 MB/sec for 30 seconds
* 100% read
* 100% write
* 70% read, 30% write
* Measure/aggregate IOPs, throughput and latency
* Bring down and cleanup
The run should take a few minutes (depending on how fast the cloud can
create the resources) and you should see the actions taken by KloudBuster
displayed on the console. Once this minimal scale test passes, you can tackle
more elaborate scale testing by increasing the scale numbers or providing
@ -169,12 +193,16 @@ KloudBuster Configuration
To create a custom scale test configuration, make a copy of the default
configuration and modify that file to satisfy our own needs. A copy of the
default configuration can be obtained by redirecting the output of
*--show-config* to a new file. Once done, provide that custom configuration
*--show-config* to a new file. For example:
kloudbuster --show-config >scale1.cfg
Once done, provide that custom configuration
file to the KloudBuster command line using the *--config <file>* option.
.. note::
Note that the default configuration is always loaded by KloudBuster and
The default configuration is always loaded by KloudBuster and
any default option can be overridden by providing a custom configuration
file that only contains modified options. So you can delete all the lines
in the configuration file that you do not intend to change
@ -185,7 +213,7 @@ General Options
Each item in cfg.scale.yaml is well documented and self-explained. Below is
just a quick-start on some important config items that need to be paid more
attention.
attention to.
* **vm_creation_concurrency**
@ -258,7 +286,7 @@ In the case of Storage testing:
capacity of storage is reached, the overall performance will start to
degrade.
e.g. In the randread and randwrite mode, for example the IOPS is limited to
e.g. In the random read and random write mode, for example the IOPS is limited to
100 IOPS/VM. In the iteration of 10 VMs, the requested IOPS for the system
is 100 * 10 = 1000. However, the measured IOPS is degraded to only 800 IOPS.
So the degraded percentile is calculated as 800/1000=20% for this set of
@ -395,14 +423,14 @@ command line.
Displaying the Results
^^^^^^^^^^^^^^^^^^^^^^
Results can be saved in a file in json format or in html format. The json format
Results can be saved in a file in json format or in HTML format. The json format
is more appropriate for usage by any post-processing tool or script while the
html file is more adapted for human usage.
HTML file is more adapted for human usage.
The KloudBuster Web UI will display the results using charts and tables when the
test is finished running. The KloudBuster CLI provides an option to generate
the html file from the results (*--html* option). It can also generate the html
file from the json results (*--charts-from-json* option).
the HTML file from the results (*--html* option). It can also generate the HTML
file from the JSON results (*--charts-from-json* option).
Examples of running KloudBuster

16
kloudbuster/cfg.scale.yaml
View File

@ -43,9 +43,9 @@ vm_creation_concurrency: 5
# example to debug)
public_key_file:
# ==========================
# SERVER SIDE CONFIG OPTIONS
# ==========================
# ==================================================
# SERVER SIDE CONFIG OPTIONS (HTTP SERVER SIDE ONLY)
# ==================================================
server:
# Flavor to use for the test images
flavor:
@ -93,9 +93,9 @@ server:
html_size: 32768
# ==========================
# CLIENT SIDE CONFIG OPTIONS
# ==========================
# ====================================================
# CLIENT SIDE CONFIG OPTIONS (HTTP CLIENT AND STORAGE)
# ====================================================
client:
# Flavor to use for the test images
flavor:
@ -136,6 +136,8 @@ client:
# {'vm_start': 0, 'vm_multiple': 1}: 1, 2, 3, 4, 5...
# {'vm_start': 0, 'vm_multiple': 5}: 5, 10, 15, 20...
vm_multiple: 2
# HTTP ONLY
# The stop condition for HTTP benchmarking, it is used for KloudBuster to
# determine when to stop the progression, and do the cleanup if needed.
# It defines as:
@ -154,6 +156,8 @@ client:
# [50, 75, 90, 99, 99.9, 99.99, 99.999]
# (3) Sets percentile to 0 to disable HTTP request timeout checks;
http_stop_limit: [50, 0]
# STORAGE ONLY
# The stop condition for storage benchmarking, it is used for KloudBuster
# to determine when to stop the progression, and do the cleanup if needed.
# In the mode of random read and random write, this value indicates the