166 lines
9.6 KiB
HTML
166 lines
9.6 KiB
HTML
|
|
<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
<meta name="description" content="StackTach.v3">
|
|
<meta name="author" content="Sandy Walsh">
|
|
<link rel="icon" href="../../favicon.ico">
|
|
|
|
<title>StackTach.v3</title>
|
|
|
|
<!-- Latest compiled and minified CSS -->
|
|
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.2/css/bootstrap.min.css">
|
|
|
|
<link href="stv3-narrow.css" rel="stylesheet">
|
|
|
|
<!-- Optional theme -->
|
|
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.2/css/bootstrap-theme.min.css">
|
|
|
|
<!-- Latest compiled and minified JavaScript -->
|
|
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.2/js/bootstrap.min.js"></script>
|
|
|
|
<style>
|
|
.bottom_padding {
|
|
padding-bottom: 20px;
|
|
}
|
|
</style>
|
|
|
|
<!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
|
|
<!--[if lt IE 9]>
|
|
<script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
|
|
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
|
|
<![endif]-->
|
|
|
|
|
|
</head>
|
|
|
|
<body>
|
|
|
|
<div class="container">
|
|
<div class="header bottom_padding">
|
|
<nav>
|
|
<ul class="nav nav-pills pull-right">
|
|
<li role="presentation"><a href="index.html">Home</a></li>
|
|
<li role="presentation"><a href="about.html">Docs</a></li>
|
|
<li role="presentation" class="active"><a href="install.html">Getting Started</a></li>
|
|
<li role="presentation"><a href="api.html">API</a></li>
|
|
<li role="presentation"><a href="screencasts.html">Screencasts</a></li>
|
|
<li role="presentation"><a href="contribute.html">Contribute</a></li>
|
|
</ul>
|
|
</nav>
|
|
<img src="StackTach_160x70.png"/>
|
|
</div>
|
|
|
|
<div class="row marketing">
|
|
<div class="col-lg-12">
|
|
<h3>A Minimal StackTach.v3 deployment</h3>
|
|
<p>The bare minimum StackTach.v3 deployment has the following components:</p>
|
|
<ul>
|
|
<li>a queuing system, like RabbitMQ, that contains the notifications.</li>
|
|
<li>a MySQL database</li>
|
|
<li>something to generate source notifications</li>
|
|
<li>a yagi-event process to read the notifications and get them into winchester</li>
|
|
<li>a winchester pipeline-worker process to do processing on ready streams</li>
|
|
</ul>
|
|
<p>If you have a machine with MySQL and RabbitMQ on it, the <a href='https://github.com/stackforge/stacktach-sandbox' target='_blank'>stacktach-sandbox</a> repo does the rest. The sandbox will set up a python virtual environment, install all the required components and run everything under a <a href="http://www.rackaid.com/blog/linux-screen-tutorial-and-how-to/" target="_blank">screen</a> session.</p>
|
|
<img src='sandbox-1.gif' class="img-rounded"/>
|
|
<p>The sandbox sets up seven screen pages that contain:</p>
|
|
<ol type="a">
|
|
<li>The REST API server, via the gunicorn web server, quincy WSGI application and quince API implementation. Quince uses winchester to talk to the database.</li>
|
|
<li>A bash shell for running the klugman command line tool.</li>
|
|
<li>Two yagi-event workers. One per screen page.</li>
|
|
<li>Two pipeline-worker workers. One per screen page.</li>
|
|
<li>The notigen notification simulator.</li>
|
|
</ol>
|
|
|
|
<p>You can see the flow of data in the diagram above:</p>
|
|
<ol>
|
|
<li>OpenStack Nova notifications are simulated by notigen and pumped into RabbitMQ via the notabene library. With the sandbox, there is no need to stand up a full OpenStack deployment.</li>
|
|
<li>The yagi-event workers consume notifications from RabbitMQ, archives them via shoebox, distills them into events and stuffs them into streams via winchester</li>
|
|
<li>The pipeline-worker workers look for ready streams and does pipeline processing on them.</li>
|
|
<li>The user can use the klugman cmdline tool to talk to the REST API to perform stream and event operations/queries</li>
|
|
</ol>
|
|
<p>In order to do all this, there are a number of configuration files required. Refer to the general documentation here or the particular libraries README file for configuration details. Of course, the names could be anything, these are just the ones we've settled on. The main configuration files include:</p>
|
|
<ul>
|
|
<li><b>yagi.conf</b> - the configuration file that tells yagi how to connect to the queue and what to do with the notifications consumed.</li>
|
|
<li><b>winchester.yaml</b> - the master configuration file for winchester. It specifies the pipeline configuration to use, the stream definitions and the triggering rules.</p>
|
|
<li><b>triggers.yaml</b> - the detailed stream definitions and pipeline triggering rules</b></li>
|
|
<li><b>pipelines.yaml</b> - the pipeline handler definitions</p>
|
|
<li><b>event_definitions.yaml</b> - how the distiller should map notifications to events. Which traits to select or transform.
|
|
</ul>
|
|
|
|
<p>Where each of these files is used is illustrated in the diagram below:</p>
|
|
|
|
<img src='sandbox-2.gif' class="img-rounded"/>
|
|
|
|
<p>This will get you going for a minimal StackTach.v3 installation. It's especially handy for dev environments as well as a way of just playing around. For low-volume/non-mission critical environments, this would be appropriate. Read up on the "build" command, below, for instructions on launching the sandbox environment. For larger deployments, you may want to look at how we deploy StackTach.v3 within Rackspace Public Cloud, below:</p>
|
|
|
|
<h3>How StackTach.v3 is deployed at Rackspace</h3>
|
|
<p>For Rackspace Public Cloud, OpenStack is deployed in many different regions and each region is comprised of many cells.</p>
|
|
|
|
<img src='deploy_diagram.gif' class="img-rounded"/>
|
|
|
|
<p>In this deployment scenario, we have a set of "worker" boxes that run the <code>yagi-event</code> workers (to consume notifications from RabbitMQ). We run one yagi-event worker per OpenStack cell. We also run the Winchester pipeline-workers on this box. The pipeline-worker manages pipeline processing for ready streams. We run as many pipeline-workers as needed to keep each core busy. Both the yagi-event and pipeline-worker processes are single threaded so we spawn more to keep the host busy. We add more servers like these to handle the load in the pipeline processing and/or the source queues.</p>
|
|
<p>Additionally, we have servers for the API nodes and the MySQL database. Load balancing the API servers and making MySQL highly available is left to the deployer.</p>
|
|
<h3>The Ansible deployment scripts</h3>
|
|
<p>In the <a href='https://github.com/stackforge/stacktach-sandbox/tree/master/ansible'>StackTach Sandbox</a> repo, there is an ansible directory that contains roles for the workers, the API nodes and the database. There is a fourth "common" role that is used by the other roles.</p>
|
|
<p>These scripts assume a multi-cell deployment where there is one StackTach.v3 deployment per region (each with it's own API servers and database). If you are not running with cells, pretend it's a single cell.</p>
|
|
|
|
<p>The servers are deployed in the following way:</p>
|
|
<pre>
|
|
# Everything is installed and deployed as:
|
|
user: stv3
|
|
group: stv3
|
|
|
|
Code:
|
|
/opt/stv3
|
|
|
|
Daemons:
|
|
/etc/init.d/
|
|
|
|
Config:
|
|
/etc/stv3
|
|
|
|
Logs:
|
|
/var/log/stv3
|
|
|
|
PIDs:
|
|
/var/run/stv3/
|
|
|
|
# To launch a service manually (logging in a different user)
|
|
sudo -s
|
|
su stv3
|
|
cd /opt/stv3/bin
|
|
. ./activate
|
|
./yagi-event --config /etc/stv3/yagi-cell-1.conf
|
|
</pre>
|
|
<h3>The Build script</h3>
|
|
<p>The <a href='https://github.com/stackforge/stacktach-sandbox'>StackTach Sandbox</a> repo is the umbrella repo for all the other StackTach modules. It:</p>
|
|
<ul>
|
|
<li>can create a local dev environment for anyone wanting to contribute to StackTach or just play around with it
|
|
<li>can package StackTach into a python virtual environment and tarball it up for larger scale deployment. The ansible scripts use this tarball approach.
|
|
<li>git clones all the dependent python modules so you can develop locally and see your changes immediately and easily.
|
|
</ul>
|
|
<p>The sandbox readme will get you started, but the gory details are in the <code>build.sh</code> script. There are essentially three options:</p>
|
|
<ul>
|
|
<li><code>./build.sh</code> - run the local StackTach.v3 dev environment in a screen session.
|
|
<li><code>./build.sh -t</code> - run the tests on all the StackTach.v3 repos.
|
|
<li><code>./build.sh -p</code> - install StackTach.v3 into the virtual env and package it up into a tarball for deployment.
|
|
<li><code>./build.sh -p -d</code> - package and deploy the tarball. Currently this just lists the ansible incantation required to do the deployment.
|
|
</ul>
|
|
<p>The <a href='screencasts.html'>screencasts</a> cover using the sandbox for local dev in more detail.</p>
|
|
<footer class="footer">
|
|
<p>© Dark Secret Software Inc. 2014</p>
|
|
</footer>
|
|
|
|
</div> <!-- /container -->
|
|
|
|
<!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
|
|
<script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
|
|
</body>
|
|
</html>
|
|
|