Fix up build script for stackforge repo rename.
Change repo paths for stv3 compionents in build script. Also fix docs for same. Change-Id: If846f6c097b28f23c697e17bb50dbc33f7c390a8
This commit is contained in:
parent
a069d95942
commit
df47784421
@ -14,7 +14,7 @@ Prerequisites:
|
|||||||
TL;DR:
|
TL;DR:
|
||||||
|
|
||||||
handle the prerequisites above.
|
handle the prerequisites above.
|
||||||
git clone https://github.com/stackforge/stacktach-sandbox
|
git clone https://github.com/openstack/stacktach-sandbox
|
||||||
cd sandbox
|
cd sandbox
|
||||||
create a mysql database to use
|
create a mysql database to use
|
||||||
set the database url appropriately in winchester.yaml
|
set the database url appropriately in winchester.yaml
|
||||||
|
2
build.sh
2
build.sh
@ -60,7 +60,7 @@ for file in shoebox simport notigen notification-utils \
|
|||||||
stackdistiller quincy quince timex \
|
stackdistiller quincy quince timex \
|
||||||
klugman winchester
|
klugman winchester
|
||||||
do
|
do
|
||||||
git clone http://git.openstack.org/stackforge/stacktach-$file
|
git clone http://git.openstack.org/openstack/stacktach-$file
|
||||||
done
|
done
|
||||||
# We still have some stragglers ...
|
# We still have some stragglers ...
|
||||||
for file in StackTach/notabene rackerlabs/yagi
|
for file in StackTach/notabene rackerlabs/yagi
|
||||||
|
@ -152,7 +152,7 @@ max_messages = 100
|
|||||||
</pre>
|
</pre>
|
||||||
|
|
||||||
<p>The important part of this configuration is the <code>[event_worker]</code> section. This says we want to use the RabbitMQ data source. The RabbitMQ connectivity information is stored in the <code>[rabbit_broker]</code> section. The name of each RabbitMQ queue to consume from is specified in the <code>[consumers]</code> section. For every queue you define there, you will need a <code>[consumer:<queue_name>]</code> section. This last section is where there real magic happens. Beyond defining the exchange, routing_key and durability characteristics, it defines the chain of <code>Yagi Handlers</code> that will run on every notification that gets consumed. </p>
|
<p>The important part of this configuration is the <code>[event_worker]</code> section. This says we want to use the RabbitMQ data source. The RabbitMQ connectivity information is stored in the <code>[rabbit_broker]</code> section. The name of each RabbitMQ queue to consume from is specified in the <code>[consumers]</code> section. For every queue you define there, you will need a <code>[consumer:<queue_name>]</code> section. This last section is where there real magic happens. Beyond defining the exchange, routing_key and durability characteristics, it defines the chain of <code>Yagi Handlers</code> that will run on every notification that gets consumed. </p>
|
||||||
<p>You can write your own Yagi handlers if you like, but there are a number that ship with StackTach.v3 to do some interesting things. The most important of these is the <a href='https://github.com/stackforge/stacktach-winchester/blob/4875e419a66974e416dbe3b43ed286017bad1ec4/winchester/yagi_handler.py#L18'>winchester.yagi_handler:WinchesterHandler</a>. This handler is your entry point into StackTach.v3 stream processing. But first, we need to convert those messy notifications into events ...</p>
|
<p>You can write your own Yagi handlers if you like, but there are a number that ship with StackTach.v3 to do some interesting things. The most important of these is the <a href='https://github.com/openstack/stacktach-winchester/blob/4875e419a66974e416dbe3b43ed286017bad1ec4/winchester/yagi_handler.py#L18'>winchester.yagi_handler:WinchesterHandler</a>. This handler is your entry point into StackTach.v3 stream processing. But first, we need to convert those messy notifications into events ...</p>
|
||||||
<h3><a id='distill'>Distilling Notifications to Events</a></h3>
|
<h3><a id='distill'>Distilling Notifications to Events</a></h3>
|
||||||
<p>Now we have notifications coming into Winchester. But, as we hinted at above, we need to take the larger notification and <i>distill</i> it down into a, more manageable, event. The stack-distiller module makes this happen. Within StackTach.v3, this is part of <code>winchester.yagi_handler:WinchesterHandler</code>.</p>
|
<p>Now we have notifications coming into Winchester. But, as we hinted at above, we need to take the larger notification and <i>distill</i> it down into a, more manageable, event. The stack-distiller module makes this happen. Within StackTach.v3, this is part of <code>winchester.yagi_handler:WinchesterHandler</code>.</p>
|
||||||
<p>A notification is a large, nested JSON data structure. But we don't need all of that data for stream processing. In fact, we generally only require a few <a href='glossary.html#trait'>Traits</a> from the notification. That's what distilling does. It pulls out the important traits, scrubs the data and uses that. Distillations are done via the distillation configuration file (specified in winchester.conf). </p>
|
<p>A notification is a large, nested JSON data structure. But we don't need all of that data for stream processing. In fact, we generally only require a few <a href='glossary.html#trait'>Traits</a> from the notification. That's what distilling does. It pulls out the important traits, scrubs the data and uses that. Distillations are done via the distillation configuration file (specified in winchester.conf). </p>
|
||||||
@ -231,7 +231,7 @@ max_messages = 100
|
|||||||
- exception.kwargs.uuid
|
- exception.kwargs.uuid
|
||||||
- instance.uuid
|
- instance.uuid
|
||||||
</pre>
|
</pre>
|
||||||
<p>For details on the distiller configuration grammar, see the associated <a href='https://github.com/stackforge/stacktach-stackdistiller/blob/master/doc/event_definitions_config.md'>docs</a>. Note that the distiller config file is in YAML format.</p>
|
<p>For details on the distiller configuration grammar, see the associated <a href='https://github.com/openstack/stacktach-stackdistiller/blob/master/doc/event_definitions_config.md'>docs</a>. Note that the distiller config file is in YAML format.</p>
|
||||||
|
|
||||||
<p>When processed, the notification would be converted into an event that looks like this:</p>
|
<p>When processed, the notification would be converted into an event that looks like this:</p>
|
||||||
<span class="label label-default">A sample event</span>
|
<span class="label label-default">A sample event</span>
|
||||||
|
@ -58,9 +58,9 @@
|
|||||||
<div class="col-lg-12">
|
<div class="col-lg-12">
|
||||||
<h3>Contributing to StackTach.v3</h3>
|
<h3>Contributing to StackTach.v3</h3>
|
||||||
<p>StackTach.v3 is licensed under the Apache 2.0 license</p>
|
<p>StackTach.v3 is licensed under the Apache 2.0 license</p>
|
||||||
<p>All the source repos for StackTach.v3 (and .v2) are available on <a href='https://github.com/stackforge?query=stacktach'>StackForge</a>. Details on contributing to StackForge projects are available <a href='https://wiki.openstack.org/wiki/How_To_Contribute'>here</a></p>
|
<p>All the source repos for StackTach.v3 (and .v2) are available on <a href='https://github.com/openstack?query=stacktach'>StackForge</a>. Details on contributing to StackForge projects are available <a href='https://wiki.openstack.org/wiki/How_To_Contribute'>here</a></p>
|
||||||
<p>The core developers are available on Freenode IRC in the <code>#stacktach</code> channel</p>
|
<p>The core developers are available on Freenode IRC in the <code>#stacktach</code> channel</p>
|
||||||
<p>These docs are available in the <a href='https://github.com/stackforge/stacktach-sandbox/tree/master/docs'>Sandbox</a> repo. Patches welcome!</p>
|
<p>These docs are available in the <a href='https://github.com/openstack/stacktach-sandbox/tree/master/docs'>Sandbox</a> repo. Patches welcome!</p>
|
||||||
|
|
||||||
<footer class="footer">
|
<footer class="footer">
|
||||||
<p>© Dark Secret Software Inc. 2014</p>
|
<p>© Dark Secret Software Inc. 2014</p>
|
||||||
|
@ -65,7 +65,7 @@
|
|||||||
<li>a yagi-event process to read the notifications and get them into winchester</li>
|
<li>a yagi-event process to read the notifications and get them into winchester</li>
|
||||||
<li>a winchester pipeline-worker process to do processing on ready streams</li>
|
<li>a winchester pipeline-worker process to do processing on ready streams</li>
|
||||||
</ul>
|
</ul>
|
||||||
<p>If you have a machine with MySQL and RabbitMQ on it, the <a href='https://github.com/stackforge/stacktach-sandbox' target='_blank'>stacktach-sandbox</a> repo does the rest. The sandbox will set up a python virtual environment, install all the required components and run everything under a <a href="http://www.rackaid.com/blog/linux-screen-tutorial-and-how-to/" target="_blank">screen</a> session.</p>
|
<p>If you have a machine with MySQL and RabbitMQ on it, the <a href='https://github.com/openstack/stacktach-sandbox' target='_blank'>stacktach-sandbox</a> repo does the rest. The sandbox will set up a python virtual environment, install all the required components and run everything under a <a href="http://www.rackaid.com/blog/linux-screen-tutorial-and-how-to/" target="_blank">screen</a> session.</p>
|
||||||
<img src='sandbox-1.gif' class="img-rounded"/>
|
<img src='sandbox-1.gif' class="img-rounded"/>
|
||||||
<p>The sandbox sets up seven screen pages that contain:</p>
|
<p>The sandbox sets up seven screen pages that contain:</p>
|
||||||
<ol type="a">
|
<ol type="a">
|
||||||
@ -106,7 +106,7 @@
|
|||||||
<p>In this deployment scenario, we have a set of "worker" boxes that run the <code>yagi-event</code> workers (to consume notifications from RabbitMQ). We run one yagi-event worker per OpenStack cell. We also run the Winchester pipeline-workers on this box. The pipeline-worker manages pipeline processing for ready streams. We run as many pipeline-workers as needed to keep each core busy. Both the yagi-event and pipeline-worker processes are single threaded so we spawn more to keep the host busy. We add more servers like these to handle the load in the pipeline processing and/or the source queues.</p>
|
<p>In this deployment scenario, we have a set of "worker" boxes that run the <code>yagi-event</code> workers (to consume notifications from RabbitMQ). We run one yagi-event worker per OpenStack cell. We also run the Winchester pipeline-workers on this box. The pipeline-worker manages pipeline processing for ready streams. We run as many pipeline-workers as needed to keep each core busy. Both the yagi-event and pipeline-worker processes are single threaded so we spawn more to keep the host busy. We add more servers like these to handle the load in the pipeline processing and/or the source queues.</p>
|
||||||
<p>Additionally, we have servers for the API nodes and the MySQL database. Load balancing the API servers and making MySQL highly available is left to the deployer.</p>
|
<p>Additionally, we have servers for the API nodes and the MySQL database. Load balancing the API servers and making MySQL highly available is left to the deployer.</p>
|
||||||
<h3>The Ansible deployment scripts</h3>
|
<h3>The Ansible deployment scripts</h3>
|
||||||
<p>In the <a href='https://github.com/stackforge/stacktach-sandbox/tree/master/ansible'>StackTach Sandbox</a> repo, there is an ansible directory that contains roles for the workers, the API nodes and the database. There is a fourth "common" role that is used by the other roles.</p>
|
<p>In the <a href='https://github.com/openstack/stacktach-sandbox/tree/master/ansible'>StackTach Sandbox</a> repo, there is an ansible directory that contains roles for the workers, the API nodes and the database. There is a fourth "common" role that is used by the other roles.</p>
|
||||||
<p>These scripts assume a multi-cell deployment where there is one StackTach.v3 deployment per region (each with it's own API servers and database). If you are not running with cells, pretend it's a single cell.</p>
|
<p>These scripts assume a multi-cell deployment where there is one StackTach.v3 deployment per region (each with it's own API servers and database). If you are not running with cells, pretend it's a single cell.</p>
|
||||||
|
|
||||||
<p>The servers are deployed in the following way:</p>
|
<p>The servers are deployed in the following way:</p>
|
||||||
@ -138,7 +138,7 @@ cd /opt/stv3/bin
|
|||||||
./yagi-event --config /etc/stv3/yagi-cell-1.conf
|
./yagi-event --config /etc/stv3/yagi-cell-1.conf
|
||||||
</pre>
|
</pre>
|
||||||
<h3>The Build script</h3>
|
<h3>The Build script</h3>
|
||||||
<p>The <a href='https://github.com/stackforge/stacktach-sandbox'>StackTach Sandbox</a> repo is the umbrella repo for all the other StackTach modules. It:</p>
|
<p>The <a href='https://github.com/openstack/stacktach-sandbox'>StackTach Sandbox</a> repo is the umbrella repo for all the other StackTach modules. It:</p>
|
||||||
<ul>
|
<ul>
|
||||||
<li>can create a local dev environment for anyone wanting to contribute to StackTach or just play around with it
|
<li>can create a local dev environment for anyone wanting to contribute to StackTach or just play around with it
|
||||||
<li>can package StackTach into a python virtual environment and tarball it up for larger scale deployment. The ansible scripts use this tarball approach.
|
<li>can package StackTach into a python virtual environment and tarball it up for larger scale deployment. The ansible scripts use this tarball approach.
|
||||||
|
@ -1,58 +1,63 @@
|
|||||||
BeautifulSoup==3.2.1
|
alembic==0.8.3
|
||||||
Mako==1.0.1
|
amqp==1.4.7
|
||||||
MarkupSafe==0.23
|
|
||||||
MySQL-python==1.2.5
|
|
||||||
PubSubHubbub-Publisher==1.0
|
|
||||||
PyYAML==3.11
|
|
||||||
Pygments==2.0.2
|
|
||||||
Routes==2.1
|
|
||||||
SQLAlchemy==1.0.2
|
|
||||||
WebOb==1.4.1
|
|
||||||
alembic==0.7.5.post2
|
|
||||||
amqp==1.4.6
|
|
||||||
amqplib==1.0.2
|
amqplib==1.0.2
|
||||||
anyjson==0.3.3
|
anyjson==0.3.3
|
||||||
argparse==1.2.1
|
BeautifulSoup==3.2.1
|
||||||
carrot==0.10.7
|
carrot==0.10.7
|
||||||
dateutils==0.6.6
|
dateutils==0.6.6
|
||||||
decorator==3.4.2
|
decorator==4.0.4
|
||||||
docopt==0.6.2
|
docopt==0.6.2
|
||||||
docutils==0.12
|
docutils==0.12
|
||||||
enum34==1.0.4
|
enum34==1.0.4
|
||||||
eventlet==0.17.3
|
eventlet==0.17.4
|
||||||
falcon==0.2
|
falcon==0.3.0
|
||||||
feedgenerator==1.7
|
feedgenerator==1.7
|
||||||
greenlet==0.4.5
|
flake8==2.2.4
|
||||||
|
greenlet==0.4.9
|
||||||
gunicorn==19.3.0
|
gunicorn==19.3.0
|
||||||
|
hacking==0.10.2
|
||||||
httpie==0.9.2
|
httpie==0.9.2
|
||||||
httplib2==0.9.1
|
httplib2==0.9.2
|
||||||
iso8601==0.1.10
|
iso8601==0.1.10
|
||||||
jsonpath-rw==1.4.0
|
jsonpath-rw==1.4.0
|
||||||
klugman==0.5.dev24
|
klugman==0.5.0.dev25
|
||||||
kombu==3.0.26
|
kombu==3.0.29
|
||||||
librabbitmq==1.6.1
|
librabbitmq==1.6.1
|
||||||
lockfile==0.10.2
|
lockfile==0.11.0
|
||||||
mysql-connector-python==2.0.3
|
Mako==1.0.3
|
||||||
notabene==0.2.dev31
|
MarkupSafe==0.23
|
||||||
notification-utils==0.3.dev6
|
mccabe==0.2.1
|
||||||
notigen==0.3.dev36
|
mysql-connector-python==2.0.4
|
||||||
pbr==0.10.8
|
MySQL-python==1.2.5
|
||||||
ply==3.6
|
notabene==0.2.0.dev30
|
||||||
|
notification-utils==0.3.0.dev7
|
||||||
|
notigen==0.3.0.dev35
|
||||||
|
pbr==0.11.0
|
||||||
|
pep8==1.5.7
|
||||||
|
ply==3.8
|
||||||
prettytable==0.7.2
|
prettytable==0.7.2
|
||||||
python-daemon==2.0.5
|
PubSubHubbub-Publisher==1.0
|
||||||
|
pyflakes==0.8.1
|
||||||
|
Pygments==2.0.2
|
||||||
|
python-daemon==2.0.6
|
||||||
python-dateutil==2.4.2
|
python-dateutil==2.4.2
|
||||||
|
python-editor==0.4
|
||||||
python-mimeparse==0.1.4
|
python-mimeparse==0.1.4
|
||||||
pytz==2015.2
|
pytz==2015.7
|
||||||
quince==0.3.dev10
|
PyYAML==3.11
|
||||||
quincy==0.3.dev24
|
quince==0.3.0.dev12
|
||||||
|
quincy==0.3.0.dev25
|
||||||
redis==2.10.3
|
redis==2.10.3
|
||||||
repoze.lru==0.6
|
repoze.lru==0.6
|
||||||
requests==2.6.2
|
requests==2.8.1
|
||||||
shoebox==0.7.dev53
|
Routes==2.2
|
||||||
simport==0.2.dev16
|
shoebox==0.7.0.dev52
|
||||||
six==1.9.0
|
simport==0.3.0.dev18
|
||||||
stackdistiller==0.11
|
six==1.10.0
|
||||||
timex==0.20.0
|
SQLAlchemy==1.0.9
|
||||||
winchester==0.55.dev34
|
stackdistiller==0.13.0.dev10
|
||||||
wsgiref==0.1.2
|
timex==0.22.0.dev6
|
||||||
yagi==0.23
|
WebOb==1.5.0
|
||||||
|
wheel==0.24.0
|
||||||
|
winchester==0.62.0.dev42
|
||||||
|
yagi==0.24
|
||||||
|
2
vagrant/Vagrantfile
vendored
2
vagrant/Vagrantfile
vendored
@ -4,5 +4,5 @@
|
|||||||
Vagrant.configure(2) do |config|
|
Vagrant.configure(2) do |config|
|
||||||
config.vm.box = "ubuntu/trusty64"
|
config.vm.box = "ubuntu/trusty64"
|
||||||
config.vm.provision "shell", path: "bootstrap.sh"
|
config.vm.provision "shell", path: "bootstrap.sh"
|
||||||
config.vm.provision "shell", inline: "git clone https://github.com/stackforge/stacktach-sandbox.git", privileged: false
|
config.vm.provision "shell", inline: "git clone https://github.com/openstack/stacktach-sandbox.git", privileged: false
|
||||||
end
|
end
|
||||||
|
Loading…
Reference in New Issue
Block a user