x/broadview-collector
Code Issues Proposed changes

Initial commit of broadview-collector

Browse Source 
Change-Id: I87a54972774870aa705bd0948dfa7582c8ac634d
This commit is contained in:
Volodymyr Samotiy 2016-03-09 16:50:48 +02:00
parent 681f013d80
commit 8ffac8794d
45 changed files with 4734 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
.coveragerc Normal file
View File

@ -0,0 +1,7 @@
[run]
branch = True
source = broadview_collector
omit = broadview_collector/openstack/*
[report]
ignore_errors = True

3
.mailmap Normal file
View File

@ -0,0 +1,3 @@
# Format is:
# <preferred e-mail> <other e-mail 1>
# <preferred e-mail> <other e-mail 2>

7
.testr.conf Normal file
View File

@ -0,0 +1,7 @@
[DEFAULT]
test_command=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-60} \
${PYTHON:-python} -m subunit.run discover -t ./ . $LISTOPT $IDOPTION
test_id_option=--load-list $IDFILE
test_list_option=--list

17
CONTRIBUTING.rst Normal file
View File

@ -0,0 +1,17 @@
If you would like to contribute to the development of OpenStack, you must
follow the steps in this page:
http://docs.openstack.org/infra/manual/developers.html
If you already have a good understanding of how the system works and your
OpenStack accounts are set up, you can skip to the development workflow
section of this documentation to learn how changes to OpenStack should be
submitted for review via the Gerrit tool:
http://docs.openstack.org/infra/manual/developers.html#development-workflow
Pull requests submitted through GitHub will be ignored.
Bugs should be filed on Launchpad, not GitHub:
https://bugs.launchpad.net/broadview-collector

4
HACKING.rst Normal file
View File

@ -0,0 +1,4 @@
broadview-collector Style Commandments
===============================================
Read the OpenStack Style Commandments http://docs.openstack.org/developer/hacking/

176
LICENSE Normal file
View File

@ -0,0 +1,176 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.

6
MANIFEST.in Normal file
View File

@ -0,0 +1,6 @@
include AUTHORS
include ChangeLog
exclude .gitignore
exclude .gitreview
global-exclude *.pyc

434
README.md Normal file
View File

@ -0,0 +1,434 @@
# Overview
`broadview-collector` is a service capable of receiving and publishing
BroadView data received from a network device that is running Broadcom's
BroadView agent. It is built on top of
[broadview-lib](https://github.com/openstack/broadview-lib/tree/master/broadview_lib).
broadview-collector is designed to support multiple metrics-gathering
infrastructures via a plugin architecture.
## Publisher Plugins
broadview-collector is designed to accept plugins that are capable of
publishing BroadView data received by the collector to metric-collecting
services such as OpenStack Monasca. These plugins are in the
form of Python classes. These classes inherit from a base class that
defines the interface the collector will invoke when data is received
from the BroadView agent.
A serializer takes BroadView data and converts it into a playload that
is suitable for a given publisher plugin.
Data coming into the collector from a BroadView agent is passed to a
list of handlers. Each handler is designed to convert the payload
that was received by the agent into an object form (currently, only BST
data is supported, but handlers for additional BroadView components will
be developed). This object form is canonical in the sense that it is
independent of any publisher plugin.
Once the payload is in an object form, a list of publisher plugins is
iterated. Each publisher is given a chance to serialize the data (it
does this by instantiating a serializer object, described below). If
the serialization is successful, then the publisher plugin will write
the data to the service that it directly supports.
## Creating Publisher Plugins
Publisher plugins are simply python source files that exist in the directory
broadview_collector/plugins. A single class inheriting BroadViewPublisherBase
is implemented in this file. This base class defines a single method,
publish(), which accepts a data object from the collector. The plugin
must provide an override to this method.
Addition directories can be configured to hold plugins. These directories,
if configured, will be searched for plugins at startup in addition to
broadview_collector/plugins. See the documentation on the searchpath config
variable in /etc/broadviewcollector.conf, below, for details and an
example.
The name of the class implemented by the plugin must be BroadViewPlublisher.
The set of plugins used at runtime is defined in configuration, and only
configured plugins will be invoked by the collector. Simply placing a
python source file in the plugins directory will not cause it to be
recognized by broadview-collector. Further details on configuration is
provided below.
The job of the publish() method is to (optionally) call a serializer to
convert the data into payload (e.g., JSON) understandable to the service
that the publisher plugin targets, and to publish the data using whatever
API that the target service provides for doing so.
Two example publishers are provided:
* monascapublisher: targets the OpenStack Monasca Python metric publishing API
* logpublisher: appends data to a configured text file
To use a publisher, it must be added to the comma-separated list of publishers
listed in /etc/broadviewcollector.conf, and the collector must be restarted.
## Serializers
Serializers are python source files that are located in the directory
broadview_collector/serializers. The broadview-collector is not directly
aware of these serializers, nor are they required to be configured in order
to be used. They are aggregated in the serializers directory simply to make
them available for whatever plugins wish to use (or not use) them.
A serializer should inherit from BroadViewSerializerBase. The intent is to
promote a uniform API among the serializers. If the target of a
publisher plugin requires a new serializer be written, consider creating a
serializer that inherits BroadViewSerializerBase and locating it in the
serializers directory so that it might be of use to developers of other
publisher plugins.
The monasca and log publishers both use the BSTToMonasca serializer located
in serializers/bst_to_monasca.py to illustrate this concept of reuse. If you
don't like the format of the messages that are written by an existing
publisher, you can write your own serializer (if one of the existing ones
does not fit your needs) and hack the publisher to import and use it instead.
## Handlers
A handler is designed to take traffic incoming to the collector, parse it,
and return an object (of some type) that represents the parsed data. The
returned object (or list of objects perhaps) is then sent to each registered
publisher.
Handlers map to BroadView components. Handlers, like publisher plugins and
serializers, inherit a base class, in this case BroadViewHandlerBase. This
class, and the handlers themselves, are located in
broadview_collector/handlers.
Handlers directly correspond to BroadView components that are supported in
broadview-lib. See https://github.com/openstack/broadview-lib for more
information on the BroadView components supported by broadview-lib.
To use a handler, it must be listed in the comma-separated list of handlers
in /etc/broadviewcollector.conf, and it must be located in the handlers
directory.
The name of the handler class must be BroadViewHandler.
See broadview_collector/handlers/bsthandler.py for an example. Handlers have
a tight coupling to parsers provided in broadview-lib, and normally they will
be added (or updated) as parsers are added (or updated) in the broadview-lib
project.
# Installing BroadView Collector
## Basic Installation and Configuration<a name="basicinstall">
Installation requires the following steps:
1. Install OpenStack, including the services to which you intend to publish
data to. We discuss below Monasca-based publishing in more detail. Steps for
other services are currently not supported, but will be similar in scope.
Your OpenStack vendor/supplier may have installation instructions specific to
products that they support, contact your supplier for more details.
2. Install broadview-lib:
$ git clone https://github.com/openstack/broadview-lib.git
$ cd broadview-lib
$ python setup.py install
3. Install broadview-collector
$ git clone https://github.com/openstack/broadview-collector.git
$ cd broadview-collector
$ python setup.py install
4. Copy the file broadview_collector/config/broadviewcollector.conf to /etc
$ sudo cp broadview_collector/config/broadviewcollector.conf /etc
5. Edit /etc/broadviewcollector.conf as needed
$ sudo vi /etc/broadviewcollector.conf
6. Copy broadview collector application to /usr/local/bin:
$ sudo cp broadview_collector/bin/bvcollect.py /usr/local/bin
$ sudo chmod 755 /usr/local/bin/bvcollect.py
7. Start the collector (assuming /usr/local/bin is in your PATH):
$ bvcollect.py &
In addition, you must configure the broadview agent. This involves two
steps: editing the device configuration so that the agent knows the IP
address and port where the collector is listening, and configuring the
agent to publish desired statistics.
To configure the agent, refer to instructions provided by your vendor.
The IP address and port of the collector is in the [network] section of
/etc/broadviewcollector.conf, for example:
[network]
ip_address: 10.14.244.143
port: 8082
Once the agent is up and running on the networking device, you can use
bv-bstclt (included in broadview-lib) to configure BST.
## Detailed Installation: Monasca API
This section details how to set up and configure a devstack-based install
of broadview-collector that publishes to Monasca API.
The following assumes you are on a host running Ubuntu 14.04. Later versions
of Ubuntu may work, but as of this writing, are not tested or supported.
1. Follow the directions for bringing up Monasca with devstack which are
located here:
https://github.com/openstack/monasca-api/blob/master/devstack/README.txt
If you already have OpenStack and Monasca working, then you probably can
skip the above step.
2. Follow the instructions above in [Basic Installation and
Configuration](#basicinstall)
3. Add monascapublisher to the list of publishers in
/etc/broadviewcollector.conf
4. Add a section, [monasca], to /etc/broadviewcollector.conf, as in the
following:
[monasca]
username: mini-mon
password: password
project_name: mini-mon
auth_url: http://10.14.245.57:35357/v3
endpoint: http://10.14.245.57:8070/v2.0
api_version: 2_0
A /etc/broadviewcollector.conf known to work with monasca-api devstack looks
like the following (you'll need to change the IP addresses to match the one
of your host system):
[plugins]
# comma separated list of plugin modules
publishers: monascapublisher, logpublisher
# handlers map to broadview components, e.g., bst, packet trace
handlers: bsthandler
[logging]
# this section sets prefs for the logging handler.
file: /tmp/broadview-bstlogging.log
[network]
ip_address: 10.14.245.57
port: 8082
[monasca]
username: mini-mon
password: password
project_name: mini-mon
auth_url: http://10.14.245.57:35357/v3
endpoint: http://10.14.245.57:8070/v2.0
api_version: 2_0
# Configuration File Syntax
This section describes the syntax of the configuration file
/etc/broadviewcollector.conf.
## [DEFAULT]
The [DEFAULT] section defines logging parameters. Refer to the example
in config/broadviewcollector.conf for details. Logs in the default
configuration are written to the file
/var/log/broadview-collector/broadview-collector.log
## [plugins]
The [plugins] section supports the following settings related to publisher
plugins:
### publishers
publishers is a list of comma-separated python modules that are located in
the plugins directory, or the plugins searchpath (see searchpath, below).
Example:
publishers: monascapublisher, logpublisher, syslogpublisher
### searchpath
searchpath is a comma-separated list of prefixes that are preprended to the
publisher names when the collector attempts to load plugins.
A search order is defined by the order of items in this comma-separated list.
The "plugins" directory associated with broadview-collector is searched
last. Each plugin is searched for in each directory, in order. If the plugin
is successfully loaded using an entry in the search path, the search ceases.
Otherwise, the search will continue in the next directory defined in the
search path.
Note: the python searchpath (e.g., PYTHONPATH) is not modified by the value
of this variable.
Items in the searchpath must be for the form a.b.c
Example:
searchpath: tmp, home.broadview.plugins
## [misc]
### handlers
The handlers setting is a list of handlers of broadview payload sent by an
agent. This setting generally is defined to reflect the capabilities of
broadview-lib. If you are only interested in a subset of the functions that
BroadView supports, reducing the list of handlers to contain only those
BroadView components you are interested in can increase the performance of
the collector, and reduce the burden on your metric collection/analytics
pipeline. Items in this setting are comma-separated.
Example:
handlers: bsthandler
## [logging]
This section supports the logpublisher publisher plugin.
### file
This setting is the absolute path of the logfile written by the publisher
plugin
Example:
file: /tmp/broadview-bstlogging.log
## [network]
This section contains the networking configuration of the collector
### ip_address
The IP address (IPV4) that the collector will listen for connections on
Example:
ip_address: 10.14.245.57
### port
The port that the collector will listen for connections on
Example:
port: 8082
## [monasca]
This section supports the monascapublisher publisher plugin. These settings
are passed by the monasca publisher as arguments to the monasca python client
constructor. An example of how a python app creates a client object:
from monascaclient import client
import monascaclient.exc as exc
api_version = '2_0'
endpoint = 'http://192.168.10.4:8080/v2.0'
auth_kwargs = {'username': 'mini-mon',
'password': 'password',
'project_name': 'mini-mon',
'auth_url': 'http://192.168.10.5:35357/v3/'}
monasca_client = client.Client(api_version, endpoint, **auth_kwargs)
The settings in this section directory correspond (in name and purpose) with
the arguments pass to the Client constructor above.
### username
The username to authenticate
Example:
username: mini-mon
### password
The password associated with the user being authenticated
Example:
password: password
### project_name
The project name for which the above user is authenticated
Example:
project_name: mini-mon
### auth_url
The URL of the authentication service (keystone)
example:
auth_url: http://10.14.245.57:35357/v3
### endpoint
The URL that defines where monasca API is running
Example:
endpoint: http://10.14.245.57:8070/v2.0
### api_version
The version of the Monasca API
Example:
api_version: 2_0
## DevStack Support
Devstack support is provided by the files located in the directory
broadview_controller/devstack. Devstack is probably the easiest way
to experiment with BroadView Collector.
Refer to the README.txt file in broadview_controller/devstack for
details on how to setup a devstack-based environment for executing
BroadView Collector.
# License
# (C) Copyright Broadcom Corporation 2016
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
#
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

2
babel.cfg Normal file
View File

@ -0,0 +1,2 @@
[python: **.py]

0
broadview_collector/__init__.py Normal file
View File

11
broadview_collector/bin/bvcollect.py Normal file
View File

@ -0,0 +1,11 @@
#!/usr/bin/python
import re
import sys
import time
from broadview_collector.broadview_collector import main
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
sys.exit(main())

182
broadview_collector/broadview_collector.py Normal file
View File

@ -0,0 +1,182 @@
#!/usr/bin/env python
# (C) Copyright Broadcom Corporation 2016
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
#
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
from SocketServer import ThreadingMixIn
import threading
import argparse
import re
import cgi
import datetime
import json
from time import sleep
import ConfigParser
import os
import sys
from importlib import import_module
import ast
from oslo_log import log as logging
from oslo_config import cfg
LOG = logging.getLogger(__name__)
class BroadViewCollector(object):
def __init__(self):
logging.register_options(cfg.CONF)
logging.set_defaults()
cfg.CONF(args=[],
project="broadview_collector",
default_config_files=["/etc/broadviewcollector.conf"])
logging.setup(cfg.CONF, 'broadview_collector')
self._publishers = []
self._handlers = []
def readConfig(self):
LOG.info("broadview_collector: readConfig")
try:
cfg = ConfigParser.ConfigParser()
cfg.read("/etc/broadviewcollector.conf")
x = cfg.get("plugins", "publishers")
self._publisherNames = [y.strip() for y in x.split(',')]
LOG.info("publishers {}".format(self._publisherNames))
self._searchpath = []
try:
x = cfg.get("plugins", "searchpath")
self._searchpath = [y.strip() for y in x.split(',')]
except:
LOG.info("plugin searchpath missing or malformed")
if not self._searchpath or len(self._searchpath) == 0:
self._searchpath = ["broadview_collector.plugins"]
else:
self._searchpath.append("broadview_collector.plugins")
LOG.info("plugin searchpath {}".format(self._searchpath))
x = cfg.get("plugins", "handlers")
self._handlerNames = [y.strip() for y in x.split(',')]
LOG.info("plugin handlers {}".format(self._handlerNames))
self._ip_address = cfg.get("network", "ip_address")
self._port = int(cfg.get("network", "port"))
except:
LOG.error("Unable to open or read /etc/broadviewcollector.conf")
exit()
def loadPublishers(self):
LOG.info("broadview_collector: loadPublishers")
self._publishers = []
for x in self._publisherNames:
for y in self._searchpath:
try:
path = "{}.{}".format(y, x)
mod = import_module(path)
classattr = getattr(mod, "BroadViewPublisher")
self._publishers.append(classattr())
LOG.info("loaded plugin %s" % (path))
break
except:
e = sys.exc_info()[0]
LOG.info("Unable to load plugin %s: %s" % (path, e))
def loadHandlers(self):
LOG.info("broadview_collector: loadHandlers")
self._handlers = []
for x in self._handlerNames:
try:
mod = import_module("broadview_collector.handlers." + x)
classattr = getattr(mod, "BroadViewHandler")
self._handlers.append(classattr())
LOG.info("imported handler %s" % ("broadview_collector.handlers." + x))
except:
e = sys.exc_info()[0]
LOG.info("Unable to load handler %s: %s" % (x, e))
exit()
def handlePOST(self, path, ctype, length, data):
'''
find a handler that can handle the request, and then if
successful, send it to all publishers
'''
handled = False
retcode = 404
LOG.info("broadview_collector: handlePOST")
for x in self._handlers:
o, handled = x.handlePOST(path, ctype, length, data)
if handled:
for y in self._publishers:
code = y.publish(o)
if not code == 200:
LOG.info("handlePOST: {} failed to publish, code: {}".format(y, code))
retcode = 200
break
LOG.info("broadview_collector: handlePOST returns %d" % (retcode))
return retcode
collector = BroadViewCollector()
class HTTPRequestHandler(BaseHTTPRequestHandler):
def do_PUT(self):
pass
def do_POST(self):
ctype, pdict = cgi.parse_header(self.headers.getheader('content-type'))
length = int(self.headers.getheader('content-length'))
data = ast.literal_eval(json.loads(self.rfile.read(length)))
code = collector.handlePOST(self.path, ctype, length, data)
data = json.dumps({})
self.send_response(code)
self.send_header('Content-Type', 'application/json')
self.end_headers()
self.wfile.write(data)
return
class ThreadedHTTPServer(ThreadingMixIn, HTTPServer):
allow_reuse_address = True
def shutdown(self):
self.socket.close()
HTTPServer.shutdown(self)
class SimpleHttpServer():
def __init__(self, ip, port):
self.server = ThreadedHTTPServer((ip,port), HTTPRequestHandler)
def start(self):
self.server_thread = threading.Thread(target=self.server.serve_forever)
self.server_thread.daemon = True
self.server_thread.start()
def waitForThread(self):
self.server_thread.join()
def stop(self):
self.server.shutdown()
self.waitForThread()
def main():
collector.readConfig()
collector.loadPublishers()
collector.loadHandlers()
server = SimpleHttpServer(collector._ip_address, collector._port)
LOG.info('HTTP Server Running...........')
server.start()
server.waitForThread()
if __name__=='__main__':
main()

46
broadview_collector/config/broadviewcollector.conf Normal file
View File

@ -0,0 +1,46 @@
[DEFAULT]
# logging, make sure that the user under whom the server runs has permission
# to write to the directory
log_file = broadview-collector.log
log_dir = /var/log/broadview-collector
debug = True
[plugins]
# comma separated list of plugin modules
# publishers receive JSON encoded reports and write them somewhere
# uncomment for monasca API
#publishers: logpublisher, monascapublisher
publishers: logpublisher
#searchpath: tmp
# handlers map to broadview components, e.g., bst, packet trace
handlers: bsthandler
[logging]
# this section sets prefs for the logging handler.
file: /tmp/broadview-bstlogging.log
[network]
ip_address: 127.0.0.1
port: 8082
# uncomment for monasca API
#[monasca]
#
#username: mini-mon
#password: password
#project_name: mini-mon
#auth_url: http://127.0.0.1:35357/v3
#endpoint: http://127.0.0.1:8070/v2.0
#api_version: 2_0

0
broadview_collector/handlers/__init__.py Normal file
View File

30
broadview_collector/handlers/broadviewhandlerbase.py Normal file
View File

@ -0,0 +1,30 @@
# (C) Copyright Broadcom Corporation 2016
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
#
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
class BroadViewHandlerBase(object):
def __init__(self):
pass
def handlePOST(self, path, ctype, length, data):
'''
return a 2-tuple (data, handled)
where data is understandable by a downstream serializer
(in the case of BST, this would be a bst_parser object),
and handled is a boolean, True if the POST was recognized,
and False if not.
'''
raise NotImplementedError

33
broadview_collector/handlers/bsthandler.py Normal file
View File

@ -0,0 +1,33 @@
# (C) Copyright Broadcom Corporation 2016
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
#
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from broadviewhandlerbase import BroadViewHandlerBase
from broadview_lib.bst.bst_parser import BSTParser
class BroadViewHandler(BroadViewHandlerBase):
def __init__(self):
pass
def handlePOST(self, path, ctype, length, data):
parser = BSTParser()
try:
handled = parser.process(data)
except:
handled = False
print handled
return (parser, handled)
def __repr__(self):
return "BST Handler"

0
broadview_collector/plugins/__init__.py Normal file
View File

21
broadview_collector/plugins/broadviewpublisherbase.py Normal file
View File

@ -0,0 +1,21 @@
# (C) Copyright Broadcom Corporation 2016
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
#
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
class BroadViewPublisherBase(object):
def __init__(self):
pass
def publish(self, data):
raise NotImplementedError

66
broadview_collector/plugins/logpublisher.py Normal file
View File

@ -0,0 +1,66 @@
# (C) Copyright Broadcom Corporation 2016
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
#
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from broadviewpublisherbase import BroadViewPublisherBase
# the Monasca serializer format is suitable for logging output,
# so use it.
from broadview_collector.serializers.bst_to_monasca import BSTToMonasca
import json
from oslo_log import log
import ConfigParser
LOG = log.getLogger(__name__)
class BroadViewPublisher(BroadViewPublisherBase):
def readConfig(self):
try:
cfg = ConfigParser.ConfigParser()
cfg.read("/etc/broadviewcollector.conf")
self._logfile = cfg.get("logging", "file")
except:
LOG.info("log publisher: unable to process log file")
try:
self._f = open(self._logfile, "w+")
except:
LOG.info("log publisher: unable to open log file {}".format(self.__logfile))
def __init__(self):
LOG.info("log publisher: init")
self._f = None
self._logfile = "/tmp/broadview-bstlogging.log"
self.readConfig()
def __del__(self):
if self._f:
self._f.close()
def publish(self, data):
LOG.info("log publisher: publish")
code = 200
success, sdata = BSTToMonasca().serialize(data)
sdata = json.loads(sdata)
if success:
for x in sdata:
print >>self._f, json.dumps(x)
self._f.flush()
else:
code = 500
return code
def __repr__(self):
return "BroadView Log Publisher"

83
broadview_collector/plugins/monascapublisher.py Normal file
View File

@ -0,0 +1,83 @@
# (C) Copyright Broadcom Corporation 2016
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
#
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from broadviewpublisherbase import BroadViewPublisherBase
from monascaclient import client
import monascaclient.exc as exc
from broadview_collector.serializers.bst_to_monasca import BSTToMonasca
import json
import ConfigParser
from oslo_log import log
LOG = log.getLogger(__name__)
class BroadViewPublisher(BroadViewPublisherBase):
def readConfig(self):
try:
bvcfg = ConfigParser.ConfigParser()
bvcfg.read("/etc/broadviewcollector.conf")
self._endpoint = bvcfg.get("monasca", "endpoint")
self._username = bvcfg.get("monasca", "username")
self._password = bvcfg.get("monasca", "password")
self._project_name = bvcfg.get("monasca", "project_name")
self._auth_url = bvcfg.get("monasca", "auth_url")
self._endpoint = bvcfg.get("monasca", "endpoint")
self._api_version = bvcfg.get("monasca", "api_version")
except:
LOG.error("BroadViewPublisher: unable to read configuration")
def __init__(self):
self.readConfig()
try:
self._auth_kwargs = {
'username': self._username,
'password': self._password,
'auth_url': self._auth_url,
'project_name': self._project_name,
}
self._monasca_client = client.Client(self._api_version, \
self._endpoint, \
**self._auth_kwargs)
except:
LOG.error("BroadViewPublisher: failed to parse config")
self._monasca_client = None
def publish(self, data):
code = 500
if self._monasca_client:
code = 200
success, sdata = BSTToMonasca().serialize(data)
sdata = json.loads(sdata)
if success:
for x in sdata:
try:
resp = self._monasca_client.metrics.create(**x)
if not resp.status_code == 200 and not resp.status_code == 204:
code = resp.status_code
break
except exc.HTTPException as he:
LOG.error('HTTPException code=%s message=%s' % (he.code, he.message))
code = he.code
break
return code
def __repr__(self):
return "BroadView Monasca Publisher {_endpoint} {_api_version}".format(**__dict__(self))

0
broadview_collector/serializers/__init__.py Normal file
View File

31
broadview_collector/serializers/broadviewserializerbase.py Normal file
View File

@ -0,0 +1,31 @@
# (C) Copyright Broadcom Corporation 2016
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
#
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
class BroadViewSerializerBase(object):
def __init__(self):
pass
def serialize(self, data):
'''
return a 2-tuple (ret, jsonret)
where coderet is True for success and False for failure,
jsonret is a json string that holds the serialized data.
For example, on success:
return (True, "{\"foo\": \"bar\"}")
'''
raise NotImplementedError

2415
broadview_collector/serializers/bst_to_monasca.py Normal file
View File

File diff suppressed because it is too large Load Diff

0
broadview_collector/tests/__init__.py Normal file
View File

23
broadview_collector/tests/base.py Normal file
View File

@ -0,0 +1,23 @@
# -*- coding: utf-8 -*-
# Copyright 2010-2011 OpenStack Foundation
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from oslotest import base
class TestCase(base.BaseTestCase):
"""Test case base class for all unit tests."""

318
broadview_collector/tests/bst_report.py Normal file
View File

@ -0,0 +1,318 @@
# (C) Copyright Broadcom Corporation 2016
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
#
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import json
import requests
import datetime
import unittest
# Change these to the host and port the collector is listening on
host = "10.14.245.57"
port = 8082
'''
To see list of metrics in Monasca after this test is run (assuming
the Monasca plugin is enabled (see the README.md file)):
$ monasca --os-username mini-mon --os-password password metric-list | grep broadview
To view an individual statistic, use metric-statistics, as this example:
#monasca --os-username mini-mon --os-password password metric-statistics --dimensions "stat=um-share-buffer-count, service-pool=6, port=3" broadview.bst.ingress-port-service-pool MIN "2016-03-01T00:00:00Z"
This program will display the UTC time associated with the metric, which
can be used to determine an appropriate time argument for the
metric-statistics command.
'''
class TestBSTCollector(unittest.TestCase):
def setUp(self):
# convert datetime string to monasca timestamp
print("{} UTC".format(datetime.datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S")))
d = str(datetime.datetime.now()).split(" ")
t = d[1].split(".")[0]
d = "{} - {}".format(d[0], t)
print("Setting timestamp to {}".format(d))
self.bst_report = {
"jsonrpc": "2.0",
"method": "get-bst-report",
"asic-id": "20",
"version": "1",
"time-stamp": d,
"report": [
{
"realm": "device",
"data": 46
}, {
"realm": "ingress-port-priority-group",
"data": [{
"port": "2",
"data": [[5, 45500, 44450]]
}, {
"port": "3",
"data": [[6, 25500, 24450]]
}]
}, {
"realm": "ingress-port-service-pool",
"data": [{
"port": "2",
"data": [[5, 324]]
}, {
"port": "3",
"data": [[6, 366]]
}]
}, {
"realm": "ingress-service-pool",
"data": [[1, 3240], [2, 3660]]
}, {
"realm": "egress-cpu-queue",
"data": [[3, 4566, 0]]
}, {
"realm": "egress-mc-queue",
"data": [[1, "1", 34, 89], [2, "4", 1244, 0], [3, "5", 0, 3]]
}, {
"realm": "egress-port-service-pool",
"data": [{
"port": "2",
"data": [[5, 0, 324, 0]]
}, {
"port": "3",
"data": [[6, 0, 366, 0]]
}]
}, {
"realm": "egress-rqe-queue",
"data": [[2, 3333, 4444], [5, 25, 45]]
}, {
"realm": "egress-service-pool",
"data": [[2, 0, 0, 3240], [3, 3660, 0, 0]]
}, {
"realm": "egress-uc-queue",
"data": [[6, "0", 1111]]
}, {
"realm": "egress-uc-queue-group",
"data": [[6, 2222]]
}]
}
self.bst_report_unknown_method = {
"jsonrpc": "2.0",
"method": "get-foo-report",
"asic-id": "20",
"version": "1",
"time-stamp": d,
"report": [
{
"realm": "device",
"data": 46
}, {
"realm": "ingress-port-priority-group",
"data": [{
"port": "2",
"data": [[5, 45500, 44450]]
}, {
"port": "3",
"data": [[6, 25500, 24450]]
}]
}, {
"realm": "ingress-port-service-pool",
"data": [{
"port": "2",
"data": [[5, 324]]
}, {
"port": "3",
"data": [[6, 366]]
}]
}, {
"realm": "ingress-service-pool",
"data": [[1, 3240], [2, 3660]]
}, {
"realm": "egress-cpu-queue",
"data": [[3, 4566, 0]]
}, {
"realm": "egress-mc-queue",
"data": [[1, "1", 34, 89], [2, "4", 1244, 0], [3, "5", 0, 3]]
}, {
"realm": "egress-port-service-pool",
"data": [{
"port": "2",
"data": [[5, 0, 324, 0]]
}, {
"port": "3",
"data": [[6, 0, 366, 0]]
}]
}, {
"realm": "egress-rqe-queue",
"data": [[2, 3333, 4444], [5, 25, 45]]
}, {
"realm": "egress-service-pool",
"data": [[2, 0, 0, 3240], [3, 3660, 0, 0]]
}, {
"realm": "egress-uc-queue",
"data": [[6, "0", 1111]]
}, {
"realm": "egress-uc-queue-group",
"data": [[6, 2222]]
}]
}
self.bst_report_unknown_realm = {
"jsonrpc": "2.0",
"method": "get-bst-report",
"asic-id": "20",
"version": "1",
"time-stamp": d,
"report": [
{
"realm": "mustard",
"data": 46
}]
}
self.bst_report_bad_timestamp = {
"jsonrpc": "2.0",
"method": "get-bst-report",
"asic-id": "20",
"version": "1",
"time-stamp": "xxxxxx ",
"report": [
{
"realm": "device",
"data": 46
}, {
"realm": "ingress-port-priority-group",
"data": [{
"port": "2",
"data": [[5, 45500, 44450]]
}, {
"port": "3",
"data": [[6, 25500, 24450]]
}]
}, {
"realm": "ingress-port-service-pool",
"data": [{
"port": "2",
"data": [[5, 324]]
}, {
"port": "3",
"data": [[6, 366]]
}]
}, {
"realm": "ingress-service-pool",
"data": [[1, 3240], [2, 3660]]
}, {
"realm": "egress-cpu-queue",
"data": [[3, 4566, 0]]
}, {
"realm": "egress-mc-queue",
"data": [[1, "1", 34, 89], [2, "4", 1244, 0], [3, "5", 0, 3]]
}, {
"realm": "egress-port-service-pool",
"data": [{
"port": "2",
"data": [[5, 0, 324, 0]]
}, {
"port": "3",
"data": [[6, 0, 366, 0]]
}]
}, {
"realm": "egress-rqe-queue",
"data": [[2, 3333, 4444], [5, 25, 45]]
}, {
"realm": "egress-service-pool",
"data": [[2, 0, 0, 3240], [3, 3660, 0, 0]]
}, {
"realm": "egress-uc-queue",
"data": [[6, "0", 1111]]
}, {
"realm": "egress-uc-queue-group",
"data": [[6, 2222]]
}]
}
self.bst_report_report_dict = {
"jsonrpc": "2.0",
"method": "get-bst-report",
"asic-id": "20",
"version": "1",
"time-stamp": d,
"report": {}
}
self.bst_report_empty_report = {
"jsonrpc": "2.0",
"method": "get-bst-report",
"asic-id": "20",
"version": "1",
"time-stamp": d,
"report": []
}
self.bst_report_missing_report = {
"jsonrpc": "2.0",
"method": "get-bst-report",
"asic-id": "20",
"version": "1",
"time-stamp": d,
}
def test_good_bst(self):
j = json.dumps(self.bst_report)
r = requests.post('http://{}:{}'.format(host, port), json=j)
self.assertEqual(r.status_code, 200)
def test_unknown_method_bst(self):
j = json.dumps(self.bst_report_unknown_method)
r = requests.post('http://{}:{}'.format(host, port), json=j)
self.assertEqual(r.status_code, 404)
def test_unknown_realm(self):
j = json.dumps(self.bst_report_unknown_realm)
r = requests.post('http://{}:{}'.format(host, port), json=j)
self.assertEqual(r.status_code, 404)
def test_bad_timestamp(self):
j = json.dumps(self.bst_report_bad_timestamp)
r = requests.post('http://{}:{}'.format(host, port), json=j)
self.assertEqual(r.status_code, 404)
def test_report_dict(self):
j = json.dumps(self.bst_report_report_dict)
r = requests.post('http://{}:{}'.format(host, port), json=j)
self.assertEqual(r.status_code, 404)
def test_empty_report(self):
j = json.dumps(self.bst_report_empty_report)
r = requests.post('http://{}:{}'.format(host, port), json=j)
self.assertEqual(r.status_code, 404)
def test_missing_report(self):
j = json.dumps(self.bst_report_missing_report)
r = requests.post('http://{}:{}'.format(host, port), json=j)
self.assertEqual(r.status_code, 404)
if __name__ == "__main__":
unittest.main()

28
broadview_collector/tests/test_broadview_collector.py Normal file
View File

@ -0,0 +1,28 @@
# -*- coding: utf-8 -*-
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""
test_broadview_collector
----------------------------------
Tests for `broadview_collector` module.
"""
from broadview_collector.tests import base
class TestBroadview_collector(base.TestCase):
def test_something(self):
pass

153
devstack/README.txt Normal file
View File

@ -0,0 +1,153 @@
# (C) Copyright Broadcom Corporation 2016
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
#
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
The broadview-collector DevStack plugin currently only works on Ubuntu 14.04
(Trusty). It may work elsewhere, we just haven't tried :-)
Directions for installing an running Devstack can be found here:
http://docs.openstack.org/developer/devstack/
To run broadview-collector in DevStack, perform the following steps:
1. Clone the DevStack repo.
git clone https://git.openstack.org/openstack-dev/devstack
2. Add the following to the end of the DevStack local.conf file in the
root of the devstack directory. You may need to create the local.conf if
it does not already exist.
# The following will enable broadview-collector
enable_plugin broadview-collector git://github.com/openstack/broadview-collector
4. Run './stack.sh' from the root of the devstack directory.
The default configuration enables the logfile publisher plugin. You can
view/modify the configuration by editing /etc/broadviewcolector.conf and
then restarting the collector.
Minimal Devstack local.conf
---------------------------
The following minimal local.conf should be all that is needed to bring
up the BroadView Collector:
[[local|localrc]]
MYSQL_PASSWORD=secretmysql
DATABASE_PASSWORD=secretdatabase
RABBIT_PASSWORD=secretrabbit
ADMIN_PASSWORD=secretadmin
SERVICE_PASSWORD=secretservice
SERVICE_TOKEN=111222333444
LOGFILE=$DEST/logs/stack.sh.log
LOGDIR=$DEST/logs
LOG_COLOR=False
disable_all_services
enable_service rabbit mysql key tempest
enable_plugin broadview-collector git://git.openstack.org/openstack/broadview-collector
Restarting the Collector
------------------------
To restart the collector:
$ sudo service broadview_collector restart
Stopping the Collector
---------------------
To stop the collector:
$ sudo service broadview_collector stop
Starting the Collector
----------------------
To start the collector:
$ sudo service broadview_collector start
Exercising The Collector
------------------------
A simple python script in broadview_collector/tests named bst_report.py can
be used to simulate data incoming from a BroadView agent. Combined with
configuration that enables the log publisher plugin (or any other plugin
for that matter), you can validate that the installation and configuration
is valid. To run it, edit the script by changing the host and port that
the collector is listening on:
# Change these to the host and port the collector is listening on
host = "172.16.170.175"
port = 8082
And then run the script:
$ python bst_report
You can then look at the configured plugin publishing targets to validate
that the data was published as expected. The script sends the data to the
collector and looks at the return code to verify the response is 200 (OK).
The Python unittest module is used to validate the return code and determine
a pass or fail status.
Enabling Monasca
----------------
Follow these steps:
1. Follow the instructions for devstack in the Monasca API project. The
README.txt file in https://github.com/openstack/monasca-api has all the
details.
2. Add the following the end of their local.conf:
enable_plugin broadview-collector git://git.openstack.org/openstack/broadview-collector
3. Run devstack as usual with ./stack.sh
4. Stop the collector
$ sudo service broadview_collector stop
5. Edit /etc/broadviewcollector.conf and add monascapublisher as a publisher
in the plugins section, e.g.,
[plugins]
# comma separated list of plugin modules
# publishers receive JSON encoded reports and write them somewhere
publishers: logpublisher, monascapublisher
6. Restart the broadview collector:
$ sudo service broadview_collector start
7. Send some metrics using the bst simulator bst_report.py
8. View the metric list in monasca
$ monasca --os-username mini-mon --os-password password metric-list
You should see some bst statistics in the report that is dislayed.
$

22
devstack/files/broadview-collector/broadview_collector.conf Normal file
View File

@ -0,0 +1,22 @@
# (C) Copyright Broadcom Corporation 2016
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
#
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Startup script for the BroadView collector
description "BroadView Collector Python app"
start on runlevel [2345]
console log
respawn
exec /opt/broadview-collector/bin/bvcollect.py

25
devstack/files/env.sh Normal file
View File

@ -0,0 +1,25 @@
#
# (C) Copyright 2015 Hewlett Packard Enterprise Development Company LP
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Environment variables for use with broadview collector. This script
# allows us to run under a virtual environment
#. /opt/broadview-collector/bin/activate
#export OS_USERNAME=mini-mon
#export OS_PASSWORD=password
#export OS_PROJECT_NAME=mini-mon
#export OS_AUTH_URL=http://127.0.0.1:35357/v3/

308
devstack/plugin.sh Normal file
View File

@ -0,0 +1,308 @@
# (C) Copyright Broadcom Corporation 2016
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
#
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# BroadView Collector DevStack plugin
#
# Install and start BroadView Collector service in devstack
#
# To enable broadview-collector in devstack add an entry to local.conf that
# looks like
#
# [[local|localrc]]
# enable_plugin broadview-collector git://github.com/openstack/broadview-collector
#
# Save trace setting
XTRACE=$(set +o | grep xtrace)
set -o xtrace
ERREXIT=$(set +o | grep errexit)
set -o errexit
# Determine if we are running in devstack-gate or devstack.
if [[ $DEST ]]; then
# We are running in devstack-gate.
export BROADVIEW_COLLECTOR_BASE=${BROADVIEW_COLLECTOR_BASE:-"${DEST}"}
else
# We are running in devstack.
export BROADVIEW_COLLECTOR_BASE=${BROADVIEW_COLLECTOR_BASE:-"/opt/stack"}
fi
function pre_install_broadview_collector {
:
}
function install_broadview_collector {
install_broadview_collector_env
install_git
install_broadview_lib
install_broadview_collector_python
}
function post_config_broadview_collector {
:
}
function unstack_broadview_collector {
sudo service broadview_collector stop || true
}
function clean_broadview_collector {
set +o errexit
unstack_broadview_collector
# XXX this comnes later...
# if is_service_enabled horizon; then
#
# clean_broadview_horizon_ui
#
# fi
clean_broadview_collector_python
clean_broadview_lib
clean_broadview_collector_env
#Restore errexit
set -o errexit
}
function install_broadview_collector_env {
echo_summary "Install BroadView Collector Virtual Environment"
sudo groupadd --system broadview_collector || true
sudo mkdir -p /opt/broadview-collector || true
sudo chown $STACK_USER:broadview_collector /opt/broadview-collector
}
function clean_broadview_collector_env {
echo_summary "Clean BroadView Collector Virtual Environment"
sudo rm -rf /opt/broadview-collector
sudo groupdel broadview_collector
}
function install_git {
echo_summary "Install git"
sudo apt-get -y install git
}
function install_broadview_lib {
echo_summary "Install broadview_lib"
if [[ ! -d /opt/broadview-lib ]]; then
sudo git clone https://git.openstack.org/openstack/broadview-lib.git /opt/broadview-lib
fi
(cd /opt/broadview-lib; sudo python setup.py sdist)
BROADVIEW_LIB_SRC_DIST=$(ls -td /opt/broadview-lib/dist/broadview-lib-*.tar.gz)
pip_install $BROADVIEW_LIB_SRC_DIST
}
function clean_broadview_lib {
echo_summary "Clean broadview_lib"
}
function install_broadview_collector_python {
echo_summary "Install broadview_collector_python"
sudo mkdir -p /opt/broadview-collector/bin
sudo chown $STACK_USER:broadview_collector /opt/broadview-collector
sudo chown $STACK_USER:broadview_collector /opt/broadview-collector/bin
(cd "${BROADVIEW_COLLECTOR_BASE}"/broadview-collector; sudo python setup.py install)
sudo cp -f "${BROADVIEW_COLLECTOR_BASE}"/broadview-collector/broadview_collector/bin/bvcollect.py /opt/broadview-collector/bin
sudo chown root:broadview_collector /opt/broadview-collector/bin/bvcollect.py
sudo chmod 755 /opt/broadview-collector/bin/bvcollect.py
sudo cp -f "${BROADVIEW_COLLECTOR_BASE}"/broadview-collector/devstack/files/broadview-collector/broadview_collector.conf /etc/init/broadview_collector.conf
sudo chown root:broadview_collector /etc/init/broadview_collector.conf
sudo chmod 0755 /etc/init/broadview_collector.conf
sudo mkdir -p /var/log/broadview-collector || true
sudo chown root:broadview_collector /var/log/broadview-collector
sudo chmod 0755 /var/log/broadview-collector
sudo cp -f "${BROADVIEW_COLLECTOR_BASE}"/broadview-collector/broadview_collector/config/broadviewcollector.conf /etc/broadviewcollector.conf
sudo chown root:broadview_collector /etc/broadviewcollector.conf
sudo chmod 0755 /etc/broadviewcollector.conf
if [[ ${SERVICE_HOST} ]]; then
sudo sed -i "s/127\.0\.0\.1/${SERVICE_HOST}/g" /etc/broadviewcollector.conf
fi
if [[ ${SERVICE_HOST} ]]; then
# set broadview_collector server listening ip address
sudo sed -i "s/host = 127\.0\.0\.1/host = ${SERVICE_HOST}/g" /etc/broadviewcollector.conf
fi
sudo start broadview_collector || sudo restart broadview_collector
}
function clean_broadview_collector_python {
echo_summary "Clean broadview_collector_python"
sudo rm /etc/init/broadview_collector.conf
sudo rm /etc/broadviewcollector.conf
sudo rm -rf /var/log/broadview-collector
sudo rm /var/log/upstart/broadview_collector.log*
sudo rm -rf /opt/broadview-collector
}
function install_broadview_collector_horizon_ui {
echo_summary "Install BroadView Collector Horizon UI"
sudo mkdir -p /opt/broadview_collector-horizon-ui || true
sudo chown $STACK_USER:broadview_collector /opt/broadview_collector-horizon-ui
(cd /opt/broadview_collector-horizon-ui ; virtualenv .)
(cd /opt/broadview_collector-horizon-ui ; sudo -H ./bin/pip install broadview_collector-ui)
sudo ln -sf /opt/broadview_collector-horizon-ui/lib/python2.7/site-packages/monitoring/enabled/_50_admin_add_monitoring_panel.py "${BROADVIEW_COLLECTOR_BASE}"/horizon/openstack_dashboard/local/enabled/_50_admin_add_monitoring_panel.py
sudo ln -sf /opt/broadview_collector-horizon-ui/lib/python2.7/site-packages/monitoring/static/monitoring "${BROADVIEW_COLLECTOR_BASE}"/horizon/monitoring
sudo PYTHONPATH=/opt/broadview_collector-horizon-ui/lib/python2.7/site-packages python "${BROADVIEW_COLLECTOR_BASE}"/horizon/manage.py compress --force
sudo service apache2 restart
}
function extra_broadview_collector {
:
}
function clean_broadview_collector_horizon_ui {
echo_summary "Clean BroadView Collector Horizon UI"
sudo rm -f "${BROADVIEW_COLLECTOR_BASE}"/horizon/openstack_dashboard/local/enabled/_50_admin_add_monitoring_panel.py
sudo rm -f "${BROADVIEW_COLLECTOR_BASE}"/horizon/monitoring
sudo rm -rf /opt/broadview_collector-horizon-ui
}
# Allows this script to be called directly outside of
# the devstack infrastructure code. Uncomment to use.
#if [[ $(type -t is_service_enabled) != 'function' ]]; then
#
# function is_service_enabled {
#
# return 0
#
# }
#fi
#if [[ $(type -t echo_summary) != 'function' ]]; then
#
# function echo_summary {
#
# echo "$*"
#
# }
#
#fi
# check for service enabled
if is_service_enabled broadview-collector; then
if [[ "$1" == "stack" && "$2" == "pre-install" ]]; then
# Set up system services
echo_summary "Configuring BroadView Collector system services"
pre_install_broadview_collector
elif [[ "$1" == "stack" && "$2" == "install" ]]; then
# Perform installation of service source
echo_summary "Installing BroadView Collector"
install_broadview_collector
elif [[ "$1" == "stack" && "$2" == "post-config" ]]; then
# Configure after the other layer 1 and 2 services have been configured
echo_summary "Configuring BroadView Collector"
post_config_broadview_collector
elif [[ "$1" == "stack" && "$2" == "extra" ]]; then
# Initialize and start the BroadView Collector service
echo_summary "Initializing BroadView Collector"
extra_broadview_collector
fi
if [[ "$1" == "unstack" ]]; then
# Shut down BroadView Collector services
echo_summary "Unstacking BroadView Collector"
unstack_broadview_collector
fi
if [[ "$1" == "clean" ]]; then
# Remove state and transient data
# Remember clean.sh first calls unstack.sh
echo_summary "Cleaning BroadView Collector"
clean_broadview_collector
fi
fi
#Restore errexit
$ERREXIT
# Restore xtrace
$XTRACE

0
devstack/post_test_hook.sh Normal file
View File

0
devstack/pre_test_hook.sh Normal file
View File

15
devstack/settings Normal file
View File

@ -0,0 +1,15 @@
# (C) Copyright Broadcom Corporation 2016
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
#
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
enable_service broadview-collector

75
doc/source/conf.py Normal file
View File

@ -0,0 +1,75 @@
# -*- coding: utf-8 -*-
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import sys
sys.path.insert(0, os.path.abspath('../..'))
# -- General configuration ----------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = [
'sphinx.ext.autodoc',
#'sphinx.ext.intersphinx',
'oslosphinx'
]
# autodoc generation is a bit aggressive and a nuisance when doing heavy
# text edit cycles.
# execute "export SPHINX_DEBUG=1" in your terminal to disable
# The suffix of source filenames.
source_suffix = '.rst'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'broadview-collector'
copyright = u'2013, OpenStack Foundation'
# If true, '()' will be appended to :func: etc. cross-reference text.
add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
add_module_names = True
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# -- Options for HTML output --------------------------------------------------
# The theme to use for HTML and HTML Help pages. Major themes that come with
# Sphinx are currently 'default' and 'sphinxdoc'.
# html_theme_path = ["."]
# html_theme = '_theme'
# html_static_path = ['static']
# Output file base name for HTML help builder.
htmlhelp_basename = '%sdoc' % project
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass
# [howto/manual]).
latex_documents = [
('index',
'%s.tex' % project,
u'%s Documentation' % project,
u'OpenStack Foundation', 'manual'),
]
# Example configuration for intersphinx: refer to the Python standard library.
#intersphinx_mapping = {'http://docs.python.org/': None}

4
doc/source/contributing.rst Normal file
View File

@ -0,0 +1,4 @@
============
Contributing
============
.. include:: ../../CONTRIBUTING.rst

25
doc/source/index.rst Normal file
View File

@ -0,0 +1,25 @@
.. broadview-collector documentation master file, created by
sphinx-quickstart on Tue Jul 9 22:26:36 2013.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to broadview-collector's documentation!
========================================================
Contents:
.. toctree::
:maxdepth: 2
readme
installation
usage
contributing
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

6
doc/source/installation.rst Normal file
View File

@ -0,0 +1,6 @@
============
Installation
============
Refer to README.md for directions on installing this software.

1
doc/source/readme.rst Normal file
View File

@ -0,0 +1 @@
.. include:: ../../README.md

6
doc/source/usage.rst Normal file
View File

@ -0,0 +1,6 @@
========
Usage
========
Refer to README.md for documentation on how to use broadview-collector

5
requirements.txt Normal file
View File

@ -0,0 +1,5 @@
# The order of packages is significant, because pip processes them in the order
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.
pbr>=1.6

43
setup.cfg Normal file
View File

@ -0,0 +1,43 @@
[metadata]
name = broadview-collector
summary = broadview-collector is a plugin-based service that collects and publishes network underlay data to services such as Monasca.
description-file =
README.md
author = OpenStack
author-email = openstack-dev@lists.openstack.org
home-page = http://www.openstack.org/
classifier =
Environment :: OpenStack
Intended Audience :: Information Technology
Intended Audience :: System Administrators
License :: OSI Approved :: Apache Software License
Operating System :: POSIX :: Linux
Programming Language :: Python
Programming Language :: Python :: 2
Programming Language :: Python :: 2.7
[files]
packages =
broadview_collector
[build_sphinx]
source-dir = doc/source
build-dir = doc/build
all_files = 1
[upload_sphinx]
upload-dir = doc/build/html
[compile_catalog]
directory = broadview_collector/locale
domain = broadview_collector
[update_catalog]
domain = broadview_collector
output_dir = broadview_collector/locale
input_file = broadview_collector/locale/broadview_collector.pot
[extract_messages]
keywords = _ gettext ngettext l_ lazy_gettext
mapping_file = babel.cfg
output_file = broadview_collector/locale/broadview_collector.pot

29
setup.py Normal file
View File

@ -0,0 +1,29 @@
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# THIS FILE IS MANAGED BY THE GLOBAL REQUIREMENTS REPO - DO NOT EDIT
import setuptools
# In python < 2.7.4, a lazy loading of package `pbr` will break
# setuptools if some other modules registered functions in `atexit`.
# solution from: http://bugs.python.org/issue15881#msg170215
try:
import multiprocessing # noqa
except ImportError:
pass
setuptools.setup(
setup_requires=['pbr'],
pbr=True)

14
test-requirements.txt Normal file
View File

@ -0,0 +1,14 @@
# The order of packages is significant, because pip processes them in the order
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.
hacking<0.11,>=0.10.0
coverage>=3.6
python-subunit>=0.0.18
sphinx!=1.2.0,!=1.3b1,<1.3,>=1.1.2
oslosphinx>=2.5.0 # Apache-2.0
oslotest>=1.10.0 # Apache-2.0
testrepository>=0.0.18
testscenarios>=0.4
testtools>=1.4.0

60
tox.ini Normal file
View File

@ -0,0 +1,60 @@
[tox]
minversion = 2.0
envlist = py34-constraints,py27-constraints,pypy-constraints,pep8-constraints
skipsdist = True
[testenv]
usedevelop = True
install_command =
constraints: {[testenv:common-constraints]install_command}
pip install -U {opts} {packages}
setenv =
VIRTUAL_ENV={envdir}
deps = -r{toxinidir}/test-requirements.txt
commands = python setup.py test --slowest --testr-args='{posargs}'
[testenv:common-constraints]
install_command = pip install -c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt} {opts} {packages}
[testenv:pep8]
commands = flake8 {posargs}
[testenv:pep8-constraints]
install_command = {[testenv:common-constraints]install_command}
commands = flake8 {posargs}
[testenv:venv]
commands = {posargs}
[testenv:venv-constraints]
install_command = {[testenv:common-constraints]install_command}
commands = {posargs}
[testenv:cover]
commands = python setup.py test --coverage --testr-args='{posargs}'
[testenv:cover-constraints]
install_command = {[testenv:common-constraints]install_command}
commands = python setup.py test --coverage --testr-args='{posargs}'
[testenv:docs]
commands = python setup.py build_sphinx
[testenv:docs-constraints]
install_command = {[testenv:common-constraints]install_command}
commands = python setup.py build_sphinx
[testenv:debug]
commands = oslo_debug_helper {posargs}
[testenv:debug-constraints]
install_command = {[testenv:common-constraints]install_command}
commands = oslo_debug_helper {posargs}
[flake8]
# E123, E125 skipped as they are invalid PEP-8.
show-source = True
ignore = E123,E125
builtins = _
exclude=.venv,.git,.tox,dist,doc,*openstack/common*,*lib/python*,*egg,build