979af9b1c4
Currently duplication part from configuration.rst file was removed (patch https://review.openstack.org/#/c/133035) but links in events.rst are still refering to removed part of docs. This patch improves links. Closes-bug: #1401444 Change-Id: I99037f99f3fcb40eb27b56e1581bf60f32ff3051
270 lines
11 KiB
ReStructuredText
270 lines
11 KiB
ReStructuredText
..
|
|
Copyright 2013 Rackspace Hosting.
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may
|
|
not use this file except in compliance with the License. You may obtain
|
|
a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
|
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
|
License for the specific language governing permissions and limitations
|
|
under the License.
|
|
|
|
=============================
|
|
Events and Event Processing
|
|
=============================
|
|
|
|
Events vs. Samples
|
|
==================
|
|
|
|
In addition to Meters, and related Sample data, Ceilometer can also process
|
|
Events. While a Sample represents a single numeric datapoint, driving a Meter
|
|
that represents the changes in that value over time, an Event represents the
|
|
state of an object in an OpenStack service (such as an Instance in Nova, or
|
|
an Image in Glance) at a point in time when something of interest has occurred.
|
|
This can include non-numeric data, such as an instance's flavor, or network
|
|
address.
|
|
|
|
In general, Events let you know when something has changed about an
|
|
object in an OpenStack system, such as the resize of an instance, or creation
|
|
of an image.
|
|
|
|
While Samples can be relatively cheap (small),
|
|
disposable (losing an individual sample datapoint won't matter much),
|
|
and fast, Events are larger, more informative, and should be handled more
|
|
consistently (you do not want to lose one).
|
|
|
|
Event Structure
|
|
===============
|
|
|
|
To facilitate downstream processing (billing and/or aggregation), a
|
|
:doc:`minimum required data set and format <format>` has been defined for
|
|
services, however events generally contain the following information:
|
|
|
|
|
|
event_type
|
|
A dotted string defining what event occurred, such as "compute.instance.resize.start"
|
|
|
|
message_id
|
|
A UUID for this event.
|
|
|
|
generated
|
|
A timestamp of when the event occurred on the source system.
|
|
|
|
traits
|
|
A flat mapping of key-value pairs.
|
|
The event's Traits contain most of the details of the event. Traits are
|
|
typed, and can be strings, ints, floats, or datetimes.
|
|
|
|
|
|
Events from Notifications
|
|
=========================
|
|
|
|
Events are primarily created via the notifications system in OpenStack.
|
|
OpenStack systems, such as Nova, Glance, Neutron, etc. will emit
|
|
notifications in a JSON format to message queue when some notable action is
|
|
taken by that system. Ceilometer will consume such notifications from the
|
|
message queue, and process them.
|
|
|
|
The general philosophy of notifications in OpenStack is to emit any and all
|
|
data someone might need, and let the consumer filter out what they are not
|
|
interested in. In order to make processing simpler and more efficient,
|
|
the notifications are stored and processed within Ceilometer as Events.
|
|
The notification payload, which can be an arbitrarily complex JSON data
|
|
structure, is converted to a flat set of key-value pairs known as Traits.
|
|
This conversion is specified by a config file, so that only the specific
|
|
fields within the notification that are actually needed for processing the
|
|
event will have to be stored as Traits.
|
|
|
|
Note that the Event format is meant for efficient processing and querying,
|
|
there are other means available for archiving notifications (i.e. for audit
|
|
purposes, etc), possibly to different datastores.
|
|
|
|
Converting Notifications to Events
|
|
----------------------------------
|
|
|
|
In order to make it easier to allow users to extract what they need,
|
|
the conversion from Notifications to Events is driven by a
|
|
configuration file (specified by the flag definitions_cfg_file_ in
|
|
ceilometer.conf).
|
|
|
|
This includes descriptions of how to map fields in the notification body
|
|
to Traits, and optional plugins for doing any programmatic translations
|
|
(splitting a string, forcing case, etc.)
|
|
|
|
The mapping of notifications to events is defined per event_type, which
|
|
can be wildcarded. Traits are added to events if the corresponding fields
|
|
in the notification exist and are non-null. (As a special case, an empty
|
|
string is considered null for non-text traits. This is due to some openstack
|
|
projects (mostly Nova) using empty string for null dates.)
|
|
|
|
If the definitions file is not present, a warning will be logged, but an empty
|
|
set of definitions will be assumed. By default, any notifications that
|
|
do not have an event definition in the definitions file for them will be
|
|
converted to events with a set of minimal, default traits. This can be
|
|
changed by setting the flag drop_unmatched_notifications_ in the
|
|
ceilometer.conf file. If this is set to True, then any notifications
|
|
that don't have events defined for them in the file will be dropped.
|
|
This can be what you want, the notification system is quite chatty by design
|
|
(notifications philosophy is "tell us everything, we'll ignore what we don't
|
|
need"), so you may want to ignore the noisier ones if you don't use them.
|
|
|
|
.. _definitions_cfg_file: http://docs.openstack.org/trunk/config-reference/content/ch_configuring-openstack-telemetry.html
|
|
.. _drop_unmatched_notifications: http://docs.openstack.org/trunk/config-reference/content/ch_configuring-openstack-telemetry.html
|
|
|
|
There is a set of default traits (all are TEXT type) that will be added to
|
|
all events if the notification has the relevant data:
|
|
|
|
* service: (All notifications should have this) notification's publisher
|
|
* tenant_id
|
|
* request_id
|
|
|
|
These do not have to be specified in the event definition, they are
|
|
automatically added, but their definitions can be overridden for a given
|
|
event_type.
|
|
|
|
Definitions file format
|
|
-----------------------
|
|
|
|
The event definitions file is in YAML format. It consists of a list of event
|
|
definitions, which are mappings. Order is significant, the list of definitions
|
|
is scanned in *reverse* order (last definition in the file to the first),
|
|
to find a definition which matches the notification's event_type. That
|
|
definition will be used to generate the Event. The reverse ordering is done
|
|
because it is common to want to have a more general wildcarded definition
|
|
(such as "compute.instance.*" ) with a set of traits common to all of those
|
|
events, with a few more specific event definitions (like
|
|
"compute.instance.exists") afterward that have all of the above traits, plus
|
|
a few more. This lets you put the general definition first, followed by the
|
|
specific ones, and use YAML mapping include syntax to avoid copying all of the
|
|
trait definitions.
|
|
|
|
Event Definitions
|
|
-----------------
|
|
|
|
Each event definition is a mapping with two keys (both required):
|
|
|
|
event_type
|
|
This is a list (or a string, which will be taken as a 1 element
|
|
list) of event_types this definition will handle. These can be
|
|
wildcarded with unix shell glob syntax. An exclusion listing
|
|
(starting with a '!') will exclude any types listed from matching.
|
|
If ONLY exclusions are listed, the definition will match anything
|
|
not matching the exclusions.
|
|
traits
|
|
This is a mapping, the keys are the trait names, and the values are
|
|
trait definitions.
|
|
|
|
Trait Definitions
|
|
-----------------
|
|
|
|
Each trait definition is a mapping with the following keys:
|
|
|
|
type
|
|
(optional) The data type for this trait. (as a string). Valid
|
|
options are: *text*, *int*, *float*, and *datetime*.
|
|
defaults to *text* if not specified.
|
|
fields
|
|
A path specification for the field(s) in the notification you wish
|
|
to extract for this trait. Specifications can be written to match
|
|
multiple possible fields, the value for the trait will be derived
|
|
from the matching fields that exist and have a non-null values in
|
|
the notification. By default the value will be the first such field.
|
|
(plugins can alter that, if they wish). This is normally a string,
|
|
but, for convenience, it can be specified as a list of
|
|
specifications, which will match the fields for all of them. (See
|
|
`Field Path Specifications`_ for more info on this syntax.)
|
|
plugin
|
|
(optional) This is a mapping (For convenience, this value can also
|
|
be specified as a string, which is interpreted as the name of a
|
|
plugin to be loaded with no parameters) with the following keys
|
|
|
|
name
|
|
(string) name of a plugin to load
|
|
|
|
parameters
|
|
(optional) Mapping of keyword arguments to pass to the plugin on
|
|
initialization. (See documentation on each plugin to see what
|
|
arguments it accepts.)
|
|
|
|
Field Path Specifications
|
|
-------------------------
|
|
|
|
The path specifications define which fields in the JSON notification
|
|
body are extracted to provide the value for a given trait. The paths
|
|
can be specified with a dot syntax (e.g. "payload.host"). Square
|
|
bracket syntax (e.g. "payload[host]") is also supported. In either
|
|
case, if the key for the field you are looking for contains special
|
|
characters, like '.', it will need to be quoted (with double or single
|
|
quotes) like so:
|
|
|
|
payload.image_meta.'org.openstack__1__architecture'
|
|
|
|
The syntax used for the field specification is a variant of JSONPath,
|
|
and is fairly flexible. (see: https://github.com/kennknowles/python-jsonpath-rw for more info)
|
|
|
|
Example Definitions file
|
|
------------------------
|
|
|
|
::
|
|
|
|
---
|
|
- event_type: compute.instance.*
|
|
traits: &instance_traits
|
|
user_id:
|
|
fields: payload.user_id
|
|
instance_id:
|
|
fields: payload.instance_id
|
|
host:
|
|
fields: publisher_id
|
|
plugin:
|
|
name: split
|
|
parameters:
|
|
segment: 1
|
|
max_split: 1
|
|
service_name:
|
|
fields: publisher_id
|
|
plugin: split
|
|
instance_type_id:
|
|
type: int
|
|
fields: payload.instance_type_id
|
|
os_architecture:
|
|
fields: payload.image_meta.'org.openstack__1__architecture'
|
|
launched_at:
|
|
type: datetime
|
|
fields: payload.launched_at
|
|
deleted_at:
|
|
type: datetime
|
|
fields: payload.deleted_at
|
|
- event_type:
|
|
- compute.instance.exists
|
|
- compute.instance.update
|
|
traits:
|
|
<<: *instance_traits
|
|
audit_period_beginning:
|
|
type: datetime
|
|
fields: payload.audit_period_beginning
|
|
audit_period_ending:
|
|
type: datetime
|
|
fields: payload.audit_period_ending
|
|
|
|
Trait plugins
|
|
-------------
|
|
|
|
Trait plugins can be used to do simple programmatic conversions on the value in
|
|
a notification field, like splitting a string, lowercasing a value, converting
|
|
a screwball date into ISO format, or the like. They are initialized with the
|
|
parameters from the trait definition, if any, which can customize their
|
|
behavior for a given trait. They are called with a list of all matching fields
|
|
from the notification, so they can derive a value from multiple fields. The
|
|
plugin will be called even if there is no fields found matching the field
|
|
path(s), this lets a plugin set a default value, if needed. A plugin can also
|
|
reject a value by returning *None*, which will cause the trait not to be
|
|
added. If the plugin returns anything other than *None*, the trait's value
|
|
will be set from whatever the plugin returned (coerced to the appropriate type
|
|
for the trait).
|
|
|