diff --git a/doc/source/index.rst b/doc/source/index.rst index 0cf0d6c6..77beebf2 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -12,6 +12,7 @@ logging (like resource id's etc). installation usage + journal migration opts configfiles/index diff --git a/doc/source/journal.rst b/doc/source/journal.rst new file mode 100644 index 00000000..997275ae --- /dev/null +++ b/doc/source/journal.rst @@ -0,0 +1,149 @@ +========================= + Systemd Journal Support +========================= + +One of the newer features in oslo.log is the ability to integrate with +the systemd journal service (journald) natively on newer Linux +systems. When using native journald support, additional metadata will +be logged on each log message in addition to the message itself, which +can later be used to do some interesting searching through your logs. + +Enabling +======== + +In order to enable the support you must have Python bindings for +systemd installed. On Red Hat based systems, run:: + + yum install systemd-python + +On Ubuntu/Debian based systems, run:: + + apt install python-systemd + +If there is no native package for your distribution, or you are +running in a virtualenv, you can install with pip.:: + + pip install systemd-python + +.. note:: + + There are also many non official systemd python modules on pypi, + with confusingly close names. Make sure you install `systemd-python + `_. + +After the package is installed, you must enable journald support +manually in all services that will be using it. Add the following to +the config files for all services: + +.. code-block:: ini + + [DEFAULT] + use_journal = True + +In all relevant config files. + +Extra Metadata +============== + +Journald supports the concept of adding structured metadata in +addition to the log message in question. This makes it much easier to +take the output of journald and push it into other logging systems +like Elastic Search, without needing to regex guess relevant data. It +also allows you to search the journal by these fields using +``journalctl``. + +We use this facility to add our own structured information, if it is +known at the time of logging the message. + +CODE_FILE=, CODE_LINE=, CODE_FUNC= + + The code location generating this message, if known. Contains the + source filename, the line number and the function name. (This is + the same as systemd uses) + +LOGGER_NAME= + + The name of the python logger that emitted the log + message. Very often this is the module where the log message was + emitted from. + +LOGGER_LEVEL= + + The name of the python logging level, which allows seeing all + 'ERROR' messages very easily without remembering how they are + translated to syslog priorities. + +SYSLOG_IDENTIFIER= + + The binary name identified for syslog compatibility. It will be the + basename of the process that emits the log messages + (e.g. ``nova-api``, ``neutron-l3-agent``) + +PRIORITY= + + The syslog priority (based on LOGGER_LEVEL), which allows syslog + style filtering of messages based on their priority (an + openstack.err log file for instance). + +REQUEST_ID= + + Most OpenStack services generate a unique ``request-id`` on every + REST API call, which is then passed between it's sub services as + that request is handled. For example, this can be very useful in + tracking the build of a nova server from the initial HTTP POST to + final VM create. + +PROJECT_ID=, PROJECT_NAME=, USER_ID=, USER_NAME= + + The keystone known user and project information about the + requestor. Both the id and name are provided for easier + searching. This can be used to understand when particular users or + projects are reporting issues in the environment. + + +Additional fields may be added over time. It is unlikely that fields +will be removed, but if so they will be deprecated for one release +cycle before that happens. + + +Using Journalctl +================ + +Because systemd is relatively new in the Linux ecosystem, it's worth +noting how one can effectively use journal control. + +If you want to follow all the journal logs you would do so with:: + + journalctl -f + +That's going to be nearly everything on your system, which you will +probably find overwhelming. You can limit this to a smaller number of +things using the ``SYSLOG_IDENTIFIER=``:: + + journalctl -f SYSLOG_IDENTIFIER=nova-compute SYSLOG_IDENTIFIER=neutron-l3-agent + +Specifying a query parameter multiple times defaults to an ``OR`` +operation, so that will show either nova-compute or neutron-l3-agent +logs. + +You can also query by request id to see the entire flow of a REST +call:: + + journalctl REQUEST_ID=req-b1903300-77a8-401d-984c-8e7d17e4a15f + + +References +========== + +- A complete list of the systemd journal fields is here, it is worth + making yourself familiar with them - + https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html + +- The complete journalctl manual is worth reading, especially the + ``-o`` parameter, as default displayed time resolution is only in + seconds (even though systemd internally is tracking microsecs) - + https://www.freedesktop.org/software/systemd/man/journalctl.html + +- The guide for using systemd in devstack provides additional examples + of effective journalctl queries - + http://git.openstack.org/cgit/openstack-dev/devstack/tree/SYSTEMD.rst diff --git a/releasenotes/notes/systemd-journal-support-fcbc34b3c5ce93ec.yaml b/releasenotes/notes/systemd-journal-support-fcbc34b3c5ce93ec.yaml new file mode 100644 index 00000000..d0b929cf --- /dev/null +++ b/releasenotes/notes/systemd-journal-support-fcbc34b3c5ce93ec.yaml @@ -0,0 +1,5 @@ +--- +features: + - | + Systemd native journal support is added. You can enable this in + services with the ``use_journal`` flag.