From fae7cbba82603e1d5bda34489c47c7097e0c29a0 Mon Sep 17 00:00:00 2001
From: Jaesuk Ahn <bluejay.ahn@gmail.com>
Date: Tue, 14 Nov 2017 14:22:29 +0900
Subject: [PATCH] Add fluent-based logging mechanism description

This PS adds logging requirements and EFK (Elasticsearch,
Fluent-bit & Fluentd, Kibana) based logging mechanism description.

Change-Id: I121c5511b9cf2ff9d752fedd31f32254e3221922
---
 doc/source/devref/fluent-logging.rst | 45 ++++++++++++++++++++++++++++
 doc/source/devref/index.rst          |  1 +
 2 files changed, 46 insertions(+)
 create mode 100644 doc/source/devref/fluent-logging.rst

diff --git a/doc/source/devref/fluent-logging.rst b/doc/source/devref/fluent-logging.rst
new file mode 100644
index 0000000000..171038d098
--- /dev/null
+++ b/doc/source/devref/fluent-logging.rst
@@ -0,0 +1,45 @@
+Logging Mechanism
+=================
+
+Logging Requirements
+--------------------
+
+OpenStack-Helm defines a centralized logging mechanism to provide insight into
+the the state of the OpenStack services and infrastructure components as
+well as underlying kubernetes platform. Among the requirements for a logging
+platform, where log data can come from and where log data need to be delivered
+are very variable. To support various logging scenarios, OpenStack-Helm should
+provide a flexible mechanism to meet with certain operation needs.
+
+
+EFK (Elasticsearch, Fluent-bit & Fluentd, Kibana) based Logging Mechanism
+-------------------------------------------------------------------------
+OpenStack-Helm provides fast and lightweight log forwarder and full featured log
+aggregator complementing each other providing a flexible and reliable solution.
+Especially, Fluent-bit is used as a log forwarder and Fluentd is used as a main
+log aggregator and processor.
+
+Fluent-bit, Fluentd meet OpenStack-Helm's logging requirements for gathering,
+aggregating, and delivering of logged events. Flunt-bit runs as a daemonset on
+each node and mounts the `/var/lib/docker/containers` directory. The Docker
+container runtime engine directs events posted to stdout and stderr to this
+directory on the host. Fluent-bit then forward the contents of that directory to
+Fluentd. Fluentd runs as deployment at the designated nodes and expose service
+for Fluent-bit to foward logs. Fluentd should then apply the Logstash format to
+the logs. Fluentd can also write kubernetes and OpenStack metadata to the logs.
+Fluentd will then forward the results to Elasticsearch and to optionally kafka.
+Elasticsearch indexes the logs in a logstash-* index by default. Kafka stores
+the logs in a ``logs`` topic by default. Any external tool can then consume the
+``logs`` topic.
+
+.. blockdiag::
+
+   diagram {
+     fluent-bit [stacked];
+     fluent-bit -> fluentd -> elasticsearch -> kibana;
+          fluentd -> kafka -> external_tool;
+   }
+
+The resulting logs can then be queried directly through Elasticsearch, or they
+can be viewed via Kibana. Kibana offers a dashboard that can create custom views
+on logged events, and Kibana integrates well with Elasticsearch by default.
diff --git a/doc/source/devref/index.rst b/doc/source/devref/index.rst
index 1de3346955..663d88a15f 100644
--- a/doc/source/devref/index.rst
+++ b/doc/source/devref/index.rst
@@ -12,3 +12,4 @@ Contents:
    oslo-config
    pod-disruption-budgets
    upgrades
+   fluent-logging