From 59ce1d902e2137bb7346a0d1f223e0ce1cb83216 Mon Sep 17 00:00:00 2001 From: Ian Wienand Date: Mon, 11 Feb 2019 12:27:10 +1100 Subject: [PATCH] Export all journal logs Currently we only export the devstack@ services, and then separately export the kernel & sudo logs to syslog.txt. This leaves a lot of logs potentially behind in the journal for various daemons. Just export the whole lot. Using this output is currently very opaque and makes use of systemd export tools that are very un-discoverable. Add a README that will appear alongside the journal explaining how to actually use it. This is a template as it would be nice to put into things like the list of services that are in the journal, or maybe other magic. Also make sure we export the logs since the start timestamp; currently during a full run we drop the initial logs. Change-Id: Id2626f9113d82c6d524039acda8a8ec74afb2081 --- roles/export-devstack-journal/README.rst | 14 +++--- roles/export-devstack-journal/tasks/main.yaml | 45 +++++++++++++------ .../templates/devstack.journal.README.txt.j2 | 33 ++++++++++++++ 3 files changed, 73 insertions(+), 19 deletions(-) create mode 100644 roles/export-devstack-journal/templates/devstack.journal.README.txt.j2 diff --git a/roles/export-devstack-journal/README.rst b/roles/export-devstack-journal/README.rst index a34e0706a9..9e3c919627 100644 --- a/roles/export-devstack-journal/README.rst +++ b/roles/export-devstack-journal/README.rst @@ -1,11 +1,15 @@ Export journal files from devstack services -Export the systemd journal for every devstack service in native -journal format as well as text. Also, export a syslog-style file with -kernal and sudo messages. +This performs a number of logging collection services -Writes the output to the ``logs/`` subdirectory of -``stage_dir``. +* Export the systemd journal in native format +* For every devstack service, export logs to text in a file named + ``screen-*`` to maintain legacy compatability when devstack services + used to run in a screen session and were logged separately. +* Export a syslog-style file with kernel and sudo messages for legacy + compatability. + +Writes the output to the ``logs/`` subdirectory of ``stage_dir``. **Role Variables** diff --git a/roles/export-devstack-journal/tasks/main.yaml b/roles/export-devstack-journal/tasks/main.yaml index 6e760c1f6f..cbec4447b8 100644 --- a/roles/export-devstack-journal/tasks/main.yaml +++ b/roles/export-devstack-journal/tasks/main.yaml @@ -6,32 +6,49 @@ state: directory owner: "{{ ansible_user }}" -# TODO: convert this to ansible -- name: Export journal files +- name: Export legacy stack screen log files become: true shell: cmd: | u="" name="" - for u in `systemctl list-unit-files | grep devstack | awk '{print $1}'`; do + for u in $(systemctl list-unit-files | grep devstack | awk '{print $1}'); do name=$(echo $u | sed 's/devstack@/screen-/' | sed 's/\.service//') journalctl -o short-precise --unit $u | gzip - > {{ stage_dir }}/logs/$name.txt.gz done - # Export the journal in export format to make it downloadable - # for later searching. It can then be rewritten to a journal native - # format locally using systemd-journal-remote. This makes a class of - # debugging much easier. We don't do the native conversion here as - # some distros do not package that tooling. - journalctl -u 'devstack@*' -o export | \ - xz --threads=0 - > {{ stage_dir }}/logs/devstack.journal.xz - - # The journal contains everything running under systemd, we'll - # build an old school version of the syslog with just the - # kernel and sudo messages. +- name: Export legacy syslog.txt + become: true + shell: + # The journal contains everything running under systemd, we'll + # build an old school version of the syslog with just the + # kernel and sudo messages. + cmd: | journalctl \ -t kernel \ -t sudo \ --no-pager \ --since="$(cat {{ devstack_base_dir }}/log-start-timestamp.txt)" \ | gzip - > {{ stage_dir }}/logs/syslog.txt.gz + +# TODO: convert this to ansible +# - make a list of the above units +# - iterate the list here +- name: Export journal + become: true + shell: + # Export the journal in export format to make it downloadable + # for later searching. It can then be rewritten to a journal native + # format locally using systemd-journal-remote. This makes a class of + # debugging much easier. We don't do the native conversion here as + # some distros do not package that tooling. + cmd: | + journalctl -o export \ + --since="$(cat {{ devstack_base_dir }}/log-start-timestamp.txt)" \ + | xz --threads=0 - > {{ stage_dir }}/logs/devstack.journal.xz + +- name: Save journal README + become: true + template: + src: devstack.journal.README.txt.j2 + dest: '{{ stage_dir }}/logs/devstack.journal.README.txt' diff --git a/roles/export-devstack-journal/templates/devstack.journal.README.txt.j2 b/roles/export-devstack-journal/templates/devstack.journal.README.txt.j2 new file mode 100644 index 0000000000..598eb7f3db --- /dev/null +++ b/roles/export-devstack-journal/templates/devstack.journal.README.txt.j2 @@ -0,0 +1,33 @@ +Devstack systemd journal +======================== + +The devstack.journal file is a copy of the systemd journal during the +devstack run. + +To use it, you will need to convert it so journalctl can read it +locally. After downloading the file: + + $ /lib/systemd/systemd-journal-remote <(xzcat ./devstack.journal.xz) -o output.journal + +Note this binary is not in the regular path. On Debian/Ubuntu +platforms, you will need to have the "sytemd-journal-remote" package +installed. + +It should result in something like: + + Finishing after writing entries + +You can then use journalctl to examine this file. For example, to see +all devstack services try: + + $ journalctl --file ./output.journal -u 'devstack@*' + +To see just cinder API server logs restrict the match with + + $ journalctl --file ./output.journal -u 'devstack@c-api' + +There may be many types of logs available in the journal, a command like + + $ journalctl --file ./output.journal --output=json-pretty | grep "_SYSTEMD_UNIT" | sort -u + +can help you find interesting things to filter on. \ No newline at end of file