Use dstat to record performance of system-config-run hosts

We have seen some poor performance from gitea which may be related to
manage project updates. Start a dstat service which logs to a csv file
on our system-config-run job hosts in order to collect performance info
from our services in pre merge testing. This will include gitea and
should help us evaluate service upgrades and other changes from a
performance perspective before they hit production.

Change-Id: I7bdaab0a0aeb9e1c00fcfcca3d114ae13a76ccc9

playbooks/roles/dstat-logger/README.rst

@@ -0,0 +1,6 @@
+Install, configure, and run a dstat logger
+
+This is primarily useful for testing where we don't have instances hooked
+up to cacti. You can use this to get a csv log file at
+/var/log/dstat-csv.log in test jobs that records similar system performance
+information.

playbooks/roles/dstat-logger/tasks/main.yaml

@@ -0,0 +1,28 @@
+- name: Install dstat
+  become: yes
+  package:
+    name: dstat
+    state: present
+- name: Install dstat unit
+  become: yes
+  copy:
+    dest: /etc/systemd/system/dstatlogger.service
+    owner: root
+    group: root
+    mode: 0644
+    content: |
+      [Unit]
+      Description=Dstat logging service
+      [Service]
+      User=root
+      ExecStart=/usr/bin/dstat -tcmndrylpg --tcp --swap --output /var/log/dstat-csv.log
+      KillMode=process
+      TimeoutStopSec=300
+      [Install]
+      WantedBy=multi-user.target
+- name: Reload systemd
+  become: yes
+  command: systemctl daemon-reload
+- name: Start dstat unit
+  become: yes
+  command: systemctl start dstatlogger

playbooks/service-dstatlogger.yaml

@@ -0,0 +1,3 @@
+- hosts: "*:!disabled"
+  roles:
+    - dstat-logger

playbooks/zuul/run-base.yaml

@@ -92,6 +92,8 @@
       command: ansible-playbook -f 50 -v /home/zuul/src/opendev.org/opendev/system-config/playbooks/base.yaml
     - name: Run bridge service playbook
       command: ansible-playbook -v /home/zuul/src/opendev.org/opendev/system-config/playbooks/service-bridge.yaml
+    - name: Run dstat logger playbook
+      command: ansible-playbook -v /home/zuul/src/opendev.org/opendev/system-config/playbooks/service-dstatlogger.yaml
     - name: Run playbook
       when: run_playbooks is defined
       loop: "{{ run_playbooks }}"

zuul.d/system-config-run.yaml

@@ -17,6 +17,7 @@
         '/var/log/messages': logs_txt
         '/var/log/docker': logs
         '/var/log/containers': logs
+        '/var/log/dstat-csv.log': logs
         '/etc/iptables/rules.v4': logs_txt
         '/etc/iptables/rules.v6': logs_txt
     host-vars: