opendev/system-config
Code Issues Proposed changes

prod-playbook : send playbook runtime/status to graphite

Browse Source 
We used to track the runtime with the old cron-based system
(I299c0ab5dc3dea4841e560d8fb95b8f3e7df89f2) and had a dashboard view,
which was often helpful to see at a glance what might be going wrong.

Restore this for Zuul CD by simply sending the nested-Ansible task
time-delta and status to graphite.  bridge.openstack.org is still
allowed to send stats to graphite from this prior work, so no ports
need to be opened.

Change-Id: I90dfb7a25cb5ab08403c89ef59ea21972cf2aae2
This commit is contained in:
Ian Wienand 2022-03-09 16:51:07 +11:00
parent c43289b75a
commit b7cdaa7fce
1 changed files with 11 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
playbooks/zuul/run-production-playbook.yaml
View File

@ -23,9 +23,20 @@
- name: Run specified playbook on bridge.o.o and redirect output
become: yes
shell: 'ansible-playbook -v -f {{ infra_prod_ansible_forks }} /home/zuul/src/opendev.org/opendev/system-config/playbooks/{{ playbook_name }} >> /var/log/ansible/{{ playbook_name }}.log'
register: _run
always:
- name: Send run stats
shell: |
# delta is in string format h:m:s.sss; convert to ms for statsd
{% set delta = _run.delta.split(':') %}
{% set delta_ms = ((delta[0]|int * 60 * 60 * 1000) + (delta[1]|int * 60 * 1000) + (delta[2]|float * 1000)) | int %}
echo 'bridge.ansible.{{ playbook_name }}.runtime:{{ delta_ms }}|ms' | nc -w 1 -u graphite.opendev.org 8125
echo 'bridge.ansible.{{ playbook_name }}.rc:{{ _run.rc }}|g' | nc -w 1 -u graphite.opendev.org 8125
args:
executable: '/bin/bash'
- name: Encrypt log
when: infra_prod_playbook_encrypt_log|default(False)
block: