From 7a2c00ddd86802d99c843b18c588f0d30ba4e42c Mon Sep 17 00:00:00 2001 From: Clark Boylan Date: Thu, 10 Dec 2015 11:04:09 -0800 Subject: [PATCH] Add doc on using jenkins restart playbook. Add documentation on how to run the jenkins restart playbook against a specific jenkins master. This is useful if a jenkins master starts to leak threads before its weekly restart. Change-Id: Ib5163589c1c83e4fcb7493daa387f42cda02bc9d --- doc/source/jenkins.rst | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/doc/source/jenkins.rst b/doc/source/jenkins.rst index f46a8be669..c9b457ed59 100644 --- a/doc/source/jenkins.rst +++ b/doc/source/jenkins.rst @@ -165,3 +165,26 @@ slave is then: #. Set appropriate labels on it #. Profit! + +Safe Master Restarts +==================== + +Jenkins masters periodically leak threads reducing their job +throughput and eventually leading to crashes. We work around this +by performing weekly rolling restarts of the Jenkins masters with +an ansible playbook. + +If you need to perform a safe restart against a single master you +can do this by running the same playbook and limiting it to a +specific jenkins master + +To do this:: + + root@puppetmaster# ansible-playbook -f1 --limit $server_fqdn \ + /opt/system-config/production/playbooks/restart_jenkins_masters.yaml \ + --extra-vars "user=hudson-openstack \ + password=$(/opt/system-config/production/tools/hieraedit.py \ + --yaml /etc/puppet/hieradata/production/fqdn/nodepool.openstack.org.yaml jenkins_api_key)" + +Consider running this in screen as the worst case run time is as +long as our longest running job.