Merge "Document Zuul's SIGUSR2 handler"

doc/source/zuul.rst

@@ -275,6 +275,31 @@ the problem. Note that ``Traceback`` messages are not prefixed with the event
 ID which means you'll have to grep with additional context, for example using
 ``grep -B20 -A20``.
 
+Another useful debugging tool is Zuul's SIGUSR2 handler. This signal handler
+produces a thread dump in the debug log and toggles the yappi python profiler.
+Each Zuul service supports the signal handler and it can be triggered via::
+
+  you@zuul02$ sudo kill -USR2 $ZUUL_PID
+
+To determine ``$ZUUL_PID`` you can run ``ps`` against the ``zuul-*`` service
+that you are interested in getting information from. For example::
+
+  you@zuul02$ ps -ef | grep zuul-scheduler
+  zuuld    1893030 1893010  0 08:33 ?        00:00:00 /usr/bin/dumb-init -- /usr/local/bin/zuul-scheduler -f
+  zuuld    1893052 1893030 69 08:33 ?        07:57:42 /usr/local/bin/python /usr/local/bin/zuul-scheduler -f
+  zuuld    1893198 1893052  0 08:33 ?        00:03:22 /usr/local/bin/python /usr/local/bin/zuul-scheduler -f
+
+All of the zuul services are run under ``dumb-init``. The process to send
+SIGUSR2 to is the child of the ``dumb-init`` process. In the example above
+``$ZUUL_PID`` would be ``1893052``.
+
+The first time you run it you will turn on the yappi profiler. This profiler
+does incur a runtime cost which can significantly slow down Zuul's processing
+of pipelines. Be sure to resend the signal once you have let Zuul run long
+enough to collect a representative set of profiler data. In most cases a minute
+or two should be sufficient. Slow memory leaks may require hours, but running
+Zuul under yappi for hours isn't practical.
+
 .. _zuul_github_projects:
 
 GitHub Projects