diff --git a/doc/source/images/reload_process_tree_1.svg b/doc/source/images/reload_process_tree_1.svg new file mode 100644 index 0000000000..d057f428f5 --- /dev/null +++ b/doc/source/images/reload_process_tree_1.svg @@ -0,0 +1,117 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Old Manager + + + + + + + + + + + + + + + + + + + + + + Old Workers + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/doc/source/images/reload_process_tree_2.svg b/doc/source/images/reload_process_tree_2.svg new file mode 100644 index 0000000000..947dd16f3e --- /dev/null +++ b/doc/source/images/reload_process_tree_2.svg @@ -0,0 +1,135 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Old Manager + + + + + + + + + + + + + + + + + + + + + + Old Workers + + + + + + + + + + + + + + + + + + + + Socket Closer + + + + + + + + + + + + + + \ No newline at end of file diff --git a/doc/source/images/reload_process_tree_3.svg b/doc/source/images/reload_process_tree_3.svg new file mode 100644 index 0000000000..df327130b8 --- /dev/null +++ b/doc/source/images/reload_process_tree_3.svg @@ -0,0 +1,167 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Re-exec’ed Manager + + + + + + + + + + + + + + + + + + + + + + Old Workers + + + + + + + + + + + + + + + + + + + + Socket Closer + + + + + + + + + + + + + + + + + + + + + + + + + + + + Old Workers + + + + + + + + + + + + + + \ No newline at end of file diff --git a/doc/source/images/reload_process_tree_4.svg b/doc/source/images/reload_process_tree_4.svg new file mode 100644 index 0000000000..a8d6a096d5 --- /dev/null +++ b/doc/source/images/reload_process_tree_4.svg @@ -0,0 +1,219 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Re-exec’ed Manager + + + + + + + + + + + + + + + + + + + + + + Old Workers + + + + + + + + + + + + + + + + + + + + Socket Closer + + + + + + + + + + + + + + + + + + + + + + + + + + + + Old Workers + + + + + + + + + + + + + + + + + + + + + + + + + + + + Old Workers + + + + + + + + + + + + + + + + + + + + + + New Workers + + + + + + + + + + + + + + \ No newline at end of file diff --git a/doc/source/images/reload_process_tree_5.svg b/doc/source/images/reload_process_tree_5.svg new file mode 100644 index 0000000000..715884df38 --- /dev/null +++ b/doc/source/images/reload_process_tree_5.svg @@ -0,0 +1,202 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Re-exec’ed Manager + + + + + + + + + + + + + + + + + + + + + + Old Workers + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Old Workers + + + + + + + + + + + + + + + + + + + + + + + + + + + + Old Workers + + + + + + + + + + + + + + + + + + + + + + New Workers + + + + + + + + + + + + + + \ No newline at end of file diff --git a/doc/source/images/reload_process_tree_6.svg b/doc/source/images/reload_process_tree_6.svg new file mode 100644 index 0000000000..ea6b4c6e71 --- /dev/null +++ b/doc/source/images/reload_process_tree_6.svg @@ -0,0 +1,152 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Re-exec’ed Manager + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Old Workers + + + + + + + + + + + + + + + + + + + + + + New Workers + + + + + + + + \ No newline at end of file diff --git a/doc/source/index.rst b/doc/source/index.rst index 41ec26e808..bddb243ce7 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -45,6 +45,7 @@ Overview and Concepts api/object_api_v1_overview overview_architecture + overview_wsgi_management overview_ring overview_policies overview_reaper diff --git a/doc/source/overview_wsgi_management.rst b/doc/source/overview_wsgi_management.rst new file mode 100644 index 0000000000..206fe3e02f --- /dev/null +++ b/doc/source/overview_wsgi_management.rst @@ -0,0 +1,84 @@ +WSGI Server Process Management +============================== + +Graceful Shutdowns with ``SIGHUP`` +---------------------------------- + +Swift has always supported graceful WSGI server shutdown via ``SIGHUP``. +This causes the manager process to fall out of its +ensure-all-workers-are-running loop, close all workers' listen sockets, +and exit. Closing the listen sockets causes all new ``accept`` calls to +fail, but does not impact any established connections. + +The workers are re-parented, likely to PID 1, and are discoverable with +``swift-orphans``. When the ``accept`` call fails, it waits for the +connection-handling ``GreenPool`` to complete, then exits. Each worker +continues processing the current request, then closes the connection. +Note that clients will get connection errors if they try to re-use a +connection for further requests. + +Prior to the introduction of seamless reloads (see below), a common +reload strategy was to perform a graceful shutdown followed by a fresh +service start. + +Seamless Reloads with ``SIGUSR1`` +--------------------------------- + +Beginning with Swift 2.24.0, WSGI servers support seamless reloads via +``SIGUSR1``. This allows servers to restart to pick up configuration or +code changes while being minimally-disruptive to clients. The process +is as follows: + +.. image:: images/reload_process_tree_1.svg + +1. Manager process receives ``USR1`` signal. This causes the process to fall + out of its loop ensuring that all workers are running and instead begin + reloading. The workers continue servicing client requests as long as + their listen sockets remain open. + +.. image:: images/reload_process_tree_2.svg + +2. Manager process forks. The new child knows about all the existing + workers and their listen sockets; it will be responsible for closing + the old worker listen sockets so they stop accepting new connections. + +.. image:: images/reload_process_tree_3.svg + +3. Manager process re-exec's itself. It picks up new configuration and + code while maintaining the same PID as the old manager process. At + this point only the socket-closer is tracking the old workers, but + everything (including old workers) remains a child of the new manager + process. As a result, old workers are *not* discoverable with + ``swift-orphans``; ``swift-oldies`` may be useful, but will also find + the manager process. + +.. image:: images/reload_process_tree_4.svg + +4. New manager process forks off new workers, each with its own listen + socket. Once all workers have started and can accept new connections, + the manager notifies the socket-closer via a pipe. The socket-closer + closes the old worker listen sockets so they stop accepting new + connections, then exits. + +.. image:: images/reload_process_tree_5.svg + +5. Old workers continue servicing any in-progress connections, while new + connections are picked up by new workers. Once an old worker completes + all of its oustanding requests, it exits. + +.. image:: images/reload_process_tree_6.svg + +6. All old workers have now exited. Only new code and configs are in use. + +``swift-reload`` +---------------- + +Beginning with Swift 2.33.0, a new ``swift-reload`` helper is included +to help validate the reload process. Given a PID, it will + +1. Validate that the PID seems to belong to a Swift WSGI server manager + process, +2. Check that the config file used by that PID is currently valid, +3. Send the ``USR1`` signal to initiate a reload, and +4. Wait for the new workers to come up (indicating the reload is complete) + before exiting.