opendev/system-config
Code Issues Proposed changes

Set an upper bound to log pusher output queue.

Browse Source 
Logstash's queues are small. During periods of high load this can cause
the log pusher to cache all of the logs it is pulling in. Set an upper
bound to its output queue size to force it to block when Logstash cannot
keep up. This should prevent OOMing and won't impact log throughput as
Logstash is blocking anyways.

Change-Id: I9ca31e6dbe454e9c4878fd7ba35bc33bc9df7d83
Reviewed-on: https://review.openstack.org/30572
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Khai Do <zaro0508@gmail.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
This commit is contained in:
Clark Boylan 2013-05-26 15:33:04 -07:00 committed by Jenkins
parent 52abfa977f
commit 3b8e20e598
1 changed files with 4 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
modules/openstack_project/files/logstash/log-pusher.py
View File

@ -288,7 +288,10 @@ class Server(object):
self.event_queues = []
self.retrievers = []
# TODO(clarkb) support multiple outputs
self.logqueue = queue.Queue()
# Logstash queues are small. Set an upper bound to our queue
# so that this doesn't turn into an unbounded cache prone to
# OOMing. But set it to a reasonable size to provide some caching.
self.logqueue = queue.Queue(16384)
self.processor = None
def setup_logging(self):