openstack/oslo.utils
Code Issues Proposed changes

Make forever_retry_uncaught_exceptions handle its own failures

Browse Source 
When an exception occurs inside 'except' clause, it is not handled.
As a result, forever_retry_uncaught_exceptions fails with exception,
while by definition it should not.

For instance, oslo.messaging's RPC server relies on that
function to process any exception. When forever_retry_... fails
to do so, the server thread dies. An example could be
found in referenced bug.

Change-Id: I415a0f49b25b80a264f0bc951f4b926d57a9c9a8
Closes-Bug: #1502092
This commit is contained in:
Dmitry Mescheryakov 2015-10-02 13:04:43 +03:00
parent 24d8950080
commit 6010fb3fff
1 changed files with 30 additions and 19 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

49
oslo_utils/excutils.py
View File

@ -204,23 +204,34 @@ def forever_retry_uncaught_exceptions(infunc):
try:
return infunc(*args, **kwargs)
except Exception as exc:
this_exc_message = six.u(str(exc))
if this_exc_message == last_exc_message:
exc_count += 1
else:
exc_count = 1
# Do not log any more frequently than once a minute unless
# the exception message changes
cur_time = int(time.time())
if (cur_time - last_log_time > 60 or
this_exc_message != last_exc_message):
logging.exception(
_LE('Unexpected exception occurred %d time(s)... '
'retrying.') % exc_count)
last_log_time = cur_time
last_exc_message = this_exc_message
exc_count = 0
# This should be a very rare event. In case it isn't, do
# a sleep.
time.sleep(1)
try:
this_exc_message = six.u(str(exc))
if this_exc_message == last_exc_message:
exc_count += 1
else:
exc_count = 1
# Do not log any more frequently than once a minute unless
# the exception message changes
cur_time = int(time.time())
if (cur_time - last_log_time > 60 or
this_exc_message != last_exc_message):
logging.exception(
_LE('Unexpected exception occurred %d time(s)... '
'retrying.') % exc_count)
last_log_time = cur_time
last_exc_message = this_exc_message
exc_count = 0
# This should be a very rare event. In case it isn't, do
# a sleep.
time.sleep(1)
except Exception:
try:
logging.exception(
_LE('Unexpected error occurred while '
'processing Exception'))
except Exception:
# In case either serialization of the last exception
# or logging failed, ignore the error
pass
return inner_func