oslo.messaging/oslo_messaging/_drivers/amqp1_driver
John Eckersberg 02a38f507d amqp1: fix race when reconnecting
Currently this is how reconnect works:

- pyngus detects failure and invokes callback
  Controller.connection_failed() which in turn calls
  Controller._handle_connection_loss()

- The first thing that _handle_connection_loss does is to set
  self.addresser to None (important later)

- Then it defers _do_reconnect after a delay (normally 1 second)

- (1 second passes)

- _do_reconnect calls _hard_reset which resets the controller state

However, there is a race here.  This can happen:

- The above, up until it defers and waits for 1 second

- Controller.send() is invoked on a task

- A new Sender is created, and critically because self.reply_link
  still exists and is active, we call sender.attach and pass in
  self.addresser.  Remember _handle_connection_loss sets
  self.addresser to None.

- Eventually Sender.attach throws an AttributeError because it
  attempts to call addresser.resolve() but addresser is None

The reason this happens is because although the connection is dead,
the controller state is still half-alive because _hard_reset hasn't
been called yet since it's deferred one second in _do_reconnect.

The fix here is to move _hard_reset out of _do_reconnect and directly
into _handle_connection_loss.  The eventloop is woken up immediately
to process _hard_reset but _do_reconnect is still deferred as before
so as to retain the desired reconnect backoff behavior.

Closes-Bug: #1941652
Change-Id: Ife62a7d76022908f0dc6a77f1ad607cb2fbd3e8f
2021-11-09 15:59:54 -05:00
..
__init__.py Reorganize the AMQP 1.0 driver source files 2016-05-13 14:04:37 -04:00
addressing.py Update messaging intermediaries for amqp1 tests 2019-03-11 08:47:46 -04:00
controller.py amqp1: fix race when reconnecting 2021-11-09 15:59:54 -05:00
eventloop.py Remove monotonic usage 2020-05-04 17:02:44 -05:00
opts.py Remove deprecated amqp1 options 2018-09-05 16:59:13 +10:00
oslo_messaging_amqp_driver_overview.rst Replace git.openstack.org URLs with opendev.org URLs 2019-04-26 10:31:26 +08:00