Tolerate 404s during setUp/tearDown in func tests

A couple times, I've seen tests fail in the gate because we got back a
404 while trying to clean out the test account. The story that gets us
here seems to be:

 - One or more object servers take too long to respond to the initial
   DELETE request, so the test client gets back a 503 and sleeps so
   it can retry.
 - Meanwhile, the servers finish writing their tombstones and want to
   respond 204 (but probably *actually* respond 408 because the proxy
   killed the connection).
 - The test client sends its retry, and since the object servers now
   have tombstones, it gets back a 404.

But the thing is, this is *outside of the test scope* anyway, we're just
trying to get back to a sane state. If it's gone, s much the better!

For an example of this, see the failures on patchset 3 of
https://review.openstack.org/#/c/534978 (which both failed for the same
reason on different tests).

Change-Id: I9ab2fd430d4800f9f55275959a20e30f09d9e1a4
This commit is contained in:
Tim Burke 2018-03-01 23:30:00 +00:00
parent 10943b89b9
commit 8b8a2a3406

View File

@ -543,7 +543,7 @@ class Container(Base):
def delete_files(self):
for f in listing_items(self.files):
file_item = self.file(f)
if not file_item.delete():
if not file_item.delete(tolerate_missing=True):
return False
return listing_empty(self.files)
@ -764,14 +764,19 @@ class File(Base):
self.conn.make_path(self.path))
return True
def delete(self, hdrs=None, parms=None, cfg=None):
def delete(self, hdrs=None, parms=None, cfg=None, tolerate_missing=False):
if hdrs is None:
hdrs = {}
if parms is None:
parms = {}
if self.conn.make_request('DELETE', self.path, hdrs=hdrs,
cfg=cfg, parms=parms) != 204:
if tolerate_missing:
allowed_statuses = (204, 404)
else:
allowed_statuses = (204,)
if self.conn.make_request(
'DELETE', self.path, hdrs=hdrs, cfg=cfg,
parms=parms) not in allowed_statuses:
raise ResponseError(self.conn.response, 'DELETE',
self.conn.make_path(self.path))