zuul/zuul-registry
Code Issues Proposed changes

Revert "Atomically concatenate blob objects"

Browse Source 
This reverts commit 0acf995175.

This changed the errors from short reads leading to manifest mismatches
to unknown blob errors. The unknown blob errors are due to not
responding to the HEAD requests until after all the data is written.
Either we get the HEAD request early and return a short size or we
return 404 and either way the end result is failure.

I'm not sure how what the solution is at this point but want to have
this revert ready in case this is desireable.

Change-Id: I6ba420e53e2091456db23c1c1cf4c76be04a4d03
This commit is contained in:
Clark Boylan 2022-02-25 13:36:37 -08:00
parent 0acf995175
commit 19ccf5990e
1 changed files with 1 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
zuul_registry/filesystem.py
View File

@ -108,18 +108,9 @@ class FilesystemDriver(storageutils.StorageDriver):
os.rename(src_path, dst_path)
def cat_objects(self, path, chunks, uuid=None):
# We cat the objects into a tmp file that we will atomically move into
# its final resting place. The reason for this is that HEAD requests
# for the blob may arrive before we have completely written the
# concatenated file out. If this occurs the presence of the half
# written file is sufficient to respond with an invalid size indicating
# to the client that the upload is complete and that the small size
# can be used to generate a manifest. When this manifest is pushed
# we fail due to a size mismatch.
path = os.path.join(self.root, path)
tmp_path = path + '.tmp'
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(tmp_path, 'wb') as outf:
with open(path, 'wb') as outf:
for chunk in chunks:
chunk_path = os.path.join(self.root, chunk['path'])
with open(chunk_path, 'rb') as inf:
@ -130,7 +121,6 @@ class FilesystemDriver(storageutils.StorageDriver):
outf.write(d)
outf.flush()
os.fsync(outf.fileno())
os.rename(tmp_path, path)
for chunk in chunks:
chunk_path = os.path.join(self.root, chunk['path'])
os.unlink(chunk_path)