openstack/swift
Code Issues Proposed changes

dlo: Update doc about manifest containing data

Browse Source 
Current behavior:
* If data/body is present in manifest file PUT request, the data/body gets
  saved onto disk, just like for a normal object.
* Generally, this data in manifest file is never served on a GET response.
  However, when the manifest object path itself is part of prefix, GET
  response would contain data present in manifest file as well.
* The query param multipart-manifest=get meant to retrieve SLO manifest
  also works in case of DLO manifest. Hence a COPY request with the
  multipart-manifest=get query param would actually copy DLO manifest.

How things should have been:
* The DLO manifest object is supposed to have no content and only have
  X-Object-Manifest metadata header.
* Query param multipart-manifest=get is SLO specific and shouldn't have
  any role in DLO.

This change intends to only document current behaviour and not change it,
assuming there are users who have previously saved some content in DLO
manifest file and/or have been using multipart-manifest=get to fetch
and/or COPY the DLO manifest file with it's content.

Change-Id: I0f6e175ad7752169ecf94df949336e0665928df7
Signed-off-by: Prashanth Pai <ppai@redhat.com>
This commit is contained in:
Prashanth Pai 2015-01-12 12:51:46 +05:30
parent bf9fd3f8f5
commit a0977ab8ca
4 changed files with 46 additions and 15 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

27
doc/source/api/large_objects.rst
View File

@ -28,11 +28,12 @@ object. The large object is comprised of two types of objects:
the segment objects in JSON format.
**Dynamic large objects**
The manifest object has no content but it has a
``X-Object-Manifest`` metadata header. The value of this header
is ``{container}/{prefix}``, where ``{container}`` is the name of
the container where the segment objects are stored, and
``{prefix}`` is a string that all segment objects have in common.
The manifest object has a ``X-Object-Manifest`` metadata header.
The value of this header is ``{container}/{prefix}``,
where ``{container}`` is the name of the container where the
segment objects are stored, and ``{prefix}`` is a string that all
segment objects have in common. The manifest object should have
no content. However, this is not enforced.
Note
~~~~
@ -288,8 +289,14 @@ both the original and new manifest objects share the same set of segment
objects.
When creating dynamic large objects, the **COPY** operation does not create
a manifest object. To duplicate a manifest object, use the **GET** operation
to read the value of ``X-Object-Manifest`` and use this value in the
``X-Object-Manifest`` request header in a **PUT** operation. This creates
a new manifest object that shares the same set of segment objects as the
original manifest object.
a manifest object but a normal object with content same as what you would
get on a **GET** request to original manifest object.
To duplicate a manifest object:
* Use the **GET** operation to read the value of ``X-Object-Manifest`` and
use this value in the ``X-Object-Manifest`` request header in a **PUT**
operation.
* Alternatively, you can include *``?multipart-manifest=get``* query
string in the **COPY** request.
This creates a new manifest object that shares the same set of segment
objects as the original manifest object.

18
doc/source/overview_large_objects.rst
View File

@ -40,7 +40,7 @@ So now, the following ``swift`` command would download the entire large object::
swift download test_container large_file
``swift`` uses a strict convention for its segmented object
``swift`` command uses a strict convention for its segmented object
support. In the above example it will upload all the segments into a
second container named test_container_segments. These segments will
have names like large_file/1290206778.25/21474836480/00000000,
@ -66,14 +66,15 @@ Direct API
You can also work with the segments and manifests directly with HTTP
requests instead of having ``swift`` do that for you. You can just
upload the segments like you would any other object and the manifest
is just a zero-byte file with an extra ``X-Object-Manifest`` header.
is just a zero-byte (not enforced) file with an extra
``X-Object-Manifest`` header.
All the object segments need to be in the same container, have a common object
name prefix, and their names sort in the order they should be concatenated.
They don't have to be in the same container as the manifest file will be, which
is useful to keep container listings clean as explained above with ``swift``.
The manifest file is simply a zero-byte file with the extra
The manifest file is simply a zero-byte (not enforced) file with the extra
``X-Object-Manifest: <container>/<prefix>`` header, where ``<container>`` is
the container the object segments are in and ``<prefix>`` is the common prefix
for all the segments.
@ -85,6 +86,17 @@ location and then update the manifest to point to this new location. During the
upload of the new segments, the original manifest will still be available to
download the first set of segments.
.. note::
The manifest file should have no content. However, this is not enforced.
If the manifest path itself conforms to container/prefix specified in
X-Object-Manifest, and if manifest has some content/data in it, it would
also be considered as segment and manifest's content will be part of the
concatenated GET response. The order of concatenation follows the usual DLO
logic which is - the order of concatenation adheres to order returned when
segment names are sorted.
Here's an example using ``curl`` with tiny 1-byte segments::
# First, upload the segments

4
test/functional/swift_test_client.py
View File

@ -722,8 +722,10 @@ class File(Base):
['content_type', 'content-type'],
['last_modified', 'last-modified'],
['etag', 'etag']]
optional_fields = [['x_object_manifest', 'x-object-manifest']]
header_fields = self.header_fields(fields)
header_fields = self.header_fields(fields,
optional_fields=optional_fields)
header_fields['etag'] = header_fields['etag'].strip('"')
return header_fields

12
test/functional/tests.py
View File

@ -1805,6 +1805,9 @@ class TestDlo(Base):
file_item = self.env.container.file('man1')
file_contents = file_item.read(parms={'multipart-manifest': 'get'})
self.assertEqual(file_contents, "man1-contents")
self.assertEqual(file_item.info()['x_object_manifest'],
"%s/%s/seg_lower" %
(self.env.container.name, self.env.segment_prefix))
def test_get_range(self):
file_item = self.env.container.file('man1')
@ -1839,6 +1842,8 @@ class TestDlo(Base):
self.assertEqual(
file_contents,
"aaaaaaaaaabbbbbbbbbbccccccccccddddddddddeeeeeeeeeeffffffffff")
# The copied object must not have X-Object-Manifest
self.assertTrue("x_object_manifest" not in file_item.info())
def test_copy_account(self):
# dlo use same account and same container only
@ -1863,9 +1868,12 @@ class TestDlo(Base):
self.assertEqual(
file_contents,
"aaaaaaaaaabbbbbbbbbbccccccccccddddddddddeeeeeeeeeeffffffffff")
# The copied object must not have X-Object-Manifest
self.assertTrue("x_object_manifest" not in file_item.info())
def test_copy_manifest(self):
# Copying the manifest should result in another manifest
# Copying the manifest with multipart-manifest=get query string
# should result in another manifest
try:
man1_item = self.env.container.file('man1')
man1_item.copy(self.env.container.name, "copied-man1",
@ -1879,6 +1887,8 @@ class TestDlo(Base):
self.assertEqual(
copied_contents,
"aaaaaaaaaabbbbbbbbbbccccccccccddddddddddeeeeeeeeee")
self.assertEqual(man1_item.info()['x_object_manifest'],
copied.info()['x_object_manifest'])
finally:
# try not to leave this around for other tests to stumble over
self.env.container.file("copied-man1").delete()