dlo: Update doc about manifest containing data

Current behavior:
* If data/body is present in manifest file PUT request, the data/body gets
  saved onto disk, just like for a normal object.
* Generally, this data in manifest file is never served on a GET response.
  However, when the manifest object path itself is part of prefix, GET
  response would contain data present in manifest file as well.
* The query param multipart-manifest=get meant to retrieve SLO manifest
  also works in case of DLO manifest. Hence a COPY request with the
  multipart-manifest=get query param would actually copy DLO manifest.

How things should have been:
* The DLO manifest object is supposed to have no content and only have
  X-Object-Manifest metadata header.
* Query param multipart-manifest=get is SLO specific and shouldn't have
  any role in DLO.

This change intends to only document current behaviour and not change it,
assuming there are users who have previously saved some content in DLO
manifest file and/or have been using multipart-manifest=get to fetch
and/or COPY the DLO manifest file with it's content.

Change-Id: I0f6e175ad7752169ecf94df949336e0665928df7
Signed-off-by: Prashanth Pai <ppai@redhat.com>

doc/source/api/large_objects.rst

@@ -28,11 +28,12 @@ object. The large object is comprised of two types of objects:
        the segment objects in JSON format.
 
    **Dynamic large objects**
-       The manifest object has no content but it has a
+       The manifest object has a ``X-Object-Manifest`` metadata header.
-       ``X-Object-Manifest`` metadata header. The value of this header
+       The value of this header is ``{container}/{prefix}``,
-       is ``{container}/{prefix}``, where ``{container}`` is the name of
+       where ``{container}`` is the name of the container where the
-       the container where the segment objects are stored, and
+       segment objects are stored, and ``{prefix}`` is a string that all
-       ``{prefix}`` is a string that all segment objects have in common.
+       segment objects have in common. The manifest object should have
+       no content. However, this is not enforced.
 
 Note
 ~~~~
@@ -288,8 +289,14 @@ both the original and new manifest objects share the same set of segment
 objects.
 
 When creating dynamic large objects, the **COPY** operation does not create
-a manifest object. To duplicate a manifest object, use the **GET** operation
+a manifest object but a normal object with content same as what you would
-to read the value of ``X-Object-Manifest`` and use this value in the
+get on a **GET** request to original manifest object.
-``X-Object-Manifest`` request header in a **PUT** operation. This creates
+
-a new manifest object that shares the same set of segment objects as the
+To duplicate a manifest object:
-original manifest object.
+* Use the **GET** operation to read the value of ``X-Object-Manifest`` and
+  use this value in the ``X-Object-Manifest`` request header in a **PUT**
+  operation.
+* Alternatively, you can include *``?multipart-manifest=get``*  query
+  string in the **COPY**  request.
+This creates a new manifest object that shares the same set of segment
+objects as the original manifest object.

doc/source/overview_large_objects.rst

@@ -40,7 +40,7 @@ So now, the following ``swift`` command would download the entire large object::
 
     swift download test_container large_file
 
-``swift`` uses a strict convention for its segmented object
+``swift`` command uses a strict convention for its segmented object
 support. In the above example it will upload all the segments into a
 second container named test_container_segments. These segments will
 have names like large_file/1290206778.25/21474836480/00000000,
@@ -66,14 +66,15 @@ Direct API
 You can also work with the segments and manifests directly with HTTP
 requests instead of having ``swift`` do that for you. You can just
 upload the segments like you would any other object and the manifest
-is just a zero-byte file with an extra ``X-Object-Manifest`` header.
+is just a zero-byte (not enforced) file with an extra
+``X-Object-Manifest`` header.
 
 All the object segments need to be in the same container, have a common object
 name prefix, and their names sort in the order they should be concatenated.
 They don't have to be in the same container as the manifest file will be, which
 is useful to keep container listings clean as explained above with ``swift``.
 
-The manifest file is simply a zero-byte file with the extra
+The manifest file is simply a zero-byte (not enforced) file with the extra
 ``X-Object-Manifest: <container>/<prefix>`` header, where ``<container>`` is
 the container the object segments are in and ``<prefix>`` is the common prefix
 for all the segments.
@@ -85,6 +86,17 @@ location and then update the manifest to point to this new location. During the
 upload of the new segments, the original manifest will still be available to
 download the first set of segments.
 
+.. note::
+
+    The manifest file should have no content. However, this is not enforced.
+    If the manifest path itself conforms to container/prefix specified in
+    X-Object-Manifest, and if manifest has some content/data in it, it would
+    also be considered as segment and manifest's content will be part of the
+    concatenated GET response. The order of concatenation follows the usual DLO
+    logic which is - the order of concatenation adheres to order returned when
+    segment names are sorted.
+
+
 Here's an example using ``curl`` with tiny 1-byte segments::
 
     # First, upload the segments

test/functional/swift_test_client.py

@@ -722,8 +722,10 @@ class File(Base):
                   ['content_type', 'content-type'],
                   ['last_modified', 'last-modified'],
                   ['etag', 'etag']]
+        optional_fields = [['x_object_manifest', 'x-object-manifest']]
 
-        header_fields = self.header_fields(fields)
+        header_fields = self.header_fields(fields,
+                                           optional_fields=optional_fields)
         header_fields['etag'] = header_fields['etag'].strip('"')
         return header_fields
 

test/functional/tests.py

@@ -1805,6 +1805,9 @@ class TestDlo(Base):
         file_item = self.env.container.file('man1')
         file_contents = file_item.read(parms={'multipart-manifest': 'get'})
         self.assertEqual(file_contents, "man1-contents")
+        self.assertEqual(file_item.info()['x_object_manifest'],
+                         "%s/%s/seg_lower" %
+                         (self.env.container.name, self.env.segment_prefix))
 
     def test_get_range(self):
         file_item = self.env.container.file('man1')
@@ -1839,6 +1842,8 @@ class TestDlo(Base):
         self.assertEqual(
             file_contents,
             "aaaaaaaaaabbbbbbbbbbccccccccccddddddddddeeeeeeeeeeffffffffff")
+        # The copied object must not have X-Object-Manifest
+        self.assertTrue("x_object_manifest" not in file_item.info())
 
     def test_copy_account(self):
         # dlo use same account and same container only
@@ -1863,9 +1868,12 @@ class TestDlo(Base):
         self.assertEqual(
             file_contents,
             "aaaaaaaaaabbbbbbbbbbccccccccccddddddddddeeeeeeeeeeffffffffff")
+        # The copied object must not have X-Object-Manifest
+        self.assertTrue("x_object_manifest" not in file_item.info())
 
     def test_copy_manifest(self):
-        # Copying the manifest should result in another manifest
+        # Copying the manifest with multipart-manifest=get query string
+        # should result in another manifest
         try:
             man1_item = self.env.container.file('man1')
             man1_item.copy(self.env.container.name, "copied-man1",
@@ -1879,6 +1887,8 @@ class TestDlo(Base):
             self.assertEqual(
                 copied_contents,
                 "aaaaaaaaaabbbbbbbbbbccccccccccddddddddddeeeeeeeeee")
+            self.assertEqual(man1_item.info()['x_object_manifest'],
+                             copied.info()['x_object_manifest'])
         finally:
             # try not to leave this around for other tests to stumble over
             self.env.container.file("copied-man1").delete()