docs: Update document types documentation

This patch set updates document types documentation
which includes expounding on control documents, layering policy
documents, and dataschema documents.

Change-Id: Id31fcb6b68ca30fdf681dec8348c2fc4237cd48d

doc/source/document-types.rst

@@ -19,31 +19,72 @@
 Document Types
 ==============
 
+.. _application-documents:
+
+Application Documents
+---------------------
+
+Application documents are those whose ``metadata.schema`` begins with
+``metadata/Document``. These documents define all the data that make up
+a site deployment, including but not limited to: networking, hardware, host,
+bare metal, software, etc. site information. Prior to ingestion by Deckhand,
+application documents are known as "raw documents". After rendering, they are
+known as "rendered documents". Application documents are subject to the
+following :ref:`rendering` operations:
+
+* :ref:`encryption`
+* :ref:`layering`
+* :ref:`substitution`
+* :ref:`replacement`
+
+.. _control-documents:
+
 Control Documents
 -----------------
 
-Control documents (documents which have ``metadata.schema=metadata/Control/v1``),
-are special, and are used to control the behavior of Deckhand at runtime. Only
-the following types of control documents are allowed.
+Control documents (documents which have ``metadata.schema`` of
+``metadata/Control/v1``), are special, and are used to control
+the behavior of Deckhand at runtime. Control documents are immutable so
+any document mutation or manipulation does not apply to them.
+
+Control documents only exist to control how :ref:`application-documents` are
+validated and rendered.
+
+.. note::
+
+  Unlike :ref:`application-documents`, control documents do not require
+  ``storagePolicy`` or ``layeringDefinition`` properties; in fact, it is
+  recommended that such properties not be used for control documents. Again,
+  this is because such documents should not themselves undergo layering,
+  substitution or encryption. It is not meaningful to treat them like normal
+  documents. See :ref:`validation-schemas` for more information on required
+  document properties.
+
+Only the following types of control documents are allowed:
 
 DataSchema
 ^^^^^^^^^^
 
 ``DataSchema`` documents are used by various services to register new schemas
 that Deckhand can use for validation. No ``DataSchema`` documents with names
-beginning with ``deckhand/`` or ``metadata/`` are allowed.  The ``metadata.name``
-field of each ``DataSchema`` document specifies the top level ``schema`` that it
-is used to validate against.
+beginning with ``deckhand/`` or ``metadata/`` are allowed. The
+``metadata.name`` field of each ``DataSchema`` document references the
+top-level ``schema`` of  :ref:`application-documents`: when there is a match
+between both values, the ``data`` section of all :ref:`application-documents`
+is validated against the JSON schema found in the matching ``DataSchema``
+document.
 
-The contents of its ``data`` key are expected to be the JSON schema definition
-for the target document type from the target's top level ``data`` key down.
+The JSON schema definition is found in the ``data`` key of each ``DataSchema``
+document. The entire ``data`` section of the target document is validated.
 
-.. TODO: give valid, tiny schema as example
+The following is an example of a sample ``DataSchema`` document, whose ``data``
+section features a simplistic JSON schema:
 
 .. code-block:: yaml
 
   ---
-  schema: deckhand/DataSchema/v1  # This specifies the official JSON schema meta-schema.
+  # This specifies the official JSON schema meta-schema.
+  schema: deckhand/DataSchema/v1
   metadata:
     schema: metadata/Control/v1
     name: promenade/Node/v1  # Specifies the documents to be used for validation.
@@ -51,8 +92,25 @@ for the target document type from the target's top level ``data`` key down.
       application: promenade
   data:  # Valid JSON Schema is expected here.
     $schema: http://blah
+    properties:
+      foo:
+        enum:
+          - bar
+          - baz
+          - qux
+    required:
+      - foo
   ...
 
+The JSON schema abvove requires that the ``data`` section of
+:ref:`application-documents` that match this ``DataSchema`` have a property
+called ``foo`` whose value must be one of: "bar", "baz", or "qux".
+
+Reference the `JSON schema`_ documentation for more information on writing
+correct schemas.
+
+.. _JSON schema: http://json-schema.org
+
 LayeringPolicy
 ^^^^^^^^^^^^^^
 
@@ -61,10 +119,19 @@ It is an error to attempt to insert a new ``LayeringPolicy`` document if it has
 a different ``metadata.name`` than the existing document. If the names match,
 it is treated as an update to the existing document.
 
+.. note::
+
+  In order to create a new ``LayeringPolicy`` document in Deckhand, submit
+  an **empty** payload via ``PUT /buckets/{bucket_name}/documents``. Afterward,
+  submit another request containing the new batch of documents, including
+  the new ``LayeringPolicy``.
+
 This document defines the strict order in which documents are merged together
 from their component parts. It should result in a validation error if a
 document refers to a layer not specified in the ``LayeringPolicy``.
 
+Below is an example of a ``LayeringPolicy`` document:
+
 .. code-block:: yaml
 
   ---
@@ -81,6 +148,20 @@ document refers to a layer not specified in the ``LayeringPolicy``.
       - force
   ...
 
+In the ``LayeringPolicy`` above, a 5-tier ``layerOrder`` is created, in which
+the topmost layer is ``global`` and the bottommost layer is ``force``. This
+means that ``global`` constitutes the "base" layer onto which other documents
+belonging to sub-layers can be layered. In practice, this means that
+documents with ``site-type`` can layer with documents with ``global`` and
+documents with ``region`` can layer with documents with ``site-type``, etc.
+
+Note that in the absence of any document belonging to an "intermediate" layer,
+base layers can layer with "interspersed" sub-layers, no matter the number of
+layers between them. This means that a document with layer ``force`` could
+layer with a document with layer ``global``, *provided* no document exists
+with a layer of ``site-type``, ``region``, or ``site``. For more information
+about document layering, reference the :ref:`layering` documentation.
+
 ValidationPolicy
 ^^^^^^^^^^^^^^^^
 

doc/source/validation.rst

@@ -247,7 +247,9 @@ ingested by Deckhand.
     :lines: 15-
     :caption: Schema for ``metadata/Document`` metadata document sections.
 
-DataSchema Schemas
+.. _validation-schemas:
+
+Validation Schemas
 ------------------
 
 DataSchema schemas validate the ``data`` section of every document ingested