airship/deckhand
Code Issues Proposed changes

[docs] Add documentation on document encryption

Browse Source 
This patchset adds documentation on document encryption. It also
adds appropriate references to other sections of the documentation
where linking to document encryption is apropos. Finally, this
patchset adds further information and structure around the document
rendering process in general, in order to underscore the pieces
of documentation that undergird the general concept of document
rendering: substitution, layering and replacement.

Change-Id: I566cac4a3374556a0e62e0e1962e153029f7ec05
This commit is contained in:
Felipe Monteiro 2018-05-05 19:08:05 -04:00
parent edc8949f87
commit fb6220f1b5
5 changed files with 247 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
doc/source/documents.rst
View File

@ -127,7 +127,8 @@ This type of metadata allows the following metadata hierarchy:
* ``storagePolicy`` - string, required - Either ``cleartext`` or ``encrypted``.
If ``encyrpted`` is specified, then the ``data`` section of the document will
be stored in a secure backend (likely via OpenStack Barbican). ``metadata``
and ``schema`` fields are always stored in cleartext.
and ``schema`` fields are always stored in cleartext. More information
on document encryption is available :ref:`here <encryption>`.
* ``layeringDefinition`` - dict, required - Specifies layering details. See the
Layering section below for details.

58
doc/source/encryption.rst Normal file
View File

@ -0,0 +1,58 @@
..
Copyright 2018 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _encryption:
Data Encryption
===============
Deckhand supports encrypting the ``data`` section of documents at-rest to
secure sensitive data. This encryption behavior is triggered by setting
``metadata.storagePolicy: encrypted``. It is solely the document author's
responsibility to decide the appropriate storagePolicy for the data contained
in the document.
.. note::
Note that encryption of document data incurs **runtime overhead** as the
price of encryption is performance. As a general rule, the more documents
with ``storagePolicy: encrypted``, the longer it will take to render the
documents, particularly because Barbican has a built-in `restriction`_
around retrieving only one encrypted payload a time. This means that
if 50 documents have ``storagePolicy: encrypted`` within a revision, then
Deckhand must perform 50 API calls to Barbican when rendering the documents
for that revision.
Encrypted documents, like cleartext documents, are stored in Deckhand's
database, except the ``data`` section of each encrypted document is replaced
with a reference to Barbican.
.. _Barbican: https://docs.openstack.org/barbican/latest/api/
.. _restriction: https://docs.openstack.org/barbican/latest/api/reference/secrets.html#get-v1-secrets
Supported Data Types
--------------------
Barbican supports encrypting `any`_ data type via its "opaque" secret type.
Thus, Deckhand supports encryption of any data type by utilizing this
secret type.
However, Deckhand will attempt to use Barbican's `other`_ secret types where
possible. For example, Deckhand will use "public" for document types with kind
``PublicKey``.
.. _any: https://github.com/openstack/barbican/blob/7991f8b4850d76d97c3482428638f788f5798a56/barbican/plugin/interface/secret_store.py#L272
.. _other: https://docs.openstack.org/barbican/latest/api/reference/secret_types.html

2
doc/source/index.rst
View File

@ -43,7 +43,9 @@ User's Guide
revision-history
documents
document-types
encryption
validation
rendering
substitution
layering
replacement

45
doc/source/rendering.rst Normal file
View File

@ -0,0 +1,45 @@
..
Copyright 2018 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _rendering:
Document Rendering
==================
Document rendering involves extracting all raw revision documents from
Deckhand's database, retrieving encrypted information from Barbican,
and applying substitution, layering and replacement algorithms on the
data.
The following algorithms are involved during the rendering process:
:ref:`substitution`
-------------------
Substitution provides an "open" data sharing model in which any source
document can be used to substitute data into any destination document.
:ref:`layering`
---------------
Layering provides a "restricted" data inheritance model intended to help
reduce duplication in configuration.
:ref:`replacement`
------------------
Replacement builds on top of layering to provide yet another mechanism
for reducing data duplication.

145
doc/source/substitution.rst
View File

@ -19,14 +19,52 @@
Document Substitution
=====================
Introduction
------------
Document substitution, simply put, allows one document to overwrite *parts* of
its own data with that of another document. Substitution involves a source
document sharing data with a destination document, which replaces its own data
with the shared data.
Substitution is primarily designed as a mechanism for inserting secrets into
configuration documents, but works for unencrypted source documents as well.
Substitution is applied at each layer after all merge actions occur.
Substitution may be leveraged as a mechanism for:
* inserting secrets into configuration documents
* reducing data duplication by declaring common data within one document and
having multiple other documents substitute data from the common location as
needed
During document rendering, substitution is applied at each layer after all
merge actions occur. For more information on the interaction between
document layering and substitution, see: :ref:`rendering`.
Requirements
------------
Substitutions between documents are not restricted by ``schema``, ``name``,
nor ``layer``. Source and destination documents do not need to share the same
``schema``.
No substitution dependency cycle may exist between a series of substitutions.
For example, if A substitutes from B, B from C, and C from A, then Deckhand
will raise an exception as it is impossible to determine the source data to use
for substitution in the presence of a dependency cycle.
Substitution works like this:
The source document is resolved via the ``src.schema`` and ``src.name``
keys and the ``src.path`` key is used relative to the source document's
``data`` section to retrieve the substitution data, which is then injected
into the ``data`` section of the destination document using the ``dest.path``
key.
If all the constraints above are correct, then the substitution source data
is injected into the destination document's ``data`` section, at the path
specified by ``dest.path``.
The injection of data into the destination document can be more fine-tuned
using a regular expression; see the :ref:`substitution-pattern` section
below for more information.
.. note::
@ -34,12 +72,18 @@ Substitution is applied at each layer after all merge actions occur.
because a document's ``metadata`` and ``schema`` sections should be
immutable within the scope of a revision, for obvious reasons.
Rendering Documents with Substitution
-------------------------------------
Concrete (non-abstract) documents can be used as a source of substitution
into other documents. This substitution is layer-independent, so given the 3
layer example above, which includes ``global``, ``region`` and ``site`` layers,
a document in the ``region`` layer could insert data from a document in the
``site`` layer.
Example
^^^^^^^
Here is a sample set of documents demonstrating substitution:
.. code-block:: yaml
@ -149,5 +193,96 @@ The rendered document will look like:
KEY DATA
...
This substitution is also ``schema`` agnostic, meaning that source and
destination documents can have a different ``schema``.
.. _substitution-pattern:
Substitution with Patterns
--------------------------
Substitution can be controlled in a more fine-tuned fashion using
``dest.pattern`` (optional) which functions as a regular expression underneath
the hood. The ``dest.pattern`` has the following constraints:
* ``dest.path`` key must already exist in the ``data`` section of the
destination document and must have an associated value.
* The ``dest.pattern`` must be a valid regular expression string.
* The ``dest.pattern`` must be resolvable in the value of ``dest.path``.
If the above constraints are met, then more precise substitution via a pattern
can be carried out.
Example
^^^^^^^
.. code-block:: yaml
---
# Source document.
schema: deckhand/Passphrase/v1
metadata:
name: example-password
schema: metadata/Document/v1
layeringDefinition:
layer: site
storagePolicy: cleartext
data: my-secret-password
---
# Destination document.
schema: armada/Chart/v1
metadata:
name: example-chart-01
schema: metadata/Document/v1
layeringDefinition:
layer: region
substitutions:
- dest:
path: .chart.values.some_url
pattern: INSERT_[A-Z]+_HERE
src:
schema: deckhand/Passphrase/v1
name: example-password
path: .
data:
chart:
details:
data: here
values:
some_url: http://admin:INSERT_PASSWORD_HERE@service-name:8080/v1
After document rendering, the output for ``example-chart-01`` (the destination
document) will be:
.. code-block:: yaml
---
schema: armada/Chart/v1
metadata:
name: example-chart-01
schema: metadata/Document/v1
layeringDefinition:
layer: region
substitutions:
- dest:
path: .chart.values.some_url
pattern: INSERT_[A-Z]+_HERE
src:
schema: deckhand/Passphrase/v1
name: example-password
path: .
data:
chart:
details:
data: here
values:
# Notice how the data from the source document is injected into the
# exact location specified by ``dest.pattern``.
some_url: http://admin:my-secret-password@service-name:8080/v1
Substitution of Encrypted Data
------------------------------
Deckhand allows :ref:`data to be encrypted using Barbican <encryption>`.
Substitution of encrypted data works the same as substitution of cleartext
data.
Note that during the rendering process, source and destination documents
receive the secrets stored in Barbican.