deckhand/doc/source/users/layering.rst

Document Layering

Introduction

Layering provides a restricted data inheritance model intended to help reduce duplication in configuration. With layering, child documents can inherit data from parent documents. Through layering-actions, child documents can control exactly what they inherit from their parent. Document layering, conceptually speaking, works much like class inheritance: A child class inherits all variables and methods from its parent, but can elect to override its parent's functionality.

Goals behind layering include:

• model site deployment data hierarchically
• lessen data duplication across site layers (as well as other conceptual layers)

Document Abstraction

Layering works with document-abstraction: child documents can inherit from abstract as well as concrete parent documents.

Pre-Conditions

A document only has one parent, but its parent is computed dynamically using the parent-selection algorithm. That is, the notion of "multiple inheritance" does not apply to document layering.

Documents with different schema values are never layered together (see the substitution section if you need to combine data from multiple types of documents).

Document layering requires a layering-policy to exist in the revision whose documents will be layered together (rendered). An error will be issued otherwise.

Terminology

Note

Whether a layer is "lower" or "higher" has entirely to do with its order of initialization in a layerOrder and, by extension, its precedence in the parent-selection algorithm described below.

• Layer - A position in a hierarchy used to control parent-selection by the layering-algorithm. It can be likened to a position in an inheritance hierarchy, where object in Python can be likened to the highest layer in a layerOrder in Deckhand and a leaf class can be likened to the lowest layer in a layerOrder.
• Child - Meaningful only in a parent-child document relationship. A document with a lower layer (but higher priority) than its parent, determined using using parent-selection.
• Parent - Meaningful only in a parent-child document relationship. A document with a higher layer (but lower priority) than its child.
• Layering Policy - A control document <control-documents> that defines the strict layerOrder in which documents are layered together. See layering-policy documentation for more information.
• Layer Order (layerOrder) - Corresponds to the data.layerOrder of the layering-policy document. Establishes the layering hierarchy for a set of layers in the system.
• Layering Definition (layeringDefinition) - Metadata in each document for controlling the following:
  • layer: the document layer itself
  • parentSelector: parent-selection
  • abstract: document-abstraction
  • actions: layering-actions
• Parent Selector (parentSelector) - Key-value pairs or labels for identifying the document's parent. Note that these key-value pairs are not unique and that multiple documents can use them. All the key-value pairs in the parentSelector must be found among the target parent's metadata.labels: this means that the parentSelector key-value pairs must be a subset of the target parent's metadata.labels key-value pairs. See parent-selection for further details.
• Layering Actions (actions) - A list of actions that control what data are inherited from the parent by the child. See layering-actions for further details.

Algorithm

Layering is applied at the bottommost layer of the layerOrder first and at the topmost layer of the layerOrder last, such that the "base" layers are processed first and the "leaf" layers are processed last. For each layer in the layerOrder, the documents that correspond to that layer are retrieved. For each document retrieved, the layerOrder hierarchy is resolved using parent-selection to identify the parent document. Finally, the current document is layered with its parent using layering-actions.

After layering is complete, the substitution algorithm is applied to the current document, if applicable.

Layering Configuration

Layering is configured in 2 places:

1. The LayeringPolicy control document (described in layering-policy), which defines the valid layers and their order of precedence.
2. In the metadata.layeringDefinition section of normal (metadata.schema=metadata/Document/v1) documents. For more information about document structure, reference document-format.

An example layeringDefinition may look like:

layeringDefinition:
  # Controls whether the document is abstract or concrete.
  abstract: true
  # A layer in the ``layerOrder``. Must be valid or else an error is raised.
  layer: region
  # Key-value pairs or labels for identifying the document's parent.
  parentSelector:
    required_key_a: required_label_a
    required_key_b: required_label_b
  # Actions which specify which data to add to the child document.
  actions:
    - method: merge
      path: .path.to.merge.into.parent
    - method: delete
      path: .path.to.delete

Layering Actions

Introduction

Layering actions allow child documents to modify data that is inherited from the parent. What if the child document should only inherit some of the parent data? No problem. A merge action can be performed, followed by delete and replace actions to trim down on what should be inherited.

Each layer action consists of an action and a path. Whenever any action is specified, all the parent data is automatically inherited by the child document. The path specifies which data from the child document to prioritize over that of the parent document. Stated differently, all data from the parent is considered while only the child data at path is considered during an action. However, whenever a conflict occurs during an action, the child data takes priority over that of the parent.

Layering actions are queued -- meaning that if a merge is specified before a replace then the merge will necessarily be applied before the replace. For example, a merge followed by a replace is not necessarily the same as a replace followed by a merge.

Layering actions can be applied to primitives, lists and dictionaries alike.

Action Types

Supported actions are:

• merge - "deep" merge child data and parent data into the child data, at the specified JSONPath

  Note

  For conflicts between the child and parent data, the child document's data is always prioritized. No other conflict resolution strategy for this action currently exists.

  merge behavior depends upon the data types getting merged. For objects and lists, Deckhand uses JSONPath resolution to retrieve data from those entities, after which Deckhand applies merge strategies (see below) to combine merge child and parent data into the child document's data section.

  Merge Strategies

  Deckhand applies the following merge strategies for each data type:

  • object: "Deep-merge" child and parent data together; conflicts are resolved by prioritizing child data over parent data. "Deep-merge" means recursively combining data for each key-value pair in both objects.
  • array: The merge strategy involves:
    • When using an index in the action path (e.g. a[0]):
      1. Copying the parent array into the child's data section at the specified JSONPath.
      2. Appending each child entry in the original child array into the parent array. This behavior is synonymous with the extend list function in Python.
    • When not using an index in the action path (e.g. a):
      1. The child's array replaces the parent's array.
  • primitives: Includes all other data types, except for null. In this case JSONPath resolution is impossible, so child data is prioritized over that of the parent.

  Examples

  Given:

  Child Data:    ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
  Parent Data:   ``{'a': {'x': 1, 'y': 2}, 'c': 9}``

  • When:

    Merge Path: ``.``

    Then:

    Rendered Data: ``{'a': {'x': 7, 'y': 2, 'z': 3}, 'b': 4, 'c': 9}``
    
    All data from parent is automatically considered, all data from child
    is considered due to ``.`` (selects everything), then both merged.

  • When:

    Merge Path: ``.a``

    Then:

    Rendered Data: ``{'a': {'x': 7, 'y': 2, 'z': 3}, 'c': 9}``
    
    All data from parent is automatically considered, all data from child
    at ``.a`` is considered, then both merged.

  • When:

    Merge Path: ``.b``

    Then:

    Rendered Data: ``{'a': {'x': 1, 'y': 2}, 'b': 4, 'c': 9}``
    
    All data from parent is automatically considered, all data from child
    at ``.b`` is considered, then both merged.

  • When:

    Merge Path: ``.c``

    Then:

    Error raised (``.c`` missing in child).

• replace - overwrite existing data with child data at the specified JSONPath.

  Examples

  Given:

  Child Data:    ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
  Parent Data:   ``{'a': {'x': 1, 'y': 2}, 'c': 9}``

  • When:

    Replace Path: ``.``

    Then:

    Rendered Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
    
    All data from parent is automatically considered, but is replaced by all
    data from child at ``.`` (selects everything), so replaces everything
    in parent.

  • When:

    Replace Path: ``.a``

    Then:

    Rendered Data: ``{'a': {'x': 7, 'z': 3}, 'c': 9}``
    
    All data from parent is automatically considered, but is replaced by all
    data from child at ``.a``, so replaces all parent data at ``.a``.

  • When:

    Replace Path: ``.b``

    Then:

    Rendered Data: ``{'a': {'x': 1, 'y': 2}, 'b': 4, 'c': 9}``
    
    All data from parent is automatically considered, but is replaced by all
    data from child at ``.b``, so replaces all parent data at ``.b``.
    
    While ``.b`` isn't in the parent, it only needs to exist in the child.
    In this case, something (from the child) replaces nothing (from the
    parent).

  • When:

    Replace Path: ``.c``

    Then:

    Error raised (``.c`` missing in child).

• delete - remove the existing data at the specified JSONPath.

  Examples

  Given:

  Child Data:    ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
  Parent Data:   ``{'a': {'x': 1, 'y': 2}, 'c': 9}``

  • When:

    Delete Path: ``.``

    Then:

    Rendered Data: ``{}``
    
    Note that deletion of everything results in an empty dictionary by
    default.

  • When:

    Delete Path: ``.a``

    Then:

    Rendered Data: ``{'c': 9}``
    
    All data from Parent Data at ``.a`` was deleted, rest copied over.

  • When:

    Delete Path: ``.c``

    Then:

    Rendered Data: ``{'a': {'x': 1, 'y': 2}}``
    
    All data from Parent Data at ``.c`` was deleted, rest copied over.

  • When:

    Replace Path: ``.b``

    Then:

    Error raised (``.b`` missing in child).

After actions are applied for a given layer, substitutions are applied (see the substitution section for details).

Parent Selection

Parent selection is performed dynamically. Unlike substitution, parent selection does not target a specific document using schema and name identifiers. Rather, parent selection respects the layerOrder, selecting the highest precedence parent in accordance with the algorithm that follows. This allows flexibility in parent selection: if a document's immediate parent is removed in a revision, then, if applicable, the grandparent (in the previous revision) can become the document's parent (in the latest revision).

Selection of document parents is controlled by the parentSelector field and works as follows:

• A given document, C, that specifies a parentSelector, will have exactly one parent, P. If comparing layering with inheritance, layering, then, does not allow multi-inheritance.
• Both C and P must have the same schema.
• Both C and P should have different metadata.name values except in the case of replacement.
• Document P will be the highest-precedence document whose metadata.labels are a superset of document C's parentSelector. Where:

  • Highest precedence means that P belongs to the lowest layer defined in the layerOrder list from the LayeringPolicy which is at least one level higher than the layer for C. For example, if C has layer site, then its parent P must at least have layer type or above in the following layerOrder:

    ---
    ...
    layerOrder:
      - global # Highest layer
      - type
      - site   # Lowest layer

  • Superset means that P at least has all the labels in its metadata.labels that child C references via its parentSelector. In other words, parent P can have more labels than C uses to reference it, but C must at least have one matching label in its parentSelector with P.

• Deckhand will select P if it belongs to the highest-precedence layer. For example, if C belongs to layer site, P belongs to layer type, and G belongs to layer global, then Deckhand will use P as the parent for C. If P is non-existent, then G will be selected instead.

For example, consider the following sample documents:

---
schema: deckhand/LayeringPolicy/v1
metadata:
  schema: metadata/Control/v1
  name: layering-policy
data:
  layerOrder:
    - global
    - region
    - site
---
schema: example/Kind/v1
metadata:
  schema: metadata/Document/v1
  name: global-1234
  labels:
    key1: value1
  layeringDefinition:
    abstract: true
    layer: global
data:
  a:
    x: 1
    y: 2
---
schema: example/Kind/v1
metadata:
  schema: metadata/Document/v1
  name: region-1234
  labels:
    key1: value1
  layeringDefinition:
    abstract: true
    layer: region
    parentSelector:
      key1: value1
    actions:
      - method: replace
        path: .a
data:
  a:
    z: 3
---
schema: example/Kind/v1
metadata:
  schema: metadata/Document/v1
  name: site-1234
  layeringDefinition:
    layer: site
    parentSelector:
      key1: value1
    actions:
      - method: merge
        path: .
data:
  b: 4

When rendering, the parent chosen for site-1234 will be region-1234, since it is the highest precedence document that matches the label selector defined by parentSelector, and the parent chosen for region-1234 will be global-1234 for the same reason. The rendered result for site-1234 would be:

---
schema: example/Kind/v1
metadata:
  name: site-1234
data:
  a:
    z: 3
  b: 4

If region-1234 were later removed, then the parent chosen for site-1234 would become global-1234, and the rendered result would become:

---
schema: example/Kind/v1
metadata:
  name: site-1234
data:
  a:
    x: 1
    y: 2
  b: 4