deckhand/doc/source/overview.rst

Overview

Deckhand is a storage service for YAML-based configuration documents. Deckhand stores the documents using version control: Each time a collection of documents is passed to Deckhand, a new revision is created. Thus, documents have a revision history, allowing complex configurations to be incrementally modified and validated. For example, if the first revision of documents fail validation, deployers can make modifications to the documents and submit them to Deckhand again, until the documents pass validation and are ready to be rendered into their finalized state.

Core Responsibilities

• revision history - improves auditability and enables services to provide functional validation of a well-defined collection of documents that are meant to operate together
• validation - allows services to implement and register different kinds of validations and report errors
• buckets - allow documents to be owned by different services, providing write protections around collections of documents
• layering - helps reduce duplication in configuration while maintaining auditability across many sites
• substitution - provides separation between secret data and other configuration data, while also providing a mechanism for documents to share data among themselves

Revision History

Like other version control software, Deckhand allows users to track incremental changes to documents via a revision history, built up through individual payloads to Deckhand, each forming a separate revision. Each revision, in other words, contains its own set of immutable documents: Creating a new revision maintains the existing revision history.

For more information, see the revision-history section.

Validation

For each created revision, built-in document-types are automatically validated. Validations are always stored in the database, including detailed error messages explaining why validation failed, to help deployers rectify syntactical or semantical issues with configuration documents. Regardless of validation failure, a new revision is always created, except when the documents are completely malformed.

Deckhand validation functionality is extensible via DataSchema documents, allowing the data sections of registered document types to be subjected to user-provided JSON schemas.

Note

While Deckhand ingests YAML documents, internally it translates them to Python objects and can use JSON schemas to validate those objects.

For more information, see the validation section.

Buckets

Collections of documents, called buckets, are managed together. All documents belong to a bucket and all documents that are part of a bucket must be fully specified together.

To create or update a new document in, e.g. bucket mop, one must PUT the entire set of documents already in mop along with the new or modified document. Any documents not included in that PUT will be automatically deleted in the created revision.

Each bucket provides write protections around a group of documents. That is, only the bucket that owns a collection of documents can manage those documents. However, documents can be read across different buckets and used together to render finalized configuration documents, to be consumed by other services like Armada, Drydock, Promenade or Shipyard.

In other words:

• Documents can be read from any bucket.

  This is useful so that documents from different buckets can be used together for layering and substitution.

• Documents can only be written to by the bucket that owns them.

  This is useful because it offers the concept of ownership to a document in which only the bucket that owns the document can manage it.

Deckhand should offer RBAC (Role-Based Access Control) around buckets. This will allow deployers to control permissions around who can write or read documents to or from buckets.

Note

The best analogy for a bucket is a folder. Like a folder, which houses files and offers read and write permissions, a bucket houses documents and offers read and write permissions around them.

A bucket is not akin to a repository, because a repository has its own distinct revision history. A bucket, on the other hand, shares its revision history with every other bucket.

Layering

Layering provides a restricted data inheritance model intended to help reduce duplication in configuration. A LayeringPolicy can be created to declare the order of inheritance via layers for documents. Parent documents can provide common data to child documents, who can override their parent data or tweak it in order to achieve more nuanced configuration that builds on top of common configurations.

For more information, see the layering section.

Substitution

Substitution is a mechanism for documents to share data among themselves. It is particularly useful for documents that possess secrets to be stored securely and on demand provide the secrets to documents that need them. However, substitution can also apply to any data, not just secrets.

For more information, see the substitution section.

Replacement

Document replacement provides an advanced mechanism for reducing the overhead with data duplication across multiple documents.

For more information, see the replacement section.