tripleo-docs/doc/source/install/containers_deployment/architecture.rst

TripleO Containers Architecture

This document explains the details around TripleO's containers architecture. The document goes into the details of how the containers are built for TripleO, how the configuration files are generated and how the containers are eventually run.

Like other areas of TripleO, the containers based deployment requires a couple of different projects to play together. The next section will cover each of the parts that allow for deploying OpenStack in containers using TripleO.

Building Containers

The containers used for TripleO are sourced from Kolla. Kolla is an OpenStack team that aims to create tools to allow for deploying OpenStack on container technologies. Kolla (or Kolla Build) is one of the tools produced by this team and it allows for building and customizing container images for OpenStack services and their dependencies.

TripleO consumes these images and takes advantage of the customization capabilities provided by the Kolla build tool to install some packages that are required by other parts of TripleO.

The following template is an example of the template used for building the base images that are consumed by TripleO. Further customization is required for some of the services, like mariadb:

{% extends parent_template %}
{% set base_centos_binary_packages_append = ['puppet'] %}
{% set nova_scheduler_packages_append = ['openstack-tripleo-common'] %}

Note

parent_template is the literal string to include. No need to replace it.

Use the following command to build an image using kolla-build and the template above (template-overrides.j2):

$ kolla-build --base centos \
  --template-override /usr/share/tripleo-common/container-images/tripleo_kolla_template_overrides.j2 \
  --template-override template-overrides.j2

TripleO maintains its complete list of kolla customization in the tripleo-common project.

Paunch

The paunch hook is used to manage containers. This hook takes json as input and uses it to create and run containers on demand. The json describes how the container will be started. Some example keys are:

• net: To specify what network to use. This is commonly set to host.
• privileged: Whether to give full access to the host's devices to the container, similar to what happens when the service runs directly on the host.
• volumes: List of host path volumes, named volumes, or dynamic volumes to bind on the container.
• environment: List of environment variables to set on the container.

Note

The list above is not exhaustive and you should refer to the paunch docs for the complete list.

The json file passed to this hook is built out of the docker_config attribute defined in the service's yaml file. Refer to the Docker specific settings section for more info on this.

TripleO Heat Templates

The TripleO Heat Templates repo is where most of the logic resides in the form of heat templates. These templates define each service, the containers' configuration and the initialization or post-execution operations.

Understanding container related files

The docker templates can be found under the docker sub directory in the tripleo-heat-templates root. The services files are under docker/service but the docker directory contains a bit more than just service files and some of them are worth diving into:

deploy-steps.j2

This file is a jinja template and it's rendered before the deployment is started. This file defines the resources that are executed before and after the container initialization.

docker-puppet.py

This script is responsible for generating the config files for each service. The script is called from the deploy-steps.j2 file and it takes a json file as configuration. The json files passed to this script are built out of the puppet_config parameter set in every service template (explained in the Docker specific settings section).

The docker-puppet.py execution results in a oneshot container being executed (usually named puppet-$service_name) to generate the configuration options or run other service specific initialization tasks. Example: Create Keystone endpoints.

Anatomy of a containerized service template

Containerized services templates inherit almost everything from the puppet based templates, with some exceptions for some services. New properties have been added to define container specific configurations, which will be covered in this section.

Docker specific settings

Each service may define output variable(s) which control config file generation, initialization, and stepwise deployment of all the containers for this service. The following sections are available:

• config_settings: This setting is generally inherited from the puppet/services templates and may be appended to if required to support the docker specific config settings.
• step_config: This setting controls the manifest that is used to create docker config files via puppet. The puppet tags below are used along with this manifest to generate a config directory for this container.
• kolla_config: Contains YAML that represents how to map config files into the kolla container. This config file is typically mapped into the container itself at the /var/lib/kolla/config_files/config.json location and drives how kolla's external config mechanisms work.
• docker_config: Data that is passed to the docker-cmd hook to configure a container, or step of containers at each step. See the available steps below and the related docker-cmd hook documentation in the heat-agents project.
• puppet_config: This section is a nested set of key value pairs that drive the creation of config files using puppet. Required parameters include:
  • puppet_tags: Puppet resource tag names that are used to generate config files with puppet. Only the named config resources are used to generate a config file. Any service that specifies tags will have the default tags of 'file,concat,file_line,augeas,cron' appended to the setting. Example: keystone_config
  • config_volume: The name of the volume (directory) where config files will be generated for this service. Use this as the location to bind mount into the running Kolla container for configuration.
  • config_image: The name of the docker image that will be used for generating configuration files. This is often the same container that the runtime service uses. Some services share a common set of config files which are generated in a common base container.
  • step_config: This setting controls the manifest that is used to create docker config files via puppet. The puppet tags below are used along with this manifest to generate a config directory for this container.
• docker_puppet_tasks: This section provides data to drive the docker-puppet.py tool directly. The task is executed only once within the cluster (not on each node) and is useful for several puppet snippets we require for initialization of things like keystone endpoints, database users, etc. See docker-puppet.py for formatting.

Docker steps

Similar to baremetal, docker containers are brought up in a stepwise manner. The current architecture supports bringing up baremetal services alongside of containers. Therefore, baremetal steps may be required depending on the service and they are always executed before the corresponding container step.

The list below represents the correlation between the baremetal and the containers steps. These steps are executed sequentially:

1. Containers config files generated per hiera settings.
2. Load Balancer configuration baremetal
   1. Step 1 baremetal
   2. Step 1 containers
3. Core Services (Database/Rabbit/NTP/etc.)
   1. Step 2 baremetal
   2. Step 2 containers
4. Early Openstack Service setup (Ringbuilder, etc.)
   1. Step 3 baremetal
   2. Step 3 containers
5. General OpenStack Services
   1. Step 4 baremetal
   2. Step 4 containers
   3. Keystone containers post initialization (tenant, service, endpoint creation)
6. Service activation (Pacemaker)
   1. Step 5 baremetal
   2. Step 5 containers

Service Bootstrap

Bootstrapping services is a one-shot operation for most services and it's done by defining a separate container that shares the same structure as the main service container commonly defined under the docker_step number 3 (see Docker steps section above).

Unlike normal service containers, the bootstrap container should be run in the foreground - detach: false - so there can be more control on when the execution is done and whether it succeeded or not.

Example taken from Glance's service file:

docker_config:
  step_3:
    glance_api_db_sync:
      image: *glance_image
      net: host
      privileged: false
      detach: false
      volumes: &glance_volumes
        - /var/lib/kolla/config_files/glance-api.json:/var/lib/kolla/config_files/config.json
        - /etc/localtime:/etc/localtime:ro
        - /lib/modules:/lib/modules:ro
        - /var/lib/config-data/glance_api/:/var/lib/kolla/config_files/src:ro
        - /run:/run
        - /dev:/dev
        - /etc/hosts:/etc/hosts:ro
      environment:
        - KOLLA_BOOTSTRAP=True
        - KOLLA_CONFIG_STRATEGY=COPY_ALWAYS
  step_4:
    glance_api:
      image: *glance_image
      net: host
      privileged: false
      restart: always
      volumes: *glance_volumes
      environment:
        - KOLLA_CONFIG_STRATEGY=COPY_ALWAYS