Merge "Ansible multi-node specification"
This commit is contained in:
Jenkins
Gerrit Code Review
commit
0c5d6575ed
317
specs/ansible-multi.rst
Normal file
317
specs/ansible-multi.rst
Normal file
@ -0,0 +1,317 @@
|
||||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
==================
|
||||
Multi-node Ansible
|
||||
==================
|
||||
|
||||
This blueprint specifies an approach to automate the deployment of OpenStack
|
||||
using Ansible and Docker best practices. The overriding principles used in
|
||||
this specification are simplicity, flexibility and optimized deployment speed.
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
Kolla can be deployed multi-node currently. To do so, the environment
|
||||
variables must be hand edited to define the hosts to connect to for various
|
||||
services.
|
||||
|
||||
To implement HA in our containers, we need multi-node deployment operational
|
||||
so we can validate the high availability implementation.
|
||||
|
||||
To meet our community approved mission, we must implement a deployment tool
|
||||
and Ansible provides the most effecient development path while ensuring ease
|
||||
of use.
|
||||
|
||||
Use cases
|
||||
---------
|
||||
|
||||
1. Deploy Kolla's Docker containers multi-node and generate a one-hundred node
|
||||
working OpenStack deployment out of the box with minimal Operator
|
||||
configuration.
|
||||
2. Offer a fully customizable configuration enabling an unopinionated
|
||||
deployment of OpenStack.
|
||||
3. Upgrade an OpenStack deployment by Operator command.
|
||||
4. Offer our container content as a building block to third party downstream
|
||||
deployment tools.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
The docker-compose tool is single node and does nearly the same job as Ansible
|
||||
would in this specification. As a result, we recommend deprecating
|
||||
docker-compose as the default deployment system for Kolla.
|
||||
|
||||
To replace it, we recommend Ansible as a technology choice. Ansible is easy
|
||||
to learn, easy to use, and offers a base set of functionality to solve
|
||||
deployment as outlined in our four use cases.
|
||||
|
||||
We recommend three models of configuration.
|
||||
|
||||
The first model is based upon internally configuring the container and having
|
||||
the container take responsibility for all container configuration including
|
||||
database setup, database synchronization, and keystone registration. This
|
||||
model uses docker-compose and docker as dependencies. Existing containers will
|
||||
be maintained but new container content will use either of the two remaining
|
||||
models. James Slagle (TripleO PTL on behalf of our downstream TripleO
|
||||
community) was very clear that he would prefer to see this model stay available
|
||||
and maintained. As TripleO enters the world of Big Tent, they don't intend to
|
||||
deploy all of the services, and as such it doesn't make sense to maintain this
|
||||
legacy operational mode for new container content except on demand of our
|
||||
downstreams, hopefully with their assistance. This model is called
|
||||
CONFIG_INSIDE.
|
||||
|
||||
The second model and third model configure the containers outside of the
|
||||
container. These models depend on Ansible and Docker. In the future, the
|
||||
OpenStack Puppet, OpenStack Chef and TripleO communities may decide to switch
|
||||
to one of these two models in which case these communities may maintain tooling
|
||||
to integrate with Kolla. The major difference between these two models is that
|
||||
one offers immutability and single source of truth (CONFIG_OUTSIDE_COPY_ONCE),
|
||||
while the third model trades these two properties to allow an Operator to
|
||||
directly modify configuration files on a system and have the configuration be
|
||||
live in the container (CONFIG_OUTSIDE_COPY_ALWAYS). Because
|
||||
CONFIG_OUTSIDE_COPY_ALWAYS requires direct Operator intervention on a node, and
|
||||
we prefer as a community Operators interact with the tools provided by Kolla,
|
||||
CONFIG_OUTSIDE_COPY_ONCE will be the default.
|
||||
|
||||
We do not have to further enhance two sets of container configuration, but
|
||||
instead can focus our development effort on the default Ansible configuration
|
||||
methods. If a defect is found in one of the containers based upon the
|
||||
CONFIG_INSIDE model, the community will repair it.
|
||||
|
||||
Finally we will implement a complete Ansible deployment system. The details
|
||||
of the implementation are covered in a later section in this specification.
|
||||
We estimate this will be approximately ~1000 LOC defining ~100 Ansible tasks.
|
||||
We further estimate the total code base when complete will be under 6 KLOC.
|
||||
|
||||
The CONFIG_INSIDE model of configuration maintains the immutable,
|
||||
declarative, and idempotent nature of the Kolla containers, as defined by our
|
||||
current Kolla best practices but is opinionated in configuration.
|
||||
|
||||
The CONFIG_OUTSIDE_COPY_ONCE model of configuration maintains the immutable and
|
||||
declarative nature of the Kolla containers, as defined by our current Kolla
|
||||
best practices while introducing completely customizable configuration.
|
||||
|
||||
The CONFIG_OUTSIDE_COPY_ALWAYS model of configuration offers the Operator
|
||||
greater flexibility in managing their deployment, at greater risk of damaging
|
||||
their deployment. It trades one set of best practices for another,
|
||||
specifically the Kolla container best practices for flexibility.
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
None.
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
Multi-node deployment speed will be rapidly improved.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
The following section uses the Keystone container as an example.
|
||||
|
||||
On container start, a simple shell script will be run.
|
||||
|
||||
Passed into the container will be the CONFIG_STRATEGY environment variable with
|
||||
the following options:
|
||||
|
||||
CONFIG_STRATEGY="CONFIG_INSIDE"
|
||||
CONFIG_STRATEGY="CONFIG_OUTSIDE_COPY_ALWAYS"
|
||||
CONFIG_STRATEGY="CONFIG_OUTSIDE_COPY_ONCE"
|
||||
|
||||
CONFIG_INSIDE will match the current crudini.sh implementation.
|
||||
|
||||
The shell script will be similar in nature to the following:
|
||||
|
||||
case $CONFIG_STRATEGY in
|
||||
CONFIG_INSIDE)
|
||||
# We exec on crudini.sh to keep the same behaviour as current
|
||||
exec /crudini.sh
|
||||
;;
|
||||
CONFIG_OUTSIDE_COPY_ONCE|CONFIG_OUTSIDE_COPY_ALWAYS)
|
||||
# We source this file to allow variables to be set if needed
|
||||
source /config_outside.sh
|
||||
;;
|
||||
*)
|
||||
echo '$CONFIG_STRATEGY is not set properly'
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
exec $CONFIG_STRATEGY_BINARY_NAME
|
||||
|
||||
The crudini.sh script would be almost identical to the existing start.sh script
|
||||
while the config_outside.sh would copy the files to the appropriate location
|
||||
and set the proper permissions on those files. The $CONFIG_STRATEGY variable
|
||||
would be checked to see if the files should be copied or it should exit early.
|
||||
|
||||
The following bindmounts would be applied to the container in the above example
|
||||
for different CONFIG_STRATEGY values:
|
||||
|
||||
CONFIG_INSIDE - no bind mount
|
||||
CONFIG_OUTSIDE_COPY_ONCE - {{ HOST_CONFIG_DIR }}/keystone:/opt/kolla/configs/keystone:ro
|
||||
CONFIG_OUTSIDE_COPY_ALWAYS - {{ HOST_CONFIG_DIR }}/keystone:/opt/kolla/configs/keystone:ro
|
||||
|
||||
{{ HOST_CONFIG_DIR }} would be an Ansible variable with the default of
|
||||
/opt/kolla/configs. This same pattern will be used for most containers, unless
|
||||
there is a compelling technical reason not to do so.
|
||||
|
||||
An Ansible role represents a service in OpenStack. The Ansible role contains
|
||||
3 major sections. This same pattern will be used for all supported
|
||||
OpenStack containers.
|
||||
|
||||
Each Ansible role has a set of default key/value pairs. An example key/value
|
||||
file for Keystone is:
|
||||
|
||||
---
|
||||
container: "keystone"
|
||||
database_password: "{{ database_keystone_password }}"
|
||||
|
||||
|
||||
The second major section of a Ansible role are the role tasks. The seven
|
||||
tasks we will implement per role (i.e. OpenStack Service):
|
||||
|
||||
* bootstrap - database initialization and add roles to keystone
|
||||
* pull - pulls the latest container from the registry
|
||||
* main - Does the main job of orchestrating the role
|
||||
* config - Joins the default configuration and the user augmented
|
||||
configuration and saves the resulting file to be bind-mounted
|
||||
* start - Similar in nature to a docker compose YAML file - defines the
|
||||
defaults for the container start operation.
|
||||
* stop - Stops the container
|
||||
* upgrade - Upgrades to the latest container content
|
||||
|
||||
The details of how these role tasks operate is an implementation detail.
|
||||
|
||||
Finally each Ansible role has a default template. An example of a default
|
||||
template for Keystone is:
|
||||
|
||||
[DEFAULT]
|
||||
verbose = {{ keystone_verbose }}
|
||||
debug = {{ keystone_debug }}
|
||||
|
||||
bind_host = {{ ansible_br_mgmt['ipv4']['address'] }}
|
||||
|
||||
admin_token = {{ keystone_admin_token }}
|
||||
|
||||
public_endpoint = http://{{ keystone_service_ip }}:{{ keystone_service_public_port }}
|
||||
admin_endpoint = http://{{ keystone_service_ip }}:{{ keystone_service_admin_port }}
|
||||
|
||||
log_file = {{ keystone_log_file }}
|
||||
log_dir = {{ keystone_log_dir }}
|
||||
|
||||
[database]
|
||||
connection = mysql://{{ keystone_db_user }}:{{ database_keystone_password }}@{{ keystone_service_ip }}/keystone
|
||||
|
||||
[revoke]
|
||||
driver = keystone.contrib.revoke.backends.sql.Revoke
|
||||
|
||||
This role default will contain sufficient mandatory configuration options to
|
||||
create a working deployment. If the Operator wishes to augment the Keystone
|
||||
configuration, an augmentation file can be added to the deployment. An example
|
||||
augmentation file in /etc/kolla/keystone.aug is:
|
||||
|
||||
[DEFAULT]
|
||||
public_endpoint = https://{{ keystone_service_ip }}:{{ keystone_service_public_port }}
|
||||
|
||||
[ipman]
|
||||
life = "Two Words. Horizontal. Vertical. Make a mistake - Horizontal. Stay standing and you win."
|
||||
|
||||
This augmentation file will keep the original default configurations but
|
||||
replace public_endpoint with an https endpoint instead of an http endpoint.
|
||||
Further the [ipman] section will be added to the file placed by Ansible in
|
||||
the target host's configuration directory.
|
||||
|
||||
The end result of the merge will be a single file on the host that is in the
|
||||
appropriate format for the Openstack service to consume containing the content
|
||||
of both the Ansible default file and the augmentation file.
|
||||
|
||||
The final implication of these Ansible best practices is that an Operator can
|
||||
deploy in 1 hour or less a one-hundred node OpenStack deployment out of the
|
||||
box using Kolla containers with Ansible deployment tooling with minimal
|
||||
configuration. If additional customization is required for the Operator's
|
||||
environment, this can be achieved via augmentation files developed by the
|
||||
Operator.
|
||||
|
||||
NB: to override any default key/value pair (the key is located in {{ }} above
|
||||
and replaced by the value by Ansible), there is one global override file to
|
||||
configure the deployment called /etc/kolla/globals.yml
|
||||
|
||||
We will implement a simple shell script called kolla-ansible which wraps
|
||||
ansible-playbook. It will implement four commands which operate on the
|
||||
OpenStack deployment globally. It will automatically load the globals.yml
|
||||
overrides and an invetory file located in /etc/kolla executing the appropriate
|
||||
roles for all of the deployed containers. The initial supported
|
||||
commands are:
|
||||
|
||||
1. kolla-ansible deploy
|
||||
2. kolla-ansible start
|
||||
3. kolla-ansible stop
|
||||
4. kolla-ansible upgrade
|
||||
|
||||
Ansible supports a model of deployment using an inventory file. The inventory
|
||||
file specifies which nodes get assigned which roles. For an example of an
|
||||
inventory file, see:
|
||||
|
||||
https://github.com/SamYaple/yaodu/blob/master/ansible/inventories/production
|
||||
|
||||
To the untrained eye, this looks like a bunch of heavy wizardy. I personally
|
||||
believe we will in some way merge our globals.yml and the inventory file into
|
||||
one master configuration file and generate the globals.yml and Ansible-specific
|
||||
inventory file on each kolla-ansible operation. The long term goal is to get
|
||||
to one configuration file with "all the things" needed to deploy OpenStack.
|
||||
This would permit a GUI to simply configure the deployment.
|
||||
|
||||
How this is done or if it is done remains an implementation detail which may
|
||||
warrant expanding this specification or a completely new specification in the
|
||||
future. As we obtain more experience with what we are developing, we will
|
||||
have a more complete picture of what this master configuration file format will be.
|
||||
|
||||
The implementation described in this section is just a sample of the
|
||||
implementation details required. We intend to refactor Sam Yaple's fantastic
|
||||
vision with yaodu (https://github.com/SamYaple/yaodu/) into Kolla to
|
||||
implement Ansible deployment of OpenStack while retaining Kolla, Docker, and
|
||||
Ansible best practices and conventions.
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignees:
|
||||
|
||||
diga
|
||||
fangfenghua
|
||||
harmw
|
||||
samyaple
|
||||
sdake
|
||||
|
||||
The kolla core team will support and execute this specification through normal
|
||||
workflow operations.
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
1. Convert all fat containers to thin containers to facilitate this work.
|
||||
2. Move all start.sh scripts to crudini.sh and create the function to execute
|
||||
the configuration strategy across containers.
|
||||
3. Rename the kolla script to kolla-compose and create a new kolla-ansible
|
||||
script to manage playbook operation.
|
||||
4. Refactor the remaining portions of yaodu that are compatible with Kolla into
|
||||
the Kolla code base.
|
||||
5. Implement our existing crudini defaults in Ansible.
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Functional tests will be implemented in the OpenStack check/gating system to
|
||||
automatically check that the Ansible deployment works for an AIO environment.
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
The developer quickstart must be augmented with instructions to use the new
|
||||
Ansible deployment methodology.
|
||||
Loading…
Reference in New Issue
Block a user