From ee68bc385f7b7494c4f19ea9adfba1990f6acc1f Mon Sep 17 00:00:00 2001 From: Mark Goddard Date: Tue, 21 Feb 2017 08:53:02 -0500 Subject: [PATCH] Improve developer documentation for Bifrost This change fills out the Bifrost developer documentation and makes a few corrections. Change-Id: I14a6eca58c440677d5de638cbc677e9ec05738af Closes-Bug: #1661013 Co-Authored-By: Sayantani Goswami --- doc/bifrost.rst | 407 ++++++++++++++++++++++++++++++++---------------- 1 file changed, 273 insertions(+), 134 deletions(-) diff --git a/doc/bifrost.rst b/doc/bifrost.rst index a379de04af..82af61d946 100644 --- a/doc/bifrost.rst +++ b/doc/bifrost.rst @@ -2,31 +2,87 @@ Bifrost Guide ============= +From the bifrost developer documentation: -Prep host -========= + Bifrost (pronounced bye-frost) is a set of Ansible playbooks that automates + the task of deploying a base image onto a set of known hardware using + Ironic. It provides modular utility for one-off operating system + deployment with as few operational requirements as reasonably possible. -Clone kolla ------------ +Kolla uses bifrost as a mechanism for bootstrapping an OpenStack control plane +on a set of baremetal servers. Kolla provides a container image for bifrost. +Kolla-ansible provides a playbook to configure and deploy the bifrost +container, as well as building a base OS image and provisioning it onto the +baremetal nodes. -:: +Hosts in the System +=================== - git clone https://github.com/openstack/kolla - cd kolla +In a system deployed by bifrost we define a number of classes of hosts. -set up kolla dependencies :doc:`quickstart` +Control host + The control host is the host on which kolla and kolla-ansible will be + installed, and is typically where the cloud will be managed from. +Deployment host + The deployment host runs the bifrost deploy container and is used to + provision the cloud hosts. +Cloud hosts + The cloud hosts run the OpenStack control plane, compute and storage + services. +Bare metal compute hosts: + In a cloud providing bare metal compute services to tenants via Ironic, + these hosts will run the bare metal tenant workloads. In a cloud with only + virtualised compute this category of hosts does not exist. -Fix hosts file --------------- +.. note:: -Docker bind mounts ``/etc/hosts`` into the container from a volume -This prevents atomic renames which will prevent ansible from fixing -the ``/etc/hosts`` file automatically. + In many cases the control and deployment host will be the same, although + this is not mandatory. -To enable bifrost to be bootstrapped correctly add the deployment -hosts hostname to 127.0.0.1 line for example: +.. note:: -:: + Bifrost supports provisioning of bare metal nodes. While kolla-ansible is + agnostic to whether the host OS runs on bare metal or is virtualised, in a + virtual environment the provisioning of VMs for cloud hosts and their base + OS images is currently out of scope. + +Cloud Deployment Procedure +========================== + +Cloud deployment using kolla and bifrost follows the following high level +steps: + +#. Install and configure kolla and kolla-ansible on the control host. +#. Deploy bifrost on the deployment host. +#. Use bifrost to build a base OS image and provision cloud hosts with this + image. +#. Deploy OpenStack services on the cloud hosts provisioned by bifrost. + +Preparation +=========== + +Prepare the Control Host +------------------------ + +Follow the **Install dependencies** section of the :doc:`quickstart` guide +instructions to set up kolla and kolla-ansible dependencies. Follow the +instructions in either the **Install kolla for development** section or the +**Install kolla for deployment or evaluation** section to install kolla and +kolla-ansible. + +Prepare the Deployment Host +--------------------------- + +RabbitMQ requires that the system's hostname resolves to the IP address that it +has been configured to use, which with bifrost will be ``127.0.0.1``. Bifrost +will attempt to modify ``/etc/hosts`` on the deployment host to ensure that +this is the case. Docker bind mounts ``/etc/hosts`` into the container from a +volume. This prevents atomic renames which will prevent Ansible from fixing +the +``/etc/hosts`` file automatically. + +To enable bifrost to be bootstrapped correctly add an entry to ``/etc/hosts`` +resolving the deployment host's hostname to ``127.0.0.1``, for example:: ubuntu@bifrost:/repo/kolla$ cat /etc/hosts 127.0.0.1 bifrost localhost @@ -40,50 +96,70 @@ hosts hostname to 127.0.0.1 line for example: ff02::3 ip6-allhosts 192.168.100.15 bifrost +Build a Bifrost Container Image +=============================== -Enable source build type -======================== +This section provides instructions on how to build a container image for +bifrost using kolla. -Via config file ---------------- +Enable Source Build Type +------------------------ -:: +Currently kolla only supports the source install type for the bifrost image. +Configuration File +~~~~~~~~~~~~~~~~~~ + +If required, generate a default configuration file for ``kolla-build``:: + + cd kolla tox -e genconfig -Modify ``kolla-build.conf`` as follows. -Set ``install_type`` to ``source`` - -:: +Modify ``kolla-build.conf``, setting ``install_type`` to ``source``:: install_type = source Command line ------------- +~~~~~~~~~~~~ -Alternatively if you do not wish to use the ``kolla-build.conf`` -you can enable a source build by appending ``-t source`` to -your ``kolla-build`` or ``tools/build.py`` command. +Alternatively, instead of using ``kolla-build.conf``, a source build can be +enabled by appending ``--type source`` to the ``kolla-build`` or +``tools/build.py`` command. -Build container -=============== +Build Container +--------------- Development ------------ +~~~~~~~~~~~ :: + cd kolla tools/build.py bifrost-deploy Production ----------- +~~~~~~~~~~ :: kolla-build bifrost-deploy +.. note:: + + By default kolla-build will build all containers using CentOS as the base + image. To change this behavior, use the following parameter with + ``kolla-build`` or ``tools/build.py`` command:: + + --base [ubuntu|centos|oraclelinux] + +Configure and Deploy a Bifrost Container +======================================== + +This section provides instructions for how to configure and deploy a container +running bifrost services. + Prepare Kolla-Ansible Inventory -=============================== +------------------------------- Kolla-ansible will deploy bifrost on the hosts in the ``bifrost`` Ansible group. In the ``all-in-one`` and ``multinode`` inventory files, a ``bifrost`` @@ -100,85 +176,131 @@ services deployed by kolla including OpenStack Ironic, MariaDB, RabbitMQ and (optionally) OpenStack Keystone. These services should not be deployed on the host on which bifrost is deployed. -Prepare bifrost configs -======================= +Prepare Kolla-Ansible Configuration +----------------------------------- -Create servers.yml ------------------- +Follow the instructions in :doc:`quickstart` to prepare kolla-ansible's global +configuration file ``globals.yml``. For bifrost, the +``bifrost_network_interface`` variable should be set to the name of the +interface that will be used to provision bare metal cloud hosts if this is +different than ``network_interface``. For example to use ``eth1``: -The ``servers.yml`` will describing your physical nodes and list IPMI -credentials. See bifrost dynamic inventory examples for more details. +.. code-block:: yaml -For example ``/etc/kolla/config/bifrost/servers.yml`` + bifrost_network_interface: eth1 + +Note that this interface should typically have L2 network connectivity with the +bare metal cloud hosts in order to provide DHCP leases with PXE boot options. + +Since bifrost only supports the source image type, ensure that this is +reflected in ``globals.yml`` + +.. code-block:: yaml + + kolla_install_type: source + +Prepare Bifrost Configuration +----------------------------- + +Kolla ansible custom configuration files can be placed in a directory given by +the ``node_custom_config`` variable, which defaults do ``/etc/kolla/config``. +Bifrost configuration files should be placed in this directory or in a +``bifrost`` subdirectory of it (e.g. ``/etc/kolla/config/bifrost``). Within +these directories the files ``bifrost.yml``, ``servers.yml`` and ``dib.yml`` +can be used to configure Bifrost. + +Create a Bifrost Inventory +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The file ``servers.yml`` defines the bifrost hardware inventory that will be +used to populate Ironic. See the `bifrost dynamic inventory examples +`_ for +further details. + +For example, the following inventory defines a single node managed via the +Ironic ``agent_ipmitool`` driver. The inventory contains credentials required +to access the node's BMC via IPMI, the MAC addresses of the node's NICs, an IP +address to configure the node's configdrive with, a set of scheduling +properties and a logical name. .. code-block:: yaml --- cloud1: - uuid: "31303735-3934-4247-3830-333132535336" - driver_info: - power: - ipmi_username: "admin" - ipmi_address: "192.168.1.30" - ipmi_password: "root" - nics: - - - mac: "1c:c1:de:1c:aa:53" - - - mac: "1c:c1:de:1c:aa:52" - driver: "agent_ipmitool" - ipv4_address: "192.168.1.10" - properties: - cpu_arch: "x86_64" - ram: "24576" - disk_size: "120" - cpus: "16" - name: "cloud1" + uuid: "31303735-3934-4247-3830-333132535336" + driver_info: + power: + ipmi_username: "admin" + ipmi_address: "192.168.1.30" + ipmi_password: "root" + nics: + - + mac: "1c:c1:de:1c:aa:53" + - + mac: "1c:c1:de:1c:aa:52" + driver: "agent_ipmitool" + ipv4_address: "192.168.1.10" + properties: + cpu_arch: "x86_64" + ram: "24576" + disk_size: "120" + cpus: "16" + name: "cloud1" -adjust as appropriate for your deployment +The required inventory will be specific to the hardware and environment in use. -Create bifrost.yml ------------------- -By default kolla mostly use bifrosts default playbook values. -Parameters passed to the bifrost install playbook can be overridden by -creating a ``bifrost.yml`` file in the kolla custom config directory or in a -bifrost sub directory. -For example ``/etc/kolla/config/bifrost/bifrost.yml`` +Create Bifrost Configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -:: +The file ``bifrost.yml`` provides global configuration for the bifrost +playbooks. By default kolla mostly uses bifrost's default variable values. +For details on bifrost's variables see the bifrost documentation. - mysql_service_name: mysql - ansible_python_interpreter: /var/lib/kolla/venv/bin/python - network_interface: < add you network interface here > - # uncomment below if needed - # dhcp_pool_start: 192.168.2.200 - # dhcp_pool_end: 192.168.2.250 - # dhcp_lease_time: 12h - # dhcp_static_mask: 255.255.255.0 +For example: -Create Disk Image Builder Config --------------------------------- -By default kolla mostly use bifrosts default playbook values when -building the baremetal os image. The baremetal os image can be customised -by creating a ``dib.yml`` file in the kolla custom config directory or in a -bifrost sub directory. -For example ``/etc/kolla/config/bifrost/dib.yml`` +.. code-block:: yaml -:: + mysql_service_name: mysql + ansible_python_interpreter: /var/lib/kolla/venv/bin/python + enabled_drivers: agent_ipmitool,agent_ipminative + # uncomment below if needed + # dhcp_pool_start: 192.168.2.200 + # dhcp_pool_end: 192.168.2.250 + # dhcp_lease_time: 12h + # dhcp_static_mask: 255.255.255.0 - dib_os_element: ubuntu +Create Disk Image Builder Configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The file ``dib.yml`` provides configuration for bifrost's image build +playbooks. By default kolla mostly uses bifrost's default variable values when +building the baremetal OS and deployment images, and will build an +**Ubuntu-based** image for deployment to nodes. For details on bifrost's +variables see the bifrost documentation. + +For example to use the ``debian`` Disk Image Builder OS element: + +.. code-block:: yaml + + dib_os_element: debian + +See the `diskimage-builder documentation +`__ for more details. Deploy Bifrost -========================= +-------------- -Ansible -------- +The bifrost container can be deployed either using kolla-ansible or manually. + +Kolla-Ansible +~~~~~~~~~~~~~ Development ___________ :: + cd kolla-ansible tools/kolla-ansible deploy-bifrost Production @@ -189,16 +311,19 @@ __________ kolla-ansible deploy-bifrost Manual ------- +~~~~~~ Start Bifrost Container _______________________ + :: - docker run -it --net=host -v /dev:/dev -d --privileged --name bifrost_deploy kolla/ubuntu-source-bifrost-deploy:3.0.1 + docker run -it --net=host -v /dev:/dev -d \ + --privileged --name bifrost_deploy \ + kolla/ubuntu-source-bifrost-deploy:3.0.1 -Copy configs -____________ +Copy Configuration Files +________________________ .. code-block:: console @@ -207,39 +332,40 @@ ____________ docker cp /etc/kolla/config/bifrost/bifrost.yml bifrost_deploy:/etc/bifrost/bifrost.yml docker cp /etc/kolla/config/bifrost/dib.yml bifrost_deploy:/etc/bifrost/dib.yml -Bootstrap bifrost +Bootstrap Bifrost _________________ :: docker exec -it bifrost_deploy bash -Generate ssh key -~~~~~~~~~~~~~~~~ +Generate an SSH Key +___________________ :: ssh-keygen -Source env variables -~~~~~~~~~~~~~~~~~~~~ +Bootstrap and Start Services +____________________________ -:: - - cd /bifrost - . env-vars - . /opt/stack/ansible/hacking/env-setup - cd playbooks/ - - -Bootstrap and start services -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: console - ansible-playbook -vvvv -i /bifrost/playbooks/inventory/localhost /bifrost/playbooks/install.yaml -e @/etc/bifrost/bifrost.yml + cd /bifrost + ./scripts/env-setup.sh + . env-vars + cat > /etc/rabbitmq/rabbitmq-env.conf << EOF + HOME=/var/lib/rabbitmq + EOF + ansible-playbook -vvvv \ + -i /bifrost/playbooks/inventory/target \ + /bifrost/playbooks/install.yaml \ + -e @/etc/bifrost/bifrost.yml \ + -e @/etc/bifrost/dib.yml \ + -e skip_package_install=true -Check ironic is running -======================= +Validate the Deployed Container +=============================== .. code-block:: console @@ -257,80 +383,93 @@ Running "ironic node-list" should return with no nodes, for example +------+------+---------------+-------------+--------------------+-------------+ +------+------+---------------+-------------+--------------------+-------------+ - Enroll and Deploy Physical Nodes ================================ -Ansible -------- +Once we have deployed a bifrost container we can use it to provision the bare +metal cloud hosts specified in the inventory file. Again, this can be done +either using kolla-ansible or manually. + +Kolla-Ansible +------------- Development -___________ +~~~~~~~~~~~ :: tools/kolla-ansible deploy-servers Production -__________ +~~~~~~~~~~ :: kolla-ansible deploy-servers - Manual ------ + .. code-block:: console docker exec -it bifrost_deploy bash cd /bifrost . env-vars export BIFROST_INVENTORY_SOURCE=/etc/bifrost/servers.yml - ansible-playbook -vvvv -i inventory/bifrost_inventory.py enroll-dynamic.yaml -e "ansible_python_interpreter=/var/lib/kolla/venv/bin/python" -e network_interface= + ansible-playbook -vvvv \ + -i /bifrost/playbooks/inventory/bifrost_inventory.py \ + /bifrost/playbooks/enroll-dynamic.yaml \ + -e "ansible_python_interpreter=/var/lib/kolla/venv/bin/python" \ + -e @/etc/bifrost/bifrost.yml docker exec -it bifrost_deploy bash cd /bifrost . env-vars export BIFROST_INVENTORY_SOURCE=/etc/bifrost/servers.yml - ansible-playbook -vvvv -i inventory/bifrost_inventory.py deploy-dynamic.yaml -e "ansible_python_interpreter=/var/lib/kolla/venv/bin/python" -e network_interface= -e @/etc/bifrost/dib.yml + ansible-playbook -vvvv \ + -i /bifrost/playbooks/inventory/bifrost_inventory.py \ + /bifrost/playbooks/deploy-dynamic.yaml \ + -e "ansible_python_interpreter=/var/lib/kolla/venv/bin/python" \ + -e @/etc/bifrost/bifrost.yml -At this point ironic should clean down your nodes and install the default -os image. +At this point Ironic should clean down the nodes and install the default +OS image. -Advanced configuration +Advanced Configuration ====================== -Bring your own image +Bring Your Own Image -------------------- + TODO -Bring your own ssh key +Bring Your Own SSH Key ---------------------- -To use your own ssh key after you have generated the ``passwords.yml`` file -update the private and public keys under bifrost_ssh_key. + +To use your own SSH key after you have generated the ``passwords.yml`` file +update the private and public keys under ``bifrost_ssh_key``. Known issues ============ SSH daemon not running ---------------------- -By default sshd is installed in the image but may not be enabled. -If you encounter this issue you will have to access the server physically in -recovery mode to enable the ssh service. If your hardware supports it, this -can be done remotely with ipmitool and serial over lan. For example + +By default ``sshd`` is installed in the image but may not be enabled. If you +encounter this issue you will have to access the server physically in recovery +mode to enable the ``sshd`` service. If your hardware supports it, this can be +done remotely with ``ipmitool`` and Serial Over LAN. For example .. code-block:: console ipmitool -I lanplus -H 192.168.1.30 -U admin -P root sol activate - References ========== -Docs: http://docs.openstack.org/developer/bifrost/ +Bifrost documentation: http://docs.openstack.org/developer/bifrost/ -Troubleshooting: http://docs.openstack.org/developer/bifrost/troubleshooting.html +Bifrost troubleshooting guide: http://docs.openstack.org/developer/bifrost/troubleshooting.html -Code: https://github.com/openstack/bifrost +Bifrost code repository: https://github.com/openstack/bifrost