ironic-python-agent/doc/source/index.rst

===============================
Welcome to Ironic Python Agent!
===============================

Overview
========

Ironic Python Agent is an agent for controlling and deploying Ironic controlled
baremetal nodes. Typically run in a ramdisk, the agent exposes a REST API for
provisioning servers.

Throughout the remainder of the document, Ironic Python Agent will be
abbreviated to IPA.

Index
=====

.. toctree::

    troubleshooting
    metrics

How it works
============

Integration with Ironic
-----------------------

Compatible Deploy Drivers
~~~~~~~~~~~~~~~~~~~~~~~~~

Agent Deploy Driver
<<<<<<<<<<<<<<<<<<<
IPA works with the agent Deploy driver in Ironic to provision nodes. Starting
with ironic-python-agent running on a ramdisk on an unprovisioned node,
Ironic makes API calls to ironic-python-agent to provision the machine. This
allows for greater control and flexibility of the entire deployment process.

PXE Deploy Driver
<<<<<<<<<<<<<<<<<
IPA may also be used with the original Ironic pxe driver as of the Kilo
OpenStack Ironic release.

Configuring Deploy Drivers
<<<<<<<<<<<<<<<<<<<<<<<<<<
For information on how to install and configure Ironic drivers, including
drivers for IPA, see the Ironic drivers documentation [0]_.

Lookup
~~~~~~
On startup, the agent performs a lookup in Ironic to determine its node UUID
by sending a hardware profile to the Ironic lookup endpoint:
``/v1/lookup``.

Heartbeat
~~~~~~~~~
After successfully looking up its node, the agent heartbeats via
``/v1/heartbeat/{node_ident}`` every N seconds, where
N is the Ironic conductor's agent.heartbeat_timeout value multiplied by a
number between .3 and .6.

For example, if your conductor's ironic.conf contains::

  [agent]
  heartbeat_timeout = 60

IPA will heartbeat between every 20 and 36 seconds. This is to ensure jitter
for any agents reconnecting after a network or API disruption.

After the agent heartbeats, the conductor performs any actions needed against
the node, including querying status of an already run command. For example,
initiating in-band cleaning tasks or deploying an image to the node.

Inspection
~~~~~~~~~~
IPA can conduct hardware inspection on start up and post data to the `Ironic
Inspector`_. Edit your default PXE/iPXE configuration or IPA options
baked in the image, and set ``ipa-inspection-callback-url`` to the
full endpoint of Ironic Inspector, for example::

    ipa-inspection-callback-url=http://IP:5050/v1/continue

Make sure your DHCP environment is set to boot IPA by default.

.. _Ironic Inspector: http://docs.openstack.org/developer/ironic-inspector/

Hardware Inventory
------------------
IPA collects various hardware information using its `Hardware Managers`_,
and sends it to Ironic on lookup and to Ironic Inspector on Inspection_.

The exact format of the inventory depends on the hardware manager used.
Here is the basic format expected to be provided by all hardware managers.
The inventory is a dictionary (JSON object), containing at least the following
fields:

``cpu``
    CPU information: ``model_name``, ``frequency``, ``count``,
    ``architecture`` and ``flags``.

``memory``
    RAM information: ``total`` (total size in bytes), ``physical_mb``
    (physically installed memory size in MiB, optional).

    .. note::
        The difference is that the latter includes the memory region reserved
        by the kernel and is always slightly bigger. It also matches what
        the Nova flavor would contain for this node and thus is used by the
        inspection process instead of ``total``.

``bmc_address``
    IP address of the node's BMC (aka IPMI address), optional.

``disks``
    list of disk block devices with fields: ``name``, ``model``,
    ``size`` (in bytes), ``rotational`` (boolean), ``wwn``, ``serial``,
    ``vendor``, ``wwn_with_extension``, ``wwn_vendor_extension``, ``hctl``.

``interfaces``
    list of network interfaces with fields: ``name``, ``mac_address``,
    ``ipv4_address``, ``lldp``, ``vendor``, ``product``, and optionally
    ``biosdevname``(BIOS given NIC name). If configuration option
    ``collect_lldp`` is set to True the ``lldp`` field will be populated
    by a list of type-length-value(TLV) fields retrieved using the
    Link Layer Discovery Protocol (LLDP).

``system_vendor``
    system vendor information from SMBIOS as reported by ``dmidecode``:
    ``product_name``, ``serial_number`` and ``manufacturer``.

``boot``
    boot information with fields: ``current_boot_mode`` (boot mode used for
    the current boot - BIOS or UEFI) and ``pxe_interface`` (interface used
    for PXE booting, if any).

Image Builders
--------------
Unlike most other python software, you must build an IPA ramdisk image before
use. This is because it's not installed in an operating system, but instead is
run from within a ramdisk.

CoreOS
~~~~~~
One way to build a ramdisk image for IPA is with the CoreOS image [1]_.
Prebuilt copies of the CoreOS image, suitable for pxe, are available on
`tarballs.openstack.org <http://tarballs.openstack.org/ironic-python-agent/coreos/files/>`__.

Build process
<<<<<<<<<<<<<
On a high level, the build steps are as follows:

1) A docker build is performed using the ``Dockerfile`` in the root of the
   ironic-python-agent project.
2) The resulting docker image is exported to a filesystem image.
3) The filesystem image, along with a cloud-config.yml [2]_, are embedded into
   the CoreOS PXE image at /usr/share/oem/.
4) On boot, the ironic-python-agent filesystem image is extracted and run
   inside a systemd-nspawn container. /usr/share/oem is mounted into this
   container as /mnt.

Customizing the image
<<<<<<<<<<<<<<<<<<<<<
There are several methods you can use to customize the IPA ramdisk:

* Embed SSH keys by putting an authorized_keys file in /usr/share/oem/
* Add your own hardware managers by modifying the Dockerfile to install
  additional python packages.
* Modify the cloud-config.yml [2]_ to perform additional tasks at boot time.

diskimage-builder
~~~~~~~~~~~~~~~~~~
Another way to build a ramdisk image for IPA is by using diskimage-builder
[3]_. The ironic-agent diskimage-builder element builds the IPA ramdisk, which
installs all the required packages and configures services as needed.

tinyipa
~~~~~~~

Ironic Python Agent repo also provides a set of scripts to build a
Tiny Core Linux-based deployment kernel and ramdisk (code name ``tinyipa``)
under ``imagebuild/tinyipa`` folder.

`Tiny Core Linux <http://tinycorelinux.net/>`_
is a very minimalistic Linux distribution.
Due to its small size and decreased RAM requirements
it is mostly suitable for usage in CI with virtualized hardware,
and is already used on a number of gate jobs in projects under
OpenStack Baremetal program.
On the other hand, due to its generally newer Linux kernel it also known to
work on real hardware if the kernel supports all necessary components
installed.

Please refer to ``imagebuild/tinyipa/README.rst`` for more information and
build instructions.

ISO Images
~~~~~~~~~~
Additionally, the IPA ramdisk can be packaged inside of an ISO for use with
supported virtual media drivers. Simply use the ``iso-image-create`` utility
packaged with IPA, pass it an initrd and kernel. e.g.::

  ./iso-image-create -o /path/to/output.iso -i /path/to/ipa.initrd -k /path/to/ipa.kernel

This is a generic tool that can be used to combine any initrd and kernel into
a suitable ISO for booting, and so should work against any IPA ramdisk created
-- both DIB and CoreOS.

IPA Flags
=========

You can pass a variety of flags to IPA on start up to change its behavior.
If you're using the CoreOS image, you can modify the
ironic-python-agent.service unit in cloud-config.yaml [5]_.

* ``--standalone``: This disables the initial lookup and heartbeats to Ironic.
  Lookup sends some information to Ironic in order to determine Ironic's node
  UUID for the node. Heartbeat sends periodic pings to Ironic to tell Ironic
  the node is still running. These heartbeats also trigger parts of the deploy
  and cleaning cycles. This flag is useful for debugging IPA without an Ironic
  installation.

* ``--debug``: Enables debug logging.


IPA and SSL
===========

During its operation IPA makes HTTP requests to a number of other services,
currently including

- ironic for lookup/heartbeats
- ironic-inspector to publish results of introspection
- HTTP image storage to fetch the user image to be written to the node's disk
  (Object storage service or other service storing user images
  when ironic is running in a standalone mode)

When these services are configured to require SSL-encrypted connections,
IPA can be configured to either properly use such secure connections or
ignore verifying such SSL connections.

Configuration mostly happens in the IPA config file
(default is ``/etc/ironic_python_agent/ironic_python_agent.conf``)
or command line arguments passed to ``ironic-python-agent``,
and it is possible to provide some options via kernel command line arguments
instead.

Available options in the ``[DEFAULT]`` config file section are:

insecure
  Whether to verify server SSL certificates.
  When not specified explicitly, defaults to the value of ``ipa-insecure``
  kernel command line argument (converted to boolean).
  The default for this kernel command line argument is taken to be ``False``.
  Overriding it to ``True`` by adding ``ipa-insecure=1`` to the value of
  ``[pxe]pxe_append_params`` in ironic configuration file will allow running
  the same IPA-based deploy ramdisk in a CI-like environment when services
  are using secure HTTPS endpoints with self-signed certificates without
  adding a custom CA file to the deploy ramdisk (see below).

cafile
  Path to the PEM encoded Certificate Authority file.
  When not specified, available system-wide list of CAs will be used to
  verify server certificates.
  Thus in order to use IPA with HTTPS endpoints of other services in
  a secure fashion (with ``insecure`` option being ``False``, see above),
  operators should either ensure that certificates of those services
  are verifiable by root CAs present in the deploy ramdisk,
  or add a custom CA file to the ramdisk and set this IPA option to point
  to this file at ramdisk build time.

certfile
  Path to PEM encoded client certificate cert file.
  This option must be used when services are configured to require client
  certificates on SSL-secured connections.
  This cert file must be added to the deploy ramdisk and path
  to it specified for IPA via this option at ramdisk build time.
  This option has an effect only when the ``keyfile`` option is also set.

keyfile
  Path to PEM encoded client certificate key file.
  This option must be used when services are configured to require client
  certificates on SSL-secured connections.
  This key file must be added to the deploy ramdisk and path
  to it specified for IPA via this option at ramdisk build time.
  This option has an effect only when the ``certfile`` option is also set.

Currently a single set of cafile/certfile/keyfile options is used for all
HTTP requests to the other services.

Securing IPA's HTTP server itself with SSL is not yet supported in default
ramdisk builds.

Hardware Managers
=================

What is a HardwareManager?
--------------------------
Hardware managers are how IPA supports multiple different hardware platforms
in the same agent. Any action performed on hardware can be overridden by
deploying your own hardware manager.

How are methods executed on HardwareManagers?
---------------------------------------------
Methods that modify hardware are dispatched to each hardware manager in
priority order. When a method is dispatched, if a hardware manager does not
have a method by that name or raises `IncompatibleHardwareMethodError`, IPA
continues on to the next hardware manager. Any hardware manager that returns
a result from the method call is considered a success and its return value
passed on to whatever dispatched the method. If the method is unable to run
successfully on any hardware managers, `HardwareManagerMethodNotFound` is
raised.

Does IPA ship with a HardwareManager?
-------------------------------------
IPA ships with GenericHardwareManager, which implements basic cleaning and
deployment methods compatible with most hardware.

Why build a custom HardwareManager?
-----------------------------------
Custom hardware managers allow you to include hardware-specific tools, files
and cleaning steps in the Ironic Python Agent. For example, you could include a
BIOS flashing utility and BIOS file in a custom ramdisk. Your custom
hardware manager could expose a cleaning step that calls the flashing utility
and flashes the packaged BIOS version (or even download it from a tested web
server).

How can I build a custom HardwareManager?
-----------------------------------------
Custom HardwareManagers should subclass hardware.HardwareManager or
hardware.GenericHardwareManager. The only required method is
evaluate_hardware_support(), which should return one of the enums
in hardware.HardwareSupport. Hardware support determines which hardware
manager is executed first for a given function (see: "`How are methods
executed on HardwareManagers?`_" for more info). Common methods you
may want to implement are list_hardware_info(), to add additional hardware
the GenericHardwareManager is unable to identify and erase_devices(), to
erase devices in ways other than ATA secure erase or shredding.

Custom HardwareManagers and Cleaning
------------------------------------
One of the reasons to build a custom hardware manager is to expose extra steps
in Ironic Cleaning [4]_. A node will perform a set of cleaning steps any
time the node is deleted by a tenant or moved from MANAGEABLE state to
AVAILABLE state. If the node is using an agent\_\* driver, Ironic will query
IPA for a list of clean steps that should be executed on the node. IPA
will dispatch a call to `get_clean_steps()` on all available hardware managers
and then return the combined list to Ironic.

To expose extra clean steps, the custom hardware manager should have a function
named `get_clean_steps()` which returns a list of dictionaries. The
dictionaries should be in the form::

    {
        // A function on the custom hardware manager
        'step': 'upgrade_firmware',
        // An integer priority. Largest priorities are executed first
        'priority': 10,
        // Should always be the deploy interface
        'interface': 'deploy',
        // Request the node to be rebooted out of band by Ironic when the
        // step completes successfully
        'reboot_requested': False
    }

Then, you should create functions which match each of the `step` keys in
the clean steps you return. The functions will take two parameters: `node`,
a dictionary representation of the Ironic node, and `ports`, a list of
dictionary representations of the Ironic ports attached to `node`.

When a clean step is executed in IPA, the `step` key will be sent to the
hardware managers in hardware support order, using
`hardware.dispatch_to_managers()`. For each hardware manager, if the manager
has a function matching the `step` key, it will be executed. If the function
returns a value (including None), that value is returned to Ironic and no
further managers are called. If the function raises
`IncompatibleHardwareMethodError`, the next manager will be called. If the
function raises any other exception, the command will be considered failed,
the command result's error message will be set to the exception's error
message, and no further managers will be called. An example step::

    def upgrade_firmware(self, node, ports):
        if self._device_exists():
            # Do the upgrade
            return 'upgraded firmware'
        else:
            raise errors.IncompatibleHardwareMethodError()

.. note::

    If two managers return steps with the same `step` key, the priority will
    be set to whichever manager has a higher hardware support level and then
    use the higher priority in the case of a tie.

Versioning
~~~~~~~~~~
Each hardware manager has a name and a version. This version is used during
cleaning to ensure the same version of the agent is used to on a node through
the entire process. If the version changes, cleaning is restarted from the
beginning to ensure consistent cleaning operations and to make
updating the agent in production simpler.

You can set the version of your hardware manager by creating a class variable
named 'HARDWARE_MANAGER_VERSION', which should be a string. The default value
is '1.0'. You should change this version string any time you update your
hardware manager. You can also change the name your hardware manager presents
by creating a class variable called HARDWARE_MANAGER_NAME, which is a string.
The name defaults to the class name. Currently IPA only compares version as a
string; any version change whatsoever will induce cleaning to restart.

Priority
~~~~~~~~
A hardware manager has a single overall priority, which should be based on how
well it supports a given piece of hardware. At load time, IPA executes
`evaluate_hardware_support()` on each hardware manager. This method should
return an int representing hardware manager priority, based on what it detects
about the platform it's running on. Suggested values are included in the
`HardwareSupport` class. Returning a value of 0 aka `HardwareSupport.NONE`,
will prevent the hardware manager from being used. IPA will never ship a
hardware manager with a priority higher than 3, aka
`HardwareSupport.SERVICE_PROVIDER`.


Generated Developer Documentation
=================================

.. toctree::
   :maxdepth: 1

   api/autoindex


References
==========
.. [0] Enabling Drivers - http://docs.openstack.org/developer/ironic/drivers/ipa.html
.. [1] CoreOS PXE Images - https://coreos.com/docs/running-coreos/bare-metal/booting-with-pxe/
.. [2] CoreOS Cloud Init - https://coreos.com/docs/cluster-management/setup/cloudinit-cloud-config/
.. [3] DIB Element for IPA - http://docs.openstack.org/developer/diskimage-builder/elements/ironic-agent/README.html
.. [4] Ironic Cleaning - http://docs.openstack.org/developer/ironic/deploy/cleaning.html
.. [5] cloud-config.yaml - https://git.openstack.org/cgit/openstack/ironic-python-agent/tree/imagebuild/coreos/oem/cloud-config.yml

Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`