openstack-ansible-os_ironic/doc/source/configure-lxc-example.rst

===================================
Example LXC based Ironic deployment
===================================

This section describes a specific deployment of Ironic using
Openstack-Ansible. A number of design choices are made which
illustrate how to configure the Ironic service for a specific
set of requirements.

Deployment design decisions:

* LXC containers are used in the openstack control plane
* A single ``bmaas`` network is used for Ironic provisioning,
  cleaning and inspection
* The ``bmaas`` network is not routable to any other networks
  nor to the internal VIP
* Multitenancy is used with Ironic servers attached to project
  networks
* networking-generic-switch will be used to control network devices
  to achieve multitenancy
* Cisco NXOS switches
* The deployment uses VXLAN project networks

A number of these design decisions are opinionated and could be
changed, e.g. allowing the ``bmaas`` network to be routed to
other networks including the internal VIP would result in some
simplification.

This example is illustrative of a specific set of deployment
requirements but is not intended to be followed rigidly. It
demonstrates some of the capabilities of Ironic and
Openstack-Ansible and how to approach most parts of a practical
deployment of Ironic at small to modest scales.

.. warning::
  Consideration should be given to the security of IPMI or other
  out-of-band interfaces which are notoriously buggy and often
  have vendor specific in-band tools which allow the BMC and its
  firmware to be manipulated from userspace. Understand the risks
  of allowing IPMI/BMC interfaces to share a physical interface
  with onboard ethernet ports, this feature will allow full access
  to the management interface of a deployed Ironic node. For high
  security environments or where the users are untrusted consult
  your hardware vendor for appropriate hardening steps.

.. image:: figures/ironic-lxc-example.drawio.png

BMAAS network address plan
~~~~~~~~~~~~~~~~~~~~~~~~~~

In this example the subnet for the ``bmaas`` network is 10.88.104.0/24.
The size of the subnet determines the maximum number of nodes
in the Ironic deployment.

===================== ==============================================
Address               Purpose
===================== ==============================================
10.88.104.0 to .10    Reserve for potential physical routers / SVI
10.88.104.11 to .29   Control plane Ironic container interfaces
10.88.104.64 to .95   Neutron DHCP allocation pool in Ironic_Network
10.88.104.96 to .127  Ironic Inspector DHCP pool range
10.88.104.128 to .254 Static IPs manually assigned to IPMI/iDRAC
===================== ==============================================

In this deployment the ``bmaas`` network combines several functions
for simplicity. It would be possible to use separate networks for
inspection, cleaning and provisioning but that is beyond the scope
of this example.

The subnet is divided into several address ranges, a reservation for
future interfaces on physical routers, or a gateway address if the
subnet is to be made routable in the future. A small number of addresses
are then reserved for the ``bmaas`` network to connect to the Ironic
containers in the control plane, and the remainder of the addresses
are shared between a neutron network for provisioning, a range for
Ironic Inspector to allocate with DHCP and finally a block of
addresses for the IPMI (or other protocol) management ports
of the Ironic nodes.

.. note::
  This example will use VLAN ID 3003 for the ``bmaas`` network but
  any available VLAN ID could be used.

.. warning::
  This example collapses the Ironic IPMI, provisioning, cleaning
  and inspection networks into the same subnet. It is possible
  to make these be separate from each other by creating a network
  for each different function and updating the container networks
  and Ironic service configuration suitably. In particular it should
  be understood that the nodes are "untrusted" during the cleaning
  phase and will be in an arbitrary state as left by the previous
  user.

IPMI Interfaces
~~~~~~~~~~~~~~~

When configuring IPMI interfaces for Ironic nodes, the following steps
are recommended:

* Use static IP allocations for the IPMI interfaces, unless there is
  already a very reliable means of allocating addresses with DHCP. The
  Ironic team do not consider the Neutron DHCP agent to be suitable
  for assigning addresses to the IPMI interfaces.

* Use dedicated IPMI ports on Ironic nodes especially if multitenancy
  is required. A node with a shared onboard ethernet/IPMI port will have
  that port moved into the tenant network when deployment is complete
  and the Ironic control plane will no longer be able to communicate with
  the management interface of the node.

Maximum size of the deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The maximum size of this Ironic deployment is limited by the
address alloction in the ``bmaas`` network. In this example there can
be a maximum of 127 server BMC interfaces in the range 10.88.104.128/25.

The maxiumum number of servers that can be simultanously provisioned is
determined by the address allocation to the Neutron DHCP pool.

The maximum number of servers that can be simultanously inspected is
determined by the address allocation to the Ironic Inspector DHCP pool.

To increase the size of the deployment, the size of the ``bmaas``
subnet should be increased and the addresses allocated to meet the
number of required nodes and maximum number of simultaneous nodes
being provisioned and inspected.

Openstack-Ansible configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Once the address plan has been decided, the Openstack-Ansible configuration
can be updated to match.

The existing ``cidr_networks`` and ``used_ips`` sections in
``/etc/openstack_deploy/openstack_user_config.yml`` must have extra
entries to describe the network range available for the Ansible inventory
to assign to Ironic control plane containers, in this example all addresses
in the ``bmaas`` network are marked as used except the range 10.88.104.11
to 10.88.104.29.

An additional network is defined in the ``provider_networks`` list which
represents the connection between the bridge ``br-bmaas`` on the
controller and ``eth15`` inside the ironic service containers.

The ``bmaas`` network must be extended from the control plane hosts
to the switch ports connected to the Ironic node IPMI interfaces,
and also to switch ports connected to the interfaces on the Ironic
nodes that will be used for PXEboot. This will typically be a VLAN
allocated specifically for Ironic provisioning.

The hosts for the Ironic control plane containers are assigned.

.. note::

  It is the responsibility of the deployer to create br-bmaas
  on the controller nodes and ensure that it is connected to
  the correct VLAN ID for the ``bmaas`` network. Configuration of
  host networking is outside the scope of Openstack-Ansible.

.. note::

  The ``range`` key in the provider network definition is not used
  but its useful as an reminder in the config file of the VLAN ID.

.. code-block:: yaml

  cidr_networks:
    <existing entries>
    bmaas:     10.88.104.0/24        # for containers on the bmaas network

  used_ips:
    <existing entries>
    # bmaas ips
    - "10.88.104.0,10.88.104.10"     # reserve for routers or other infrastructure
    - "10.88.104.30,10.88.104.255"   # reserve for ironic IPMI and provisioning

  provider_networks:
    <existing entries>
    # Network definition to connect Ironic LXC containers to the bmaas network
    # on the infra hosts
    - network:
        net_name: physnet_neutron
        container_type: "veth"
        container_bridge: "br-bmaas"
        container_interface: "eth15"
        ip_from_q: bmaas
        type: "vlan"
        range: "3003:3003"
        group_binds:
          - ironic_api_container
          - ironic_compute_container
          - ironic_inspector_container


  # ironic API and conductor
  ironic-infra_hosts:
    infra1: *_infra1_
    infra2: *_infra2_
    infra3: *_infra3_

  # nova used by ironic for machine state management
  ironic-compute_hosts:
    infra1: *_infra1_
    infra2: *_infra2_
    infra3: *_infra3_

  # Ironic-inspector can only support a single instance at the moment
  # High availability for ironic-inspector is not yet implemented
  ironic-inspector_hosts:
    infra1: *_infra1_

.. note::
  This example uses YAML Anchors to simplify ``openstack_user_config.yml``
  allowing the IP addresses of the infra nodes to be defined only once.
  See https://yaml.org/spec/1.2.2/#alias-nodes.

Create the Neutron configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Enable the Neutron ``baremetal`` and ``genericswitch`` mechanism drivers
by updating ``/etc/openstack_deploy/group_vars/neutron_server.yml``

.. code-block:: yaml

  ---
  neutron_plugin_types:
    - ml2.genericswitch
    - ml2.baremetal

  # keep the ml2 drivers in this order
  # see https://storyboard.openstack.org/#!/story/2008686
  neutron_ml2_mechanism_drivers: "genericswitch,baremetal"

Configure neutron networking-generic-switch to know about the switches
that the Ironic nodes are connected to in ``/etc/openstack_deploy/user_variables.yml``.
These switches are programmed by neutron to switch the Ironic nodes between the
provisioning and project networks once deployment is complete. This is enabling
multitenancy for Ironic.

This example is for a Cisco NXOS based switch, which uses the same command
set as a Cisco IOS based switch for the functions needed by
networking-generic-switch. There is no specific ``device_type`` for NXOS.

.. note::

  A MAC address for the switch must be specified in the neutron config, but Cisco
  and some other switch vendors present a unique MAC address per port so the MAC
  address as seen from the client cannot be used to identify the switch. For IOS/NXOS
  networking-generic-switch uses the field ``switch_info`` from the Ironic node
  ``local_link_connection`` information rather than match a MAC address when
  choosing which switch to configure for a particular node.

.. code-block:: yaml

  neutron_neutron_conf_overrides:
    genericswitch:my-switch-name:            # This should match the hostname configured on the switch
      device_type: netmiko_cisco_ios         # It is really NXOS but the commands are the same
      ngs_mac_address: "cc:46:d6:64:4b:41"   # Doesn't seem to matter but is required - this is taken from an SVI on the mgmt network
      ip: "10.80.240.3"                      # An IP on the switch which has ssh access from the br-mgmt network, loopback, SVI or mgmt0 as needed
      username: "neutron"                    # The user that Neutron will SSH to the switch as
      password: "supersecret"                # The password that Neutron will use to SSH to the switch
      ##  key_file: <ssh key file>           # An SSH key may be used instead of a password
      ngs_manage_vlans: "False"              # VLANs are already provisioned on the switch so tell neutron not to create/delete VLANs

.. note::

  The configuration for networking-generic-switch is added to neturon.conf
  rather than ml2_conf_genericswitch.ini as the config needs to be read by
  both neutron-rpc-server and neutron-server. neutron-server is a uwsgi
  service in openstack-ansible so is only passed one config file,
  see https://bugs.launchpad.net/openstack-ansible/+bug/1987405

.. note::
  If there is already an override in place for this variable
  then extend the existing override rather than making a
  second one.

Configure switch to allow ssh from Neutron
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To achieve multitenancy, Neutron will connect to the specified switch and
configure the port for the Ironic node being provisioned to be in the
correct project VLAN once the deployment is complete. During deployment
Neutron will ensure that the node is in the ``bmaas`` provisioning
network as specified in the Ironic config.

A suitable user and credential must exist on the switch. The SSH
connection will originate from the Neutron processes running on the
OpenStack control plane, on the ``mgmt`` network. There must be an IP
route from the ``mgmt`` network to an interface on the switch which
permits SSH login. That interface could be a physical management port
(``mgmt0`` on NXOS), a loopback interface, an SVI or another interface
with an IP address. SSH communication with the switch can happen either
in-band or out-of-band depending on the requirements of the deployment.

This example config is for a ``neutron`` user using password authentication
on an NXOS switch as seen by ``show run``. The config applied on the
switch gives the ``neutron`` user access to a minimal set of commands
for configuring VLAN membership on specific ports.

To control the commands that the ``neutron`` user is allowed to issue
on the Cisco Nexus switch create a role:

.. code-block:: bash

  role name neutron-role
    rule 3 permit command configure t
    rule 2 permit read-write feature interface
    rule 1 permit read
    vlan policy deny
      permit vlan 3003-3003
      permit vlan 3100-3200
    interface policy deny
      permit interface Ethernet1/1
      permit interface Ethernet1/2
      permit interface Ethernet1/3
      permit interface Ethernet1/4
      permit interface Ethernet1/5
      permit interface Ethernet1/6
      permit interface Ethernet1/7
      permit interface Ethernet1/8

This role allows the ``neutron`` user assign a port to VLAN 3003
which is the ``bmaas`` network and is used during node provisioning.
Any project VLANS that nodes should be able to be moved into after
deployment should also be permitted, range 3100-3200 here.

The interfaces which the ``neutron`` user is permitted to modify are
listed, in this case individually but consult the switch documentation
for other options such as a regular expression.

A similar config can be made on an Arista switch, where a much more
explicit list of allowed CLI commands must be defined using regular
expressions.

.. code-block:: bash

  role neutron-role
     10 permit mode exec command configure
     20 permit mode exec command terminal width 511
     30 permit mode exec command terminal length 0
     40 permit mode exec command enable
     50 permit mode exec command copy running-config startup-config
     60 permit mode config command interface
     70 permit mode if-Et([1-9]|27|29)\/1 command switchport mode access
     80 permit mode if-Et([1-9]|27|29)\/1 command (no )*switchport access vlan (3003|3966)
     90 permit mode if-Et([1-9]|27|29)\/1 command no switchport mode trunk
     100 permit mode if-Et([1-9]|27|29)\/1 command switchport trunk allowed vlan none
     110 permit mode config command copy running-config startup-config

Create the user and password, which must match those in the
``neutron.conf / genericswitch`` config file options:

.. code-block:: bash

  username neutron password 5 <ENCRYPTED-PASSWORD-HERE> role neutron-role

Allow SSH to the switch from the expected
IP addresses, for example a pair out of band management hosts
192.168.0.100/31 and the OpenStack ``mgmt`` network 10.80.240.0/24.

.. code-block:: bash

  ip access-list ACL_ALLOW_SSH_VTY
    10 permit tcp 192.168.0.100/31 any eq 22
    20 permit tcp 10.80.240.0/22 any eq 22

  line vty
    session-limit 5
    exec-timeout 10
    access-class ACL_ALLOW_SSH_VTY in

Create the Neutron network for Ironic provisioning, cleaning and inspection
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: bash

  openstack network create \
    --internal \
    --provider-network-type vlan \
    --provider-physical-network physnet_neutron \
    --provider-segment 3003 \
    Ironic_Network

  openstack subnet create \
    --allocation-pool 10.88.104.64-10.88.104.95 \
    --dhcp \
    --subnet-range 10.88.104.0/24
    --gateway none
    Ironic_Subnet

Create the Ironic configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In ``/etc/openstack_deploy/user_variables_ironic.yml``

.. code-block:: yaml

  ## IRONIC ##

  ironic_ipxe_enabled: yes                  # use HTTP image download from the ironic conductor container
  ironic_enable_web_server_for_images: yes  # use same web server to cache user images

  # Ensure values used during PXEboot refer directly to the correct interface on Ironic API container
  # instead of the internal VIP
  ironic_http_url: "{{ ironic_ipxe_proto }}://{{ container_networks['bmaas_address']['address'] }}:{{ ironic_ipxe_port }}"
  ironic_tftp_server_address: "{{ container_networks['bmaas_address']['address'] }}"

  # Enable ironic drivers
  ironic_drivers_enabled:     # Use PXE boot and IPMItool
    - agent_ipmitool
    - pxe_ipmitool
    - pxe_drac                # enables drivers for Dell iDrac interface

  # Configure Ironic to use Neutron networking
  ironic_enabled_network_interfaces_list: "noop,neutron"
  ironic_default_network_interface: neutron

  # Enable the default set of cleaning steps
  ironic_automated_clean: yes

  # Configure the neutron networks that Ironc should use
  ironic_neutron_provisioning_network_name: "Ironic_Network"
  ironic_neutron_cleaning_network_name:     "Ironic_Network"
  ironic_neutron_inspection_network_name:   "Ironic_Network"

  # Ensure ironic API (using uwsgi) listens on br-bmaas for agent callbacks
  # as well as the mgmt interface for the loadbalancer
  ironic_uwsgi_bind_address: 0.0.0.0

  # Add ipa-insecure=1 to kernel parameters
  # Needed when ironic endpoint is available over https with self-signed cert.
  ironic_kernel_append_params: "ipa-debug=1 systemd.journald.forward_to_console=yes ipa-insecure=1"

  # INI file overrides
  ironic_ironic_conf_overrides:
    # Disable full device erasure (slow) and just metadata erasure, and replace with "Express erasure"
    # which tries to use firmware secure-erase command, but if that fails, reverts to metadata erasure.
    # See: https://docs.openstack.org/ironic/yoga/admin/cleaning.html#storage-cleaning-options
    deploy:
      erase_devices_priority: 0
      erase_devices_metadata_priority: 0
    conductor:
      clean_step_priority_override: "deploy.erase_devices_express:5"

    # Direct IPA to callback directly to deploying ironic container (via BMAAS network)
    # instead of going via HAProxy on mgmt network. Only applies when bmaas network is isolated.
    service_catalog:
      endpoint_override: "http://{{ container_networks['bmaas_address']['address'] }}:6385"

    # Enable ipmitool's Serial-over-LAN terminal console for baremetal nodes
    DEFAULT:
      enabled_console_interfaces: "ipmitool-socat,no-console"

  ## IRONIC INSPECTOR ##

  # Direct Inspector to callback directly to deploying ironic container (via BMAAS network)
  # instead of going via HAProxy on mgmt network. Only applies when bmaas network is isolated.
  ironic_inspector_callback_url: "{{ ironic_inspector_service_internaluri_proto }}://{{ container_networks['bmaas_address']['address'] }}:{{ ironic_inspector_service_port }}/v1/continue"

  # Add ipa-insecure=1 to kernel parameters
  # Needed when inspector is available over https with self-signed cert.
  ironic_inspector_extra_callback_parameters: "ipa-collect-lldp=1 ipa-insecure=1"

  # Ensure inspector API (using uwsgi) listens on br-bmaas for agent callbacks
  # as well as the mgmt interface for the loadbalancer
  ironic_inspector_service_address: "0.0.0.0"

  # dnsmasq/dhcp information for inspector
  ironic_inspector_dhcp_pool_range: 10.88.104.96 10.88.104.127
  ironic_inspector_dhcp_subnet: 10.88.104.0/24
  ironic_inspector_dhcp_subnet_mask: 255.255.255.0
  ironic_inspector_dhcp_enable_gateway: False
  ironic_inspector_dhcp_enable_nameservers: False

  ironic_inspector_dhcp_interface: eth15   # connected to br-bmaas on the host

Deploy Neutron changes
~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: bash

  openstack-ansible playbooks/os-neutron-install.yml

Deploy the ironic-specific nova services
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This deploys nova compute and nova console services
to the ironic compute containers.

.. code-block::

  playbooks/os-nova-install.yml --limit ironic_all

Deploy changes to HAProxy
~~~~~~~~~~~~~~~~~~~~~~~~~

This will bring up the required Ironic, Inspector, and console
endpoints.

.. code-block:: bash

  openstack-ansible playbooks/haproxy-install.yml --tags haproxy_server-config

Deploy the Ironic and Inspector services
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block::

  openstack-ansible playbooks/os-ironic-install.yml

Deploy the Horizon dashbaords for Ironic
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: bash

  openstack-ansible playbooks/os-horizon-install.yml

Using Ironic
~~~~~~~~~~~~

Please refer to the general instructions in the Configuring Ironic
section of this documentation.

VXLAN project networks
~~~~~~~~~~~~~~~~~~~~~~

In this example Ironic multitenancy is implemented using
VLANs. In an OpenStack deployment where project networks
are implemented using an overlay such as VXLAN, it will not
be possible to attach Ironic nodes directly to these networks.
In addition, it is not possible for an end user to request that
the underlying implementation is VLAN when creating a project
network.

In a cloud using overlay project networks it will be
necessary for the cloud administrator to create VLAN provider
networks for users to attach Ironic nodes to and to share these
into individual projects using Neutron RBAC.