=== TLS === This guide describes how to configure Kolla Ansible to deploy OpenStack with TLS enabled. Enabling TLS on the provided internal and/or external VIP address allows OpenStack clients to authenticate and encrypt network communication with OpenStack services. When an OpenStack service exposes an API endpoint, Kolla Ansible will configure HAProxy for that service to listen on the internal and/or external VIP address. The HAProxy container load-balances requests on the VIPs to the nodes running the service container. There are two different layers of TLS configuration for OpenStack APIs: 1. Enabling TLS on the internal and/or external VIP, so communication between an OpenStack client and the HAProxy listening on the VIP is secure. 2. Enabling TLS on the backend network, so communication between HAProxy and the backend API services is secure. .. note:: TLS authentication is based on certificates that have been signed by trusted Certificate Authorities. Examples of commercial CAs are Comodo, Symantec, GoDaddy, and GlobalSign. Letsencrypt.org is a CA that will provide trusted certificates at no charge. If using a trusted CA is not possible for your project, you can use a private CA, e.g. Hashicorp Vault, to create a certificate for your domain, or see :ref:`admin-tls-generating-a-private-ca` to use a Kolla Ansible generated private CA. For details on ACME-enabled CAs, such as letsencrypt.org, please see :doc:`acme`. Quick Start ~~~~~~~~~~~ .. note:: The certificates generated by Kolla Ansible use a simple Certificate Authority setup and are not suitable for a production deployment. Only certificates signed by a trusted Certificate Authority should be used in a production deployment. To deploy OpenStack with TLS enabled for the external, internal and backend APIs, configure the following in ``globals.yml``: .. code-block:: yaml kolla_enable_tls_internal: "yes" kolla_enable_tls_external: "yes" kolla_enable_tls_backend: "yes" kolla_copy_ca_into_containers: "yes" If deploying on Debian or Ubuntu: .. code-block:: yaml openstack_cacert: "/etc/ssl/certs/ca-certificates.crt" If on CentOS or Rocky: .. code-block:: yaml openstack_cacert: "/etc/pki/tls/certs/ca-bundle.crt" The Kolla Ansible ``certificates`` command generates a private test Certificate Authority, and then uses the CA to sign the generated certificates for the enabled VIP(s) to test TLS in your OpenStack deployment. Assuming you are using the ``multinode`` inventory: .. code-block:: console kolla-ansible -i ~/multinode certificates TLS Configuration for internal/external VIP ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The configuration variables that control TLS for the internal and/or external VIP are: - ``kolla_enable_tls_external`` - ``kolla_enable_tls_internal`` - ``kolla_internal_fqdn_cert`` - ``kolla_external_fqdn_cert`` .. note:: If TLS is enabled only on the internal or external network, then ``kolla_internal_vip_address`` and ``kolla_external_vip_address`` must be different. If there is only a single network configured in your topology (as opposed to separate internal and external networks), TLS can only be enabled using the internal network configuration variables. The default state for TLS networking is disabled. To enable external TLS encryption: .. code-block:: yaml kolla_enable_tls_external: "yes" To enable internal TLS encryption: .. code-block:: yaml kolla_enable_tls_internal: "yes" Two certificate files are required to use TLS securely with authentication, which will be provided by your Certificate Authority: * server certificate with private key * CA certificate with any intermediate certificates The combined server certificate and private key needs to be provided to Kolla Ansible, with the path configured via ``kolla_external_fqdn_cert`` or ``kolla_internal_fqdn_cert``. These paths default to ``{{ kolla_certificates_dir }}/haproxy.pem`` and ``{{ kolla_certificates_dir }}/haproxy-internal.pem`` respectively, where ``kolla_certificates_dir`` is ``/etc/kolla/certificates`` by default. If the server certificate provided is not already trusted by clients, then the CA certificate file will need to be distributed to the clients. This is discussed in more detail in :ref:`admin-tls-openrc` and :ref:`admin-tls-ca-in-containers`. .. _admin-tls-openrc: Configuring the OpenStack Client for TLS ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The location for the CA certificate for the ``admin-openrc.sh`` file is configured with the ``kolla_admin_openrc_cacert`` variable, which is not set by default. This must be a valid path on all hosts where ``admin-openrc.sh`` is used. When TLS is enabled on a VIP, and ``kolla_admin_openrc_cacert`` is set to ``/etc/pki/tls/certs/ca-bundle.crt``, an OpenStack client will have settings similar to this configured by ``admin-openrc.sh``: .. code-block:: shell export OS_PROJECT_DOMAIN_NAME=Default export OS_USER_DOMAIN_NAME=Default export OS_PROJECT_NAME=admin export OS_TENANT_NAME=admin export OS_USERNAME=admin export OS_PASSWORD=demoPassword export OS_AUTH_URL=https://mykolla.example.net:5000 export OS_INTERFACE=internal export OS_ENDPOINT_TYPE=internalURL export OS_MISTRAL_ENDPOINT_TYPE=internalURL export OS_IDENTITY_API_VERSION=3 export OS_REGION_NAME=RegionOne export OS_AUTH_PLUGIN=password # os_cacert is optional for trusted certificates export OS_CACERT=/etc/pki/tls/certs/ca-bundle.crt .. _admin-tls-ca-in-containers: Adding CA Certificates to the Service Containers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To copy CA certificate files to the service containers: .. code-block:: yaml kolla_copy_ca_into_containers: "yes" When ``kolla_copy_ca_into_containers`` is configured to "yes", the CA certificate files in ``/etc/kolla/certificates/ca`` will be copied into service containers to enable trust for those CA certificates. This is required for any certificates that are either self-signed or signed by a private CA, and are not already present in the service image trust store. Kolla will install these certificates in the container system wide trust store when the container starts. All certificate file names will have the ``kolla-customca-`` prefix prepended to them when they are copied into the containers. For example, if a certificate file is named ``internal.crt``, it will be named ``kolla-customca-internal.crt`` in the containers. For Debian and Ubuntu containers, the certificate files will be copied to the ``/usr/local/share/ca-certificates/`` directory. For CentOS and Rocky containers, the certificate files will be copied to the ``/etc/pki/ca-trust/source/anchors/`` directory. In both cases, valid certificates will be added to the system trust store - ``/etc/ssl/certs/ca-certificates.crt`` on Debian and Ubuntu, and ``/etc/pki/tls/certs/ca-bundle.crt`` on CentOS and Rocky. Configuring a CA bundle ~~~~~~~~~~~~~~~~~~~~~~~ OpenStack services do not always trust CA certificates from the system trust store by default. To resolve this, the ``openstack_cacert`` variable should be configured with the path to the CA Certificate in the container. To use the system trust store on Debian or Ubuntu: .. code-block:: yaml openstack_cacert: /etc/ssl/certs/ca-certificates.crt For CentOS or Rocky: .. code-block:: yaml openstack_cacert: /etc/pki/tls/certs/ca-bundle.crt Back-end TLS Configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~ Enabling TLS on the backend services secures communication between the HAProxy listing on the internal/external VIP and the OpenStack services. It also enables secure end-to-end communication between OpenStack services that support TLS termination. The OpenStack services that support backend TLS termination in Victoria are: Nova, Ironic, Neutron, Keystone, Glance, Heat, Placement, Horizon, Barbican, and Cinder. The configuration variables that control back-end TLS for service endpoints are: - ``kolla_enable_tls_backend`` - ``kolla_tls_backend_cert`` - ``kolla_tls_backend_key`` - ``haproxy_backend_cacert`` - ``haproxy_backend_cacert_dir`` The default state for back-end TLS is disabled. To enable TLS for the back-end communication: .. code-block:: yaml kolla_enable_tls_backend: "yes" It is also possible to enable back-end TLS on a per-service basis. For example, to enable back-end TLS for Keystone, set ``keystone_enable_tls_backend`` to ``yes``. The default values for ``haproxy_backend_cacert`` and ``haproxy_backend_cacert_dir`` should suffice if the certificate is in the system trust store. Otherwise, they should be configured to a location of the CA certificate installed in the service containers. Each backend service requires a certificate and private key. In many cases it is necessary to use a separate certificate and key for each host, or even per-service. The following precedence is used for the certificate: * ``{{ kolla_certificates_dir }}/{{ inventory_hostname }}/{{ project_name }}-cert.pem`` * ``{{ kolla_certificates_dir }}/{{ inventory_hostname }}-cert.pem`` * ``{{ kolla_certificates_dir }}/{{ project_name }}-cert.pem`` * ``{{ kolla_tls_backend_cert }}`` And for the private key: * ``{{ kolla_certificates_dir }}/{{ inventory_hostname }}/{{ project_name }}-key.pem`` * ``{{ kolla_certificates_dir }}/{{ inventory_hostname }}-key.pem`` * ``{{ kolla_certificates_dir }}/{{ project_name }}-key.pem`` * ``{{ kolla_tls_backend_key }}`` The default for ``kolla_certificates_dir`` is ``/etc/kolla/certificates``. ``kolla_tls_backend_cert`` and ``kolla_tls_backend_key``, default to ``{{ kolla_certificates_dir }}/backend-cert.pem`` and ``{{ kolla_certificates_dir }}/backend-key.pem`` respectively. ``project_name`` is the name of the OpenStack service, e.g. ``keystone`` or ``cinder``. .. note:: The back-end TLS cert/key can be the same certificate that is used for the VIP, as long as those certificates are configured to allow requests from both the VIP and internal networks. By default, the TLS certificate will be verified as trustable by the OpenStack services. Although not recommended for production, it is possible to disable verification of the backend certificate: .. code-block:: yaml kolla_verify_tls_backend: "no" .. _admin-tls-generating-a-private-ca: Generating TLS certificates with Let's Encrypt ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Let's Encrypt is a free, automated, and open certificate authority. To enable OpenStack to deploy the Let's Encrypt container to fetch certificates from the Let's Encrypt certificate authority, the following must be configured in ``globals.yml``: .. code-block:: yaml enable_letsencrypt: "yes" letsencrypt_email: "" The Let's Encrypt container will attempt to renew your certificates every 12 hours. If the certificates are renewed, they will automatically be deployed to the HAProxy containers using SSH. .. note:: If ``letsencrypt_email`` is not valid email, letsencrypt role will not work correctly. .. note:: If ``enable_letsencrypt`` is set to true, haproxy's socket will run with admin access level. This is needed so Let's Encrypt can interact with HAProxy. Generating a Private Certificate Authority ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. note:: The certificates generated by Kolla Ansible use a simple Certificate Authority setup and are not suitable for a production deployment. Only certificates signed by a trusted Certificate Authority should be used in a production deployment. It's not always practical to get a certificate signed by a trusted CA. In a development or internal test OpenStack deployment, it can be useful to generate certificates locally to enable TLS. For convenience, the ``kolla-ansible`` command will generate the necessary certificate files based on the information in the ``globals.yml`` configuration file and the inventory file: .. code-block:: console kolla-ansible -i multinode certificates The ``certificates`` role performs the following actions: 1. Generates a test root Certificate Authority 2. Generates the internal/external certificates which are signed by the root CA. 3. If back-end TLS is enabled, generate the back-end certificate signed by the root CA. The combined certificate and key file ``haproxy.pem`` (which is the default value for ``kolla_external_fqdn_cert``) will be generated and stored in the ``/etc/kolla/certificates/`` directory, and a copy of the CA certificate (``root.crt``) will be stored in the ``/etc/kolla/certificates/ca/`` directory. Generating your certificates without kolla-ansible ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you want to manage your TLS certificates outside kolla-ansible directly on your hosts, you can do it by setting ``kolla_externally_managed_cert`` to ``true``. This will make kolla-ansible ignore any copy of certificate from the operator to kolla-ansible managed hosts and will keep other configuration options for TLS as is. If using this option, make sure that all certificates are present on the appropriate hosts in the appropriate location.