Merge "Add password rotation docs page"

doc/source/admin/index.rst

@@ -12,3 +12,4 @@ Admin Guides
    etcd
    production-architecture-guide
    deployment-philosophy
+   password-rotation

doc/source/admin/password-rotation.rst

@@ -0,0 +1,295 @@
+=================
+Password Rotation
+=================
+
+This guide describes how to change the internal secrets from ``passwords.yml``
+used by Kolla-Ansible. It does not cover every possible ``passwords.yml``
+variable, only the most common ones.
+
+.. warning::
+
+   Always back up your ``passwords.yml`` file before making any changes.
+   Otherwise, it is easy to make unrecoverable mistakes.
+
+.. warning::
+
+   This guide relies on recent changes to Kolla and Kolla-Ansible. You may
+   encounter errors if applying this guide to older deployments. It is
+   recommended that you update your containers and kolla-ansible to the latest
+   available versions before proceeding.
+
+Regenerating secrets
+--------------------
+
+Passwords can be quickly re-generated using ``kolla-genpwd``.
+
+Assuming an existing ``/etc/kolla/passwords.yml`` file, make a backup:
+
+.. code-block:: bash
+
+   cp /etc/kolla/passwords.yml ./passwords.yml.bak
+
+Edit the ``passwords.yml`` file to remove the password strings for any secrets
+that need to be regenerated i.e. change ``foo: "bar"`` to ``foo:``.
+
+Regenerate the removed passwords:
+
+.. code-block:: bash
+
+   kolla-genpwd -p /etc/kolla/passwords.yml
+
+Applying regenerated secrets
+----------------------------
+
+The majority of the secrets can be applied by simply reconfiguring services
+with ``kolla-ansible reconfigure``. Below is a list of secrets that can be
+applied this way.
+
+
+* ``*_keystone_password``
+* ``*_database_password`` (excluding ``nova_database_password``)
+* ``*_ssh_key`` (excluding ``kolla_ssh_key``)
+* ``keystone_admin_password``
+* ``designate_rndc_key``
+* ``keepalived_password``
+* ``libvirt_sasl_password``
+* ``metadata_secret``
+* ``opensearch_dashboards_password``
+* ``osprofiler_secret``
+* ``prometheus_alertmanager_password``
+* ``qdrouterd_password``
+* ``redis_master_password``
+
+It is possible to change more secrets however some require manual steps. The
+manual steps vary depending on the secret. They are listed below in the order
+they should be applied if they are to be changed at the same time. Once all
+manual steps are complete, reconfigure services (``kolla-ansible
+reconfigure``).
+
+For simplicity, this guide assumes Docker is being used. The same commands
+should also work for Podman deployments by replacing instances of ``docker``
+with ``podman`` in all relevant commands.
+
+Kolla SSH key
+^^^^^^^^^^^^^
+There is currently no mechanism within Kolla-Ansible to rotate
+``kolla_ssh_key``. It is however a relatively simple task to perform using a
+standard Ansible playbook, or can be performed by hand on smaller deployments.
+
+Horizon Secret Key
+^^^^^^^^^^^^^^^^^^
+The Horizon secret key (``horizon_secret_key``) is unique because it explicitly
+supports rotation. In reality, it is a Django secret key, and is used for
+cryptographic signing e.g. generating password recovery links. To minimise user
+impact, it is possible to set two secret keys at once. The new one will be used
+for generating new artifacts, while the old one will still be accepted for
+existing artifacts.
+
+Take note of the old password, generate a new one, and take note of it as well.
+
+Add it to the ``passwords.yml`` file, along with the old secret, in this
+exact format (including quotes in the middle):
+
+.. code:: bash
+
+   horizon_secret_key: newsecret' 'oldsecret
+
+It is important to remember to remove the old key and reconfigure services
+again, after all old artifacts have expired e.g. after approximately one to two
+weeks.
+
+Grafana Admin Password
+^^^^^^^^^^^^^^^^^^^^^^
+The Grafana admin password (``grafana_admin_password``) must be rotated
+manually.
+
+#. Generate a new Grafana Admin password.
+
+#. Replace the old password in ``passwords.yml``.
+
+#. Exec into any Grafana container:
+
+   .. code:: bash
+
+      docker exec -it grafana bash
+
+#. Run the password reset command, then enter the new password:
+
+   .. code:: bash
+
+      grafana-cli admin reset-admin-password --password-from-stdin
+
+Database Password
+^^^^^^^^^^^^^^^^^
+The database administrator password (``database_password``) must be rotated
+manually.
+
+#. Generate a new database password.
+
+#. Replace the old password in ``passwords.yml``, take note of both the old and
+   new passwords.
+
+#. SSH to a host running a MariaDB container.
+
+#. Exec into the MariaDB container:
+
+   .. code-block:: bash
+
+      docker exec -it mariadb bash
+
+#. Log in to the database. You will be prompted for the password. Use the
+   old value of ``database_password``:
+
+   .. code:: bash
+
+      mysql --batch -uroot -p
+
+#. Check the current state of the ``root`` user:
+
+   .. code:: bash
+
+      SELECT Host,User,Password FROM mysql.user WHERE User='root';
+
+#. Update the password for the ``root`` user:
+
+   .. code:: bash
+
+      SET PASSWORD FOR 'root'@'%' = PASSWORD('newpassword');
+
+#. Check that the password hash has changed in the user list:
+
+   .. code:: bash
+
+      SELECT Host,User,Password FROM mysql.user WHERE User='root';
+
+#. If there are any remaining root users with the old password e.g.
+   ``root@localhost``, change the password for them too.
+
+Nova Database Password
+^^^^^^^^^^^^^^^^^^^^^^
+The nova database admin user password (``nova_database_password``) must be
+rotated manually.
+
+.. warning::
+
+   From this point onward, API service may be disrupted.
+
+#. Generate a new Nova database password.
+
+#. Replace the old password in ``passwords.yml``.
+
+#. Exec into the ``nova_conductor`` container:
+
+   .. code:: bash
+
+      docker exec -it nova_conductor bash
+
+#. List the cells:
+
+   .. code:: bash
+
+      nova-manage cell_v2 list_cells --verbose
+
+#. Find the entry for ``cell0``, copy the Database Connection value,
+   replace the password in the string with the new value, and update it
+   with the following command:
+
+   .. code:: bash
+
+      nova-manage cell_v2 update_cell --cell_uuid 00000000-0000-0000-0000-000000000000 --database_connection "CONNECTION WITH NEW PASSWORD HERE" --transport-url "none:///"
+
+   (If the ``cell_uuid`` for ``cell0`` is not
+   ``00000000-0000-0000-0000-000000000000``, change the above command
+   accordingly)
+
+Heat Domain Admin Password
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+The keystone password for the heat domain admin service user
+(``heat_domain_admin_password``) must be rotated manually.
+
+It can be changed by an administrator just like any other standard OpenStack
+user password. Generate a new password, replace the old password in
+``passwords.yml``, then apply the change manually:
+
+.. code-block:: bash
+
+   openstack user set --password <password> heat_domain_admin --domain heat_user_domain
+
+RabbitMQ Secrets
+^^^^^^^^^^^^^^^^
+RabbitMQ uses two main secrets. An Erlang cookie for cluster membership
+(``rabbitmq_cluster_cookie``), and a RabbitMQ management user password
+(``rabbitmq_password``). There is currently no documented process for
+seamlessly rotating these secrets. Many OpenStack services use RabbitMQ for
+communication and reconfiguring them with the new credentials can take some
+time, resulting in a relatively long API outage.
+
+It is recommended that you stop all services, then stop and destroy the
+RabbitMQ containers and volumes. Because the RabbitMQ containers are destroyed,
+``kolla-ansible deploy`` should be used to restart services rather than
+``kolla-ansible reconfigure``. Detailed steps are listed below:
+
+#. Generate a new ``rabbitmq_cluster_cookie`` and ``rabbitmq_password``.
+
+#. Replace the old values in ``passwords.yml``.
+
+#. Stop OpenStack services:
+
+   .. code-block:: bash
+
+      kolla-ansible -i inventory stop
+
+#. On each node running RabbitMQ, destroy its containers and volumes:
+
+   .. code-block:: bash
+
+      docker stop rabbitmq
+      docker rm rabbitmq
+      docker volume rm rabbitmq
+
+#. Redeploy services:
+
+   .. code-block:: bash
+
+      kolla-ansible -i inventory deploy
+
+Post-redeploy changes
+^^^^^^^^^^^^^^^^^^^^^
+Once services have been redeployed, the existing Memcached data should be
+flushed. The old Memcached password will no longer be used so any data stored
+using it will be inaccessible.
+
+The instructions below must be run from a host that has access to the network
+the Memcached containers are using. If you are not sure, run them from a host
+that is running Memcached.
+
+#. Install a telnet client:
+
+   .. code-block:: bash
+
+      apt/dnf install telnet
+
+#. Check the config for the IP and port used by Memcached (on every host
+   running Memcached):
+
+   .. code:: bash
+
+      sudo grep command /etc/kolla/memcached/config.json
+
+   The IP and port will be printed after ``-l`` and ``-p`` respectively
+
+#. For each container start a Telnet session, clear all data, then
+   exit:
+
+   .. code:: bash
+
+      telnet <ip> <port>
+      flush_all
+      quit
+
+Known out-of-scope secrets
+--------------------------
+Below is a list of passwords that are known to be outside the scope of this
+guide.
+
+* ``docker_registry_password`` - kolla-ansible cannot manage docker registries.