From d6d82e2a883f9d8a5a73b89b1b8357631b9b643e Mon Sep 17 00:00:00 2001 From: Alex-Welsh Date: Thu, 21 Dec 2023 09:26:18 +0000 Subject: [PATCH] Add password rotation docs page Closes-Bug: #1793323 Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/903178 Depends-On: https://review.opendev.org/c/openstack/kolla/+/902057 Change-Id: Ibebd6e04de215e1a1aaff52c55d28c4741af98f2 --- doc/source/admin/index.rst | 1 + doc/source/admin/password-rotation.rst | 295 +++++++++++++++++++++++++ 2 files changed, 296 insertions(+) create mode 100644 doc/source/admin/password-rotation.rst diff --git a/doc/source/admin/index.rst b/doc/source/admin/index.rst index f668312514..eb259951c1 100644 --- a/doc/source/admin/index.rst +++ b/doc/source/admin/index.rst @@ -12,3 +12,4 @@ Admin Guides etcd production-architecture-guide deployment-philosophy + password-rotation diff --git a/doc/source/admin/password-rotation.rst b/doc/source/admin/password-rotation.rst new file mode 100644 index 0000000000..0e43c6f88f --- /dev/null +++ b/doc/source/admin/password-rotation.rst @@ -0,0 +1,295 @@ +================= +Password Rotation +================= + +This guide describes how to change the internal secrets from ``passwords.yml`` +used by Kolla-Ansible. It does not cover every possible ``passwords.yml`` +variable, only the most common ones. + +.. warning:: + + Always back up your ``passwords.yml`` file before making any changes. + Otherwise, it is easy to make unrecoverable mistakes. + +.. warning:: + + This guide relies on recent changes to Kolla and Kolla-Ansible. You may + encounter errors if applying this guide to older deployments. It is + recommended that you update your containers and kolla-ansible to the latest + available versions before proceeding. + +Regenerating secrets +-------------------- + +Passwords can be quickly re-generated using ``kolla-genpwd``. + +Assuming an existing ``/etc/kolla/passwords.yml`` file, make a backup: + +.. code-block:: bash + + cp /etc/kolla/passwords.yml ./passwords.yml.bak + +Edit the ``passwords.yml`` file to remove the password strings for any secrets +that need to be regenerated i.e. change ``foo: "bar"`` to ``foo:``. + +Regenerate the removed passwords: + +.. code-block:: bash + + kolla-genpwd -p /etc/kolla/passwords.yml + +Applying regenerated secrets +---------------------------- + +The majority of the secrets can be applied by simply reconfiguring services +with ``kolla-ansible reconfigure``. Below is a list of secrets that can be +applied this way. + + +* ``*_keystone_password`` +* ``*_database_password`` (excluding ``nova_database_password``) +* ``*_ssh_key`` (excluding ``kolla_ssh_key``) +* ``keystone_admin_password`` +* ``designate_rndc_key`` +* ``keepalived_password`` +* ``libvirt_sasl_password`` +* ``metadata_secret`` +* ``opensearch_dashboards_password`` +* ``osprofiler_secret`` +* ``prometheus_alertmanager_password`` +* ``qdrouterd_password`` +* ``redis_master_password`` + +It is possible to change more secrets however some require manual steps. The +manual steps vary depending on the secret. They are listed below in the order +they should be applied if they are to be changed at the same time. Once all +manual steps are complete, reconfigure services (``kolla-ansible +reconfigure``). + +For simplicity, this guide assumes Docker is being used. The same commands +should also work for Podman deployments by replacing instances of ``docker`` +with ``podman`` in all relevant commands. + +Kolla SSH key +^^^^^^^^^^^^^ +There is currently no mechanism within Kolla-Ansible to rotate +``kolla_ssh_key``. It is however a relatively simple task to perform using a +standard Ansible playbook, or can be performed by hand on smaller deployments. + +Horizon Secret Key +^^^^^^^^^^^^^^^^^^ +The Horizon secret key (``horizon_secret_key``) is unique because it explicitly +supports rotation. In reality, it is a Django secret key, and is used for +cryptographic signing e.g. generating password recovery links. To minimise user +impact, it is possible to set two secret keys at once. The new one will be used +for generating new artifacts, while the old one will still be accepted for +existing artifacts. + +Take note of the old password, generate a new one, and take note of it as well. + +Add it to the ``passwords.yml`` file, along with the old secret, in this +exact format (including quotes in the middle): + +.. code:: bash + + horizon_secret_key: newsecret' 'oldsecret + +It is important to remember to remove the old key and reconfigure services +again, after all old artifacts have expired e.g. after approximately one to two +weeks. + +Grafana Admin Password +^^^^^^^^^^^^^^^^^^^^^^ +The Grafana admin password (``grafana_admin_password``) must be rotated +manually. + +#. Generate a new Grafana Admin password. + +#. Replace the old password in ``passwords.yml``. + +#. Exec into any Grafana container: + + .. code:: bash + + docker exec -it grafana bash + +#. Run the password reset command, then enter the new password: + + .. code:: bash + + grafana-cli admin reset-admin-password --password-from-stdin + +Database Password +^^^^^^^^^^^^^^^^^ +The database administrator password (``database_password``) must be rotated +manually. + +#. Generate a new database password. + +#. Replace the old password in ``passwords.yml``, take note of both the old and + new passwords. + +#. SSH to a host running a MariaDB container. + +#. Exec into the MariaDB container: + + .. code-block:: bash + + docker exec -it mariadb bash + +#. Log in to the database. You will be prompted for the password. Use the + old value of ``database_password``: + + .. code:: bash + + mysql --batch -uroot -p + +#. Check the current state of the ``root`` user: + + .. code:: bash + + SELECT Host,User,Password FROM mysql.user WHERE User='root'; + +#. Update the password for the ``root`` user: + + .. code:: bash + + SET PASSWORD FOR 'root'@'%' = PASSWORD('newpassword'); + +#. Check that the password hash has changed in the user list: + + .. code:: bash + + SELECT Host,User,Password FROM mysql.user WHERE User='root'; + +#. If there are any remaining root users with the old password e.g. + ``root@localhost``, change the password for them too. + +Nova Database Password +^^^^^^^^^^^^^^^^^^^^^^ +The nova database admin user password (``nova_database_password``) must be +rotated manually. + +.. warning:: + + From this point onward, API service may be disrupted. + +#. Generate a new Nova database password. + +#. Replace the old password in ``passwords.yml``. + +#. Exec into the ``nova_conductor`` container: + + .. code:: bash + + docker exec -it nova_conductor bash + +#. List the cells: + + .. code:: bash + + nova-manage cell_v2 list_cells --verbose + +#. Find the entry for ``cell0``, copy the Database Connection value, + replace the password in the string with the new value, and update it + with the following command: + + .. code:: bash + + nova-manage cell_v2 update_cell --cell_uuid 00000000-0000-0000-0000-000000000000 --database_connection "CONNECTION WITH NEW PASSWORD HERE" --transport-url "none:///" + + (If the ``cell_uuid`` for ``cell0`` is not + ``00000000-0000-0000-0000-000000000000``, change the above command + accordingly) + +Heat Domain Admin Password +^^^^^^^^^^^^^^^^^^^^^^^^^^ +The keystone password for the heat domain admin service user +(``heat_domain_admin_password``) must be rotated manually. + +It can be changed by an administrator just like any other standard OpenStack +user password. Generate a new password, replace the old password in +``passwords.yml``, then apply the change manually: + +.. code-block:: bash + + openstack user set --password heat_domain_admin --domain heat_user_domain + +RabbitMQ Secrets +^^^^^^^^^^^^^^^^ +RabbitMQ uses two main secrets. An Erlang cookie for cluster membership +(``rabbitmq_cluster_cookie``), and a RabbitMQ management user password +(``rabbitmq_password``). There is currently no documented process for +seamlessly rotating these secrets. Many OpenStack services use RabbitMQ for +communication and reconfiguring them with the new credentials can take some +time, resulting in a relatively long API outage. + +It is recommended that you stop all services, then stop and destroy the +RabbitMQ containers and volumes. Because the RabbitMQ containers are destroyed, +``kolla-ansible deploy`` should be used to restart services rather than +``kolla-ansible reconfigure``. Detailed steps are listed below: + +#. Generate a new ``rabbitmq_cluster_cookie`` and ``rabbitmq_password``. + +#. Replace the old values in ``passwords.yml``. + +#. Stop OpenStack services: + + .. code-block:: bash + + kolla-ansible -i inventory stop + +#. On each node running RabbitMQ, destroy its containers and volumes: + + .. code-block:: bash + + docker stop rabbitmq + docker rm rabbitmq + docker volume rm rabbitmq + +#. Redeploy services: + + .. code-block:: bash + + kolla-ansible -i inventory deploy + +Post-redeploy changes +^^^^^^^^^^^^^^^^^^^^^ +Once services have been redeployed, the existing Memcached data should be +flushed. The old Memcached password will no longer be used so any data stored +using it will be inaccessible. + +The instructions below must be run from a host that has access to the network +the Memcached containers are using. If you are not sure, run them from a host +that is running Memcached. + +#. Install a telnet client: + + .. code-block:: bash + + apt/dnf install telnet + +#. Check the config for the IP and port used by Memcached (on every host + running Memcached): + + .. code:: bash + + sudo grep command /etc/kolla/memcached/config.json + + The IP and port will be printed after ``-l`` and ``-p`` respectively + +#. For each container start a Telnet session, clear all data, then + exit: + + .. code:: bash + + telnet + flush_all + quit + +Known out-of-scope secrets +-------------------------- +Below is a list of passwords that are known to be outside the scope of this +guide. + +* ``docker_registry_password`` - kolla-ansible cannot manage docker registries.