openstack/kolla-ansible
Code Issues Proposed changes

Merge "Add password rotation docs page"

Browse Source
This commit is contained in:
Zuul 2024-02-29 11:37:52 +00:00 committed by Gerrit Code Review
commit d42cdef420
2 changed files with 296 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
doc/source/admin/index.rst
View File

@ -12,3 +12,4 @@ Admin Guides
etcd etcd
production-architecture-guide production-architecture-guide
deployment-philosophy deployment-philosophy
password-rotation

295
doc/source/admin/password-rotation.rst Normal file
View File

@ -0,0 +1,295 @@
=================
Password Rotation
=================
This guide describes how to change the internal secrets from ``passwords.yml``
used by Kolla-Ansible. It does not cover every possible ``passwords.yml``
variable, only the most common ones.
.. warning::
Always back up your ``passwords.yml`` file before making any changes.
Otherwise, it is easy to make unrecoverable mistakes.
.. warning::
This guide relies on recent changes to Kolla and Kolla-Ansible. You may
encounter errors if applying this guide to older deployments. It is
recommended that you update your containers and kolla-ansible to the latest
available versions before proceeding.
Regenerating secrets
--------------------
Passwords can be quickly re-generated using ``kolla-genpwd``.
Assuming an existing ``/etc/kolla/passwords.yml`` file, make a backup:
.. code-block:: bash
cp /etc/kolla/passwords.yml ./passwords.yml.bak
Edit the ``passwords.yml`` file to remove the password strings for any secrets
that need to be regenerated i.e. change ``foo: "bar"`` to ``foo:``.
Regenerate the removed passwords:
.. code-block:: bash
kolla-genpwd -p /etc/kolla/passwords.yml
Applying regenerated secrets
----------------------------
The majority of the secrets can be applied by simply reconfiguring services
with ``kolla-ansible reconfigure``. Below is a list of secrets that can be
applied this way.
* ``*_keystone_password``
* ``*_database_password`` (excluding ``nova_database_password``)
* ``*_ssh_key`` (excluding ``kolla_ssh_key``)
* ``keystone_admin_password``
* ``designate_rndc_key``
* ``keepalived_password``
* ``libvirt_sasl_password``
* ``metadata_secret``
* ``opensearch_dashboards_password``
* ``osprofiler_secret``
* ``prometheus_alertmanager_password``
* ``qdrouterd_password``
* ``redis_master_password``
It is possible to change more secrets however some require manual steps. The
manual steps vary depending on the secret. They are listed below in the order
they should be applied if they are to be changed at the same time. Once all
manual steps are complete, reconfigure services (``kolla-ansible
reconfigure``).
For simplicity, this guide assumes Docker is being used. The same commands
should also work for Podman deployments by replacing instances of ``docker``
with ``podman`` in all relevant commands.
Kolla SSH key
^^^^^^^^^^^^^
There is currently no mechanism within Kolla-Ansible to rotate
``kolla_ssh_key``. It is however a relatively simple task to perform using a
standard Ansible playbook, or can be performed by hand on smaller deployments.
Horizon Secret Key
^^^^^^^^^^^^^^^^^^
The Horizon secret key (``horizon_secret_key``) is unique because it explicitly
supports rotation. In reality, it is a Django secret key, and is used for
cryptographic signing e.g. generating password recovery links. To minimise user
impact, it is possible to set two secret keys at once. The new one will be used
for generating new artifacts, while the old one will still be accepted for
existing artifacts.
Take note of the old password, generate a new one, and take note of it as well.
Add it to the ``passwords.yml`` file, along with the old secret, in this
exact format (including quotes in the middle):
.. code:: bash
horizon_secret_key: newsecret' 'oldsecret
It is important to remember to remove the old key and reconfigure services
again, after all old artifacts have expired e.g. after approximately one to two
weeks.
Grafana Admin Password
^^^^^^^^^^^^^^^^^^^^^^
The Grafana admin password (``grafana_admin_password``) must be rotated
manually.
#. Generate a new Grafana Admin password.
#. Replace the old password in ``passwords.yml``.
#. Exec into any Grafana container:
.. code:: bash
docker exec -it grafana bash
#. Run the password reset command, then enter the new password:
.. code:: bash
grafana-cli admin reset-admin-password --password-from-stdin
Database Password
^^^^^^^^^^^^^^^^^
The database administrator password (``database_password``) must be rotated
manually.
#. Generate a new database password.
#. Replace the old password in ``passwords.yml``, take note of both the old and
new passwords.
#. SSH to a host running a MariaDB container.
#. Exec into the MariaDB container:
.. code-block:: bash
docker exec -it mariadb bash
#. Log in to the database. You will be prompted for the password. Use the
old value of ``database_password``:
.. code:: bash
mysql --batch -uroot -p
#. Check the current state of the ``root`` user:
.. code:: bash
SELECT Host,User,Password FROM mysql.user WHERE User='root';
#. Update the password for the ``root`` user:
.. code:: bash
SET PASSWORD FOR 'root'@'%' = PASSWORD('newpassword');
#. Check that the password hash has changed in the user list:
.. code:: bash
SELECT Host,User,Password FROM mysql.user WHERE User='root';
#. If there are any remaining root users with the old password e.g.
``root@localhost``, change the password for them too.
Nova Database Password
^^^^^^^^^^^^^^^^^^^^^^
The nova database admin user password (``nova_database_password``) must be
rotated manually.
.. warning::
From this point onward, API service may be disrupted.
#. Generate a new Nova database password.
#. Replace the old password in ``passwords.yml``.
#. Exec into the ``nova_conductor`` container:
.. code:: bash
docker exec -it nova_conductor bash
#. List the cells:
.. code:: bash
nova-manage cell_v2 list_cells --verbose
#. Find the entry for ``cell0``, copy the Database Connection value,
replace the password in the string with the new value, and update it
with the following command:
.. code:: bash
nova-manage cell_v2 update_cell --cell_uuid 00000000-0000-0000-0000-000000000000 --database_connection "CONNECTION WITH NEW PASSWORD HERE" --transport-url "none:///"
(If the ``cell_uuid`` for ``cell0`` is not
``00000000-0000-0000-0000-000000000000``, change the above command
accordingly)
Heat Domain Admin Password
^^^^^^^^^^^^^^^^^^^^^^^^^^
The keystone password for the heat domain admin service user
(``heat_domain_admin_password``) must be rotated manually.
It can be changed by an administrator just like any other standard OpenStack
user password. Generate a new password, replace the old password in
``passwords.yml``, then apply the change manually:
.. code-block:: bash
openstack user set --password <password> heat_domain_admin --domain heat_user_domain
RabbitMQ Secrets
^^^^^^^^^^^^^^^^
RabbitMQ uses two main secrets. An Erlang cookie for cluster membership
(``rabbitmq_cluster_cookie``), and a RabbitMQ management user password
(``rabbitmq_password``). There is currently no documented process for
seamlessly rotating these secrets. Many OpenStack services use RabbitMQ for
communication and reconfiguring them with the new credentials can take some
time, resulting in a relatively long API outage.
It is recommended that you stop all services, then stop and destroy the
RabbitMQ containers and volumes. Because the RabbitMQ containers are destroyed,
``kolla-ansible deploy`` should be used to restart services rather than
``kolla-ansible reconfigure``. Detailed steps are listed below:
#. Generate a new ``rabbitmq_cluster_cookie`` and ``rabbitmq_password``.
#. Replace the old values in ``passwords.yml``.
#. Stop OpenStack services:
.. code-block:: bash
kolla-ansible -i inventory stop
#. On each node running RabbitMQ, destroy its containers and volumes:
.. code-block:: bash
docker stop rabbitmq
docker rm rabbitmq
docker volume rm rabbitmq
#. Redeploy services:
.. code-block:: bash
kolla-ansible -i inventory deploy
Post-redeploy changes
^^^^^^^^^^^^^^^^^^^^^
Once services have been redeployed, the existing Memcached data should be
flushed. The old Memcached password will no longer be used so any data stored
using it will be inaccessible.
The instructions below must be run from a host that has access to the network
the Memcached containers are using. If you are not sure, run them from a host
that is running Memcached.
#. Install a telnet client:
.. code-block:: bash
apt/dnf install telnet
#. Check the config for the IP and port used by Memcached (on every host
running Memcached):
.. code:: bash
sudo grep command /etc/kolla/memcached/config.json
The IP and port will be printed after ``-l`` and ``-p`` respectively
#. For each container start a Telnet session, clear all data, then
exit:
.. code:: bash
telnet <ip> <port>
flush_all
quit
Known out-of-scope secrets
--------------------------
Below is a list of passwords that are known to be outside the scope of this
guide.
* ``docker_registry_password`` - kolla-ansible cannot manage docker registries.