e1b59dde35
Updated Patshet 2 comments Updated Patchset 1 comments Signed-off-by: Juanita-Balaraj <juanita.balaraj@windriver.com> Change-Id: I764232e87edb856e9efd8e353b2527a91b03b70b
421 lines
25 KiB
ReStructuredText
421 lines
25 KiB
ReStructuredText
.. _manual-kubernetes-root-ca-certificate-update-8e9df2cd7fb9:
|
|
|
|
============================================
|
|
Manual Kubernetes Root CA Certificate Update
|
|
============================================
|
|
|
|
.. rubric:: |context|
|
|
|
|
You can update Kubernetes Root |CA| certificate on a running system, with
|
|
either an uploaded certificate or an auto generated certificate.
|
|
|
|
A set of 'system' |CLI| subcommands, prefixed by ``kube-rootca-update-``, are
|
|
provided to accomplish this task. These |CLI| commands need to be executed in
|
|
certain order and phases, with most of the commands to be executed host by
|
|
host.
|
|
|
|
.. warning::
|
|
Do **not** let the Kubernetes Root |CA| certificate expire on your system
|
|
and ensure that certificates with valid/adequate expiry dates are used
|
|
during renewal as there is no easy way to recover a system if the
|
|
Kubernetes Root |CA| certificate expires.
|
|
|
|
Special care should be taken when updating the Root |CA| certificate.
|
|
|
|
.. warning::
|
|
During the Kubernetes Root |CA| update, ``deployments``, ``daemonsets``, and
|
|
``statefulsets`` present in the cluster are rolling restarted. This impacts
|
|
services provided by the application. It is highly recommended to schedule
|
|
a Kubernetes Root |CA| update during planned maintenance windows.
|
|
|
|
.. rubric:: |prereq|
|
|
|
|
- The system must be clear of alarms \(with the exception of alarms for locked
|
|
hosts, stopped instances, certificate expiring soon, certificate expired,
|
|
and Kubernetes root ca update in progress\).
|
|
|
|
- All hosts must be unlocked, enabled and available.
|
|
|
|
- All Kubernetes pods must be ready.
|
|
|
|
- Cert-manager app is applied.
|
|
|
|
- A file containing a self-signed certificate and corresponding private key
|
|
if choose to upload a new Root |CA| certificate.
|
|
|
|
.. rubric:: |proc|
|
|
|
|
Before starting the update, it is highly recommended to backup the existing
|
|
kubernetes Root |CA| certificate and key, i.e. ``/etc/kubernetes/pki/ca.crt``
|
|
and ``/etc/kubernetes/pki/ca.key``.
|
|
|
|
#. Start the update.
|
|
|
|
.. code-block::
|
|
|
|
system kube-rootca-update-start
|
|
|
|
The command supports ``--force`` option to force the update to start if
|
|
there are non mgmt_affecting alarms.
|
|
|
|
.. code-block::
|
|
|
|
~(keystone_admin)]$ system kube-rootca-update-start
|
|
|
|
+-------------------+---------------------------------------+
|
|
| Property | Value |
|
|
+-------------------+---------------------------------------+
|
|
| uuid | 95afbd19-d159-407b-abe0-9ecb6ba4eb56 |
|
|
| state | update-started |
|
|
| from_rootca_cert | 8503e172a63b23e6-12808492498813125379 |
|
|
| to_rootca_cert | None |
|
|
| created_at | 2021-08-26T14:37:31.710407+00:00 |
|
|
| updated_at | None |
|
|
+-------------------+---------------------------------------+
|
|
|
|
#. Upload or generate a new Kubernetes Root |CA| certificate.
|
|
|
|
To generate a new Kubernetes Root |CA| certificate:
|
|
|
|
.. code-block::
|
|
|
|
~(keystone_admin)]$ system kube-rootca-update-generate-cert --expiry-date="2031-08-25" --subject="C=CA ST=ON L=Ottawa O=company OU=sale CN=kubernetes"
|
|
Generated new rootca certificate: 66b4daf7bb4ca6dd-248389040031382177497040389244094812254
|
|
|
|
The value returned is the ID of the new Root |CA| certificate.
|
|
|
|
``--expiry-date``
|
|
|
|
Optional argment to specify the expiry date of the new certificate. It has
|
|
to be in the "YYYY-MM-DD" format. If not specified, the new certificate
|
|
will have the same valid period as the existing one (normally 10 years).
|
|
|
|
``--subject``
|
|
|
|
Optional argment to specify the distinguished name of the new certificate.
|
|
It has to be in the format ``C=<Country> ST=<State/Province> L=<Locality>
|
|
O=<Organization> OU=<OrganizationUnit> CN=<commonName>``. If not specified,
|
|
the new certificate will have "Kubernetes" as default.
|
|
|
|
Alternatively, to upload a self-signed certificate as the new Root |CA|
|
|
certificate:
|
|
|
|
.. code-block::
|
|
|
|
~(keystone_admin)]$ system kube-rootca-update-upload-cert ca_with_key.crt
|
|
Uploaded new rootca certificate: 8503e172a63b23e6-12808492498813125379
|
|
|
|
The value returned is the ID of the uploaded Root |CA| certificate. The
|
|
``ca_with_key.crt`` file contains both the new certifcate and the
|
|
corresponding private key.
|
|
|
|
#. Show the cluster overall update status using the
|
|
:command:`kube-rootca-update-show` command.
|
|
|
|
This command can be run anytime during the update.
|
|
|
|
For example, run the command after new certificate generated.
|
|
|
|
.. code-block::
|
|
|
|
~(keystone_admin)]$ system kube-rootca-update-show
|
|
|
|
+----------------------------------------+------------------------------------+-----------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
|
|
| uuid | state | from_rootca_cert | to_rootca_cert | created_at | updated_at |
|
|
+----------------------------------------+------------------------------------+-----------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
|
|
| 95afbd19-d159-407b-abe0-9ecb6ba4eb56 | update-new-rootca-cert-generated | 8503e172a63b23e6-12808492498813125379 | 66b4daf7bb4ca6dd-248389040031382177497040389244094812254 | 2021-08-26T14:37:31.710407+00:00 | 2021-08-26T14:47:50.728284+00:00 |
|
|
+----------------------------------------+------------------------------------+-----------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
|
|
|
|
#. Update Kubernetes components of each host to trust both the old and new
|
|
Root |CA| certificates using the :command:`system kube-rootca-host-update <hostname> --phase=trust-both-cas`
|
|
command.
|
|
|
|
The command needs to be executed sequentially on each of the nodes in the
|
|
cluster (except dedicated storage nodes if there are any).
|
|
|
|
For example, update controller-1 to trust both old and new Root |CA|
|
|
certificates.
|
|
|
|
.. code-block::
|
|
|
|
~(keystone_admin)]$ system kube-rootca-host-update controller-1 --phase=trust-both-cas
|
|
|
|
+------------------------+----------------------------------------------------------+
|
|
| Property | Value |
|
|
+------------------------+----------------------------------------------------------+
|
|
| uuid | 513d626c-559e-4df7-8e15-f92481dc190f |
|
|
| state | updating-host-trust-both-cas |
|
|
| effective_rootca_cert | 8503e172a63b23e6-12808492498813125379 |
|
|
| target_rootca_cert | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
|
|
| created_at | 2021-08-26T15:48:39.903793+00:00 |
|
|
| updated_at | 2021-08-26T15:50:14.299276+00:00 |
|
|
+------------------------+----------------------------------------------------------+
|
|
|
|
#. List the update status of all the hosts in the cluster system
|
|
``kube-rootca-host-update-list``.
|
|
|
|
The :command:`kube-rootca-host-update-list` command can be run anytime
|
|
during the update.
|
|
|
|
For example, list the status of hosts after controller-0, controller-1 have
|
|
been updated to trust both old and new |CAs|.
|
|
|
|
.. code-block::
|
|
|
|
~(keystone_admin)]$ system kube-rootca-host-update-list
|
|
|
|
+--------------+-------------+------------------------------+----------------------------------------+-----------------------------------------------------------+-----------------------------------+-----------------------------------+
|
|
| hostname | personality | state | effective_rootca_cert | target_rootca_cert | created_at | updated_at |
|
|
+--------------+-------------+------------------------------+----------------------------------------+-----------------------------------------------------------+-----------------------------------+-----------------------------------+
|
|
| controller-0 | controller | updated-host-trust-both-cas | 8503e172a63b23e6-12808492498813125379 | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 | 2021-08-26T15:48:39.888935+00:00 | 2021-08-26T15:51:47.343297+00:00 |
|
|
| controller-1 | controller | updated-host-trust-both-cas | 8503e172a63b23e6-12808492498813125379 | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 | 2021-08-26T15:48:39.903793+00:00 | 2021-08-26T15:50:30.716854+00:00 |
|
|
| worker-0 | worker | None | 8503e172a63b23e6-12808492498813125379 | None | 2021-08-26T15:48:39.915956+00:0 | None |
|
|
+--------------+-------------+------------------------------+----------------------------------------+-----------------------------------------------------------+-----------------------------------+-----------------------------------+
|
|
|
|
#. Update pods deployed by Deployments and Daemonsets to trust both old and
|
|
new Root |CA| certificates.
|
|
|
|
Run this command only once on active controller. It takes a few minutes for
|
|
all the pods to restart after updated.
|
|
|
|
.. code-block::
|
|
|
|
~(keystone_admin)]$ system kube-rootca-pods-update --phase=trust-both-cas
|
|
|
|
+-------------------+----------------------------------------------------------+
|
|
| Property | Value |
|
|
+-------------------+----------------------------------------------------------+
|
|
| uuid | 6bc2ff57-e82c-4da1-af69-4d52c67917f7 |
|
|
| state | updating-pods-trust-both-cas |
|
|
| from_rootca_cert | 8503e172a63b23e6-12808492498813125379 |
|
|
| to_rootca_cert | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
|
|
| created_at | 2021-08-26T15:48:39.860160+00:00 |
|
|
| updated_at | 2021-08-26T15:59:53.851646+00:00 |
|
|
+-------------------+----------------------------------------------------------+
|
|
|
|
#. Update client and server certificates for Kubernetes components of each
|
|
host using the new Root |CA| certificate.
|
|
|
|
The command needs to be executed sequentially on each of the nodes in the
|
|
cluster (except dedicated storage nodes if there are any).
|
|
|
|
For example, update Kubernetes client and server certificates on
|
|
controller-0 using the new Root |CA| certificate.
|
|
|
|
.. code-block::
|
|
|
|
~(keystone_admin)]$ system kube-rootca-host-update controller-0 --phase=update-certs
|
|
|
|
+------------------------+----------------------------------------------------------+
|
|
| Property | Value |
|
|
+------------------------+----------------------------------------------------------+
|
|
| uuid | 18c5b474-8d7a-4b15-bee8-06d4feb704dd |
|
|
| state | updating-host-update-certs |
|
|
| effective_rootca_cert | 8503e172a63b23e6-12808492498813125379 |
|
|
| target_rootca_cert | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
|
|
| created_at | 2021-08-26T15:48:39.888935+00:00 |
|
|
| updated_at | 2021-08-26T16:13:22.064894+00:00 |
|
|
+------------------------+----------------------------------------------------------+
|
|
|
|
#. Update Kubernetes components of each host to trust only the new Root |CA|
|
|
certificate.
|
|
|
|
The command needs to be executed sequentially on each of the nodes in the
|
|
cluster (except dedicated storage nodes if there are any).
|
|
|
|
For example, update controller-0 to trust only the new Root |CA|
|
|
certificate.
|
|
|
|
.. code-block::
|
|
|
|
~(keystone_admin)]$ system kube-rootca-host-update controller-0 --phase=trust-new-ca
|
|
|
|
+------------------------+----------------------------------------------------------+
|
|
| Property | Value |
|
|
+------------------------+----------------------------------------------------------+
|
|
| uuid | 18c5b474-8d7a-4b15-bee8-06d4feb704dd |
|
|
| state | updating-host-trust-new-ca |
|
|
| effective_rootca_cert | 8503e172a63b23e6-12808492498813125379 |
|
|
| target_rootca_cert | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
|
|
| created_at | 2021-08-26T15:48:39.888935+00:00 |
|
|
| updated_at | 2021-08-26T19:19:19.366338+00:00 |
|
|
+------------------------+----------------------------------------------------------+
|
|
|
|
#. Update pods deployed by Deployments and Daemonsets to trust only the new
|
|
Root |CA| certificate.
|
|
|
|
Run this command only once on active controller. It takes a few minutes for
|
|
all the pods to restart after updated.
|
|
|
|
.. code-block::
|
|
|
|
~(keystone_admin)]$ system kube-rootca-pods-update --phase=trust-new-ca
|
|
|
|
+-------------------+----------------------------------------------------------+
|
|
| Property | Value |
|
|
+-------------------+----------------------------------------------------------+
|
|
| uuid | 6bc2ff57-e82c-4da1-af69-4d52c67917f7 |
|
|
| state | updating-pods-trust-new-ca |
|
|
| from_rootca_cert | 8503e172a63b23e6-12808492498813125379 |
|
|
| to_rootca_cert | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
|
|
| created_at | 2021-08-26T15:48:39.860160+00:00 |
|
|
| updated_at | 2021-08-26T19:26:34.347519+00:00 |
|
|
+-------------------+----------------------------------------------------------+
|
|
|
|
#. Complete the update.
|
|
|
|
.. code-block::
|
|
|
|
~(keystone_admin)]$ system kube-rootca-update-complete
|
|
|
|
+-------------------+----------------------------------------------------------+
|
|
| Property | Value |
|
|
+-------------------+----------------------------------------------------------+
|
|
| uuid | 6bc2ff57-e82c-4da1-af69-4d52c67917f7 |
|
|
| state | update-completed |
|
|
| from_rootca_cert | 8503e172a63b23e6-12808492498813125379 |
|
|
| to_rootca_cert | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
|
|
| created_at | 2021-08-26T15:48:39.860160+00:00 |
|
|
| updated_at | 2021-08-26T20:19:36.579505+00:00 |
|
|
+-------------------+----------------------------------------------------------+
|
|
|
|
#. Abort an on-going update.
|
|
|
|
This command aborts the on-going update at any step. When an update is
|
|
aborted, alarm **900.009** will be raised and the overall update status
|
|
will be in ``update-aborted``. A new update should be started, and run to
|
|
complete to fully update kubernetes certificates.
|
|
|
|
.. code-block::
|
|
|
|
system kube-rootca-update-abort
|
|
|
|
For example, the update is aborted when:
|
|
|
|
- controller-0, controller-1 and worker-0 have been updated to trust both
|
|
|CA| certificates,
|
|
|
|
- client and server certificates have been updated on controller-0 (a step
|
|
further than controller-1, worker-0),
|
|
|
|
- overall update is in ``updating-host-update-certs`` state.
|
|
|
|
.. code-block::
|
|
|
|
~(keystone_admin)]$ system kube-rootca-update-show
|
|
|
|
+----------------------------------------+------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
|
|
| uuid | state | from_rootca_cert | to_rootca_cert | created_at | updated_at |
|
|
+----------------------------------------+------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
|
|
| 04863d56-2f36-404b-ad9d-a0b1d967939e | updating-host-update-certs | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 | d70efa2daaee06f8-18974112351299353303834821971390931938 | 2021-08-26T20:28:09.383461+00:00 | 2021-08-26T20:42:40.673674+00:00 |
|
|
+----------------------------------------+------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
|
|
|
|
.. code-block::
|
|
|
|
~(keystone_admin)]$ system kube-rootca-host-update-list
|
|
|
|
+--------------+-------------+------------------------------+-----------------------------------------------------------+----------------------------------------------------------+-----------------------------------+-----------------------------------+
|
|
| hostname | personality | state | effective_rootca_cert | target_rootca_cert | created_at | updated_at |
|
|
+--------------+-------------+------------------------------+-----------------------------------------------------------+----------------------------------------------------------+-----------------------------------+-----------------------------------+
|
|
| controller-0 | controller | updated-host-update-certs | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 | d70efa2daaee06f8-18974112351299353303834821971390931938 | 2021-08-26T20:28:09.404809+00:00 | 2021-08-26T20:43:49.577920+00:00 |
|
|
| controller-1 | controller | updated-host-trust-both-cas | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 | d70efa2daaee06f8-18974112351299353303834821971390931938 | 2021-08-26T20:28:09.417891+00:00 | 2021-08-26T20:33:03.754760+00:00 |
|
|
| worker-0 | worker | updated-host-trust-both-cas | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 | d70efa2daaee06f8-18974112351299353303834821971390931938 | 2021-08-26T20:28:09.430753+00:00 | 2021-08-26T20:34:13.390571+00:00 |
|
|
+--------------+-------------+------------------------------+-----------------------------------------------------------+----------------------------------------------------------+-----------------------------------+-----------------------------------+
|
|
|
|
.. code-block::
|
|
|
|
~(keystone_admin)]$ system kube-rootca-update-abort
|
|
|
|
+-------------------+----------------------------------------------------------+
|
|
| Property | Value |
|
|
+-------------------+----------------------------------------------------------+
|
|
| uuid | 04863d56-2f36-404b-ad9d-a0b1d967939e |
|
|
| state | update-aborted |
|
|
| from_rootca_cert | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
|
|
| to_rootca_cert | d70efa2daaee06f8-18974112351299353303834821971390931938 |
|
|
| created_at | 2021-08-26T20:28:09.383461+00:00 |
|
|
| updated_at | 2021-08-26T20:53:04.493889+00:00 |
|
|
+-------------------+----------------------------------------------------------+
|
|
|
|
.. code-block::
|
|
|
|
~(keystone_admin)]$ fm alarm-list
|
|
|
|
+-----------+------------------------------------------------------------------+-----------------+---------+---------------+
|
|
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
|
|
+-----------+------------------------------------------------------------------+-----------------+---------+---------------+
|
|
| 900.009 | Kubernetes root CA update aborted, certificates may not be fully | host=controller | minor | 2021-08-26T20 |
|
|
+-----------+------------------------------------------------------------------+-----------------+---------+---------------+
|
|
| | updated | | | :53:04.577578 |
|
|
+-----------+------------------------------------------------------------------+-----------------+---------+---------------+
|
|
|
|
.. code-block::
|
|
|
|
~(keystone_admin)]$ system kube-rootca-update-show
|
|
|
|
+----------------------------------------+------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
|
|
| uuid | state | from_rootca_cert | to_rootca_cert | created_at | updated_at |
|
|
+----------------------------------------+------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
|
|
| 04863d56-2f36-404b-ad9d-a0b1d967939e | update-aborted | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 | d70efa2daaee06f8-18974112351299353303834821971390931938 | 2021-08-26T20:28:09.383461+00:00 | 2021-08-26T20:53:04.493889+00:00 |
|
|
+----------------------------------------+------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
|
|
|
|
|
|
**States of the update procedure**
|
|
|
|
``update-started``: semantic checks passed, health check passed, update started.
|
|
|
|
``update-new-rootca-cert-uploaded``: the new Root |CA| certificate is uploaded.
|
|
|
|
``update-new-rootca-cert-generated``: the new Root |CA| certificate is
|
|
generated.
|
|
|
|
``updating-host-trust-both-cas``: new Root |CA| certificate is being added to
|
|
Kubernetes components' trusted |CAs|.
|
|
|
|
``updated-host-trust-both-cas``: new Root |CA| certificate has been added to
|
|
Kubernetes components' trusted |CAs|.
|
|
|
|
``updating-host-trust-both-cas-failed``: new Root |CA| certificate failed to be
|
|
added to Kubernetes components' trusted |CAs|.
|
|
|
|
``updating-pods-trust-both-cas``: new Root |CA| certificate is being added to
|
|
pods' trusted |CA| list.
|
|
|
|
``updated-pods-trust-both-cas``: new Root |CA| certificate has been added to
|
|
pods' trusted |CA| list.
|
|
|
|
``updating-pods-trust-both-cas-failed``: new Root |CA| certificate failed to be
|
|
added to pods' trusted |CA| list.
|
|
|
|
``updating-host-update-certs``: server and client certificates is being updated
|
|
for Kubernetes components.
|
|
|
|
``updated-host-update-certs``: server and client certificates have been updated
|
|
for Kubernetes components.
|
|
|
|
``updating-host-update-certs-failed``: server and client certificates failed to
|
|
be updated for Kubernetes components.
|
|
|
|
``updating-host-trust-new-ca``: old Root |CA| certificate is being removed,
|
|
only new cert will be trusted for Kubernetes components.
|
|
|
|
``updated-host-trust-new-ca``: old Root |CA| certificate has been removed, only
|
|
new cert is trusted for Kubernetes components.
|
|
|
|
``updating-host-trust-new-ca-failed``: old Root |CA| certificate failed to be
|
|
removed, both old and new certs are trusted for Kubernetes components.
|
|
|
|
``updating-pods-trust-new-ca``: old Root |CA| certificate is being removed from
|
|
pods' trusted |CA| list.
|
|
|
|
``updated-pods-trust-new-ca``: old Root |CA| certificate has been removed from
|
|
pods' trusted |CA| list.
|
|
|
|
``updating-pods-trust-new-ca-failed``: old Root |CA| certificate failed to be
|
|
removed from pods' trusted |CA| list.
|
|
|
|
``update-compete``: Kubernetes components and pods are healthy, update
|
|
completed.
|
|
|
|
``update-aborted``: a Kubernetes Root |CA| update is aborted.
|