Juanita-Balaraj e1b59dde35 Added warning for K8 Root CA update impact on services

Updated Patshet 2 comments
Updated Patchset 1 comments
Signed-off-by: Juanita-Balaraj <juanita.balaraj@windriver.com>
Change-Id: I764232e87edb856e9efd8e353b2527a91b03b70b

2022-01-10 17:24:01 -05:00

25 KiB

Raw Blame History

Manual Kubernetes Root CA Certificate Update

You can update Kubernetes Root certificate on a running system, with either an uploaded certificate or an auto generated certificate.

A set of 'system' subcommands, prefixed by kube-rootca-update-, are provided to accomplish this task. These commands need to be executed in certain order and phases, with most of the commands to be executed host by host.

Warning

Do not let the Kubernetes Root certificate expire on your system and ensure that certificates with valid/adequate expiry dates are used during renewal as there is no easy way to recover a system if the Kubernetes Root certificate expires.

Special care should be taken when updating the Root certificate.

Warning

During the Kubernetes Root update, deployments, daemonsets, and statefulsets present in the cluster are rolling restarted. This impacts services provided by the application. It is highly recommended to schedule a Kubernetes Root update during planned maintenance windows.

The system must be clear of alarms (with the exception of alarms for locked hosts, stopped instances, certificate expiring soon, certificate expired, and Kubernetes root ca update in progress).
All hosts must be unlocked, enabled and available.
All Kubernetes pods must be ready.
Cert-manager app is applied.
A file containing a self-signed certificate and corresponding private key if choose to upload a new Root certificate.

Before starting the update, it is highly recommended to backup the existing kubernetes Root certificate and key, i.e. /etc/kubernetes/pki/ca.crt and /etc/kubernetes/pki/ca.key.

Start the update.

system kube-rootca-update-start

The command supports --force option to force the update to start if there are non mgmt_affecting alarms.

~(keystone_admin)]$ system kube-rootca-update-start

+-------------------+---------------------------------------+
| Property          | Value                                 |
+-------------------+---------------------------------------+
| uuid              | 95afbd19-d159-407b-abe0-9ecb6ba4eb56  |
| state             | update-started                        |
| from_rootca_cert  | 8503e172a63b23e6-12808492498813125379 |
| to_rootca_cert    | None                                  |
| created_at        | 2021-08-26T14:37:31.710407+00:00      |
| updated_at        | None                                  |
+-------------------+---------------------------------------+

Upload or generate a new Kubernetes Root certificate.

To generate a new Kubernetes Root certificate:
```
~(keystone_admin)]$ system kube-rootca-update-generate-cert --expiry-date="2031-08-25" --subject="C=CA ST=ON L=Ottawa O=company OU=sale CN=kubernetes"
Generated new rootca certificate: 66b4daf7bb4ca6dd-248389040031382177497040389244094812254
```
The value returned is the ID of the new Root certificate.

--expiry-date

Optional argment to specify the expiry date of the new certificate. It has to be in the "YYYY-MM-DD" format. If not specified, the new certificate will have the same valid period as the existing one (normally 10 years).

--subject

Optional argment to specify the distinguished name of the new certificate. It has to be in the format C=<Country> ST=<State/Province> L=<Locality> O=<Organization> OU=<OrganizationUnit> CN=<commonName>. If not specified, the new certificate will have "Kubernetes" as default.

Alternatively, to upload a self-signed certificate as the new Root certificate:
```
~(keystone_admin)]$ system kube-rootca-update-upload-cert ca_with_key.crt
Uploaded new rootca certificate: 8503e172a63b23e6-12808492498813125379
```
The value returned is the ID of the uploaded Root certificate. The ca_with_key.crt file contains both the new certifcate and the corresponding private key.

Show the cluster overall update status using the kube-rootca-update-show command.

This command can be run anytime during the update.

For example, run the command after new certificate generated.

~(keystone_admin)]$ system kube-rootca-update-show

+----------------------------------------+------------------------------------+-----------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
| uuid                                   | state                              | from_rootca_cert                        | to_rootca_cert                                             | created_at                         | updated_at                         |
+----------------------------------------+------------------------------------+-----------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
| 95afbd19-d159-407b-abe0-9ecb6ba4eb56   | update-new-rootca-cert-generated   | 8503e172a63b23e6-12808492498813125379   | 66b4daf7bb4ca6dd-248389040031382177497040389244094812254   | 2021-08-26T14:37:31.710407+00:00   | 2021-08-26T14:47:50.728284+00:00   |
+----------------------------------------+------------------------------------+-----------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+

Update Kubernetes components of each host to trust both the old and new Root certificates using the system kube-rootca-host-update <hostname> --phase=trust-both-cas command.

The command needs to be executed sequentially on each of the nodes in the cluster (except dedicated storage nodes if there are any).

For example, update controller-1 to trust both old and new Root certificates.

~(keystone_admin)]$ system kube-rootca-host-update controller-1 --phase=trust-both-cas

+------------------------+----------------------------------------------------------+
| Property               | Value                                                    |
+------------------------+----------------------------------------------------------+
| uuid                   | 513d626c-559e-4df7-8e15-f92481dc190f                     |
| state                  | updating-host-trust-both-cas                             |
| effective_rootca_cert  | 8503e172a63b23e6-12808492498813125379                    |
| target_rootca_cert     | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
| created_at             | 2021-08-26T15:48:39.903793+00:00                         |
| updated_at             | 2021-08-26T15:50:14.299276+00:00                         |
+------------------------+----------------------------------------------------------+

List the update status of all the hosts in the cluster system kube-rootca-host-update-list.

The kube-rootca-host-update-list command can be run anytime during the update.

For example, list the status of hosts after controller-0, controller-1 have been updated to trust both old and new .

~(keystone_admin)]$ system kube-rootca-host-update-list

+--------------+-------------+------------------------------+----------------------------------------+-----------------------------------------------------------+-----------------------------------+-----------------------------------+
| hostname     | personality | state                        | effective_rootca_cert                  | target_rootca_cert                                        | created_at                        | updated_at                        |
+--------------+-------------+------------------------------+----------------------------------------+-----------------------------------------------------------+-----------------------------------+-----------------------------------+
| controller-0 | controller  | updated-host-trust-both-cas  | 8503e172a63b23e6-12808492498813125379  | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610  | 2021-08-26T15:48:39.888935+00:00  | 2021-08-26T15:51:47.343297+00:00  |
| controller-1 | controller  | updated-host-trust-both-cas  | 8503e172a63b23e6-12808492498813125379  | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610  | 2021-08-26T15:48:39.903793+00:00  | 2021-08-26T15:50:30.716854+00:00  |
| worker-0     | worker      | None                         | 8503e172a63b23e6-12808492498813125379  | None                                                      | 2021-08-26T15:48:39.915956+00:0   | None                              |
+--------------+-------------+------------------------------+----------------------------------------+-----------------------------------------------------------+-----------------------------------+-----------------------------------+

Update pods deployed by Deployments and Daemonsets to trust both old and new Root certificates.

Run this command only once on active controller. It takes a few minutes for all the pods to restart after updated.

~(keystone_admin)]$ system kube-rootca-pods-update --phase=trust-both-cas

+-------------------+----------------------------------------------------------+
| Property          | Value                                                    |
+-------------------+----------------------------------------------------------+
| uuid              | 6bc2ff57-e82c-4da1-af69-4d52c67917f7                     |
| state             | updating-pods-trust-both-cas                             |
| from_rootca_cert  | 8503e172a63b23e6-12808492498813125379                    |
| to_rootca_cert    | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
| created_at        | 2021-08-26T15:48:39.860160+00:00                         |
| updated_at        | 2021-08-26T15:59:53.851646+00:00                         |
+-------------------+----------------------------------------------------------+

Update client and server certificates for Kubernetes components of each host using the new Root certificate.

The command needs to be executed sequentially on each of the nodes in the cluster (except dedicated storage nodes if there are any).

For example, update Kubernetes client and server certificates on controller-0 using the new Root certificate.

~(keystone_admin)]$ system kube-rootca-host-update controller-0 --phase=update-certs

+------------------------+----------------------------------------------------------+
| Property               | Value                                                    |
+------------------------+----------------------------------------------------------+
| uuid                   | 18c5b474-8d7a-4b15-bee8-06d4feb704dd                     |
| state                  | updating-host-update-certs                               |
| effective_rootca_cert  | 8503e172a63b23e6-12808492498813125379                    |
| target_rootca_cert     | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
| created_at             | 2021-08-26T15:48:39.888935+00:00                         |
| updated_at             | 2021-08-26T16:13:22.064894+00:00                         |
+------------------------+----------------------------------------------------------+

Update Kubernetes components of each host to trust only the new Root certificate.

The command needs to be executed sequentially on each of the nodes in the cluster (except dedicated storage nodes if there are any).

For example, update controller-0 to trust only the new Root certificate.

~(keystone_admin)]$ system kube-rootca-host-update controller-0 --phase=trust-new-ca

+------------------------+----------------------------------------------------------+
| Property               | Value                                                    |
+------------------------+----------------------------------------------------------+
| uuid                   | 18c5b474-8d7a-4b15-bee8-06d4feb704dd                     |
| state                  | updating-host-trust-new-ca                               |
| effective_rootca_cert  | 8503e172a63b23e6-12808492498813125379                    |
| target_rootca_cert     | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
| created_at             | 2021-08-26T15:48:39.888935+00:00                         |
| updated_at             | 2021-08-26T19:19:19.366338+00:00                         |
+------------------------+----------------------------------------------------------+

Update pods deployed by Deployments and Daemonsets to trust only the new Root certificate.

Run this command only once on active controller. It takes a few minutes for all the pods to restart after updated.

~(keystone_admin)]$ system kube-rootca-pods-update --phase=trust-new-ca

+-------------------+----------------------------------------------------------+
| Property          | Value                                                    |
+-------------------+----------------------------------------------------------+
| uuid              | 6bc2ff57-e82c-4da1-af69-4d52c67917f7                     |
| state             | updating-pods-trust-new-ca                               |
| from_rootca_cert  | 8503e172a63b23e6-12808492498813125379                    |
| to_rootca_cert    | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
| created_at        | 2021-08-26T15:48:39.860160+00:00                         |
| updated_at        | 2021-08-26T19:26:34.347519+00:00                         |
+-------------------+----------------------------------------------------------+

Complete the update.

~(keystone_admin)]$ system kube-rootca-update-complete

+-------------------+----------------------------------------------------------+
| Property          | Value                                                    |
+-------------------+----------------------------------------------------------+
| uuid              | 6bc2ff57-e82c-4da1-af69-4d52c67917f7                     |
| state             | update-completed                                         |
| from_rootca_cert  | 8503e172a63b23e6-12808492498813125379                    |
| to_rootca_cert    | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
| created_at        | 2021-08-26T15:48:39.860160+00:00                         |
| updated_at        | 2021-08-26T20:19:36.579505+00:00                         |
+-------------------+----------------------------------------------------------+

Abort an on-going update.

This command aborts the on-going update at any step. When an update is aborted, alarm 900.009 will be raised and the overall update status will be in update-aborted. A new update should be started, and run to complete to fully update kubernetes certificates.

system kube-rootca-update-abort

For example, the update is aborted when:

controller-0, controller-1 and worker-0 have been updated to trust both certificates,
client and server certificates have been updated on controller-0 (a step further than controller-1, worker-0),
overall update is in updating-host-update-certs state.

~(keystone_admin)]$ system kube-rootca-update-show

+----------------------------------------+------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
| uuid                                   | state                              | from_rootca_cert                                           | to_rootca_cert                                             | created_at                         | updated_at                         |
+----------------------------------------+------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
| 04863d56-2f36-404b-ad9d-a0b1d967939e   | updating-host-update-certs         | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610   | d70efa2daaee06f8-18974112351299353303834821971390931938    | 2021-08-26T20:28:09.383461+00:00   | 2021-08-26T20:42:40.673674+00:00   |
+----------------------------------------+------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+

~(keystone_admin)]$ system kube-rootca-host-update-list

+--------------+-------------+------------------------------+-----------------------------------------------------------+----------------------------------------------------------+-----------------------------------+-----------------------------------+
| hostname     | personality | state                        | effective_rootca_cert                                     | target_rootca_cert                                       | created_at                        | updated_at                        |
+--------------+-------------+------------------------------+-----------------------------------------------------------+----------------------------------------------------------+-----------------------------------+-----------------------------------+
| controller-0 | controller  | updated-host-update-certs    | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610  | d70efa2daaee06f8-18974112351299353303834821971390931938  | 2021-08-26T20:28:09.404809+00:00  | 2021-08-26T20:43:49.577920+00:00  |
| controller-1 | controller  | updated-host-trust-both-cas  | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610  | d70efa2daaee06f8-18974112351299353303834821971390931938  | 2021-08-26T20:28:09.417891+00:00  | 2021-08-26T20:33:03.754760+00:00  |
| worker-0     | worker      | updated-host-trust-both-cas  | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610  | d70efa2daaee06f8-18974112351299353303834821971390931938  | 2021-08-26T20:28:09.430753+00:00  | 2021-08-26T20:34:13.390571+00:00  |
+--------------+-------------+------------------------------+-----------------------------------------------------------+----------------------------------------------------------+-----------------------------------+-----------------------------------+

~(keystone_admin)]$ system kube-rootca-update-abort

+-------------------+----------------------------------------------------------+
| Property          | Value                                                    |
+-------------------+----------------------------------------------------------+
| uuid              | 04863d56-2f36-404b-ad9d-a0b1d967939e                     |
| state             | update-aborted                                           |
| from_rootca_cert  | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
| to_rootca_cert    | d70efa2daaee06f8-18974112351299353303834821971390931938  |
| created_at        | 2021-08-26T20:28:09.383461+00:00                         |
| updated_at        | 2021-08-26T20:53:04.493889+00:00                         |
+-------------------+----------------------------------------------------------+

~(keystone_admin)]$ fm alarm-list

+-----------+------------------------------------------------------------------+-----------------+---------+---------------+
| Alarm ID  | Reason Text                                                      | Entity ID      | Severity | Time Stamp    |
+-----------+------------------------------------------------------------------+-----------------+---------+---------------+
| 900.009   | Kubernetes root CA update aborted, certificates may not be fully | host=controller | minor   | 2021-08-26T20 |
+-----------+------------------------------------------------------------------+-----------------+---------+---------------+
|           | updated                                                          |                 |         | :53:04.577578 |
+-----------+------------------------------------------------------------------+-----------------+---------+---------------+

~(keystone_admin)]$ system kube-rootca-update-show

+----------------------------------------+------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
| uuid                                   | state                              | from_rootca_cert                                           | to_rootca_cert                                             | created_at                         | updated_at                         |
+----------------------------------------+------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
| 04863d56-2f36-404b-ad9d-a0b1d967939e   | update-aborted                     | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610   | d70efa2daaee06f8-18974112351299353303834821971390931938    | 2021-08-26T20:28:09.383461+00:00   | 2021-08-26T20:53:04.493889+00:00   |
+----------------------------------------+------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+

States of the update procedure

update-started: semantic checks passed, health check passed, update started.

update-new-rootca-cert-uploaded: the new Root certificate is uploaded.

update-new-rootca-cert-generated: the new Root certificate is generated.

updating-host-trust-both-cas: new Root certificate is being added to Kubernetes components' trusted .

updated-host-trust-both-cas: new Root certificate has been added to Kubernetes components' trusted .

updating-host-trust-both-cas-failed: new Root certificate failed to be added to Kubernetes components' trusted .

updating-pods-trust-both-cas: new Root certificate is being added to pods' trusted list.

updated-pods-trust-both-cas: new Root certificate has been added to pods' trusted list.

updating-pods-trust-both-cas-failed: new Root certificate failed to be added to pods' trusted list.

updating-host-update-certs: server and client certificates is being updated for Kubernetes components.

updated-host-update-certs: server and client certificates have been updated for Kubernetes components.

updating-host-update-certs-failed: server and client certificates failed to be updated for Kubernetes components.

updating-host-trust-new-ca: old Root certificate is being removed, only new cert will be trusted for Kubernetes components.

updated-host-trust-new-ca: old Root certificate has been removed, only new cert is trusted for Kubernetes components.

updating-host-trust-new-ca-failed: old Root certificate failed to be removed, both old and new certs are trusted for Kubernetes components.

updating-pods-trust-new-ca: old Root certificate is being removed from pods' trusted list.

updated-pods-trust-new-ca: old Root certificate has been removed from pods' trusted list.

updating-pods-trust-new-ca-failed: old Root certificate failed to be removed from pods' trusted list.

update-compete: Kubernetes components and pods are healthy, update completed.

update-aborted: a Kubernetes Root update is aborted.

25 KiB Raw Blame History

Manual Kubernetes Root CA Certificate Update

25 KiB

Raw Blame History