integ/kubernetes/k8s-pod-recovery
Andre Kantek 1c628f50a6 Test kube-api server availability for longer time
It was observed that during dead office recovery (when all nodes
restart simultaneously) that the availability of kube-api-server can
be larger than the current wait loops used to check pod stabilization.
This can lead to a failed execution of important pod operations to
recover them to normal operation.

This change adds an initial specific loop to wait for the API server
availability, in case of failure it exits the script and now the
service is marked as failed which will trigger new attempts to execute
in case of failure (with a 10s interval between service executions).

A 5 minutes time was chosen to have a better chance of finishing the
script without needing to have systemd to restart the service.

The monitoring is done using the instructions in:
https://kubernetes.io/docs/reference/using-api/health-checks/

Test Plan:
install AIO-DX and:
[PASS] execute nodes lock/unlock and observe the service correct
       execution, kube-api is immediately available.
[PASS] execute simultaneous node reboots and observe that the service
       remains waiting for kube-api server for some time (in vbox
       it was 2.5min) and after that it executes the script

Closes-Bug: 2089864

Change-Id: I0d77da1735ecb829ab1da013fe93431688e4cb97
Signed-off-by: Andre Kantek <andrefernandozanella.kantek@windriver.com>
2024-11-29 15:04:33 -03:00
..
debian Remove CentOS/OpenSUSE build support 2024-05-01 16:39:19 -04:00
files Test kube-api server availability for longer time 2024-11-29 15:04:33 -03:00