It was observed that during dead office recovery (when all nodes
restart simultaneously) that the availability of kube-api-server can
be larger than the current wait loops used to check pod stabilization.
This can lead to a failed execution of important pod operations to
recover them to normal operation.
This change adds an initial specific loop to wait for the API server
availability, in case of failure it exits the script and now the
service is marked as failed which will trigger new attempts to execute
in case of failure (with a 10s interval between service executions).
A 5 minutes time was chosen to have a better chance of finishing the
script without needing to have systemd to restart the service.
The monitoring is done using the instructions in:
https://kubernetes.io/docs/reference/using-api/health-checks/
Test Plan:
install AIO-DX and:
[PASS] execute nodes lock/unlock and observe the service correct
execution, kube-api is immediately available.
[PASS] execute simultaneous node reboots and observe that the service
remains waiting for kube-api server for some time (in vbox
it was 2.5min) and after that it executes the script
Closes-Bug: 2089864
Change-Id: I0d77da1735ecb829ab1da013fe93431688e4cb97
Signed-off-by: Andre Kantek <andrefernandozanella.kantek@windriver.com>
1c628f50a6