starlingx/integ
Code Issues Proposed changes

Armada: Fix tiller stuck connecting to postgres database

Browse Source 
Tiller may start executing before IPv6 network is fully initialized.
This will result in tiller not being fully functional.
The liveness probe will detect that tiller didn't start properly and
restart it. But this might happen an unlimited number of times in a row.

Wait until ping is succesful to the ip of the postgres database.
This ensures that networking finished setting up.
Credits to Cole Walker <cole.walker@windriver.com> for proposing the
idea.

Depends-On: I177bb628497611eb64472291a04d635856c26590
Closes-Bug: 1928141
Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>
Change-Id: I9c5be3f30fad2650e6aa53fb80ef44f7798813ed
This commit is contained in:
Dan Voiculeasa 2021-05-11 17:04:01 +00:00
parent 1974b3f570
commit 764cac1642
2 changed files with 68 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
kubernetes/armada/centos/armada.spec
View File

@ -16,6 +16,7 @@ URL: https://airship-armada.readthedocs.io/
Source0: %{name}-%{git_sha}.tar.gz
Patch1: 0001-Add-Helm-v2-client-initialization-using-tiller-postS.patch
Patch2: 0002-Tiller-wait-for-postgres-database-ping.patch
BuildArch: noarch
@ -29,6 +30,7 @@ BuildRequires: chartmuseum
%prep
%setup -n armada
%patch1 -p1
%patch2 -p1
%build
# Package the armada chart tarball using methodology derived from:

66
kubernetes/armada/centos/files/0002-Tiller-wait-for-postgres-database-ping.patch Normal file
View File

@ -0,0 +1,66 @@
From 96e49fcc6d6b988d03a61261511abf64a0af2e2a Mon Sep 17 00:00:00 2001
From: Dan Voiculeasa <dan.voiculeasa@windriver.com>
Date: Tue, 11 May 2021 21:04:18 +0300
Subject: [PATCH] Tiller wait for postgres database ping
Networking might not be correctly initialized when tiller starts.
Modify the pod command to wait for networking to be available before
starting up tiller.
Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>
---
charts/armada/templates/deployment-api.yaml | 31 +++++++++++++--------
1 file changed, 19 insertions(+), 12 deletions(-)
diff --git a/charts/armada/templates/deployment-api.yaml b/charts/armada/templates/deployment-api.yaml
index 69036c0..3816366 100644
--- a/charts/armada/templates/deployment-api.yaml
+++ b/charts/armada/templates/deployment-api.yaml
@@ -167,24 +167,31 @@ spec:
- name: TILLER_HISTORY_MAX
value: {{ .Values.conf.tiller.history_max | quote }}
command:
- - /tiller
+ - sh
+ - -c
+ - |
+ /bin/sh <<'EOF'
{{- if .Values.conf.tiller.storage }}
- - --storage={{ .Values.conf.tiller.storage }}
{{- if and (eq .Values.conf.tiller.storage "sql") (.Values.conf.tiller.sql_dialect) (.Values.conf.tiller.sql_connection) }}
- - --sql-dialect={{ .Values.conf.tiller.sql_dialect }}
- - --sql-connection-string={{ .Values.conf.tiller.sql_connection }}
+ while ! /bin/busybox nc -vz -w 1 {{ .Values.conf.tiller.sql_endpoint_ip}} 5432; do continue; done;
{{- end }}
{{- end }}
- - -listen
- - ":{{ .Values.conf.tiller.port }}"
- - -probe-listen
- - ":{{ .Values.conf.tiller.probe_port }}"
- - -logtostderr
- - -v
- - {{ .Values.conf.tiller.verbosity | quote }}
+ /tiller \
+{{- if .Values.conf.tiller.storage }}
+ --storage={{ .Values.conf.tiller.storage }} \
+{{- if and (eq .Values.conf.tiller.storage "sql") (.Values.conf.tiller.sql_dialect) (.Values.conf.tiller.sql_connection) }}
+ --sql-dialect={{ .Values.conf.tiller.sql_dialect }} \
+ --sql-connection-string={{ .Values.conf.tiller.sql_connection }} \
+{{- end }}
+{{- end }}
+ -listen ":{{ .Values.conf.tiller.port }}" \
+ -probe-listen ":{{ .Values.conf.tiller.probe_port }}" \
+ -logtostderr \
+ -v {{ .Values.conf.tiller.verbosity | quote }} \
{{- if .Values.conf.tiller.trace }}
- - -trace
+ -trace
{{- end }}
+ EOF
lifecycle:
postStart:
exec:
--
2.30.0