Add toleration to vault Pod objects

A toleration needs to be added to all resources that create pods since
the node-role.kubernetes.io/master taint will be restored to all master
nodes. This ensures that the pods will run on the master node.

This adds toleration to vault statefulset and deployment objects

Test cases:

PASSED: Verify that vault pods are able to run on a tainted node

PASSED: Verify that other pods, without the taint toleration on,
fail to schedule at the tainted node and that a 'kubectl describe'
of them shows a Warning of 'node(s) had taint
{node-role.kubernetes.io/master: }, that the pod didn't tolerate.'

PASSED: Verify that system application-update from a previous
version to this version works fine

PASSED: Verify that disabling the taint has no effect on vault
running pods

PASSED: Verify that enabling the taint has no effect on vault
running pods

PASSED: Verify that vault is working by creating a vault secret
using vault's '/secret/basic-secret' api

PASSED: Verify that valut is working by reading a vault secret
using vault's '/secret/basic-secret' api

Story: 2009232
Task: 43386

Signed-off-by: Rei Oliveira <Reinildes.JoseMateusOliveira@windriver.com>
Change-Id: Ida9787e059e8c8b97f8b45d829c531f4cee1115a

stx-vault-helm/stx-vault-helm/manifests/vault_manifest.yaml

@@ -60,6 +60,10 @@ data:
       image:
         repository: starlingx/stx-vault-manager
         tag: stx.5.0-v1.18.3
+      tolerations: |
+        - key: "node-role.kubernetes.io/master"
+          operator: "Exists"
+          effect: "NoSchedule"
     injector:
       enabled: true
       nodeSelector: |
@@ -70,6 +74,10 @@ data:
       agentImage:
         repository: vault
         tag: 1.4.2
+      tolerations: |
+        - key: "node-role.kubernetes.io/master"
+          operator: "Exists"
+          effect: "NoSchedule"
     server:
       affinity: |
         podAntiAffinity:
@@ -83,6 +91,10 @@ data:
       image:
         repository: vault
         tag: 1.4.2
+      tolerations: |
+        - key: "node-role.kubernetes.io/master"
+          operator: "Exists"
+          effect: "NoSchedule"
       auditStorage:
         enabled: true
         size: 10Gi

vault-helm/vault-helm/helm-charts/vault-init.yaml

@@ -199,6 +199,10 @@ spec:
       imagePullSecrets:
         {{- toYaml .Values.global.imagePullSecrets | nindent 8 }}
       {{- end }}
+      {{- if .Values.manager.tolerations }}
+      tolerations:
+        {{- tpl .Values.manager.tolerations . | nindent 8 }}
+      {{- end }}
       containers:
         - name: manager
           image: "{{ .Values.manager.image.repository }}:{{ .Values.manager.image.tag }}"