From 69c525d791fcec03d7a0ea975db54b3357115ff7 Mon Sep 17 00:00:00 2001 From: "Neely, Travis (tn720x)" Date: Fri, 29 Jan 2021 10:30:52 -0600 Subject: [PATCH] Added detailed FiXME for ks-service script bug and code changes in [0] On somewhat rare occasions the openstack service list call fails with a connection aborted OSError 104 ECONNRESET. During an upgrade this failure causes the script to think that the service it is checking for does not exist and therefore it recreates the script. In turn this causes further issues when other services try to use this duplicate service. This is a temporary change in order to alliviate the issue while the root cause is investigated. [0] https://review.opendev.org/c/openstack/openstack-helm-infra/+/772416 Change-Id: Id0971a95eb54eca9486a9811f7ec6f603a007cbb --- helm-toolkit/Chart.yaml | 2 +- .../templates/scripts/_ks-service.sh.tpl | 18 +++++++++++++++--- 2 files changed, 16 insertions(+), 4 deletions(-) diff --git a/helm-toolkit/Chart.yaml b/helm-toolkit/Chart.yaml index 46962abcd..780e151f4 100644 --- a/helm-toolkit/Chart.yaml +++ b/helm-toolkit/Chart.yaml @@ -15,7 +15,7 @@ apiVersion: v1 appVersion: v1.0.0 description: OpenStack-Helm Helm-Toolkit name: helm-toolkit -version: 0.2.3 +version: 0.2.4 home: https://docs.openstack.org/openstack-helm icon: https://www.openstack.org/themes/openstack/images/project-mascots/OpenStack-Helm/OpenStack_Project_OpenStackHelm_vertical.png sources: diff --git a/helm-toolkit/templates/scripts/_ks-service.sh.tpl b/helm-toolkit/templates/scripts/_ks-service.sh.tpl index eb9b9314d..3b48ac330 100644 --- a/helm-toolkit/templates/scripts/_ks-service.sh.tpl +++ b/helm-toolkit/templates/scripts/_ks-service.sh.tpl @@ -37,9 +37,21 @@ OS_SERVICE_DESC="${OS_REGION_NAME}: ${OS_SERVICE_NAME} (${OS_SERVICE_TYPE}) serv # Get Service ID if it exists unset OS_SERVICE_ID -# If OS_SERVICE_ID is blank (due to the service not being ready yet) -# then wait a few seconds to give it additional time to be ready -# and try again +# FIXME - There seems to be an issue once in a while where the +# openstack service list fails and encounters an error message such as: +# Unable to establish connection to +# https://keystone-api.openstack.svc.cluster.local:5000/v3/auth/tokens: +# ('Connection aborted.', OSError("(104, 'ECONNRESET')",)) +# During an upgrade scenario, this would cause the OS_SERVICE_ID to be blank +# and it would attempt to create a new service when it was not needed. +# This duplciate service would sometimes be used by other services such as +# Horizon and would give an 'Invalid Service Catalog' error. +# This loop allows for a 'retry' of the openstack service list in an +# attempt to get the service list as expected if it does ecounter an error. +# This loop and recheck can be reverted once the underlying issue is addressed. + +# If OS_SERVICE_ID is blank then wait a few seconds to give it +# additional time and try again for i in {1...3} do OS_SERVICE_ID=$( openstack service list -f csv --quote none | \