From befd8424e2efd4e1bebe89b5085032bf120de148 Mon Sep 17 00:00:00 2001
From: Dmitriy Rabotyagov <dmitriy.rabotyagov@citynetwork.eu>
Date: Thu, 12 Jan 2023 13:52:52 +0100
Subject: [PATCH] Skip haproxy with setup-infrastructure for upgrades

As we've enabled /healtcheck URIs for services [1], that we want to
verify, we need to apply haproxy backend changes only when all services
are re-configured, otherwise haproxy will fail to poll them and
mark all backends as down, which would mean continious downtime until
corresponsive roles will run to fix service api-paste configuration.

[1] https://review.opendev.org/q/topic:osa/healthcheck
Closes-Bug: #2002645

Change-Id: I004f1a86b4dba3c6e356ac14bb70a43abc17f538
---
 doc/source/admin/upgrades/major-upgrades.rst | 15 ++++++++++++++-
 scripts/run-upgrade.sh                       |  3 ++-
 2 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/doc/source/admin/upgrades/major-upgrades.rst b/doc/source/admin/upgrades/major-upgrades.rst
index e4f57d08b8..b112fe5fa0 100644
--- a/doc/source/admin/upgrades/major-upgrades.rst
+++ b/doc/source/admin/upgrades/major-upgrades.rst
@@ -200,10 +200,13 @@ Upgrade infrastructure
 
 We can now go ahead with the upgrade of all the infrastructure components. To
 ensure that rabbitmq and mariadb are upgraded, we pass the appropriate flags.
+We also skip upgrading haproxy re-configuration at this step, since some
+services will become unavailable right after running haproxy role, which we
+want to avoid
 
 .. code-block:: console
 
-    # openstack-ansible setup-infrastructure.yml -e 'galera_upgrade=true' -e 'rabbitmq_upgrade=true' -e package_state=latest
+    # openstack-ansible setup-infrastructure.yml -e 'galera_upgrade=true' -e 'rabbitmq_upgrade=true' -e 'package_state=latest' --skip-tags haproxy-config
 
 With this complete, we can now restart the mariadb containers one at a time,
 ensuring that each is started, responding, and synchronized with the other
@@ -224,6 +227,16 @@ We can now go ahead with the upgrade of all the OpenStack components.
 
     # openstack-ansible setup-openstack.yml -e package_state=latest
 
+Run haproxy re-configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Now, once all services are being re-configured, we can run haproxy
+role to apply new configuration for the backends
+
+.. code-block:: console
+
+    # openstack-ansible haproxy-install.yml -e package_state=latest
+
 Upgrade Ceph
 ~~~~~~~~~~~~
 
diff --git a/scripts/run-upgrade.sh b/scripts/run-upgrade.sh
index dee0fd42c9..16b4890bfa 100755
--- a/scripts/run-upgrade.sh
+++ b/scripts/run-upgrade.sh
@@ -184,11 +184,12 @@ function main {
                      openstack-ansible setup-hosts.yml -e 'lxc_container_allow_restarts=false' --limit 'galera_all:rabbitmq_all' || \
                      openstack-ansible setup-hosts.yml -e package_state=latest")
         # upgrade infrastructure
-        RUN_TASKS+=("setup-infrastructure.yml -e 'galera_upgrade=true' -e 'rabbitmq_upgrade=true' -e package_state=latest")
+        RUN_TASKS+=("setup-infrastructure.yml -e 'galera_upgrade=true' -e 'rabbitmq_upgrade=true' -e 'package_state=latest' --skip-tags haproxy-config")
         # explicitly perform controlled galera cluster restart with new lxc config
         RUN_TASKS+=("${SCRIPTS_PATH}/upgrade-utilities/galera-cluster-rolling-restart.yml")
         # upgrade openstack
         RUN_TASKS+=("setup-openstack.yml -e package_state=latest")
+        RUN_TASKS+=("haproxy-install.yml -e package_state=latest")
         # Run the tasks in order
         for item in ${!RUN_TASKS[@]}; do
           echo "### NOW RUNNING: ${RUN_TASKS[$item]}"