[doc] Reffer need of haproxy backend configuration in upgrade guide

Sicne 2023.1 we have started configuring haproxy backends per service
during specific playbook runtime. While it works in general, it requires
some extra steps to be run for upgrades to avoid race conditions,
when keepalived selects host as active that does not have any
backends configured.

To avoid this we suggest running haproxy with skipping keepalived
installation and proceed with backends configuration.

Change-Id: I37d9b25ae996bc66f69a7aad7a0ba13c32abc929

doc/source/admin/upgrades/distribution-upgrades.rst

@@ -142,6 +142,12 @@ Deploying Infrastructure Hosts
 
    for each API or service instance you wish to disable.
 
+   You can also use a playbook from `OPS repository`_ like this:
+
+   .. code:: console
+
+      openstack-ansible set-haproxy-backends-state.yml -e hostname=<infrahost> -e backend_state=disabled
+
    Or if you've enabled haproxy_stats as described above, you can visit
    https://admin:password@external_lb_vip_address:1936/ and select them and
    'Set state to MAINT'
@@ -170,11 +176,46 @@ Deploying Infrastructure Hosts
          rabbitmqctl cluster_status
          rabbitmqctl forget_cluster_node rabbit@removed_host_rabbitmq_container
 
-#. If it is NOT a 'primary', install everything on the new host
+#. Do generic preparation of reinstalled host
 
    .. code:: console
 
       openstack-ansible setup-hosts.yml --limit localhost,reinstalled_host*
+
+#. This step should be executed when you are re-configuring one of haproxy
+   hosts
+
+   Since configuration of haproxy backends happens during individual service
+   provisioning, we need to ensure that all backends are configured before
+   enabling keepalived to select this host.
+
+   Commands below will configure all required backends on haproxy nodes:
+
+   .. code:: console
+
+      openstack-ansible haproxy-install.yml --limit localhost,reinstalled_host --skip-tags keepalived
+      openstack-ansible repo-install.yml --tags haproxy-service-config
+      openstack-ansible galera-install.yml --tags haproxy-service-config
+      openstack-ansible rabbitmq-install.yml --tags haproxy-service-config
+      openstack-ansible setup-openstack.yml --tags haproxy-service-config
+
+   Once this is done, you can deploy keepalived again:
+
+   .. code:: console
+
+      openstack-ansible haproxy-install.yml --tags keepalived --limit localhost,reinstalled_host
+
+   After that you might want to ensure that "local" backends remain disabled.
+   You can also use a playbook from `OPS repository`_ for this:
+
+   .. code:: console
+
+      openstack-ansible set-haproxy-backends-state.yml -e hostname=<infrahost> -e backend_state=disabled --limit reinstalled_host
+
+#. If it is NOT a 'primary', install everything on the new host
+
+   .. code:: console
+
       openstack-ansible setup-infrastructure.yml --limit localhost,repo_all,rabbitmq_all,reinstalled_host*
       openstack-ansible setup-openstack.yml --limit localhost,keystone_all,reinstalled_host*
 
@@ -182,11 +223,7 @@ Deploying Infrastructure Hosts
 
 #. If it IS a 'primary', do these steps
 
-   .. code:: console
-
-      openstack-ansible setup-hosts.yml --limit localhost,reinstalled_host*
-
-   Temporarily set your primary Galera in MAINT in HAProxy
+   #. Temporarily set your primary Galera in MAINT in HAProxy
 
       .. code:: console
 
@@ -210,7 +247,7 @@ Deploying Infrastructure Hosts
       Everything should be sync'ed and in order now. You can take your
       primary Galera from MAINT to READY
 
-   We can move on to RabbitMQ primary
+   #. We can move on to RabbitMQ primary
 
       .. code:: console
 
@@ -226,7 +263,7 @@ Deploying Infrastructure Hosts
          rabbitmqctl start_app
          rabbitmqctl cluster_status
 
-   Everything should now be in a working state and we can finish it off with
+   #. Everything should now be in a working state and we can finish it off with
 
       .. code:: console
 
@@ -242,6 +279,13 @@ Deploying Infrastructure Hosts
    set to READY in HAProxy, and any which remain on the old operating system
    are set to 'MAINT'.
 
+   You can also use a playbook from `OPS repository`_ to re-enable all backends from the host:
+
+   .. code:: console
+
+      openstack-ansible set-haproxy-backends-state.yml -e hostname=<infrahost> -e backend_state=enabled
+
+
 Deploying Compute & Network Hosts
 =================================
 
@@ -275,3 +319,5 @@ Deploying Compute & Network Hosts
    Additionally, BGP speakers (used for IPv6) had to be re-initialised from the
    command line. These steps were necessary before reinstalling further network
    nodes to prevent HA Router interruptions.
+
+.. _OPS repository: https://opendev.org/openstack/openstack-ansible-ops/src/branch/master/ansible_tools/playbooks/set-haproxy-backends-state.yml