Horizon access becomes intermittent during Zed->Antelope upgrade

[Andrew Bonney]

After running setup-infrastructure (specifically haproxy-install), the new 'base-front' service configuration is in place and active, but at that point the Horizon install has not run (and won't do for a long time).

As a result of this, both the 'base-front' and 'horizon-front' services are bound to port 80/443, and HAProxy will determine at random which one serves your request. If the request is routed to 'base-front' an HTTP 503 will result as 'base-front' doesn't yet have the 'base_regex' configuration in place for Horizon (which comes from https://github.com/openstack/openstack-ansible/blob/stable/2023.1/inventory/group_vars/horizon_all/haproxy_service.yml#L44 during horizon-install).

Either the 'base-front' service needs to operate at a lower preference to 'horizon-front' until the Horizon playbook runs (if that's something HAProxy can do), or the upgrade process needs to add the Horizon entry to the map file earlier on as an interim workaround.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/891203

[Damian Dąbrowski (damiandabrowski)]

I proposed a fix [1], it's more a dirty hack but after spending few hours on this problem, I was not able to come up with anything better.

[1] https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/891203

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/openstack-ansible/+/891209

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Damian Dąbrowski <email address hidden>" on branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/891203
Reason: abandoned in favor of: https://review.opendev.org/c/openstack/openstack-ansible/+/891209

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible/+/891209
Committed: https://opendev.org/openstack/openstack-ansible/commit/7f2af146140c126c926b900bbcabf610ad89bcaf
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 7f2af146140c126c926b900bbcabf610ad89bcaf
Author: Damian Dabrowski <email address hidden>
Date: Fri Aug 11 17:38:07 2023 +0200

    Avoid breaking horizon during Antelope upgrade

    During Antelope upgrade, haproxy-install.yml configures base service
    listening on ports 80 and 443.
    But at this point, horizon frontend is not removed yet so haproxy has
    two frontends listening on 80/443.
    During os-horizon-install.yml, horizon frontend is removed and
    appropriate haproxy maps config is applied.
    As a result, between haproxy-install.yml and os-horizon-install.yml,
    horizon will encounter intermittent issues.
    This patch triggers horizon service reconfiguration during Antelope
    upgrade when running haproxy-install.yml playbook.
    To be precise: when horizon is already deployed in an environement but
    its maps config is not there yet, haproxy will trigger reconfiguration
    of horizon service.
    This change is intendent to land only in 2023.1. Upgrades from 2023.1
    to 2023.2 or 2024.1 are not affected.

    Closes-Bug: #2028944
    Change-Id: I9df78f286c1fcc3204b35928829a981addf876e8

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 27.1.0 release.