upgrade charm on aodh from 18.11 to 21.01 did not trigger wsgi template render for apache2

[Drew Freiberger]

After upgrading aodh to 21.01 latest charm on a xenial-queens cloud, I saw aodh exhibit the following symptom:

Blocked status with details: Ports which should be open, but are not: 8032

When I investigated the unit, I found haproxy and apache were both trying to listen on 8042.

The file /etc/apache2/sites-available/aodh-api.conf was missing the following bits from the template/ocata/aodh-api.conf in this version of the charm:

Listen 8032

VirtualHost *:8032

group=aodh

The Listen and virtualHost ports were 8042, and the group=aodh was missing from the WSGIProcess line, and no amount of config-changed seems to be trying to re-generate the rendered wsgi file for apache2.

Here is a private link to the unit debug log during a config-changed hook for setting debug=true:
https://pastebin.canonical.com/p/SxqyCbQzfH/

[Drew Freiberger (afreiberger)]

Workaround is to perform the following on each unit of aodh:

systemctl stop pacemaker
systemctl stop corosync
systemctl stop haproxy
systemctl stop apache2
vi /etc/apache2/sites-available/aodh-api.conf
  # change 8042 port references to 8032)
  # add "group=aodh" after user=aodh on the WSGIDaemonProcess line
systemctl start corosync
systemctl start pacemaker
systemctl start apache2

[Drew Freiberger (afreiberger)]

If you only update the ports to resolve the charm blocked state, without the group setting of aodh on the WSGI process, the /etc/aodh/aodh.conf file can't be read by the apache2 wsgi process, which causes 500 errors when calling the aodh api.

[Drew Freiberger (afreiberger)]

another note, the only charms I've upgraded are aodh, and hacluster-aodh. If there's perhaps something that keystone would signal to aodh or other HAOpenstackCharms in a version of keystone newer than cs:keystone-294, that may also be affecting this outcome.

[Alex Kavanagh (ajkavanagh)]

Could you check the haproxy configs and ensure that the backends are configured properly? (i.e. they are consistent and in the same networks for each backend as needed). We've seen some issues where the local address is used on one, but other networks are used for the other haproxy units. (this could be a red-herring, but just want to check).