Ports which should be open, but are not: 8032

[Alex Kavanagh]

mojo designate_ha xenial-newton
juju 2.8.10
openstack-charmers-next charms.

The apache2 server failed to start on all three units due to port 8032 already been held:

From syslog on an affected unit:

Apr 19 11:32:25 juju-adf1a1-auto-osci-sv07-0 systemd[1]: Stopping LSB: Apache2 web server...
Apr 19 11:32:25 juju-adf1a1-auto-osci-sv07-0 apache2[6607]: * Stopping Apache httpd web server apache2
Apr 19 11:32:27 juju-adf1a1-auto-osci-sv07-0 apache2[6607]: *
Apr 19 11:32:27 juju-adf1a1-auto-osci-sv07-0 systemd[1]: Stopped LSB: Apache2 web server.
Apr 19 11:32:27 juju-adf1a1-auto-osci-sv07-0 systemd[1]: Starting LSB: Apache2 web server...
Apr 19 11:32:27 juju-adf1a1-auto-osci-sv07-0 apache2[6642]: * Starting Apache httpd web server apache2
Apr 19 11:32:28 juju-adf1a1-auto-osci-sv07-0 apache2[6642]: (98)Address already in use: AH00072: make_sock: could not bind to address [::]:8032
Apr 19 11:32:28 juju-adf1a1-auto-osci-sv07-0 apache2[6642]: (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:8032
Apr 19 11:32:28 juju-adf1a1-auto-osci-sv07-0 apache2[6642]: no listening sockets available, shutting down
Apr 19 11:32:28 juju-adf1a1-auto-osci-sv07-0 apache2[6642]: AH00015: Unable to open logs
Apr 19 11:32:28 juju-adf1a1-auto-osci-sv07-0 apache2[6642]: Action 'start' failed.
Apr 19 11:32:28 juju-adf1a1-auto-osci-sv07-0 apache2[6642]: The Apache error log may have more information.
Apr 19 11:32:28 juju-adf1a1-auto-osci-sv07-0 apache2[6642]: *
Apr 19 11:32:28 juju-adf1a1-auto-osci-sv07-0 apache2[6666]: * Stopping Apache httpd web server apache2
Apr 19 11:32:28 juju-adf1a1-auto-osci-sv07-0 apache2[6666]: *

Nothing is listening:

tcp 0 0 0.0.0.0:8042 0.0.0.0:* LISTEN 29106/haproxy
tcp 0 0 252.12.0.1:53 0.0.0.0:* LISTEN 3046/dnsmasq
tcp 0 0 0.0.0.0:8022 0.0.0.0:* LISTEN 7625/aodh-api.log
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1320/sshd
tcp 0 0 127.0.0.1:8888 0.0.0.0:* LISTEN 29106/haproxy
tcp6 0 0 fe80::1:13128 :::* LISTEN 4042/lxd-bridge-pro
tcp6 0 0 :::8042 :::* LISTEN 29106/haproxy
tcp6 0 0 fe80::bc67:22ff:fe1d:53 :::* LISTEN 3046/dnsmasq
tcp6 0 0 :::22 :::* LISTEN 1320/sshd

Juju blocked status:

2:35:51 aodh/0* blocked idle 0 172.17.107.12 8042/tcp Ports which should be open, but are not: 8032
12:35:51 aodh-hacluster/1 active idle 172.17.107.12 Unit is ready and clustered
12:35:51 aodh/1 blocked idle 1 172.17.107.7 8042/tcp Ports which should be open, but are not: 8032
12:35:51 aodh-hacluster/0* active idle 172.17.107.7 Unit is ready and clustered
12:35:51 aodh/2 blocked idle 2 172.17.107.21 8042/tcp Ports which should be open, but are not: 8032
12:35:51 aodh-hacluster/2 active idle 172.17.107.21 Unit is ready and clustered

Issue: I suspect that the apache2 stop script returns before all the processes have been reaped OR the previous owner of 8032 may still have been holding the port when apache2 was started. i.e. we have a race somewhere.