New Prometheus-openstack-exporter snap refresh breaks exporter

[Tejeev Patel]

Edge Prometheus snap refreshed on a few managed clouds and saw them all loose communication with the exporter. The primary alert we received was the following:
HTTP CRITICAL - No data received from host

jhebden rolled beta, edge, and candidate channels back to revision 16 to fix.
recreating the bug was more difficult than expected.

Once the edge snap had been rolled back, I ran 'snap refresh' on all 4 clouds and saw the majority of alerts change to:
connect to address 127.0.0.1 and port 9183: Connection refused

I see nothing listening on port 9183 and I see multiple exporter processes running (this might be normal but not sure):
systemctl list-units|grep openstack
  run-snapd-ns-prometheus\x2dopenstack\x2dexporter.mnt.mount loaded active mounted /run/snapd/ns/prometheus-openstack-exporter.mnt
  snap-prometheus\x2dopenstack\x2dexporter-10.mount loaded active mounted Mount unit for prometheus-openstack-exporter
  snap-prometheus\x2dopenstack\x2dexporter-16.mount loaded active mounted Mount unit for prometheus-openstack-exporter
  snap-prometheus\x2dopenstack\x2dexporter-18.mount loaded active mounted Mount unit for prometheus-openstack-exporter
  jujud-unit-prometheus-openstack-exporter-0.service loaded active running juju unit agent for prometheus-openstack-exporter/0

[Marian Gasparovic (marosg)]

This is from log when restarting prometheus-openstack-exporter service

Dec 6 12:39:33 xxx-prometheus-1 prometheus-openstack-exporter.prometheus-openstack-exporter[11099]: python2: relocation error: /snap/prometheus-openstack-exporter/16/lib/x86_64-linux-gnu/libdl.so.2: symbol _dl_catch_error, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference
Dec 6 12:39:33 xxx-prometheus-1 systemd[1]: snap.prometheus-openstack-exporter.prometheus-openstack-exporter.service: Main process exited, code=exited, status=127/n/a
Dec 6 12:39:33 xxx-prometheus-1 systemd[1]: snap.prometheus-openstack-exporter.prometheus-openstack-exporter.service: Unit entered failed state.
Dec 6 12:39:33 xxx-prometheus-1 systemd[1]: snap.prometheus-openstack-exporter.prometheus-openstack-exporter.service: Failed with result 'exit-code'.
Dec 6 12:39:33 xxx-prometheus-1 systemd[1]: snap.prometheus-openstack-exporter.prometheus-openstack-exporter.service: Service hold-off time over, scheduling restart.
Dec 6 12:39:33 xxx-prometheus-1 systemd[1]: Stopped Service for snap application prometheus-openstack-exporter.prometheus-openstack-exporter.
Dec 6 12:39:33 xxx-prometheus-1 systemd[1]: Started Service for snap application prometheus-openstack-exporter.prometheus-openstack-exporter.
Dec 6 12:39:33 xxx-prometheus-1 /usr/bin/snap[11117]: cmd.go:118: DEBUG: restarting into "/snap/core/current/usr/bin/snap"