This is using python3 from the system and hitting systemd tasks limit

[Shunde Zhang]

This is affecting the snap. It was tested with 0.1.9 latest/edge but should affect also previous releases.

2 issues:

1. python3 symlinks are broken inside the snap, and when installed on bionic it uses python3 from the system (python 3.6)

2. Snap service is hitting systemd tasks limit and then throwing BlockingIOError

[Shunde Zhang (shunde-zhang)]

The new prometheus-openstack-exporter version 0.1.9 (in snap's latest/edge channel currently) requires Python3.8, which is not the default Python on Bionic but it is the default Python on Focal.

# ls -l /snap/prometheus-openstack-exporter/current/bin/
...
lrwxrwxrwx 1 root root 7 Jul 30 12:32 python -> python3
lrwxrwxrwx 1 root root 18 Jul 30 12:33 python3 -> /usr/bin/python3.8

When running on Bionic, it throws a lot of errors in syslog.

Nov 30 06:28:44 juju-34915b-openstack-39 prometheus-openstack-exporter.prometheus-openstack-exporter[8211]: File "/usr/lib/python3.8/socketserver.py", line 316, in _handle_request_noblock
Nov 30 06:28:44 juju-34915b-openstack-39 prometheus-openstack-exporter.prometheus-openstack-exporter[8211]: self.process_request(request, client_address)
Nov 30 06:28:44 juju-34915b-openstack-39 prometheus-openstack-exporter.prometheus-openstack-exporter[8211]: File "/usr/lib/python3.8/socketserver.py", line 603, in process_request
Nov 30 06:28:44 juju-34915b-openstack-39 prometheus-openstack-exporter.prometheus-openstack-exporter[8211]: pid = os.fork()
Nov 30 06:28:44 juju-34915b-openstack-39 prometheus-openstack-exporter.prometheus-openstack-exporter[8211]: BlockingIOError: [Errno 11] Resource temporarily unavailable

To solve that, Python3.8 needs to be installed on Bionic.

apt install python3.8 libpython3.8

However snap doesn't do this, neither does the charm.
So maybe we need to add this to the charm? e.g. if running exporter 0.1.9 on Bionic, the charm should install Python3.8 and its lib.

[Shunde Zhang (shunde-zhang)]

Looks like 'snap refresh' will break prometheus-openstack-exporter 0.1.9 [1].
I have to restart the exporter service in systemd to bring it back again.

[1] https://pastebin.ubuntu.com/p/2zKJxP9MsQ/

[Giuseppe Petralia (peppepetra)]

Thanks for reporting this. I am not able to reproduce the BlockingIOError reported, but I have proposed a change to fix the broken symlink for python3 here:

https://code.launchpad.net/~peppepetra/prometheus-openstack-exporter-snap/+git/prometheus-openstack-exporter-snap/+merge/434246

[Giuseppe Petralia (peppepetra)]

I am able to reproduce the BlockingIOError. This is happening because the exporter uses a ForkingHTTPServer which creates a new thread on each call increasing the tasks count until it reaches the limit and then starts throwing the BlockingIOError.

I have proposed a fix on the same MP:
https://code.launchpad.net/~peppepetra/prometheus-openstack-exporter-snap/+git/prometheus-openstack-exporter-snap/+merge/434246