lava-server-gunicorn fails to start with python-eventlet > 0.30.3

[Brian Murray]

autopkgtest [16:28:47]: test management: [-----------------------
+ cd /root
+ lava-server manage check
System check identified no issues (0 silenced).
+ lava-server manage check --deploy
SystemCheckError: System check identified some issues:

ERRORS:
lava services: lava-server-gunicorn service is not active.

INFOS:
debian pkg: 'lava-coordinator' not installed from a Debian package

System check identified 2 issues (2 silenced).
autopkgtest [16:28:49]: test management: -----------------------]
management FAIL non-zero exit status 1

ssh'ing into a test system with the management test failing didn't reveal much information:

Aug 3 13:52:36 autopkgtest systemd[1]: lava-server-gunicorn.service: Main process exited, code=exited, status=1/FAILURE
Aug 3 13:52:36 autopkgtest systemd[1]: lava-server-gunicorn.service: Failed with result 'exit-code'.
Aug 3 13:52:36 autopkgtest systemd[1]: lava-server-gunicorn.service: Scheduled restart job, restart counter is at 1.
Aug 3 13:52:37 autopkgtest systemd[1]: lava-server-gunicorn.service: Main process exited, code=exited, status=1/FAILURE
Aug 3 13:52:37 autopkgtest systemd[1]: lava-server-gunicorn.service: Failed with result 'exit-code'.
Aug 3 13:52:37 autopkgtest systemd[1]: lava-server-gunicorn.service: Scheduled restart job, restart counter is at 2.
Aug 3 13:52:37 autopkgtest systemd[1]: lava-server-gunicorn.service: Main process exited, code=exited, status=1/FAILURE
Aug 3 13:52:37 autopkgtest systemd[1]: lava-server-gunicorn.service: Failed with result 'exit-code'.
Aug 3 13:52:37 autopkgtest systemd[1]: lava-server-gunicorn.service: Scheduled restart job, restart counter is at 3.
Aug 3 13:52:38 autopkgtest systemd[1]: lava-server-gunicorn.service: Main process exited, code=exited, status=1/FAILURE
Aug 3 13:52:38 autopkgtest systemd[1]: lava-server-gunicorn.service: Failed with result 'exit-code'.
Aug 3 13:52:38 autopkgtest systemd[1]: lava-server-gunicorn.service: Scheduled restart job, restart counter is at 4.
Aug 3 13:52:38 autopkgtest systemd[1]: lava-server-gunicorn.service: Main process exited, code=exited, status=1/FAILURE
Aug 3 13:52:38 autopkgtest systemd[1]: lava-server-gunicorn.service: Failed with result 'exit-code'.
Aug 3 13:52:38 autopkgtest systemd[1]: lava-server-gunicorn.service: Scheduled restart job, restart counter is at 5.
Aug 3 13:52:38 autopkgtest systemd[1]: lava-server-gunicorn.service: Start request repeated too quickly.
Aug 3 13:52:38 autopkgtest systemd[1]: lava-server-gunicorn.service: Failed with result 'exit-code'.

[Brian Murray (brian-murray)]

Manually running the ExecStart line from the service revealed the following error:

ubuntu@autopkgtest:/tmp/autopkgtest.HFgA6D/build.41y/src$ sudo /usr/bin/gunicorn3 lava_server.wsgi --log-level DEBUG --log-file /var/log/lava-server/gunicorn.log -u lavaserver -g lavaserver --worker-class eventlet --workers 4

Error: class uri 'eventlet' invalid or not found:

[Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/gunicorn/util.py", line 99, in load_class
    mod = importlib.import_module('.'.join(components))
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/lib/python3/dist-packages/gunicorn/workers/geventlet.py", line 20, in <module>
    from eventlet.wsgi import ALREADY_HANDLED as EVENTLET_ALREADY_HANDLED
ImportError: cannot import name 'ALREADY_HANDLED' from 'eventlet.wsgi' (/usr/lib/python3/dist-packages/eventlet/wsgi.py)

[Brian Murray (brian-murray)]

This upstream merge proposal, for gunicorn, should fix the issue.

https://github.com/benoitc/gunicorn/pull/2581/files

[Brian Murray (brian-murray)]

I cowbody'ed the patch in the test environment and the service was able to start. I was then able to run 'debian/tests/management' and that passed.

[Brian Murray (brian-murray)]

The problematic version of python-eventlet has been removed from the archive for Impish[1] but this will still likely become a problem later so let's leave the bug open for people doing +1 maintenance in the future.

[1] https://irclogs.ubuntu.com/2021/08/04/%23ubuntu-release.txt
[19:07] <coreycb> ubuntu-archive: hello, please can we have python-eventlet removed from impish-proposed? it is blocked with an autopkgtest failure and we've decided to keep gunicorn in sync with debian for now instead of patching it, and the eventlet delta does not pick up too much new.
[19:54] <apw> coreycb, gone.
[19:54] <coreycb> apw: thanks