Magnum fails with "ModuleNotFoundError: No module named 'encodings'"

[Andreas Florath]

The installation of openstack magnum with openstack-ansible succeeds.

But it looks that the magnum-api.service is left in an failure state.

# systemctl status magnum-api.service
● magnum-api.service - magnum-api service
   Loaded: loaded (/etc/systemd/system/magnum-api.service; enabled; vendor preset: enabled)
   Active: failed (Result: signal) since Tue 2019-12-31 18:42:53 UTC; 23h ago
  Process: 2087 ExecStart=/openstack/venvs/uwsgi-20.0.0-python3/bin/uwsgi --autoload --ini /etc/uwsgi/magnum-api.ini (code=killed, signal=ABRT)
 Main PID: 2087 (code=killed, signal=ABRT)
      CPU: 20ms

Dec 31 18:42:51 controller0-magnum-container-3b7d2d08 systemd[1]: magnum-api.service: Consumed 20ms CPU time
Dec 31 18:42:53 controller0-magnum-container-3b7d2d08 systemd[1]: magnum-api.service: Service hold-off time over, scheduling restart.
Dec 31 18:42:53 controller0-magnum-container-3b7d2d08 systemd[1]: magnum-api.service: Scheduled restart job, restart counter is at 30.
Dec 31 18:42:53 controller0-magnum-container-3b7d2d08 systemd[1]: Stopped magnum-api service.
Dec 31 18:42:53 controller0-magnum-container-3b7d2d08 systemd[1]: magnum-api.service: Consumed 20ms CPU time
Dec 31 18:42:53 controller0-magnum-container-3b7d2d08 systemd[1]: magnum-api.service: Start request repeated too quickly.
Dec 31 18:42:53 controller0-magnum-container-3b7d2d08 systemd[1]: magnum-api.service: Failed with result 'signal'.
Dec 31 18:42:53 controller0-magnum-container-3b7d2d08 systemd[1]: Failed to start magnum-api service.

A bit of research revealed that the root cause might be a problem within uwsgi:

# /openstack/venvs/uwsgi-20.0.0-python3/bin/uwsgi --autoload --ini /etc/uwsgi/magnum-api.ini
[uWSGI] getting INI configuration from /etc/uwsgi/magnum-api.ini
*** Starting uWSGI 2.0.18 (64bit) on [Wed Jan 1 17:54:27 2020] ***
compiled with version: 7.4.0 on 31 December 2019 14:38:20
os: Linux-4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019
nodename: controller0-magnum-container-3b7d2d08
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 48
current working directory: /root
detected binary path: /openstack/venvs/uwsgi-20.0.0-python3/bin/uwsgi
*** dumping internal routing table ***
[rule: 0] subject: user_agent regexp: ^osa-haproxy-healthcheck$ action: donotlog:
*** end of the internal routing table ***
setgid() to 999
setuid() to 999
your processes number limit is 768209
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: enabled
uwsgi socket 0 bound to TCP address 0.0.0.0:9511 fd 3
Python version: 3.6.9 (default, Nov 7 2019, 10:44:02) [GCC 8.3.0]
Set PythonHome to /openstack/venvs/magnum-20.0.0/bin
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007ff521e27240 (most recent call first):
Aborted

Using openstack-ansible 20.0.0 on Ubuntu 18.04 LTS systems.

N.B.:
I'm not really sure if this is a problem of openstack-ansible or magnum.

[Andreas Florath (ansreas)]

It looks that this might be a mixture between python3 and python2?

/var/log/python_venv_build.log:
Ignoring argparse: markers 'python_version == "2.7"' don't match your environment

and also in /etc/uwsgi/magnum-api.ini:

wsgi-file = /openstack/venvs/magnum-20.0.0/lib/python2.7/site-packages/magnum/api/app.wsgi

but /openstack/venvs/magnum-20.0.0/lib/python2.7 does not exists at all - only the 3.6 version.

# /openstack/venvs/magnum-20.0.0/bin/python --version
Python 3.6.9

[Andreas Florath (ansreas)]

Workaround / Possible Fix?

It looks that the /etc/uwsgi/magnum-api.ini is wrong at two positions.

As mentioned above, there is the need to use python3.6 instead of 2.7 so change from:

wsgi-file = /openstack/venvs/magnum-20.0.0/lib/python2.7/site-packages/magnum/api/app.wsgi

to

wsgi-file = /openstack/venvs/magnum-20.0.0/lib/python3.6/site-packages/magnum/api/app.wsgi

In addition the virtualenv is not set correctly. This needs to be changed from:

virtualenv = /openstack/venvs/magnum-20.0.0/bin

to

virtualenv = /openstack/venvs/magnum-20.0.0

At least for me the service is now up and running:

● magnum-api.service - magnum-api service
   Loaded: loaded (/etc/systemd/system/magnum-api.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-01-01 18:34:46 UTC; 5min ago
 Main PID: 3666 (uwsgi)
    Tasks: 17 (limit: 7372)
   Memory: 1.0G
      CPU: 45.718s
   CGroup: /uwsgi.slice/magnum-api.service
           ├─3666 /openstack/venvs/uwsgi-20.0.0-python3/bin/uwsgi --autoload --ini /etc/uwsgi/magnum-api.ini
           ├─3667 /openstack/venvs/uwsgi-20.0.0-python3/bin/uwsgi --autoload --ini /etc/uwsgi/magnum-api.ini
[...]

[Andreas Florath (ansreas)]

BTW: magnum connects to keystone which fails, because magnum cannot check the certificate (there is currently none configured).

As a workaround for non-production systems you ca set 'insecure = True' twice in /etc/magnum/magnum.conf

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/699726
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_magnum/commit/?id=2e118940f1120c804f7dfcd890f6ff4ba565656b
Submitter: Zuul
Branch: master

commit 2e118940f1120c804f7dfcd890f6ff4ba565656b
Author: Jonathan Rosser <email address hidden>
Date: Wed Dec 18 16:32:23 2019 +0000

    Fix magnum-api wsgi name

    This should not be a hardcoded path and certainly not to python2.7

    Closes-Bug: 1858034
    Change-Id: Id22502d0fbe33b3d07ad738f39aff8367ef52074

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_magnum ussuri-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_magnum yoga-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_magnum victoria-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_magnum wallaby-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_magnum xena-eom release.