================== fails with python3.8 ================== # systemctl status nova-api-metadata ● nova-api-metadata.service - OpenStack Compute metadata API Loaded: loaded (/lib/systemd/system/nova-api-metadata.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2020-02-12 19:56:57 UTC; 2s ago Main PID: 20958 (nova-api-metada) Tasks: 5 (limit: 13942) Memory: 134.4M CGroup: /system.slice/nova-api-metadata.service ├─20958 /usr/bin/python3 /usr/bin/nova-api-metadata --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-api-metadata.log ├─20965 /usr/bin/python3 /usr/bin/nova-api-metadata --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-api-metadata.log ├─20966 /usr/bin/python3 /usr/bin/nova-api-metadata --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-api-metadata.log ├─20967 /usr/bin/python3 /usr/bin/nova-api-metadata --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-api-metadata.log └─20968 /usr/bin/python3 /usr/bin/nova-api-metadata --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-api-metadata.log Feb 12 19:56:59 autopkgtest-lxd-osfnks nova-api-metadata[20967]: Exception ignored in: Feb 12 19:56:59 autopkgtest-lxd-osfnks nova-api-metadata[20967]: Traceback (most recent call last): Feb 12 19:56:59 autopkgtest-lxd-osfnks nova-api-metadata[20967]: File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork Feb 12 19:56:59 autopkgtest-lxd-osfnks nova-api-metadata[20967]: assert len(_active) == 1 Feb 12 19:56:59 autopkgtest-lxd-osfnks nova-api-metadata[20967]: AssertionError: Feb 12 19:56:59 autopkgtest-lxd-osfnks nova-api-metadata[20968]: Exception ignored in: Feb 12 19:56:59 autopkgtest-lxd-osfnks nova-api-metadata[20968]: Traceback (most recent call last): Feb 12 19:56:59 autopkgtest-lxd-osfnks nova-api-metadata[20968]: File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork Feb 12 19:56:59 autopkgtest-lxd-osfnks nova-api-metadata[20968]: assert len(_active) == 1 Feb 12 19:56:59 autopkgtest-lxd-osfnks nova-api-metadata[20968]: AssertionError: ======================== switch to python 3.7 ================== # ls -al /usr/bin/python3 lrwxrwxrwx 1 root root 18 Feb 12 19:56 /usr/bin/python3 -> /usr/bin/python3.8 # sudo rm -rf /usr/bin/python3 # sudo ln -s /usr/bin/python3.7 /usr/bin/python3 # ls -al /usr/bin/python3 lrwxrwxrwx 1 root root 18 Feb 12 19:57 /usr/bin/python3 -> /usr/bin/python3.7 ======================== seems ok with python 3.7 ================== # sudo systemctl restart nova-api-metadata # systemctl status nova-api-metadata ● nova-api-metadata.service - OpenStack Compute metadata API Loaded: loaded (/lib/systemd/system/nova-api-metadata.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2020-02-12 19:57:50 UTC; 8s ago Main PID: 21023 (nova-api-metada) Tasks: 5 (limit: 13942) Memory: 147.1M CGroup: /system.slice/nova-api-metadata.service ├─21023 /usr/bin/python3 /usr/bin/nova-api-metadata --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-api-metadata.log ├─21032 /usr/bin/python3 /usr/bin/nova-api-metadata --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-api-metadata.log ├─21033 /usr/bin/python3 /usr/bin/nova-api-metadata --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-api-metadata.log ├─21034 /usr/bin/python3 /usr/bin/nova-api-metadata --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-api-metadata.log └─21035 /usr/bin/python3 /usr/bin/nova-api-metadata --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-api-metadata.log Feb 12 19:57:50 autopkgtest-lxd-osfnks systemd[1]: Started OpenStack Compute metadata API. # systemctl status nova-api-metadata ● nova-api-metadata.service - OpenStack Compute metadata API Loaded: loaded (/lib/systemd/system/nova-api-metadata.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2020-02-12 19:57:50 UTC; 51s ago Main PID: 21023 (nova-api-metada) Tasks: 5 (limit: 13942) Memory: 147.3M CGroup: /system.slice/nova-api-metadata.service ├─21023 /usr/bin/python3 /usr/bin/nova-api-metadata --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-api-metadata.log ├─21032 /usr/bin/python3 /usr/bin/nova-api-metadata --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-api-metadata.log ├─21033 /usr/bin/python3 /usr/bin/nova-api-metadata --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-api-metadata.log ├─21034 /usr/bin/python3 /usr/bin/nova-api-metadata --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-api-metadata.log └─21035 /usr/bin/python3 /usr/bin/nova-api-metadata --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-api-metadata.log Feb 12 19:57:50 autopkgtest-lxd-osfnks systemd[1]: Started OpenStack Compute metadata API. ======================== patch nova's eventlet code ================== # sudo vi /usr/lib/python3/dist-packages/nova/monkey_patch.py ## apply the following patch ## diff --git a/nova/monkey_patch.py b/nova/monkey_patch.py index a07ff91dac..bb7252c643 100644 --- a/nova/monkey_patch.py +++ b/nova/monkey_patch.py @@ -59,6 +59,9 @@ def _monkey_patch(): else: eventlet.monkey_patch() + import __original_module_threading + import threading + __original_module_threading.current_thread.__globals__['_active'] = threading._active # NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet # hub use a monotonic clock to avoid issues with drifts of system time (see ======================== and switch to python 3.8 ================== # sudo rm -rf /usr/bin/python3 # sudo ln -s /usr/bin/python3.8 /usr/bin/python3 # ls -al /usr/bin/python3 lrwxrwxrwx 1 root root 18 Feb 12 20:08 /usr/bin/python3 -> /usr/bin/python3.8 # LP 1510234 for details) ======================== seems ok with python 3.8 now ================== # sudo systemctl restart nova-api-metadata # sudo systemctl status nova-api-metadata ● nova-api-metadata.service - OpenStack Compute metadata API Loaded: loaded (/lib/systemd/system/nova-api-metadata.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2020-02-12 20:10:19 UTC; 2min 46s ago Main PID: 21788 (nova-api-metada) Tasks: 5 (limit: 13942) Memory: 134.3M CGroup: /system.slice/nova-api-metadata.service ├─21788 /usr/bin/python3 /usr/bin/nova-api-metadata --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-api-metadata.log ├─21893 /usr/bin/python3 /usr/bin/nova-api-metadata --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-api-metadata.log ├─21894 /usr/bin/python3 /usr/bin/nova-api-metadata --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-api-metadata.log ├─21895 /usr/bin/python3 /usr/bin/nova-api-metadata --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-api-metadata.log └─21896 /usr/bin/python3 /usr/bin/nova-api-metadata --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-api-metadata.log Feb 12 20:10:19 autopkgtest-lxd-osfnks systemd[1]: Started OpenStack Compute metadata API.