privsep helper command exited non-zero (1): oslo_privsep.daemon.FailedToDropPrivileges

[Chris L]

On a new install of Microstack --devmode --beta on 2 nodes on Ubuntu 20.04 LTS, one is control, second is compute. I can launch a VM on the second compute node, but the VM fail to build on the control node with the following error in nova.log

There's nothing different on the /etc/sudoers file on both nodes. I've searched bug reports and have not been able to find anything similar.

17d5cb0b8f6ce - default default] Running privsep helper: ['sudo', 'privsep-helper', '--config-file', '/snap/microstack/233/etc/nova/nova.conf', '--config-dir', '/var/snap/microstack/common/etc/nova/nova.conf.d', '--privsep_context', 'nova.privsep.sys_admin_pctxt', '--privsep_sock_path', '/tmp/tmp0d4p31kw/privsep.sock']
2021-07-02 07:29:09.311 12145 INFO nova.osapi_compute.wsgi.server [req-b813e1d7-10ed-4f29-9417-513c16b10d37 b0e76a1afc5745cebc4b887913a94a87 6100575b7f95425ab8017d5cb0b8f6ce - default default] 172.16.6.111 "GET /v2.1/limits?reserved=1 HTTP/1.1" status: 200 len: 914 time: 0.0204794
2021-07-02 07:29:09.451 12145 INFO nova.osapi_compute.wsgi.server [req-f2057bdc-121f-4a32-b06d-89b5c8390052 b0e76a1afc5745cebc4b887913a94a87 6100575b7f95425ab8017d5cb0b8f6ce - default default] 172.16.6.111 "GET /v2.1/limits?reserved=1 HTTP/1.1" status: 200 len: 914 time: 0.0182354
2021-07-02 07:29:09.482 15349 WARNING oslo.privsep.daemon [-] privsep log: Traceback (most recent call last):
2021-07-02 07:29:09.523 15349 WARNING oslo.privsep.daemon [-] privsep log: File "/usr/lib/python3.8/pkgutil.py", line 415, in get_importer
2021-07-02 07:29:09.523 15349 WARNING oslo.privsep.daemon [-] privsep log: importer = sys.path_importer_cache[path_item]
2021-07-02 07:29:09.524 15349 WARNING oslo.privsep.daemon [-] privsep log: KeyError: ''
2021-07-02 07:29:09.524 15349 WARNING oslo.privsep.daemon [-] privsep log:
2021-07-02 07:29:09.524 15349 WARNING oslo.privsep.daemon [-] privsep log: During handling of the above exception, another exception occurred:
2021-07-02 07:29:09.525 15349 WARNING oslo.privsep.daemon [-] privsep log:
2021-07-02 07:29:09.525 15349 WARNING oslo.privsep.daemon [-] privsep log: Traceback (most recent call last):
2021-07-02 07:29:09.525 15349 WARNING oslo.privsep.daemon [-] privsep log: File "/snap/microstack/233/bin/privsep-helper", line 5, in <module>
2021-07-02 07:29:09.525 15349 WARNING oslo.privsep.daemon [-] privsep log: from oslo_privsep.daemon import helper_main
2021-07-02 07:29:09.526 15349 WARNING oslo.privsep.daemon [-] privsep log: File "/snap/microstack/233/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 61, in <module>
2021-07-02 07:29:09.526 15349 WARNING oslo.privsep.daemon [-] privsep log: from oslo_config import cfg
2021-07-02 07:29:09.526 15349 WARNING oslo.privsep.daemon [-] privsep log: File "/snap/microstack/233/lib/python3.8/site-packages/oslo_config/cfg.py", line 45, in <module>
2021-07-02 07:29:09.526 15349 WARNING oslo.privsep.daemon [-] privsep log: from oslo_config import types
2021-07-02 07:29:09.527 15349 WARNING oslo.privsep.daemon [-] privsep log: File "/snap/microstack/233/lib/python3.8/site-packages/oslo_config/types.py", line 28, in <module>
2021-07-02 07:29:09.527 15349 WARNING oslo.privsep.daemon [-] privsep log: from debtcollector import removals
2021-07-02 07:29:09.527 15349 WARNING oslo.privsep.daemon [-] privsep log: File "/snap/microstack/233/lib/python3.8/site-packages/debtcollector/__init__.py", line 19, in <module>
2021-07-02 07:29:09.528 15349 WARNING oslo.privsep.daemon [-] privsep log: __version__ = pbr.version.VersionInfo(
2021-07-02 07:29:09.528 15349 WARNING oslo.privsep.daemon [-] privsep log: File "/snap/microstack/233/lib/python3.8/site-packages/pbr/version.py", line 467, in version_string
2021-07-02 07:29:09.528 15349 WARNING oslo.privsep.daemon [-] privsep log: return self.semantic_version().brief_string()
2021-07-02 07:29:09.528 15349 WARNING oslo.privsep.daemon [-] privsep log: File "/snap/microstack/233/lib/python3.8/site-packages/pbr/version.py", line 462, in semantic_version
2021-07-02 07:29:09.529 15349 WARNING oslo.privsep.daemon [-] privsep log: self._semantic = self._get_version_from_pkg_resources()
2021-07-02 07:29:09.529 15349 WARNING oslo.privsep.daemon [-] privsep log: File "/snap/microstack/233/lib/python3.8/site-packages/pbr/version.py", line 439, in _get_version_from_pkg_resources
2021-07-02 07:29:09.529 15349 WARNING oslo.privsep.daemon [-] privsep log: import pkg_resources
2021-07-02 07:29:09.530 15349 WARNING oslo.privsep.daemon [-] privsep log: File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3254, in <module>
2021-07-02 07:29:09.530 15349 WARNING oslo.privsep.daemon [-] privsep log: def _initialize_master_working_set():
2021-07-02 07:29:09.530 15349 WARNING oslo.privsep.daemon [-] privsep log: File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3237, in _call_aside
2021-07-02 07:29:09.530 15349 WARNING oslo.privsep.daemon [-] privsep log: f(*args, **kwargs)
2021-07-02 07:29:09.531 15349 WARNING oslo.privsep.daemon [-] privsep log: File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3266, in _initialize_master_working_set
2021-07-02 07:29:09.531 15349 WARNING oslo.privsep.daemon [-] privsep log: working_set = WorkingSet._build_master()
2021-07-02 07:29:09.531 15349 WARNING oslo.privsep.daemon [-] privsep log: File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 575, in _build_master
2021-07-02 07:29:09.532 15349 WARNING oslo.privsep.daemon [-] privsep log: ws = cls()
2021-07-02 07:29:09.532 15349 WARNING oslo.privsep.daemon [-] privsep log: File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 568, in __init__
2021-07-02 07:29:09.532 15349 WARNING oslo.privsep.daemon [-] privsep log: self.add_entry(entry)
2021-07-02 07:29:09.532 15349 WARNING oslo.privsep.daemon [-] privsep log: File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 624, in add_entry
2021-07-02 07:29:09.533 15349 WARNING oslo.privsep.daemon [-] privsep log: for dist in find_distributions(entry, True):
2021-07-02 07:29:09.533 15349 WARNING oslo.privsep.daemon [-] privsep log: File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1970, in find_distributions
2021-07-02 07:29:09.533 15349 WARNING oslo.privsep.daemon [-] privsep log: importer = get_importer(path_item)
2021-07-02 07:29:09.533 15349 WARNING oslo.privsep.daemon [-] privsep log: File "/usr/lib/python3.8/pkgutil.py", line 419, in get_importer
2021-07-02 07:29:09.534 15349 WARNING oslo.privsep.daemon [-] privsep log: importer = path_hook(path_item)
2021-07-02 07:29:09.534 15349 WARNING oslo.privsep.daemon [-] privsep log: File "<frozen importlib._bootstrap_external>", line 1522, in path_hook_for_FileFinder
2021-07-02 07:29:09.534 15349 WARNING oslo.privsep.daemon [-] privsep log: File "<frozen importlib._bootstrap_external>", line 107, in _path_isdir
2021-07-02 07:29:09.535 15349 WARNING oslo.privsep.daemon [-] privsep log: FileNotFoundError: [Errno 2] No such file or directory
2021-07-02 07:29:09.535 15349 CRITICAL oslo.privsep.daemon [req-aa6cc2a2-f6bd-429f-b2a4-0b4f70bf706b b0e76a1afc5745cebc4b887913a94a87 6100575b7f95425ab8017d5cb0b8f6ce - default default] privsep helper command exited non-zero (1)
2021-07-02 07:29:09.536 15349 ERROR nova.compute.manager [req-aa6cc2a2-f6bd-429f-b2a4-0b4f70bf706b b0e76a1afc5745cebc4b887913a94a87 6100575b7f95425ab8017d5cb0b8f6ce - default default] [instance: 4b2b6467-00a5-41ab-87d3-5c92e30ec87a] Instance failed to spawn: oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1)