FWIW we are also seeing this in the nova-lvm job where n-cpu calls lvs via privsep:
https://a59f59e90e7507f14599-2cecd17ff9ea5f4d02c12f6d5fc5aedd.ssl.cf1.rackcdn.com/776681/7/check/nova-lvm/a19ff22/controller/logs/screen-n-cpu.txt
Mar 09 16:45:48.496630 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1525, in _cleanup Mar 09 16:45:48.496630 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR oslo_messaging.rpc.server self._cleanup_lvm(instance, block_device_info) Mar 09 16:45:48.496630 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1615, in _cleanup_lvm Mar 09 16:45:48.496630 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR oslo_messaging.rpc.server disks = self._lvm_disks(instance) Mar 09 16:45:48.496630 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1633, in _lvm_disks Mar 09 16:45:48.496630 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR oslo_messaging.rpc.server logical_volumes = lvm.list_volumes(vg) Mar 09 16:45:48.496630 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/storage/lvm.py", line 111, in list_volumes Mar 09 16:45:48.496630 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR oslo_messaging.rpc.server out, err = nova.privsep.fs.lvlist(vg) Mar 09 16:45:48.496630 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 247, in _wrap Mar 09 16:45:48.496630 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR oslo_messaging.rpc.server return self.channel.remote_call(name, args, kwargs) Mar 09 16:45:48.496630 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 224, in remote_call Mar 09 16:45:48.496630 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR oslo_messaging.rpc.server raise exc_type(*result[2]) Mar 09 16:45:48.496630 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. Mar 09 16:45:48.496630 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR oslo_messaging.rpc.server Command: lvs --noheadings -o lv_name /dev/stack-volumes-default Mar 09 16:45:48.496630 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR oslo_messaging.rpc.server Exit code: -11 Mar 09 16:45:48.496630 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR oslo_messaging.rpc.server Stdout: '' Mar 09 16:45:48.496630 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR oslo_messaging.rpc.server Stderr: ' WARNING: Failed to get udev device handler for device /dev/sda1.\n /dev/sda15: stat failed: No such file or directory\n Path /dev/sda15 no longer valid for device(8,15)\n /dev/sda15: stat failed: No such file or directory\n Path /dev/sda15 no longer valid for device(8,15)\n Device open /dev/sda 8:0 failed errno 2\n Device open /dev/sda 8:0 failed errno 2\n Device open /dev/sda1 8:1 failed errno 2\n Device open /dev/sda1 8:1 failed errno 2\n WARNING: Scan ignoring device 8:0 with no paths.\n WARNING: Scan ignoring device 8:1 with no paths.\n'
FWIW we are also seeing this in the nova-lvm job where n-cpu calls lvs via privsep:
https:/ /a59f59e90e7507 f14599- 2cecd17ff9ea5f4 d02c12f6d5fc5ae dd.ssl. cf1.rackcdn. com/776681/ 7/check/ nova-lvm/ a19ff22/ controller/ logs/screen- n-cpu.txt
Mar 09 16:45:48.496630 ubuntu- focal-rax- ord-0023393373 nova-compute[ 91725]: ERROR oslo_messaging. rpc.server File "/opt/stack/ nova/nova/ virt/libvirt/ driver. py", line 1525, in _cleanup focal-rax- ord-0023393373 nova-compute[ 91725]: ERROR oslo_messaging. rpc.server self._cleanup_ lvm(instance, block_device_info) focal-rax- ord-0023393373 nova-compute[ 91725]: ERROR oslo_messaging. rpc.server File "/opt/stack/ nova/nova/ virt/libvirt/ driver. py", line 1615, in _cleanup_lvm focal-rax- ord-0023393373 nova-compute[ 91725]: ERROR oslo_messaging. rpc.server disks = self._lvm_ disks(instance) focal-rax- ord-0023393373 nova-compute[ 91725]: ERROR oslo_messaging. rpc.server File "/opt/stack/ nova/nova/ virt/libvirt/ driver. py", line 1633, in _lvm_disks focal-rax- ord-0023393373 nova-compute[ 91725]: ERROR oslo_messaging. rpc.server logical_volumes = lvm.list_ volumes( vg) focal-rax- ord-0023393373 nova-compute[ 91725]: ERROR oslo_messaging. rpc.server File "/opt/stack/ nova/nova/ virt/libvirt/ storage/ lvm.py" , line 111, in list_volumes focal-rax- ord-0023393373 nova-compute[ 91725]: ERROR oslo_messaging. rpc.server out, err = nova.privsep. fs.lvlist( vg) focal-rax- ord-0023393373 nova-compute[ 91725]: ERROR oslo_messaging. rpc.server File "/usr/local/ lib/python3. 8/dist- packages/ oslo_privsep/ priv_context. py", line 247, in _wrap focal-rax- ord-0023393373 nova-compute[ 91725]: ERROR oslo_messaging. rpc.server return self.channel. remote_ call(name, args, kwargs) focal-rax- ord-0023393373 nova-compute[ 91725]: ERROR oslo_messaging. rpc.server File "/usr/local/ lib/python3. 8/dist- packages/ oslo_privsep/ daemon. py", line 224, in remote_call focal-rax- ord-0023393373 nova-compute[ 91725]: ERROR oslo_messaging. rpc.server raise exc_type( *result[ 2]) focal-rax- ord-0023393373 nova-compute[ 91725]: ERROR oslo_messaging. rpc.server oslo_concurrenc y.processutils. ProcessExecutio nError: Unexpected error while running command. focal-rax- ord-0023393373 nova-compute[ 91725]: ERROR oslo_messaging. rpc.server Command: lvs --noheadings -o lv_name /dev/stack- volumes- default focal-rax- ord-0023393373 nova-compute[ 91725]: ERROR oslo_messaging. rpc.server Exit code: -11 focal-rax- ord-0023393373 nova-compute[ 91725]: ERROR oslo_messaging. rpc.server Stdout: '' focal-rax- ord-0023393373 nova-compute[ 91725]: ERROR oslo_messaging. rpc.server Stderr: ' WARNING: Failed to get udev device handler for device /dev/sda1.\n /dev/sda15: stat failed: No such file or directory\n Path /dev/sda15 no longer valid for device(8,15)\n /dev/sda15: stat failed: No such file or directory\n Path /dev/sda15 no longer valid for device(8,15)\n Device open /dev/sda 8:0 failed errno 2\n Device open /dev/sda 8:0 failed errno 2\n Device open /dev/sda1 8:1 failed errno 2\n Device open /dev/sda1 8:1 failed errno 2\n WARNING: Scan ignoring device 8:0 with no paths.\n WARNING: Scan ignoring device 8:1 with no paths.\n'
Mar 09 16:45:48.496630 ubuntu-
Mar 09 16:45:48.496630 ubuntu-
Mar 09 16:45:48.496630 ubuntu-
Mar 09 16:45:48.496630 ubuntu-
Mar 09 16:45:48.496630 ubuntu-
Mar 09 16:45:48.496630 ubuntu-
Mar 09 16:45:48.496630 ubuntu-
Mar 09 16:45:48.496630 ubuntu-
Mar 09 16:45:48.496630 ubuntu-
Mar 09 16:45:48.496630 ubuntu-
Mar 09 16:45:48.496630 ubuntu-
Mar 09 16:45:48.496630 ubuntu-
Mar 09 16:45:48.496630 ubuntu-
Mar 09 16:45:48.496630 ubuntu-
Mar 09 16:45:48.496630 ubuntu-
Mar 09 16:45:48.496630 ubuntu-