We're also seeing this fail in nova, since that has calls (via privsep) to 'lvs'. https://a59f59e90e7507f14599-2cecd17ff9ea5f4d02c12f6d5fc5aedd.ssl.cf1.rackcdn.com/776681/7/check/nova-lvm/a19ff22/controller/logs/screen-n-cpu.txt Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [None req-4c1200aa-cc35-4249-a6ba-538d7a763b07 tempest-SecurityGroupsTestJSON-739885543 tempest-SecurityGroupsTestJSON-739885543-project] [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] Setting instance vm_state to ERROR: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: Command: lvs --noheadings -o lv_name /dev/stack-volumes-default Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: Exit code: -11 Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: Stdout: '' Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: Stderr: ' WARNING: Failed to get udev device handler for device /dev/sda1.\n /dev/sda15: stat failed: No such file or directory\n Path /dev/sda15 no longer valid for device(8,15)\n /dev/sda15: stat failed: No such file or directory\n Path /dev/sda15 no longer valid for device(8,15)\n Device open /dev/sda 8:0 failed errno 2\n Device open /dev/sda 8:0 failed errno 2\n Device open /dev/sda1 8:1 failed errno 2\n Device open /dev/sda1 8:1 failed errno 2\n WARNING: Scan ignoring device 8:0 with no paths.\n WARNING: Scan ignoring device 8:1 with no paths.\n' Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] Traceback (most recent call last): Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] File "/opt/stack/nova/nova/compute/manager.py", line 3040, in do_terminate_instance Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] self._delete_instance(context, instance, bdms) Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] File "/opt/stack/nova/nova/compute/manager.py", line 2975, in _delete_instance Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] self._shutdown_instance(context, instance, bdms) Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] File "/opt/stack/nova/nova/compute/manager.py", line 2867, in _shutdown_instance Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] self._try_deallocate_network(context, instance, Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] self.force_reraise() Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] raise self.value Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] File "/opt/stack/nova/nova/compute/manager.py", line 2854, in _shutdown_instance Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] self.driver.destroy(context, instance, network_info, Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1389, in destroy Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] self.cleanup(context, instance, network_info, block_device_info, Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1457, in cleanup Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] return self._cleanup( Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1525, in _cleanup Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] self._cleanup_lvm(instance, block_device_info) Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1615, in _cleanup_lvm Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] disks = self._lvm_disks(instance) Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1633, in _lvm_disks Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] logical_volumes = lvm.list_volumes(vg) Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] File "/opt/stack/nova/nova/virt/libvirt/storage/lvm.py", line 111, in list_volumes Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] out, err = nova.privsep.fs.lvlist(vg) Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 247, in _wrap Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] return self.channel.remote_call(name, args, kwargs) Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 224, in remote_call Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] raise exc_type(*result[2]) Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] Command: lvs --noheadings -o lv_name /dev/stack-volumes-default Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] Exit code: -11 Mar 09 16:45:48.078761 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] Stdout: '' Mar 09 16:45:48.082199 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] Stderr: ' WARNING: Failed to get udev device handler for device /dev/sda1.\n /dev/sda15: stat failed: No such file or directory\n Path /dev/sda15 no longer valid for device(8,15)\n /dev/sda15: stat failed: No such file or directory\n Path /dev/sda15 no longer valid for device(8,15)\n Device open /dev/sda 8:0 failed errno 2\n Device open /dev/sda 8:0 failed errno 2\n Device open /dev/sda1 8:1 failed errno 2\n Device open /dev/sda1 8:1 failed errno 2\n WARNING: Scan ignoring device 8:0 with no paths.\n WARNING: Scan ignoring device 8:1 with no paths.\n' Mar 09 16:45:48.082199 ubuntu-focal-rax-ord-0023393373 nova-compute[91725]: ERROR nova.compute.manager [instance: 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43] Same apparent segfault root cause: https://a59f59e90e7507f14599-2cecd17ff9ea5f4d02c12f6d5fc5aedd.ssl.cf1.rackcdn.com/776681/7/check/nova-lvm/a19ff22/controller/logs/syslog.txt Mar 09 16:45:45 ubuntu-focal-rax-ord-0023393373 kernel: sda: sda1 sda15 Mar 09 16:45:45 ubuntu-focal-rax-ord-0023393373 kernel: sda: sda1 sda15 Mar 09 16:45:46 ubuntu-focal-rax-ord-0023393373 kernel: blk_update_request: I/O error, dev sda, sector 2097088 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Mar 09 16:45:46 ubuntu-focal-rax-ord-0023393373 kernel: blk_update_request: I/O error, dev sda, sector 2097088 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Mar 09 16:45:46 ubuntu-focal-rax-ord-0023393373 kernel: Buffer I/O error on dev sda, logical block 262136, async page read Mar 09 16:45:46 ubuntu-focal-rax-ord-0023393373 kernel: sd 2:0:0:1: [sda] Synchronizing SCSI cache Mar 09 16:45:46 ubuntu-focal-rax-ord-0023393373 kernel: device tapeba84e63-49 entered promiscuous mode Mar 09 16:45:46 ubuntu-focal-rax-ord-0023393373 kernel: lvs[111426]: segfault at 800 ip 00007f6a0554d860 sp 00007ffff0079fa8 error 4 in libc-2.31.so[7f6a053f9000+178000]