vgcfgbackup stuck in kernel, causing lvm and udevd block forever

[Aleksandr Kichigin]

Hello

I am using lvm snapshots over sofware raid with 4 devices for rsync backups.

Recently I found out that it is stuck and any operaton with lvm is stuck as well.
This also causes 1000+ processes of systemd-udevd to be in interpretable sleep and cannot be killed.

My backup script looks like this:

/sbin/vgcfgbackup -qq main
/sbin/lvcreate -qq --size 20G --snapshot --name backup /dev/mapper/main-root
mount /dev/mapper/main-backup /mnt/root_snapshot
*rsync stuff*
umount /mnt/root_snapshot
/sbin/lvremove -yqq /dev/mapper/main-backup

Launching as not root `pvs --unbuffered --readonly -t -vvvv` is stuck at:
#filters/filter-persistent.c:51 Wiping cache of LVM-capable devices

Launching as not root `lvs -vvvv --unbuffered --readonly -vvvv` us stuck at:
#toollib.c:3736 Get list of VGs on system

I have a lot of processes stucked (total number ~1000):

root 756 0.0 0.0 56400 8428 ? D фев26 0:00 /sbin/vgcfgbackup -qq main
root 21723 0.0 0.0 36940 1860 ? D мар13 0:00 /lib/systemd/systemd-udevd
root 21724 0.0 0.0 36940 1808 ? D мар13 0:00 /lib/systemd/systemd-udevd
root 21728 0.0 0.0 36940 1808 ? D мар13 0:00 /lib/systemd/systemd-udevd
root 14541 0.0 0.0 7780 2524 ? D 06:25 0:00 /sbin/mdadm --monitor --scan --oneshot
root 8792 0.0 0.0 56548 8308 ? D 05:59 0:00 lvs
root 11343 0.0 0.0 119424 7632 ? S 06:10 0:00 sudo lvs -vd
admin 12112 0.0 0.0 56596 8368 ? D 06:14 0:00 lvs -vvvv -dddddd --readonly -t --unbuffered

In dmesg I have following logs:

[16983388.190818] general protection fault: 0000 [#41] SMP PTI
[16987988.464353] general protection fault: 0000 [#42] SMP PTI

In system journal:

Mar 14 06:47:35 - systemd[1]: systemd-udevd.service: Found left-over process 17886 (systemd-udevd) in control group while starting unit. Ignoring.
Mar 14 06:47:35 - systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Mar 14 06:47:35- systemd[1]: systemd-udevd.service: Found left-over process 17887 (systemd-udevd) in control group while starting unit. Ignoring.
Mar 14 06:50:28 - systemd-udevd[20282]: seq 1116768 '/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda/sda1' is taking a long time
Mar 14 06:50:28 - systemd-udevd[20282]: seq 1116780 '/devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:1/1:0:1:0/block/sdc/sdc1' is taking a long time

lvmdump hangs at `Gathering /dev listening`, but I was able to terminate it.

mdstat:
md0 : active raid10 sdb1[1] sdc1[2] sda1[0] sdd1[3]
      1953259520 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
      bitmap: 1/15 pages [4KB], 65536KB chunk

OS: Ubuntu 18.04.1 LTS
Kernel: 4.15.0-29-generic
lvm2: 2.02.176-4.1ubuntu3

[Aleksandr Kichigin (mrkich)]

ps axu | awk '$8 ~ /^D/'

[Aleksandr Kichigin (mrkich)]

It seems that simple

`ls -la /dev/disk/by-path`

freezes `ls` process.

However, /dev listing normally, as well as /dev/disk/{by-id,by-label,etc} (except `by-path`).