[regression] lingering pvscan during boot

[Simon Déziel]

Since lvm2 was updated to 2.02.176-4.1ubuntu3.18.04.2 on Bionic (LP: #1854981), we notice that some of our machines have lingering pvscan processes apparently running from the initramfs's root that persist/never finish/exit.

On the affected servers, this is visible as there are 2 instances of systemd-udevd, one in the init.scope and another in system.slice:

$ systemd-cgls | cat
Control group /:
-.slice
├─user.slice
│ ├─user-501.slice
│ │ ├─session-7.scope
│ │ │ ├─12324 sshd: foo [priv]
│ │ │ ├─12353 sshd: foo@pts/2
│ │ │ ├─12354 script --quiet --return --command /bin/bash -l /dev/null
│ │ │ ├─12356 /bin/bash -l
│ │ │ ├─12375 sudo -i
│ │ │ └─12385 -bash
│ │ └─[e-mail]
│ │ └─init.scope
│ │ ├─12326 /lib/systemd/systemd --user
│ │ └─12327 (sd-pam)
│ └─user-500.slice
│ ├─[e-mail]
│ │ └─init.scope
│ │ ├─12185 /lib/systemd/systemd --user
│ │ └─12186 (sd-pam)
│ └─session-3.scope
│ ├─12170 sshd: sdeziel [priv]
│ ├─12254 sshd: sdeziel@pts/0
│ ├─12255 script --quiet --return --command /bin/bash -l /dev/null
│ ├─12257 /bin/bash -l
│ ├─14409 systemd-cgls
│ └─14410 cat
├─init.scope
│ ├─ 1 /sbin/init kaslr nosplash
│ ├─1451 /lib/systemd/systemd-udevd --daemon --resolve-names=never
│ ├─1466 /sbin/lvm pvscan --cache --activate ay --major 8 --minor 3
│ ├─1470 /sbin/lvm pvscan --cache --activate ay --major 8 --minor 2
│ ├─1481 /sbin/lvm pvscan --cache --activate ay --major 253 --minor 1
│ └─1551 /sbin/lvm pvscan --cache --activate ay --major 253 --minor 1
└─system.slice
  ├─irqbalance.service
  │ └─2796 /usr/sbin/irqbalance --foreground
  ├─systemd-networkd.service
  │ └─2833 /lib/systemd/systemd-networkd
  ├─systemd-udevd.service
  │ └─1627 /lib/systemd/systemd-udevd
  ├─cron.service
  │ ├─ 2786 /usr/sbin/cron -f
  │ └─12308 /usr/bin/python /usr/sbin/ganeti-noded -b 172.20.30.212
  ├─system-serial\x2dgetty.slice
  │ └─<email address hidden>
  │ └─12112 /sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS0 vt220
  ├─systemd-journald.service
  │ └─1601 /lib/systemd/systemd-journald
  ├─ssh.service
  │ └─12115 /usr/sbin/sshd -D
  ├─rsyslog.service
  │ └─2781 /usr/sbin/rsyslogd -n
  ├─nagios-nrpe-server.service
  │ └─12108 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f
  ├─lvm2-lvmetad.service
  │ └─1630 /sbin/lvmetad -f
  ├─systemd-resolved.service
  │ └─3280 /lib/systemd/systemd-resolved
  ├─dbus.service
  │ └─2762 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidf...
  ├─systemd-timesyncd.service
  │ └─2491 /lib/systemd/systemd-timesyncd
  ├─system-getty.slice
  │ └─<email address hidden>
  │ └─12113 /sbin/agetty -o -p -- \u --noclear tty1 linux
  ├─systemd-logind.service
  │ └─2797 /lib/systemd/systemd-logind
  └─ganeti.service
    ├─ 3892 /usr/sbin/ganeti-confd
    ├─12173 /usr/sbin/ganeti-kvmd
    └─12276 /usr/sbin/ganeti-mond

The systemd-udevd in init.scope and the many /sbin/lvm pvscan it launched during boot are left there forever.

This trips our monitoring for binaries using deleted binaries/libs and is also visible like that:

# pgrep systemd-udevd
1451
1627

# ls -l /proc/1451/exe
lrwxrwxrwx 1 root root 0 Feb 19 10:35 /proc/1451/exe -> '/lib/systemd/systemd-udevd (deleted)'

# ls -l /proc/1627/exe
lrwxrwxrwx 1 root root 0 Feb 19 10:35 /proc/1627/exe -> /lib/systemd/systemd-udevd

This can be compared with another server *NOT AFFECTED*, where the init.scope is cleaner (/sbin/init only) and a single systemd-udevd process:

# systemd-cgls | cat
Control group /:
-.slice
├─user.slice
│ └─user-0.slice
│ ├─session-313.scope
│ │ ├─90332 sshd: root@pts/0
│ │ ├─90406 script --quiet --return --command /bin/bash -l /dev/null
│ │ ├─90410 /bin/bash -l
│ │ ├─90436 systemd-cgls
│ │ └─90437 cat
│ └─[e-mail]
│ └─init.scope
│ ├─90334 /lib/systemd/systemd --user
│ └─90335 (sd-pam)
├─init.scope
│ └─1 /sbin/init kaslr nosplash
└─system.slice
...
  ├─systemd-udevd.service
  │ └─2622 /lib/systemd/systemd-udevd
...
  ├─lvm2-lvmetad.service
  │ └─2621 /sbin/lvmetad -f
...

Additional information:

# lsb_release -rd
Description: Ubuntu 18.04.4 LTS
Release: 18.04
# apt-cache policy lvm2 systemd
lvm2:
  Installed: 2.02.176-4.1ubuntu3.18.04.2
  Candidate: 2.02.176-4.1ubuntu3.18.04.2
  Version table:
 *** 2.02.176-4.1ubuntu3.18.04.2 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     2.02.176-4.1ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
systemd:
  Installed: 237-3ubuntu10.39
  Candidate: 237-3ubuntu10.39
  Version table:
 *** 237-3ubuntu10.39 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     237-3ubuntu10.38 500
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     237-3ubuntu10 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages

[Steve Langasek (vorlon)]

Eric, this is reported as a regression in an SRU you did, can you please follow through?

[Steve Dodd (anarchetic)]

Do you also see slow shutdowns? One of my servers which has other problems with this patch (bug #1870783) has been seen to get stuck shutting down / rebooting showing a message about (I think) lvmetad (hard to tell due to very small server console truncating message) .. systemd eventually times it out (after, a bit randomly, 105 secs)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Simon Déziel (sdeziel)]

@narchetic, I did not notice any slowdown during shutdown but those servers are headless and I didn't time their reboot.