update-motd-fsck-at-reboot stalls ssh logins for minutes if there is a slow/busy USB disk attached

[Peter Maydell]

I found recently that some ssh logins to my machine could stall for minutes, and tracked this down to the update-motd-fsck-at-reboot hook.

Here's the relevant bit of 'ps wafux' output:

root 3659433 0.0 0.0 56920 14836 ? Ss 14:29 0:00 \_ sshd: petmay01 [priv]
root 3659435 0.0 0.0 2608 592 ? S 14:29 0:00 \_ sh -c /usr/bin/env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin run-parts --lsbsysinit /etc/update-motd.d > /run/motd.dynamic.new
root 3659437 0.0 0.0 2496 1400 ? S 14:29 0:00 \_ run-parts --lsbsysinit /etc/update-motd.d
root 3659488 0.0 0.0 2608 1824 ? S 14:29 0:00 \_ /bin/sh /usr/lib/update-notifier/update-motd-fsck-at-reboot
root 3659535 0.0 0.0 3480 896 ? D 14:29 0:00 \_ dumpe2fs -h /dev/mapper/backup2

It's currently 14:40 and the 'dumpe2fs -h' has been stalled in the D state for 10 minutes; the ssh login is blocked by this. (In fact, as I was writing up this bug report the dumpe2fs finished at 14:42 and the ssh login completed successfully.)

The 'backup2' disk, as the name suggests, is a backup disk, and it's an external USB disk. The backups are currently running, so the disk is currently busy doing a lot of I/O, which is why the dumpe2fs command blocks for ages.

I think that anything like this that gets run during logins should not be performing operations which could potentially take a long time to complete. I don't know if there's some mechanism for identifying "fast disks" and only looking at those.

The workaround is to add an "exit 0" to "/etc/update-motd.d/98-fsck-at-reboot", disabling the hook.

This is with Ubuntu 20.04, and update-notifier-common version 3.192.30.11.