Timed out Availability of block devices service

[Andras Tim]

On a fresh Ubuntu 18.04 (64 bit) Server install, the blk-availability.service always killed by timeout at shutdown.

I think the dependency of this service is bad, but I don't know what would be the good shutdown order for this.

An almost same config has been worked on Xenial...

The disk sub-system is
  * NVMe SSD
    * EFI partition
    * boot partition
    * LVM phisical volume
        * root logical volume
        * swap logical volume
        * data logical volume
  * 2 pcs. HDDs in RAID1 (but the md0 is not used yet)

######
# Versions
#

# Ubuntu
Description: Ubuntu 18.04 LTS
Release: 18.04

# Kernel
Linux server 4.15.0-24-generic #26-Ubuntu SMP Wed Jun 13 08:44:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

# LVM
lvm2 2.02.176-4.1ubuntu3

######
# Journal
#
systemd[1]: Stopped Load/Save Random Seed.
systemd[1]: Stopped Update UTMP about System Boot/Shutdown.
systemd[1]: Stopped Network Time Synchronization.
kernel: usb 1-13: USB disconnect, device number 2
systemd[1]: blk-availability.service: Stopping timed out. Terminating.
systemd[1]: blk-availability.service: Control process exited, code=killed status=15
blkdeactivate[2447]: [MD]: deactivating raid1 device md0... resync action in progress...
systemd[1]: blk-availability.service: Failed with result 'timeout'.
systemd[1]: Stopped Availability of block devices.
systemd[1]: Stopping iSCSI initiator daemon (iscsid)...
iscsid[951]: iscsid shutting down.
systemd[1]: Stopped iSCSI initiator daemon (iscsid).
systemd[1]: Stopped target Network is Online.
systemd[1]: Stopped Wait for Network to be Configured.
systemd[1]: Stopped target Network.
systemd[1]: Stopping Network Name Resolution...
systemd[1]: Stopped Network Name Resolution.
systemd[1]: Stopping Network Service...
systemd[1]: Stopped Create Volatile Files and Directories.
systemd[1]: Stopped target Local File Systems.
systemd[1]: Unmounting /boot/efi...
systemd[1]: Unmounting /run/user/1000...
systemd[1]: Unmounted /run/user/1000.
systemd[1]: Stopped target Swap.
systemd[1]: Deactivating swap /dev/disk/by-uuid/8a3d99d0-683e-xxxx-xxxx-9765162cc070...
systemd[1]: Unmounted /boot/efi.
systemd[1]: Unmounting /boot...
systemd[1]: Stopped File System Check on /dev/disk/by-uuid/AB9E-5A2C.
systemd[1]: Deactivated swap /dev/xxx_server_ssd/swap.
systemd[1]: Deactivated swap /dev/disk/by-uuid/8a3d99d0-683e-xxxx-xxxx-9765162cc070.
systemd[1]: Deactivated swap /dev/disk/by-id/dm-uuid-LVM-eatHKVdbDuw5L5RXXXXXXX2yJn7QosUWvV2IIApCOG73Bc2CbrUabBC6L3dh0N2D.
systemd[1]: Deactivated swap /dev/disk/by-id/dm-name-xxx_server_ssd-swap.
systemd[1]: Deactivated swap /dev/dm-1.
systemd[1]: Stopped Network Service.
systemd[1]: Deactivated swap /dev/mapper/xxx_server_ssd-swap.
systemd[1]: Unmounted /boot.
systemd[1]: Stopped File System Check on /dev/disk/by-uuid/d7cc70cc-0f33-xxxx-xxxx-cc21d3ed1232.
systemd[1]: Removed slice system-systemd\x2dfsck.slice.
systemd[1]: Stopped target Local File Systems (Pre).
systemd[1]: Stopped Remount Root and Kernel File Systems.
systemd[1]: Stopped Create Static Device Nodes in /dev.
systemd[1]: Stopping Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
systemd[1]: Stopped target Network (Pre).
systemd[1]: Stopped Apply Kernel Variables.
systemd[1]: Stopped Load Kernel Modules.
systemd[1]: Reached target Shutdown.
systemd[1]: Reached target Unmount All Filesystems.
systemd[1]: Reached target Final Step.
systemd[1]: Starting Power-Off...

######
# systemctl cat blk-availability.service
#
# /lib/systemd/system/blk-availability.service
[Unit]
Description=Availability of block devices
After=lvm2-activation.service lvm2-lvmetad.service iscsi-shutdown.service iscsi.service iscsid.service fcoe.service
DefaultDependencies=no
Conflicts=shutdown.target

[Service]
Type=oneshot
ExecStart=/bin/true
ExecStop=/sbin/blkdeactivate -u -l wholevg -m disablequeueing -r wait
RemainAfterExit=yes

[Install]
WantedBy=sysinit.target

######
# Other
#
I tried to ask help on https://askubuntu.com/questions/1051856/timed-out-availability-of-block-devices, but I haven't got any response.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: lvm2 2.02.176-4.1ubuntu3
ProcVersionSignature: Ubuntu 4.15.0-24.26-generic 4.15.18
Uname: Linux 4.15.0-24-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
Date: Thu Jul 5 09:24:01 2018
InstallationDate: Installed on 2018-07-03 (1 days ago)
InstallationMedia: Ubuntu-Server 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
SourcePackage: lvm2
UpgradeStatus: No upgrade log present (probably fresh install)

[Andras Tim (tia)]

I found the commit where introduced this blkdeactivate: https://sourceware.org/git/?p=lvm2.git;a=commit;h=c698ee14bbb1310cf2383c8977d14a8e29139f8c

> Traverses the tree of block devices and tries to deactivate them.
> Currently, it supports device-mapper-based devices together with LVM.
> See man/blkdeactivate.8 for more info.
>
> It is targeted for use during shutdown to properly deactivate the
> whole block device stack - systemd and init scripts are provided as
> well. However, it might be used directly on command line too.

In this case, this stuff runs too early, as I think.

[Andras Tim (tia)]

FYI: This problem can happen when the md0 device still synchronizing. Now, when the md0 is in sync, the service stopped as well:

# journalctl -b -1 -u blk-availability.service

systemd[1]: Starting Availability of block devices...
systemd[1]: Started Availability of block devices.
systemd[1]: Stopping Availability of block devices...
blkdeactivate[1440]: Deactivating block devices:
blkdeactivate[1440]: [UMOUNT]: unmounting paint_server_ssd-system (dm-2) mounted on /var/datastore/system... done
blkdeactivate[1440]: [SKIP]: unmount of paint_server_ssd-swap (dm-1) mounted on [SWAP]
blkdeactivate[1440]: [SKIP]: unmount of paint_server_ssd-root (dm-0) mounted on /
blkdeactivate[1440]: [MD]: deactivating raid1 device md0... done
systemd[1]: Stopped Availability of block devices.

[Andras Tim (tia)]

I think it can be fixed in 2 ways:

* blkdeactivate handles properly the syncing software-raid arrays
* re-order this service, to be after the "standard" software-raid deinit

[Benj FitzPatrick (benjfitz)]

I also ran into this problem with Ubuntu Server 18.04 and a RAID 1 array (shutdown would wait 1:30 for the array to be unmounted). Manually unmounting the array before shutdown removed the problem.

[Dimitri John Ledkov (xnox)]

mdadm package ships mdadm-shutdown.service which should aid with unmounting raid arrays on shutdown.

However, for it to work it requires an installation of package dracut-core

Could you please try

$ apt install dracut-core

and let me know if that resolves long shut down times for you?

In the future a similar functionality will be built in, but it is awaiting archive admin review in the NEW queue.

[Dimitri John Ledkov (xnox)]

$ sudo apt install dracut-core

[Steve Langasek (vorlon)]

finalrd accepted from NEW queue. unassigning self.

[MM (bmn.one)]

@xnox
sudo apt install dracut-core did not work for me.

In my case:
raid md1 was unmounted but raid md0 was not.
raid md0 was mounted to /

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Simen (simentho)]

I have the same problem on Ubuntu Server 22.04 Minimal