Something is holding a lock on unmounted partition

[Josef Wolf]

Following instructions from https://discourse.maas.io/t/deploying-servers-with-full-disk-encryption-luks2/3286 to get full disk encryption with ubuntu-22.04-server

   # mount -o remount,ro /boot
   # install -m0600 /dev/null /tmp/boot.tar
   # tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
   # umount -v /boot/efi
   umount: /boot/efi unmounted
   # umount -v /boot
   umount: /boot unmounted
   # cryptsetup -y luksFormat --type luks1 /dev/nvme0n1p3
   WARNING: Device /dev/nvme0n1p3 already contains a 'ext4' superblock signature.

   WARNING!
   ========
   This will overwrite data on /dev/nvme0n1p3 irrevocably.

   Are you sure? (Type 'yes' in capital letters); YES
   Enter Passphrase for /dev/nvme0n1p3:
   Verify Passphrase:
   Device /dev/nvme0n1p3 is in use. Cannot proceed with format operation.

Uh, not what I expected. So i start to investigate:

   # grep nvme0n1p3 /proc/mounts
   # lsof | grep nvme0n1p3
   # lsof nvme0n1p3
   # fuser nvme0n1p3

So:
- umount succeeded
- /proc/mounts, lsof, fuser don's show any sign that the partition is in use

BUT:

    strace cryptsetup luksFormat --type luks1 /dev/nvme0n1p3 2>&1 | egrep '(nvme0n1p3|close)'

shows

    openat(AT_FDCWD, "/dev/nvme0n1p3", O_RDONLY|ODIRECT) = 3
    close(3)
    openat(AT_FDCWD, "/dev/nvme0n1p3", O_RDONLY|O_NONBLOCK|OCLOEXEC) = 3
    close(3)
    openat(AT_FDCWD, "/dev/nvme0n1p3", O_RDWR|OEXCL) = -1 EBUSY (Device or resource busy)

So there must be something holding a hand on the partition. But there's no trace of what it might be

=======================================================

Another try (after fresh boot):

   # mount -o remount,ro /boot
   # install -m0600 /dev/null /tmp/boot.tar
   # tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
   # umount /boot/efi
   # umount /boot
   # mount /boot
   mount: /boot: /dev/nvme0n1p3 already mounted or mount point busy

and again: no sign that the partition is in use:

   # grep nvme0n1p3 /proc/mounts
   # lsof | grep nvme0n1p3
   # lsof nvme0n1p3
   # fuser nvme0n1p3

=======================================================

So another try (reboot again):

   # mount -o remount,ro /boot
   # install -m0600 /dev/null /tmp/boot.tar
   # tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
   # umount /boot/efi
   # mount -o remount,rw /boot # <<<<------- notice this one!
   # umount /boot
   # mount /boot # <<<<------- Now it succeeds!

Now do the same, but without the "mount -oremount,rw /boot"

   # mount -o remount,ro /boot
   # install -m0600 /dev/null /tmp/boot.tar
   # tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
   # umount /boot
   # mount /boot
   mount: /boot: /dev/nvme0n1p3 already mounted or mount point busy

Oooops! Now system is messed up again. No way to "clean up the mess.

Getting closer? Partition originally was mounted read-write. umount while remounted read-only seems to mess up things and mount looses track of was is going on..

Thus, another try:

=====================================================

Again freshly booted. Omit the "mount -oremount,ro /boot" this time:

   # install -m0600 /dev/null /tmp/boot.tar
   # tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
   # umount -v /boot/efi
   umount: /boot/efi unmounted
   # umount -v /boot
   umount: /boot unmounted
   # cryptsetup -y luksFormat --type luks1 /dev/nvme0n1p3
   WARNING: Device /dev/nvme0n1p3 already contains a 'ext4' superblock signature.

   WARNING!
   ========
   This will overwrite data on /dev/nvme0n1p3 irrevocably.

   Are you sure? (Type 'yes' in capital letters); YES
   Enter Passphrase for /dev/nvme0n1p3:
   Verify Passphrase:
   Device /dev/nvme0n1p3 is in use. Cannot proceed with format operation.

failed again, but

   # mount /boot # <<<<------- Now it succeeds!
   # umount /boot # <<<<------- Now it succeeds!
   # mount /boot # <<<<------- Now it succeeds!

So, how can I track this one down further?