snap preseed failing to umount /tmp/snapd-preseed

[Paul Mars]

When building a classic image, the preseed_image steps is failing with the following error:

[8] preseed_image
ubuntu classic preseeding
starting to preseed root: /home/paul/scratch/debootstrap-2/chroot
using snapd binary: /tmp/snapd-preseed/usr/lib/snapd/snapd (2.61.2)
2024/03/21 09:55:31.100621 main.go:53: running for preseeding
2024/03/21 09:55:31.160617 overlord.go:271: Acquiring state lock file
2024/03/21 09:55:31.160652 overlord.go:276: Acquired state lock file
2024/03/21 09:55:31.215560 daemon.go:247: started snapd/2.61.2 (series 16; classic) ubuntu/24.04 (amd64) linux/6.5.0-27-generic.
2024/03/21 09:55:31.234105 daemon.go:340: adjusting startup timeout by 30s (pessimistic estimate of 30s plus 5s per snap)
2024/03/21 09:55:31.258774 backends.go:58: AppArmor status: apparmor is enabled and all features are available (using snapd provided apparmor_parser)
2024/03/21 09:55:31.458153 helpers.go:146: error trying to compare the snap system key: system-key missing on disk
2024/03/21 09:55:32.573216 daemon.go:425: stopping snapd as requested
2024/03/21 09:55:32.618884 daemon.go:519: gracefully waiting for running hooks
2024/03/21 09:55:32.618921 daemon.go:521: done waiting for running hooks
2024/03/21 09:55:35.574257 overlord.go:515: Released state lock file
unmounting: /tmp/snapd-preseed
exit status 32remove /tmp/snapd-preseed: device or resource busy
umount: /home/paul/scratch/debootstrap-2/chroot/sys/fs/cgroup: target is busy.
umount: /home/paul/scratch/debootstrap-2/chroot/sys/kernel/security: target is busy.
umount: /home/paul/scratch/debootstrap-2/chroot/proc: target is busy.
umount: /home/paul/scratch/debootstrap-2/chroot/dev/pts: target is busy.
umount: /home/paul/scratch/debootstrap-2/chroot/dev/pts: target is busy.
duration: 5.286563398s
Error: teardown failed: teardown command "/usr/bin/umount --recursive debootstrap-2/chroot/sys/fs/cgroup" failed. Output:
umount: /home/paul/scratch/debootstrap-2/chroot/sys/fs/cgroup: target is busy.

teardown command "/usr/bin/umount --recursive debootstrap-2/chroot/sys/kernel/security" failed. Output:
umount: /home/paul/scratch/debootstrap-2/chroot/sys/kernel/security: target is busy.

teardown command "/usr/bin/umount --recursive debootstrap-2/chroot/proc" failed. Output:
umount: /home/paul/scratch/debootstrap-2/chroot/proc: target is busy.

teardown command "/usr/bin/umount --recursive debootstrap-2/chroot/dev/pts" failed. Output:
umount: /home/paul/scratch/debootstrap-2/chroot/dev/pts: target is busy.

teardown command "/usr/bin/umount --recursive debootstrap-2/chroot/dev" failed. Output:
umount: /home/paul/scratch/debootstrap-2/chroot/dev/pts: target is busy.

The call to snap-preseed is failing because the umounting of /tmp/snapd-preseed is failing.

sudo mount -l | grep "debootstrap-2"

devtmpfs-build on /home/paul/scratch/debootstrap-2/chroot/dev type devtmpfs (rw,relatime,size=15723176k,nr_inodes=3930794,mode=755,inode64)
devpts-build on /home/paul/scratch/debootstrap-2/chroot/dev/pts type devpts (rw,nosuid,nodev,relatime,mode=600,ptmxmode=000)
proc-build on /home/paul/scratch/debootstrap-2/chroot/proc type proc (rw,relatime)
none on /home/paul/scratch/debootstrap-2/chroot/sys/kernel/security type securityfs (rw,relatime)
none on /home/paul/scratch/debootstrap-2/chroot/sys/fs/cgroup type cgroup2 (rw,relatime)
/home/paul/scratch/debootstrap-2/chroot/var/lib/snapd/seed/snaps/snapd_21184.snap on /home/paul/scratch/debootstrap-2/chroot/tmp/snapd-preseed type squashfs (ro,relatime,errors=continue,threads=single)

I suspect a race condition because once the build has failed, I can unmount these mountpoints.

The preseed_image state is executed after prepare_image, which is also calling /usr/lib/snapd/snap-preseed (but with the --reset flag).

To reproduce: ubuntu-image --verbose --debug classic --workdir debootstrap-2 image_definition.yaml

Image definition used:

name: ubuntu-server-amd64
display-name: Ubuntu Server amd64
revision: 1
architecture: amd64
series: noble
class: preinstalled
kernel: linux-image-generic
gadget:
  url: "https://git.launchpad.net/~canonical-foundations/snap-pc/+git/github-mirror-amd64"
  branch: classic
  type: "git"
rootfs:
  components:
    - main
    - restricted
  seed:
    urls:
      - "git://git.launchpad.net/~ubuntu-core-dev/ubuntu-seeds/+git/"
    branch: noble
    names:
      - server
      - minimal
      - standard
      - cloud-image
customization:
  cloud-init:
    user-data: |
      #cloud-config
      chpasswd:
        expire: true
        users:
          - name: ubuntu
            password: ubuntu
            type: text
    meta-data: |
      dsmode: local
      instance_id: ubuntu-server
  extra-snaps:
    - name: snapd
artifacts:
  img:
    -
      name: ubuntu-24.04-preinstalled-server-amd64.img
  manifest:
    name: "filesystem-manifest.txt"
  filelist:
    name: "filesystem-filelist.txt"

[Paul Mars (upils)]

I was previously able to reproduce reliably this behavior but I am not anymore, which is at the same time a good and a bad news. Meanwhile, I now it a similar issue, but at the update_bootloader stage.

[18] update_bootloader
Adding 'local diversion of /etc/grub.d/30_os-prober to /etc/grub.d/30_os-prober.dpkg-divert'
Sourcing file `/etc/default/grub'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.5.0-9-generic
Found initrd image: /boot/initrd.img-6.5.0-9-generic
Adding boot menu entry for UEFI Firmware Settings ...
done
Removing 'local diversion of /etc/grub.d/30_os-prober to /etc/grub.d/30_os-prober.dpkg-divert'
umount: /home/paul/scratch/debootstrap-2/scratch/loopback/proc: target is busy.
umount: /home/paul/scratch/debootstrap-2/scratch/loopback: target is busy.
duration: 1.053857891s
Error: teardown failed: teardown command "/usr/bin/umount --recursive /home/paul/scratch/debootstrap-2/scratch/loopback/proc" failed. Output:
umount: /home/paul/scratch/debootstrap-2/scratch/loopback/proc: target is busy.

teardown command "/usr/bin/umount /home/paul/scratch/debootstrap-2/scratch/loopback" failed. Output:
umount: /home/paul/scratch/debootstrap-2/scratch/loopback: target is busy.

This is also not always happening and mounted dirs can be umounted manually after the build failed. So I suspect a race condition again.