snap prepare-image fails with qemu-user-static

[Ondrej Kubik]

snap prepare-image with preseed option have to run natively as part of the preseed steps is executing snapd from the image itself. This means cross image build are not possible.
This brings complication how to build images. One of the proposed solutions is use of the emu-user-static

This itself has some other problems as auto binfmts detection is not enabled by default for arm and aarch64 (on x86_64) machines.
So I get following error:
preseeding part fails with:
starting to preseed UC20+ system: /tmp/preseed-784705990
chroot: failed to run command ‘/usr/lib/snapd/snapd’: No such file or directory
error: error running snapd in preseed mode: exit status 127

After enabling auto detecting with following steps: https://paste.ubuntu.com/p/SdwqJNxJPS/
I get following error:

starting to preseed UC20+ system: /tmp/preseed-226448566
chroot: failed to run command ‘/usr/lib/snapd/snapd’: Exec format error
error: error running snapd in preseed mode: exit status 126

[Ondrej Kubik (ondrak)]

More testing and this is a bit more complicated, mainly because our chroot is based on core snap.
with the following change I was able to at least start the chroot
https://github.com/kubiko/snapd/commit/7bebc919789e3696e14eb1df6c45dfd70eb20c2b

however chroot is still not fully working, it fails when executing other binaries, see log:

2022/06/16 16:25:20.241252 daemon.go:247: started snapd/2.56+git1056.gdb27d5e (series 16; devmode) ubuntu-core/20 (armhf) linux/5.4.0-113-generic.
2022/06/16 16:25:20.252976 main.go:129: system does not fully support snapd: cannot mount squashfs image using "squashfs": fork/exec /usr/bin/mount: no such file or directory
2022/06/16 16:25:20.253874 daemon.go:340: adjusting startup timeout by 30s (pessimistic estimate of 30s plus 5s per snap)
2022/06/16 16:25:20.268111 devicemgr.go:321: no ubuntu-save mount
2022/06/16 16:25:20.554639 devicemgr.go:789: preload verified gadget metadata from seed failed: cannot run unsquashfs: fork/exec /usr/bin/unsquashfs: no such file or directory
cannot run daemon: state startup errors: [cannot obtain snap-seccomp version information: fork/exec /usr/lib/snapd/snap-seccomp: no such file or directory]

[Ondrej Kubik (ondrak)]

I made progress to at least start the choor with following change:
https://github.com/kubiko/snapd/commit/7bebc919789e3696e14eb1df6c45dfd70eb20c2b
however this fails later on when snapd tries to run mount and unsquashfs binaries

patch.go:64: Patching system state level 6 to sublevel 3...
daemon.go:247: started snapd/2.56+git1056.gdb27d5e (series 16; devmode) ubuntu-core/20 (armhf) linux/5.4.0-113-generic.
main.go:129: system does not fully support snapd: cannot mount squashfs image using "squashfs": fork/exec /usr/bin/mount: no such file or directory
daemon.go:340: adjusting startup timeout by 30s (pessimistic estimate of 30s plus 5s per snap)
devicemgr.go:321: no ubuntu-save mount
devicemgr.go:789: preload verified gadget metadata from seed failed: cannot run unsquashfs: fork/exec /usr/bin/unsquashfs: no such file or directory
cannot run daemon: state startup errors: [cannot obtain snap-seccomp version information: fork/exec /usr/lib/snapd/snap-seccomp: no such file or directory]
error: error running snapd in preseed mode: exit status 1

So only way here to really get it working would be mounting binfmt_misc
mount binfmt_misc -t binfmt_misc <chroot>/proc/sys/fs/binfmt_misc/
and then making qemu-*-static available inside the chroot. Bad thing it needs to be at the path set in binfmt_misc, so without wrecking host machine this would mean /usr/bin/qemu-*-static
but since chroot root is base snap, it's read only, so not really sure what is the right approach here....