ubuntu-image snap is failing on mkfs.ext4 when ran on a zesty system

[Steve Langasek]

The ubuntu-image 1.1 snap was supposed to be fixed such that only internal versions of python modules, mkfs.ext4, fakeroot, etc. are used.

Instead users are reporting issues:

  https://bugs.launchpad.net/ubuntu-image/+bug/1644131/comments/7
  https://forum.snapcraft.io/t/ubuntu-image-snap-fails-with-invalid-filesystem-options-set/1275

The environment is correct, but it seems there's some incompatibility when using the snap 'xenial' mkfs.ext4 and a zesty host system. The snap should be built on a zesty env so that the newer mkfs.ext4 is included. Also, some arch-specific libraries are not properly moved to non-multiarch paths which causes some of the apps using host system libraries.

[Oliver Grawert (ogra)]

Looking at https://github.com/CanonicalLtd/ubuntu-image/blob/master/snapcraft.yaml
your "environment:" block needs to live in the "apps:" section, not somewhere globally ...

An example is at:
https://github.com/snapcore/snapcraft/blob/master/demos/git/snap/snapcraft.yaml

The PR that landed the environment feature is at https://github.com/snapcore/snapcraft/pull/1103/files

[Oliver Grawert (ogra)]

(globally set "environment" applies to the build (vars are shared in all parts at build time)... when specifically set for each app in the apps: section it becomes part of the generated wrapper)

[Łukasz Zemczak (sil2100)]

Ok, I can reproduce the issue on a zesty machine. During testing it all seemed to work, but it seems my test case was not sufficient. What I mean is: for instance, I just did this on my zesty VM:

ubuntu@zesty-test:~$ snap run --shell ubuntu-image
ubuntu@zesty-test:~$ echo $PATH
/snap/ubuntu-image/64/usr/bin:/snap/ubuntu-image/64/bin:/snap/ubuntu-image/64/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

So it seems snap run --shell is not a valid test case, since here it clearly shows the PATH env being correctly passed as per my 'environment:' section.

Another thing. I don't understand this at all, but it seems that the LD_LIBRARY_PATH that I set in the current environment: also seems to work correctly. If it didn't, the ubuntu-image snap would die horribly much earlier if libparted2 was removed from the system (as it dies when trying to import pyparted in the python code). But if you remove libparted2 and run the snap, it all works. So how is the PATH variable special? Why is it cleared somewhere when running the snap normally, but works when doing run --shell? This is not reliable at all.

[Łukasz Zemczak (sil2100)]

@ogra: just tested your fix and it doesn't seem to help, the issue seems to be somewhere else. I will be investigating further in a bit.
As per my previous experiments it just seems that the environment is set-up correctly in both cases.

On a side note: a related issue is that if you have ubuntu-image as a deb installed before of ubuntu-image as a snap, the environment of the snap will be completely busted. At least this was the case for Steve on his zesty machine. Still, this doesn't fix the mkfs part.

[Łukasz Zemczak (sil2100)]

I have now triple-confirmed that the environment is correct and that the correct mkfs is actually called when running the snap in zesty. The issue is unrelated to that so the bug title needs to be changed.

The issue is completely different. I don't know the exact 'why' but here's the story:
It seems our snap autobuilding infra is based off xenial. This means all the things we have bundled in the snap are xenial-based, meaning: we have the old mkfs.ext4 that does not yet support -d. This means that the xenial way of preppring the filesystem contents is being used, i.e. first the FS is prepared and then we copy in the contents with cp.
Somehow, possibly related to how classic snaps work, the 'xenial way' doesn't work on non-xenial systems. Somehow the xenial mkfs.ext4 is incompatible with the zesty machine and we don't get all the 'features' that are necessary for it to work.

This also solves the question why re-building the snap on zesty just works. This is also how Barry fixed the issue on #1644131. I will be rebuilding it and re-publishing.
Anyway: xenial is not the right place for building the ubuntu-image snap anyway, so yeah.

[Łukasz Zemczak (sil2100)]

Ok, I see the actual problem... somehow our install: mangling doesn't quite do its job and not all libraries are moved out of the arch-dependent lib directories. This causes the 'right' mkfs.ext4 being called with the wrong ext4 libraries.

[Łukasz Zemczak (sil2100)]

I have modified the snapcraft.yaml to make sure ALL multiarch libraries are being used from the snap. I created one on my zesty instance and published that to the store - meaning, with the new mkfs.ext4 as well. This is now in the beta channel. I tested both on xenial and zesty (with libparted2 removed from the host) to see if the snap works and images are being built successfully as expected.