snapd fails to autopkgtest on mksquashfs, which is looking for libgcc_s

[Dan Bungert]

During current snapd autopkgtest, a failure can be observed:

https://autopkgtest.ubuntu.com/results/autopkgtest-impish/impish/amd64/s/snapd/20210907_175258_5451b@/log.gz

error: cannot pack "/home/gopath/src/github.com/snapcore/snapd/tests/smoke/sandbox/test-snapd-sandbox": mksquashfs call failed:
-----
libgcc_s.so.1 must be installed for pthread_cancel to work
Parallel mksquashfs: Using 1 processor
Creating 4.0 filesystem on /home/gopath/src/github.com/snapcore/snapd/tests/smoke/sandbox/test-snapd-sandbox/test-snapd-sandbox_1.0_all.snap, block size 131072.
-----
error: cannot read snap file: "/var/lib/snapd/snaps/.local-install-136125945"
       is not a snap or snapdir

I traced the underlying call to the following:
"/snap/snapd/current/lib/x86_64-linux-gnu/ld-2.23.so" "--library-path"
"/snap/snapd/current/usr/local/lib:/snap/snapd/current/lib/x86_64-linux-gnu:/snap/snapd/current/usr/lib/x86_64-linux-gnu"
"/snap/snapd/current/usr/bin/mksquashfs" asdf asdf.squashfs

(the real call has more arguments, this is a simplified version that
produces the failure)

To observe this, one can create a VM based a cloud image from
https://cloud-images.ubuntu.com/impish/current, then run the above command in
the resulting environment.
Doing so will test against snapd v2.51.4 (12883).
Retesting with edge 2.52+git635.gada2d87 (13323) produces an equivalent result.

Running a more bland `/snap/snapd/current/usr/bin/mksquashfs asdf
asdf.squashfs` without messing with library paths has mksquashfs behaving as
expected. Also, getting a v2.51.3 snapd seems to behave itself. Updating from
2.51.3 -> 2.51.4 also works.

One can see a passing mksquashfs by taking libgcc_s.so.1 from hirsute
and placing it in the library path for the above call.

In the working scenario, it appears that libgcc_s is being obtained
from outside of /snap (and also libz). In the failing scenario, this
external copy of libgcc_s is not being loaded (per gdb info sharedlibrary),
but libz still is.

[Dan Bungert (dbungert)]

Added openssh and squashfs-tools so that this bug shows on the proposed-migration report.

[Michael Hudson-Doyle (mwhudson)]

So what is going on here is that libc is trying to load libgcc_s.so.1 and as it is not in the snap it ends up loading it from the host. Apparently in all releases from xenial through to hirsute this has worked but for whatever reason it does not work on impish. I think the appropriate fix is for the snapd snap to bundle libgcc_s.so.1 too.

[Michael Hudson-Doyle (mwhudson)]

(I confirmed by bind mount shenanigans that putting the libgcc_s.so.1 from the core snapd into the snapd snap makes this work)

[Michael Hudson-Doyle (mwhudson)]

https://github.com/snapcore/snapd/pull/10757

[Dan Bungert (dbungert)]

Retesting with snapd 2.52+git658.g005f554-dirty (13328), mksquashfs is able to work as expected. I believe that when we are able to autopkgtest against a snap with that fix, that this problem will be resolved.

[Dan Bungert (dbungert)]

Changing states - the change was committed on snapd, not squashfs-tools.

[Sebastien Bacher (seb128)]

The issue is impacting impish daily

$ sudo snap pack src
libgcc_s.so.1 must be installed for pthread_cancel to work
erreur : cannot pack "src": mksquashfs call failed

$ snap version
snap 2.53~pre1.git19b68f708
snapd 2.53~pre1.git19b68f708
series 16
ubuntu 21.10
kernel 5.13.0-16-generic

$ dpkg -l snapd
ii snapd 2.53~pre1.git19b68f708 amd64 Daemon and tooling that >

Using SNAPD_DEBUG=1 gives
2021/09/29 17:42:27.460907 tool_linux.go:93: DEBUG: snap (at "/snap/snapd/current") is older ("2.51.7") than distribution package ("2.53~pre1.git19b68f708")

Sounds like an issue we should fix for release so tagging rls incoming

[Sebastien Bacher (seb128)]

unsure if the bug should be made to affect to snapd ubuntu package?

[Ian Johnson (anonymouse67)]

As seb128 found, the issue is also affecting the debian package of snapd since when we call mksquashfs, we actually have code which specifically calls mksquashfs from the "system" snap which is either the snapd snap or the core snap, this is presumably to ensure that a consistent mksquashfs is used always and to avoid snaps from being packed with too old or new mksquashfs's.

So the net effect is that the fix for this that was landed into the snapd snap will also probably need to be added to the core snap for completeness, and also means that the only way to avoid snap pack from failing is to have a snapd snap with the fix which is only available on the edge channel currently.

So long story short, if you are affected by this today, a short-term workaround is to use the edge channel of the snapd snap by running

snap refresh snapd --edge

We have already included the fix into the snapd snap that is built on edge and it will be included in the release branch for 2.52, meaning that the next bugfix release on 2.52 will have the fix, we do plan on doing a 2.52.1 release in short order after 2.52 is complete, but it may be a few weeks from now before that is on stable.

[Dan Bungert (dbungert)]

Snapd team advises that this will be fixed in 2.52.1, and that affected users should pick up that version when available or, in the mean time, pickup a newer snapd. Currently the beta channel has this 2.52.1 version, which does appear to resolve the issue. Based on this advise I'm removing 'rls-ii-incoming'

Note that this version being on stable does not necessarily fix autopkgtest, as this deb version of snapd goes out of its way to NOT actually install the snap version of snapd.
https://github.com/snapcore/snapd/blob/master/packaging/ubuntu-16.04/tests/integrationtests#L38

[Ian Johnson (anonymouse67)]

Dan, note that although we set the SNAP_REEXEC environment variable in that integration test, that environment variable is actually not obeyed when it comes to the mksquashfs command which is called here:

https://github.com/snapcore/snapd/blob/dfba7de59a41bc22786d87f53b20deea14240713/snap/squashfs/squashfs.go#L524

and the function snapdtool.CommandFromSystemSnap does not observe the SNAP_REEXEC environment variable at all, instead just checking if the snapd or core snaps exist at all, and using the mksquashfs from those snaps instead of what's on the host on $PATH (though we do fallback and if there is no mksquashfs found in the core or snapd snaps (somehow???), then we use $PATH).

As I mentioned on IRC, we could potentially someday add a workaround where if mksquashfs from the system snap specifically segfaults then we will fall back to using the host's mksquashfs, but we have chosen for now not to work on that patch.

Hope this helps explain things.

[Christian Ehrhardt  (paelzer)]

We ahve that in Jammy
 snapd | 2.53+21.10ubuntu1 | jammy | source, amd64, arm64, armhf, ppc64el, riscv64, s390x

Removing openssl task to not show this outdated update-excuse

[Sergio Durigan Junior (sergiodj)]

Removing the tasks for openssh and squashfs-tools since this bug has been fixed and doesn't affect those packages anymore.