Snapcraft 4.5.1 - cannot pack “/data/prime”: mksquashfs call failed: signal: segmentation fault (core dumped)

[Phillip Hernandez]

Since 4.5.1 was released builds of arm64 have been failing with the error below. Builds of amd64 and armhf seem unaffected with the same snapcraft.yaml file. This effects 2 projects that I work with.

The snapcraft build files can be found:
https://github.com/sct/overseerr/blob/develop/snap/snapcraft.yaml
https://github.com/Tautulli/Tautulli/blob/master/snap/snapcraft.yaml

```
Sorry, an error occurred in Snapcraft:
Failed to create snap, snap command failed:
stdout:

stderr:
error: cannot pack "/data/prime": mksquashfs call failed: signal: segmentation fault (core dumped)

We would appreciate it if you anonymously reported this issue.
No other data than the traceback and the version of snapcraft in use will be sent.
Would you like to send this error data? (Yes/No/Always/View) [no]:
```

-- added by diddledan:

This is reproducible locally in docker for an affected app with diddledan's snapcraft container image using the following steps:

1. edit /etc/docker/daemon.json, or /snap/docker/current/config/daemon.json to ensure that `"experimental": true` is set
2. if you changed daemon.json restart your docker daemon, e.g.:
   systemctl restart snap.docker.dockerd.service
3. docker run --rm --privileged tonistiigi/binfmt:latest --install all
4. sudo mkdir /sys/kernel/security/apparmor/policy/namespaces/docker-snapcraft
5. docker run --rm -it --platform=linux/arm64 --privileged --security-opt apparmor=":docker-snapcraft:unconfined" -v $PWD:/data -w /data diddledan/snapcraft:core18
6. sudo rmdir /sys/kernel/security/apparmor/policy/namespaces/docker-snapcraft

The container image is built using diddledan's code at https://github.com/diddlesnaps/snapcraft-container

[Sergio Schvezov (sergiusens)]

Thanks for the report. I would need some more information order to correctly triage:

- how snapcraft was invoked
- host system this was run on
- version of snapd
- is this CI, are there CI build logs accessible?

If running with "snapcraft --debug", does running "snap pack prime" cause the same segmentation fault?

[Phillip Hernandez (samwiseg0)]

- how snapcraft was invoked
It was invoked from a GitHub workflow found here.
https://github.com/sct/overseerr/actions/runs/561122420/workflow

- host system this was run on
The build were created using QEMU and Ubuntu 20.04 via GitHub

- version of snapd
2.49

- is this CI, are there CI build logs accessible?
Yes. The build logs can be found here.
https://github.com/sct/overseerr/runs/1887508955?check_suite_focus=true

After some further testing it seems to be an issue between the latest QEMU version and snapcraft.
After reverting to the previous QEMU version. The build process works again.
```
  - name: Set Up QEMU
        uses: docker/setup-qemu-action@v1
        with:
          image: tonistiigi/binfmt@sha256:df15403e06a03c2f461c1f7938b171fda34a5849eb63a70e2a2109ed5a778bde
```
I am not sure what QEMU may have changed that has caused an incompatibility with snapcraft/snapd and the latest version of QEMU.

[Lucy Llewellyn (lucyllewy)]

I've amended the description to include reproduction steps.

[Maciej Borzecki (maciek-borzecki)]

Am I reading this correctly that the builds are done on an amd64 host actually, but there is a binfmt handler that uses qemu for executing arm64 binaries?

Does `snap pack` also fail on an actual arm64 host, eg. Raspberry Pi 4 or in arm64 VM?

[Lucy Llewellyn (lucyllewy)]

Yes, you're reading correctly. The build is executing using qemu-user via binfmt to emulate arm64 on an amd64 host, which is set-up by the tonistiigi/binfmt container image. This is required for execution on GitHub Actions because GitHub has no arm-native hosts.

[Lucy Llewellyn (lucyllewy)]

This appears to be a bug in qemu-user/qemu-user-static and/or ld.so in xenial (and therefore core and snapd snaps) - it does *not* require snapd to be involved to trigger. However, the scenario that triggers the bug is being used by snapd explicitly.

The problem arises with qemu-user-static emulating aarch64 on x86_64 whenever you directly call /lib/ld-linux-aarch64.so.1 with the path of an executable to spawn:

# /lib/ld-linux-aarch64.so.1 /bin/bash
Segmentation fault (core dumped)
# /lib/ld-linux-aarch64.so.1 --verify /bin/bash
Segmentation fault (core dumped)
# /lib/ld-linux-aarch64.so.1 --list /bin/bash
Segmentation fault (core dumped)
# /lib/ld-linux-aarch64.so.1 --inhibit-cache /bin/bash
Segmentation fault (core dumped)

This issue only affects the ld-linux-aarch64.so.1 that is present in xenial, which core, and snapd are both built upon. It can be fixed by bumping snapd's build-base to core18.

[jcat (jcat)]

Hi,

I also hit this issue.

Can I ask what version of QEMU-user seemed to work ?

Is there any plan to bump the build-base for snapd ?

Cheers,
Just

[jcat (jcat)]

I managed to print the info from a container, QEMU version from an unspecified commit hash: version 5.0.0 (-dirty). Pretty old.

[jcat (jcat)]

I just found your version of containerised snapcraft @diddledani - looks like it works around this issue and has some other ice features - am checking it out now :)

https://github.com/diddlesnaps/snapcraft-container