32 bit applications on 64 bit system fail due to seccomp

[Jamie Strandboge]

[Impact]
Snaps currently can't leverage compatibility architectures such as 32 bit binaries on a 64 bit system (eg, shipping 32 bit wine applications to amd64 system or 32 bit skype on amd64) or when using 64 bit kernels with a 32 bit userspace.

[Test Case]
1. On an amd64 system, download the attached snap

2. sudo snap install --dangerous ./snap-ls_2_amd64.snap

3. test 64bit binary works (ie, normal case):
$ snap-ls.ls64 --version
ls (GNU coreutils) 8.25
...

4. test 32bit binary works (this bug):
$ snap-ls.ls32 --version
ls (GNU coreutils) 8.25
...

With an unpatched snap-confine, you see this instead:
$ snap-ls.ls32 --version
Bad system call

with a seccomp denial in /var/log/syslog:
audit: type=1326 audit(1478814614.068:180980): auid=4294967295 uid=1000 gid=1000 ses=4294967295 pid=21910 comm="ld-linux.so.2" exe="/snap/snap-ls/x1/lib/i386-linux-gnu/ld-2.23.so" sig=31 arch=40000003 syscall=45 compat=1 ip=0x56633547 code=0x0

(note 'syscall=45 compat=1')

[Regression Potential]
Regression potential is considered low since we are only adding to the architectures for systems that support the compatibility architectures. snap-confine and snapd also have extensive testsuites.

= Original description =
Someone trying to snap a 32 bit application and run it on amd64 had the following seccomp denial:

auid=4294967295 uid=1000 gid=1000 ses=4294967295 pid=26210 comm="wine" exe="/snap/wine/100001/bin/wine" sig=31 arch=40000003 syscall=45 compat=1 ip=0xf7794547 code=0x0

What is happening is that the seccomp policy loaded into the kernel is for x86_64, not x86 but the 32 bit app is using the 32 bit syscall table. I need to verify with upstream seccomp, but AIUI, fixing this requires kernel work.

WORKAROUND: install the snap with --devmode
WORKAROUND #2: install 64bit snaps on 64 bit systems and 32 bit snaps on 32 bit systems

Due to the initial intended purpose and design of seccomp, the kernel doesn't do remapping of syscalls in the manner needed, but it might be possible to run a 32 bit launcher on a 64 bit system to launch a 64 bit application. Preliminary testing shows this does not work, but more investigate is needed. It may be that we need to simply say that for the foreseeable future snaps need to ship binaries for the specified architecture and not a personality of the architecture.

[John Lenton (chipaca)]

Would this mean that we could let packages with arch:i386 install in arch:amd64?

[Jamie Strandboge (jdstrand)]

@John, if it could be made to work, yes, but as of today, no. Shipping a 32 bit binary in an arch:amd64 snap currently does not work and we must (again, currently) treat it as the same error as shipping an armhf binary in a ppc64el snap.

[Geoff Teale (tealeg)]

Currently every program compiled with the version of SBCL (Steel Bank Common Lisp) that ships with Ubuntu 16.04 amd64 falls foul of this problem when you try to run it in a confined snap. This, of course, might be considered a problem with SBCL, but I thought I should raise it anyway. For what it's worth the dmesg output looks like this:

[24387.111897] audit: type=1326 audit(1470036026.101:49): auid=4294967295 uid=1000 gid=1000 ses=4294967295 pid=16056 comm="hello" exe="/snap/hello/x1/hello" sig=31 arch=c000003e syscall=135 compat=0 ip=0x7fe4a065fea7 code=0x0

I'm going to see if there's a way to compile sbcl from source that doesn't do this.

[Jamie Strandboge (jdstrand)]

@Geoff - syscall=135 is the personality syscall. I just added it to the default policy to fix bug #1614269.

[Jamie Strandboge (jdstrand)]

I've looked into this issue and the 'syscall=45 compat=1' denial has to do with the fact that the launcher is not setting up the compat architecture via seccomp_arch_add(). Once that is done, this denial goes away and I am able to launch i386 binaries in an amd64 snap[1].

Related, snappy-debug is not considering 'compat=1' and calling scmp_sys_resolver with the correct arguments.

[1] This is not as easy as it could be:
- you have to 'stage-packages' the i386 libraries (eg, stage-packages: [ libc6:i386, libselinux1:i386 ]) and snapcraft doesn't make that as easy as it could (that may be fixed in master now)
- you have to create a wrapper to add '$SNAP/lib/i386-linux-gnu:$SNAP/usr/lib/i386-linux-gnu' to LD_LIBRARY_PATH (snapcraft could do this for the developer)
- you have to invoke the 32 bit binary (ie, via the above wrapper) with '$SNAP/lib/ld-linux.so.2 <your 32 bit binary>' (snapcraft could perhaps auto-detect this and do it, but maybe that would be better as a part)

[Jamie Strandboge (jdstrand)]

This bug is in snap-confine, not snapd so marking the snappy task as Invalid.

[Jamie Strandboge (jdstrand)]

https://github.com/snapcore/snap-confine/pull/181

[Jamie Strandboge (jdstrand)]

Adding snapcraft task to address the rough edges outlined in https://bugs.launchpad.net/snappy/+bug/1592022/comments/5.

[Jamie Strandboge (jdstrand)]

FYI, snappy-debug now considers 'compat'.

[Jamie Strandboge (jdstrand)]

2.20 fixes this issue.

[Kyle Fazzari (kyrofa)]

This should make the snapcraft side of things easier: https://github.com/snapcore/snapcraft/pull/1050