asciinema 1.3.0 snap is segfaulting on 14.04

[Dave Morley]

davmor2@davmor2-HP-G62-Notebook-PC:~$ sudo snap install --classic asciinema
[sudo] password for davmor2:
asciinema 1.3.0 from 'sabdfl' installed

davmor2@davmor2-HP-G62-Notebook-PC:~$ asciinema rec
Segmentation fault (core dumped)

davmor2@davmor2-HP-G62-Notebook-PC:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty

davmor2@davmor2-HP-G62-Notebook-PC:~$ snap list
Name Version Rev Developer Notes
asciinema 1.3.0 1 sabdfl classic
core 16.04.1 714 canonical -
hello 2.10 20 canonical -
lxd 2.7 887 canonical -

Not sure if there are logs somewhere that will be of use. But in general this is the only place the snap is failing to run.

This machine will be up a while longer if there are things I can do to help get info from it for you.

[Thomas Voß (thomas-voss)]

For completeness, you might want to attach syslog and the output of 'sudo journalctl -x'.

[Dave Morley (davmor2)]

Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci 0000:03:00.0: can't claim
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci 0000:00:1c.0: PCI bridge
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci 0000:00:1c.0: bridge wi
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci 0000:00:1c.0: bridge wi
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci 0000:00:1c.0: bridge wi
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci 0000:03:00.0: BAR 6: assi
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci 0000:00:1c.1: PCI bridge
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci 0000:00:1c.1: bridge wi
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci 0000:00:1c.1: bridge wi
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci 0000:00:1c.1: bridge wi
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci 0000:00:1e.0: PCI bridge
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci_bus 0000:00: resource 4 [
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci_bus 0000:00: resource 5 [
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci_bus 0000:00: resource 6 [
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci_bus 0000:00: resource 7 [
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci_bus 0000:02: resource 0 [
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci_bus 0000:02: resource 1 [
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci_bus 0000:02: resource 2 [
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci_bus 0000:03: resource 0 [
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci_bus 0000:03: resource 1 [
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci_bus 0000:03: resource 2 [
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci_bus 0000:04: resource 4 [
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci_bus 0000:04: resource 5 [
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci_bus 0000:04: resource 6 [
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci_bus 0000:04: resource 7 [
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: NET: Registered protocol fami
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: TCP established hash table en
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: TCP bind hash table entries:
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: TCP: Hash tables configured (
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: UDP hash table entries: 2048
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: UDP-Lite hash table entries:
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: NET: Registered protocol fami
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: pci 0000:00:02.0: Video devic
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: PCI: CLS 64 bytes, default 64
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: Trying to unpack rootfs image
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: Freeing initrd memory: 21764K
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: Scanning for low memory corru
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: futex hash table entries: 204
Jan 18 11:39:54 davmor2-HP-G62-Notebook-PC kernel: audit: initializing netlink s
Jan 18 11:39:54 davmor2-HP-G62-Notebook-P...

[Dave Morley (davmor2)]

hope that helps Thomas

[Sergio Schvezov (sergiusens)]

I am reassining this to snapd and snapcraft as this is more likely a problem from the snappy world than an Ubuntu packaging one for asciinema.

[Sergio Schvezov (sergiusens)]

Adding what was shared over an email I sent to the list:

Looking at the snap itself, I see it is python3 and using the in-core python3 which leads to

$ readelf -a /snap/core/current/usr/bin/python3 |grep INTERP -A2
  INTERP 0x0000000000000238 0x0000000000400238 0x0000000000400238
                 0x000000000000001c 0x000000000000001c R 1
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]

`stage-packages` that provide runnables will also have this problem

[Dave Morley (davmor2)]

Thanks Sergio Wasn't entirely sure where it should live :)

[Michael Hudson-Doyle (mwhudson)]

So the point here is that classic snaps should not execute any binaries from the core snap? (I tried to think if there was some way we could have the interpreter for all binaries in the core snap be /snap/core/current/lib/x86_64-linux-gnu/ld-2.23.so but can't think of a sane one)

[Mark Shuttleworth (sabdfl)]

Wait, that doesn't make sense, the core snap is supposed to provide core
binaries :)

[Zygmunt Krynicki (zyga)]

The problem is that binaries from the core snap don't refer to the core snap for the dynamic linker or the required libraries. I can investigate if it is possible without any changes to the linker or to the core snap and report back.

[Sergio Schvezov (sergiusens)]

There is also the problem that in a snap shell `python3` returns /usr/bin/python3 instead of /snap/core/current/usr/bin/python. This is a desirable trait for anything that isn't python as we want to call things from the system (IDE's calling out to things like python for syntax checking and such).

[Sergio Schvezov (sergiusens)]

By adding my own python and some variable tricks I got it to work on trusty

parts:
  asciinema:
    plugin: python
    python-packages:
        - asciinema
    after: [python3]
  python3:
    source: https://www.python.org/ftp/python/3.4.6/Python-3.4.6.tar.xz
    plugin: autotools

There are some tricks required for the python plugin to consume from the added part though which could simplify these tasks a lot. We could do things by convention if not by declaration. Say if a python interpreter is found on ./stage/bin/python3 then don't pull in python (same logic for other plugins).

[Sergio Schvezov (sergiusens)]

First, sorry, my statement about using classic's python instead of core's was wrong, I forgot to source the variables the wrapper sets which led me to...

...in the original snap (asciinema.sabdfl) if you do `unset LD_LIBRARY_PATH` in `snap-wrapper.sh` you get it to no create a Segmentation Fault but libraries aren't found (as they are in core).

tput: /lib/x86_64-linux-gnu/libtinfo.so.5: no version information available (required by tput)

If we can make the linker loader available on classic (/snap/bin/ld-...) we may be able to rewrite the elf headers without needed to rebase all the offsets.

[Sergio Schvezov (sergiusens)]

This solves environment variable leaking of PYTHONHOME and PYTHONUSERBASE
https://github.com/snapcore/snapcraft/pull/1080

[Sergio Schvezov (sergiusens)]

ok, I gone as far as I can on the snapcraft side and have some branches ready for landing. There seems to be a general problem with getaddrinfo when running on trusty though:

trusty
$SNAP/bin/test google.com 443
getaddrinfo: Success

xenial
./test google.com 443
Host: google.com
IPv4 address: 181.111.164.248 (google.com)
IPv4 address: 181.111.164.250 ((null))
IPv4 address: 181.111.164.244 ((null))
IPv4 address: 181.111.164.251 ((null))
IPv4 address: 181.111.164.246 ((null))
IPv4 address: 181.111.164.247 ((null))
IPv4 address: 181.111.164.245 ((null))
IPv4 address: 181.111.164.249 ((null))
IPv6 address: 2800:3f0:4003:c00::8a ((null))

[Sergio Schvezov (sergiusens)]

These are the sources for the previous comment:

snapcraft.yaml http://paste.ubuntu.com/23893384/
main.c http://paste.ubuntu.com/23893386/
Makefile http://paste.ubuntu.com/23893392/

[Michael Hudson-Doyle (mwhudson)]

On 31 January 2017 at 01:25, Sergio Schvezov <email address hidden>
wrote:

> ok, I gone as far as I can on the snapcraft side and have some branches
> ready for landing. There seems to be a general problem with getaddrinfo
> when running on trusty though:
>
> trusty
> $SNAP/bin/test google.com 443
> getaddrinfo: Success
>

Tingly spider sense says that this will be because it's loading NSS modules
off the trusty filesystem, not from the core snap.

mwhudson@aeglos:/opt/opensource/snaps/gai$ strace -e open $SNAP/bin/test
google.com 443
[...]
open("/lib/x86_64-linux-gnu/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2", O_RDONLY|O_CLOEXEC)
= 3
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libresolv.so.2", O_RDONLY|O_CLOEXEC) = 3
[...]

Seems tingly spider sense is right. But according to the dlopen docs:

       o (ELF only) If the executable file for the calling program
contains a DT_RUNPATH tag, then the directories
           listed in that tag are searched.

and $SNAP/bin/test sure contains a DT_RUNPATH. However, looking at the
glibc source suggests this bit of the docs is a lie: it's not checking the
executable of the calling process but instead it's looking for RUNPATH's in
the module that's calling dlopen which in this case is libc, which of
course does not have DT_RUNPATHs. It checks for DT_RPATH tags in both the
module calling dlopen and the executable, so I wonder if it's just a bug
that it doesn't do this for DT_RUNPATH. Either way, the docs are wrong.

[Michael Hudson-Doyle (mwhudson)]

FWIW, the relevant code in glibc is around here https://sourceware.org/git/?p=glibc.git;a=blob;f=elf/dl-load.c;h=a5318f9c8d1d42745a254479cf6bb1cd2acd516f;hb=HEAD#l2035

[Sergio Schvezov (sergiusens)]

https://asciinema.org/a/bjzuw6ujlg3l39cx1ai1bn6ii \o/ thanks to mwhudson! rpath vs runpath

[Dave Morley (davmor2)]

On Mon, 30 Jan 2017 22:18:11 -0000
Sergio Schvezov <email address hidden> wrote:

> https://asciinema.org/a/bjzuw6ujlg3l39cx1ai1bn6ii \o/ thanks to
> mwhudson! rpath vs runpath
>

\o/

--
You Make It, I'll Break It!

I Love My Job :)

http://www.canonical.com
http://www.ubuntu.com

[Sergio Schvezov (sergiusens)]

https://github.com/snapcore/snapcraft/pull/1094

[Sergio Schvezov (sergiusens)]

https://github.com/snapcore/snapcraft/pull/1093

[Zygmunt Krynicki (zyga)]

I'm marking the snapd task as invalid as this was fixed with changes to snapcraft.