python3 ctypes can't find libs on due to ld cache

[Jamie Strandboge]

python3-magic 'undefined symbol: magic_open' on UC devices but not classic (ctypes issue)

With the snap in https://code.launchpad.net/~jdstrand/+git/test-python3-magic the snap works fine on classic distro, but not on UC16 or UC18:

On classic distro (eg, Ubuntu 19.10):
$ snapcraft
$ sudo snap install --dangerous ./test-python3-magic_0_amd64.snap
$ test-python3-magic /bin/ls
/bin/ls: application/x-executable; charset=binary

On UC16 and UC18:
$ test-python3-magic /bin/ls
Traceback (most recent call last):
  File "/snap/test-python3-magic/x1/bin/test-python3-magic", line 3, in <module>
    import magic
  File "/snap/test-python3-magic/x1/usr/lib/python3/dist-packages/magic/__init__.py", line 361, in <module>
    add_compat(globals())
  File "/snap/test-python3-magic/x1/usr/lib/python3/dist-packages/magic/__init__.py", line 325, in add_compat
    from magic import compat
  File "/snap/test-python3-magic/x1/usr/lib/python3/dist-packages/magic/compat.py", line 61, in <module>
    _open = _libraries['magic'].magic_open
  File "/snap/test-python3-magic/x1/usr/lib/python3.6/ctypes/__init__.py", line 361, in __getattr__
    func = self.__getitem__(name)
  File "/snap/test-python3-magic/x1/usr/lib/python3.6/ctypes/__init__.py", line 366, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: python3: undefined symbol: magic_open

Using 'base: core' or 'base: core18' makes no difference. Snapcraft 3.9.7 and 3.9.8 have the same behavior.

[Jamie Strandboge (jdstrand)]

Adding a review-tools task since due to this bug, they cannot be run on UC.

[Chris Patterson (cjp256)]

python3-magic uses ctypes.util.find_library('magic') , which actually uses `ldconfig -p` to find installed libraries. That's not honoring the LD_LIBRARY_PATH and it is getting your host-installed library list. With the "fake" hit of your host's /usr/lib/<arch-triplet>/libmagic.so.1 (which gets distilled down to "libmagic.so.1"), it ends up loading the library fine afterwards.

Related: https://forum.snapcraft.io/t/issue-when-snap-uses-pythons-ctype-util-find-library/9690

So, in short:
(1) ctypes.util.find_library() is currently broken for snaps.
(2) I don't know that it's an issue, but I wonder if the host ldconfig cache shouldn't be leaking for strict snaps? In this case, it's clearly triggering system-specific behavior changes...

If `ldconfig` were to provide the correct library list in a snap, it would presumably address this without having to patch ctypes. Something like what is discussed at: https://forum.snapcraft.io/t/effects-of-dynamic-library-caching-on-snap-startup-performance/14454

[Jamie Strandboge (jdstrand)]

Thanks Chris! We discussed this more on irc and found that python has fallbacks. It will use ldconfig -p, but also gcc and ld to try to find things. I found that by looking at strace, I was able to workaround this by adjusting the snapcraft.yaml to have:

  buildme:
    source: .
    plugin: python
    override-build: |
      snapcraftctl build
      # cd into the gnu triplet then create libmagic.so symlink for both
      # 'ld -t -L ... -lmagic' (from the staged binutils) and
      # 'stat(.../magic.so)'. LP: #1861026
      orig=$(pwd)
      libmagic=$(find "$SNAPCRAFT_PART_INSTALL"/usr/lib -name libmagic.so.1.0.0)
      cd $(dirname $libmagic)
      ln -sf libmagic.so.1.0.0 libmagic.so
      cd $orig
    stage-packages:
    - binutils
    - libmagic1
    - python3-magic

Now it works:

$ cat /etc/lsb-release
DISTRIB_ID="Ubuntu Core"
DISTRIB_RELEASE=16
DISTRIB_DESCRIPTION="Ubuntu Core 16"

$ test-python3-magic /bin/ls
/bin/ls: application/x-sharedlib

I discovered this by looking at strace (eg, snap run --strace="-o ./trace" test-python3-magic /bin/ls) and found things like:

4902 execve("/snap/test-python3-magic/x5/usr/sbin/ld", ["ld", "-t", "-L", "/var/lib/snapd/lib/gl", "-L", "/var/lib/snapd/lib/gl32", "-L", "/var/lib/snapd/void", "-L", "", "-L", "/snap/test-python3-magic/x5/lib", "-L", "/snap/test-python3-magic/x5/usr/"..., "-L", "/snap/test-python3-magic/x5/lib/"..., "-L", "/snap/test-python3-magic/x5/usr/"..., "-o", "/dev/null", "-lmagic"], 0x7ffde43b0578 /* 44 vars */) = -1 ENOENT (No such file or directory)

The trick was noticing that I needed *both* an ld in PATH in the snap (so I stage binutils) *and* the magic.so symlink to point to what is shipped (the override-build). This could possibly be achieved with a layout (does $SNAPCRAFT_ARCH_TRIPLET work with layouts?).

[Ian Johnson (anonymouse67)]

Adding this as a low priority bug to snapd for mounting a snap-private /etc/ld.so.cache file and making that file writable in the AppArmor policy. That will at least make it so that snaps don't need to use a layout on /etc/ld.so.cache to modify things.

[Jamie Strandboge (jdstrand)]

786d0b8eab91c3be850f07195b4cd2c52dbdfbf7 contains a workaround for this.