click-reviewers-tools fails on all python3-based snaps

[Thomi Richards]

I'm trying to build a simple python3 app. My snapcraft.yaml file is:

```
name: gmailfilter
version: 0.0.1a
summary: Programmatically filter gmail messages
description: Filter messages in a gmail inbox by writing Python code - obviously!

apps:
  gmailfilter:
    command: usr/bin/gmailfilter
    plugs: [network, network-bind]

parts:
  gmailfilter:
    plugin: python3
    source: <email address hidden>:thomir/gmailfilter.git
```

The .snap file built with snapcraft in xenial (version 2.8.4) is installable locally, and works fine. However, upon uploading it it fails review with the following error:

```
package contains external symlinks: usr/lib/x86_64-linux-gnu/libmvec.so lint-snap-v2_external_symlinks
```

I can see that file listed in the snapcraft working directory:

```
$ find . -name "libmvec.*"
./stage/usr/lib/x86_64-linux-gnu/libmvec.so
./stage/usr/lib/x86_64-linux-gnu/libmvec.a
./snap/usr/lib/x86_64-linux-gnu/libmvec.so
./snap/usr/lib/x86_64-linux-gnu/libmvec.a
./parts/gmailfilter/install/usr/lib/x86_64-linux-gnu/libmvec.so
./parts/gmailfilter/install/usr/lib/x86_64-linux-gnu/libmvec.a
```

But it's not a dependency of the application itself (i.e.- grabbing the source and installing it in a virtualenv doesn't pull in that library).

I can only imagine that this is caused by the python3 plugin.

[Thomi Richards (thomir-deactivatedaccount)]

I've confirmed that this also happens with the 'py3-project' example from the snapcraft git repo.

jdstrand says:

9:14 AM <jdstrand> may I see the review tools error?
9:14 AM <thomi> jdstrand: sure, one sec
9:14 AM <Facu> jdstrand, "package contains external symlinks: usr/lib/x86_64-linux-gnu/libmvec.so lint-snap-v2_external_symlinks"
9:14 AM <thomi> beat me to it :D
9:15 AM <jdstrand> right
9:15 AM <jdstrand> that indicates a real problem with your snap
9:15 AM <jdstrand> it is pointing to something outside of itself
9:15 AM <thomi> jdstrand: this is using the example py3-project snap from the snapcraft repo
9:16 AM <jdstrand> sounds like a bug in the snapcraft repo

[Thomi Richards (thomir-deactivatedaccount)]

9:20 AM <Facu> jdstrand, thomi, I opened the snap: it looks like there are a lot of symlinks pointing to outside: http://linkode.org/y6AKZ3Iq9Hzsoi0Ic0nui4
9:20 AM <jdstrand> iirc, some are required
9:21 AM <jdstrand> there is a list and the review tools won't complain about ones in that list
9:21 AM <Facu> in any case, the libmvec looks like pointing outside, indeed: libmvec.so -> /lib/x86_64-linux-gnu/libmvec.so.1
9:22 AM <Facu> so, snapcraft is building badly the .snap
9:22 AM <jdstrand> but the libmvec.so might need to be one of those
9:22 AM <jdstrand> I see that it is part of libc6

[Jamie Strandboge (jdstrand)]

@Sergio: can you comment on if libmvec.so should be pointing to the system libmvec.so? Is snapcraft keeping track of which system libraries need to be used? I see ./libraries/16.04 but it is a lot longer than I expected.

If snapcraft is doing the right thing here, please feel free to close the snapcraft task and I'll update the review tools accordingly.

[Kyle Fazzari (kyrofa)]

Jamie, the `libraries/16.04` file isn't for symlinks, but for determining what should or shouldn't be automatically copied into the snap when crawling for ELFs and parsing the output of ldd.

In this case, it seems that the libc6 deb is getting staged, which has absolute symlinks contained within it. My first take on this is that libc6 shouldn't be staged at all, since it's available on all systems (host and target).

[Sergio Schvezov (sergiusens)]

During our sprint I asked wgrant and nessita to get some time and fix the python and python3 plugins as they are not the best.

Given python3 is part of ubuntu core itself I want to be able to leverage that and use some of the new goodies that made it to python3 (or at least the patched python3 that made it into debian/ubuntu).

For now I will try and see how this can be solved.

[Sergio Schvezov (sergiusens)]

To answer kyrofa, python3-dev brings in libc6-dev

[Jamie Strandboge (jdstrand)]

@Sergio, should libmvec.so be added to the review tools as a valid path or is this a bug in snapcraft?

[Sergio Schvezov (sergiusens)]

OK, I've played around a bit and am sort of happy with how I modified the plugin (feel free to review btw). The pull request is up here:

https://github.com/ubuntu-core/snapcraft/pull/510

This has some advantages:

- smaller snaps as we use the system python
- no weird -dev packages being staged that need manual tunning to get out of what was brought in (your actual issue).

$ ls -lh gmailfilter_0.0.1a_amd64.snap
-rw-rw-r-- 1 sergiusens sergiusens 144K may 23 18:33 gmailfilter_0.0.1a_amd64.snap

[Thomi Richards (thomir-deactivatedaccount)]

The PR generates *much* smaller snaps, which is really nice (50MB -> 150KB). The snaps install locally and work, but fail package review, this time because of a checksum mismatch. On IRC we discussed the fact that there's an outstanding bug (in squashfs perhaps) that needs to be fixed. However, until we can get a passing snap review I can't say if your PR fixes the issue or not.

[Sergio Schvezov (sergiusens)]

thomi can you add your snap to LP: #1579931 please?

[Evan (ev)]

"Given python3 is part of ubuntu core itself ..."

$ find /snap/ubuntu-core/current -name '*python*'
$

And wasn't it the plan from very early days to remove Python from the OS snap as fast as possible?

[Oliver Grawert (ogra)]

this desire hasn't changed ... but sadly the dependency on cloud-init will be tied even closer which will keep python there until someone re-writes it in another language ...

[Sergio Schvezov (sergiusens)]

Is this fixed since the latest python plugin came into existence?

[Launchpad Janitor (janitor)]

[Expired for Snapcraft because there has been no activity for 60 days.]