Can't have two python3 parts using setuptools

[Björn Tillenius]

If two python3 parts use setuptools, building a snap will fail since some common files have different contents:

  Error: parts treac and pysmbus have the following file paths in common which have different contents:
    usr/lib/python3/dist-packages/easy-install.pth
    usr/lib/python3/dist-packages/pip-7.1.2-py3.4.egg/EGG-INFO/entry_points.txt
    usr/lib/python3/dist-packages/pysmbus-0.1.egg-info/SOURCES.txt

The latter two are intermittent. They don't always conflict. But the first one will always conflict.

An example of a snapcraft.yaml that produces this error:

name: treac
version: 0
vendor: Bjorn Tillenius <email address hidden>
binaries:
  treacd:
    exec: usr/bin/treac
summary: TREAC - The Treadmill Controller
description: Provides an API for controlling a treadmill on a Raspberry Pi
icon: icon.png

parts:
    treac:
        plugin: python3
        source: git://github.com/bjornt/treac
    pysmbus:
        plugin: python3
        source: git://github.com/bjornt/pysmbus

[Kyle Fazzari (kyrofa)]

I'm not sure there's a way for the python3 plugin to intelligently determine whether or not to include those files. Which one's files should be included? There's no good way for it to know, as far as I can see. You need to tell it by using the `stage:` keyword and exclude those files from one of the parts.

[Sergio Schvezov (sergiusens)]

On Fri, Jan 8, 2016 at 1:44 PM, Kyle Fazzari <email address hidden> wrote:

> I'm not sure there's a way for the python3 plugin to intelligently
> determine whether or not to include those files. Which one's files
> should be included? There's no good way for it to know, as far as I can
> see. You need to tell it by using the `stage:` keyword and exclude those
> files from one of the parts.
>

Agreed, this is the intended use for filesets

[Björn Tillenius (bjornt)]

So you're basically saying that having to python parts that use setuptools isn't supported? I can't see how using a stage: keyword will use, since easy-install.pth needs to contain both eggs, so you can't just pick one of them, you need to merge them.

I'm reopening this bug, since it's still a bug. If it's not supported, could you at least document it? Or is there a way to use the stage: keyword to get this to work?

FWIW, one solution to the problem would be to have the python plugin accept multiple paths, which all would use the same python installation. That way you would have python do the merging for you. Or maybe introduce a new option which can point to another python plugin to install things into?

[Kyle Fazzari (kyrofa)]

Björn, do me a favor and try this snapcraft.yaml. Does it produce a .snap that works?

name: treac
version: 0
vendor: Bjorn Tillenius <email address hidden>
binaries:
  treacd:
    exec: usr/bin/treac
summary: TREAC - The Treadmill Controller
description: Provides an API for controlling a treadmill on a Raspberry Pi
icon: icon.png

parts:
    treac:
        plugin: python3
        source: git://github.com/bjornt/treac
        filesets:
          no-easy-install-files:
            - -usr/lib/python3/dist-packages/easy-install.pth
            - -usr/lib/python3/dist-packages/pysmbus-0.1.egg-info/SOURCES.txt
            - -usr/lib/python3/dist-packages/pip-8.0.0-py3.4.egg/EGG-INFO/entry_points.txt
        stage:
          - $no-easy-install-files
        snap:
          - $no-easy-install-files
    pysmbus:
        plugin: python3
        source: git://github.com/bjornt/pysmbus
        filesets:
          no-easy-install-files:
            - -usr/lib/python3/dist-packages/easy-install.pth
            - -usr/lib/python3/dist-packages/pysmbus-0.1.egg-info/SOURCES.txt
            - -usr/lib/python3/dist-packages/pip-8.0.0-py3.4.egg/EGG-INFO/entry_points.txt
        stage:
          - $no-easy-install-files
        snap:
          - $no-easy-install-files

[Björn Tillenius (bjornt)]

It seems like your snapcraft.yaml does work. At least as far as I can see. I'm considering this a workaround, though. I think the bug is still valid, although maybe it should say "Very cumbersome" instead of "Can't" in the title.

For example, I'm not clear on why easy-install.pth isn't needed. And if it's not needed, why not exclude it automatically? The same is true for the other files that were excluded. I don't think you should have to ask a Snappy developer for help for simply things like this.

Another reason I don't think this is the solution is that your snapcraft.yaml actually didn't work. I had to update the version of pip to match the latest one.

I'm still of the opinion that you shouldn't have to do this manually. Python is perfectly capable installing multiple packages in the same location. I shouldn't have to reimplement that logic myself if I want to build a snap. Imagine maintaining such a snapcraft.yaml in an automated build system. You would have to implement tooling to keep it up to date.

[Sergio Schvezov (sergiusens)]

I am almost sure pth files are useless for the purposes of how we are doing things
http://stackoverflow.com/questions/15208615/using-pth-files

Maybe we just need to make sure those .pth files do not get transitioned to stage or prime at all.

[Sergio Schvezov (sergiusens)]

The latest snapcraft (master) does not have this problem, can you come up with another example that would maybe fail?

[Sergio Schvezov (sergiusens)]

ftr

sergiusens@snapcraft:/tmp/tmp.7WFt7WZZBk$ cat > snapcraft.yaml
name: treac
version: "0"
summary: TREAC - The Treadmill Controller
description: Provides an API for controlling a treadmill on a Raspberry Pi

parts:
    treac:
        plugin: python3
        source: git://github.com/bjornt/treac
    pysmbus:
        plugin: python3
        source: git://github.com/bjornt/pysmbus
sergiusens@snapcraft:/tmp/tmp.7WFt7WZZBk$ snapcraft
"confinement" property not specified: defaulting to "strict"
"grade" property not specified: defaulting to "stable"
DEPRECATED: The 'python3' plugin's functionality has been replaced by the 'python' plugin, and it will soon be removed.
DEPRECATED: The 'python3' plugin's functionality has been replaced by the 'python' plugin, and it will soon be removed.
Preparing to pull treac
Hit http://archive.ubuntu.com/ubuntu xenial InRelease
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [94.5 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [95.7 kB]
Get:3 http://security.ubuntu.com/ubuntu xenial-security/main Sources [40.2 kB]
Hit http://archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:4 http://archive.ubuntu.com/ubuntu xenial-updates/main Sources [191 kB]
Get:5 http://security.ubuntu.com/ubuntu xenial-security/universe Sources [9,712 B]
Get:6 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [143 kB]
Get:7 http://archive.ubuntu.com/ubuntu xenial-updates/universe Sources [96.4 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [391 kB]
Get:9 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [329 kB]
Get:10 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [41.9 kB]
Get:11 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [25.7 kB]
Get:12 http://archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [114 kB]
Fetched 1,572 kB in 6s (72.3 kB/s)
Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libssl1.0.0 amd64 1.0.2g-1ubuntu4.5 [1,082 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libpython3.5-minimal amd64 3.5.2-2~16.04 [527 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libexpat1 amd64 2.1.0-7ubuntu0.16.04.2 [71.3 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 python3.5-minimal amd64 3.5.2-2~16.04 [1,599 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial/main amd64 python3-minimal amd64 3.5.1-3 [23.3 kB]
Get:6 http://archive.ubuntu.com/ubuntu xenial/main amd64 mime-support all 3.59ubuntu1 [31.0 kB]
Get:7 http://archive.ubuntu.com/ubuntu xenial/main amd64 libmpdec2 amd64 2.4.2-1 [82.6 kB]
Get:8 http://archiv...