Python libraries installed by python path are not found in sys.path

[Alberto Donato]

snapcraft 4.0, on a core20 snap

Python libraries and apps installed by a python part (via pip or setup.pyt dependencies) are installed in $SNAP/lib/python3.8/site-packages, but the path is not added to sys.path:

sys.path = [
    '/',
    '/usr/lib/python38.zip',
    '/usr/lib/python3.8',
    '/usr/lib/python3.8/lib-dynload',
    '/usr/lib/python3/dist-packages',
]

As such, those modules are not found when running the app.

[Chris Patterson (cjp256)]

Can you share your yaml?

If using stage-packages, currently core20 will require the user to set PYTHONPATH in the app's environment to inform python where the staged-packages are. E.g.:

https://github.com/snapcore/snapcraft/blob/master/tests/spread/plugins/v2/snaps/python-hello-with-stage-package-dep/snap/snapcraft.yaml#L15

If using python-packages with the python-plugin, it should handle the case described above, e.g.:

https://github.com/snapcore/snapcraft/blob/master/tests/spread/plugins/v2/snaps/python-hello-with-python-package-dep/snap/snapcraft.yaml

['', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/snap/python-hello-with-python-package-dep/x2/lib/python3.8/site-packages']

Can you confirm/deny the following:

(1) `snap run --shell <snap>` then `which python3` yields $SNAP/bin/python3 ?

(2) that your script has a shebang of `#!/usr/bin/env python3` ?

Thanks!

[Alberto Donato (ack)]

Yes to both your questions.

This is the relevant python part:

  crbs:
    plugin: python
    source: .
    source-type: local
    requirements:
      - requirements.txt
    build-packages:
      - git
      - libffi-dev
      - libpq-dev
      - python3-setuptools
    stage-packages:
      - libpq5
      - libxml2
    filesets:
      bins:
        - bin/alembic
        - bin/crbs-admin
        - bin/crbs-asyncapi
        - bin/uwsgi
        - bin/snap-helpers*
        - bin/python
        - bin/python3
    prime:
      - $bins
      - etc
      - lib
      - usr/lib
    override-build: |
      set -e
      snapcraftctl build
      snapcraftctl set-version $($SNAPCRAFT_PROJECT_DIR/snap/local/snap-version)
      snap-helpers write-hooks

With snapcraft 4.0.1+git1.ge9d758fa (4730) if I put the followig in the configure hook:

which python3
python3 -c 'import sys; print(sys.path)'

I get:

/snap/canonical-rbac/x1/bin/python3
['', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/usr/lib/python3/dist-packages']

If I append $SNAP/lib/python3.8/site-packages to sys.path at the beginning of the python script called by the hook, it works.

[Sergio Schvezov (sergiusens)]

What shebang comes out of

       snap-helpers write-hooks
?

The shebang rewrite logic in the plugin is done inside "snapcraftctl build", so if this ends up not being "#!/usr/bin/env ..." then that could explain it.

[Alberto Donato (ack)]

The shebang for "snap-helpers" itself is "#!/usr/bin/env python3".

The files created by "snap-helpers write-hooks" are bash scripts that get placed in snap/hooks, with the following content:

#!/bin/sh
exec "$SNAP/snap/command-chain/snapcraft-runner" \\
    "$SNAP/bin/snap-helpers-hook" "{hookname}"

"snap-helpers-hook" is a python script, with "#!/usr/bin/env python3" shebang as well. (those scripts are standard package entry_points that get installed as dependencies of a python part).

[Sergio Schvezov (sergiusens)]

pyvevnv.cfg is being filtered out, so far seems like the root cause of the issue (https://github.com/python/cpython/blob/8510f430781118d9b603c3a2f06945d6ebc5fe42/Lib/site.py)

This file is required as the v2 python plugin used in core20 behaves more like a virtual environment. There may be legitimate cases to filter this out, so my suggestion is for the plugin V2 API to be extended to have a "get_highly_recommended_files_not_to_filter_out" call (not with that name) to be able to surface this on stage and prime filtering as a warning.

[Alberto Donato (ack)]

Thanks, by adding the pyvenv.cfg to the primed fileset it all works!

[Alberto Donato (ack)]

So, I found a seemingly related issue.

After adding the pyvenv.cfg to the snap, it works fine when installed on focal, but the same snap installed on a bionic container fails, again due to sys.path missing libraries.

This is what I get for sys.path from a `snap run --shell`:

bionic:

# env python3 -c 'import sys; print(sys.path)'
['', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/usr/lib/python3/dist-packages']

focal:

# env python3 -c 'import sys; print(sys.path)'
['', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/snap/canonical-rbac/x1/lib/python3.8/site-packages']

[Alberto Donato (ack)]

My current part looks like this:

parts:
  crbs:
    plugin: python
    source: .
    source-type: local
    requirements:
      - requirements.txt
    build-packages:
      - git
      - libffi-dev
      - libpq-dev
      - python3-setuptools
    stage-packages:
      - libpq5
      - libpython3.8
      - libxml2
    filesets:
      bins:
        - bin/alembic
        - bin/crbs-admin
        - bin/crbs-asyncapi
        - bin/uwsgi
        - bin/snap-helpers*
        - bin/python
        - bin/python3
        - pyvenv.cfg
    prime:
      - $bins
      - etc
      - lib
      - usr/lib
    override-build: |
      set -e
      snapcraftctl build
      snapcraftctl set-version $($SNAPCRAFT_PROJECT_DIR/snap/local/snap-version)
      snap-helpers write-hooks

[Alberto Donato (ack)]

I've also tested removing the filesets/prime entries and copying everything, with the same result

[Sergio Schvezov (sergiusens)]

This should be impossible unless classic confinement is used.

[Sergio Schvezov (sergiusens)]

From a runtime perspective, this should not matter for confined snaps. The only thing I can think of is that the configure hook might not be running under core20 as a base or that an apparmor/seccomp denial is getting in the way (the latter debunked as the issue seems to be sys.path, so running is fine).

[Sergio Schvezov (sergiusens)]

FWIW, I tried this snap on Debian Bullseye (hit a different issue, but got past the import) and A fresh instance of 18.04 through multipass and had no issues.

[Alberto Donato (ack)]

It seems the issue only happens if I build the snap with --destructive-mode, on a focal machine, then install on bionic

[Alberto Donato (ack)]

I think I've found the culprit.
When building with `--destructive-mode` (on focal) the content of the resulting pyvenv.cfg ends up containing:

home = /home/ack/canonical/src/crbs/parts/crbs/build/.ve/bin
include-system-site-packages = false
version = 3.8.2

whereas with a `--use-lxd` build it has

home = /root/stage/bin
include-system-site-packages = false
version = 3.8.2

Note that in the former the .ve/ dir is a virtualenv for development which gets picked up because of the part `source-type: local`.
If I remove it and build with `--destructive-mode`, the pyvenv.cfg is like this:

home = /usr/bin
include-system-site-packages = false
version = 3.8.2

Shouldn't `home` in there always point to /usr/bin as that's where python is in the base used by the snap? Even in the `--use-lxd` case which seems to always work, if there was a different python in /root/stage/bin, it could break things. Or perhaps the home setting is not needed at all, and it can be always set to nothing?

[Alberto Donato (ack)]

So actually the main reason this is breaking is that the `bin/python3` symlink inside the snap is actually pointing to the python3 from the .ve virtualenv.

By changing that to point to `/usr/bin/python3` and updating the `home = /usr/bin` in the pyvenv.cfg, the snap works fine.

[Sergio Schvezov (sergiusens)]

Sorry for not following up on the bug, but this was my suspicion when you mentioned you were using --destructive-mode.

The symlink depends on having staged python or not.
Setting home may not be required, I will need to research more.

Considering that this is a tainted environment, I will mark this bug as low priority. Can you try this work around:

    PATH="/usr/bin:$PATH" snapcraft --destructive-mode

?

[Alberto Donato (ack)]

Yes, removing the virtualenv from the path fixes it, thanks.

I wonder if snapcraft could detect which python to use based on the part or default to /usr/bin/python3, as that's the path in core?

As for the "home" setting, it doesn't seem it actually makes a difference, whether it points to a valid or not valid path.

[Sergio Schvezov (sergiusens)]

Alberto Donato <email address hidden> writes:

> Yes, removing the virtualenv from the path fixes it, thanks.
>
> I wonder if snapcraft could detect which python to use based on the part
> or default to /usr/bin/python3, as that's the path in core?

It could, this was low priority (I personally ran into this myself when working
on the feature). The other interpreter origin to consider is "stage".

> As for the "home" setting, it doesn't seem it actually makes a
> difference, whether it points to a valid or not valid path.

Will look into this more.