python plugin munges too many shebangs causing conflicts

[Stuart Bishop]

The Python plugin munges the shebang lines of all files under the installation directory. This causes conflicts with other parts, which have the unmunged versions of the files pulled in from packages.

Excluding usr/lib/python3.5 would probably fix this.

Parts 'postgresql' and 'pgsnap' have the following file paths in common which have different contents:
    usr/lib/python3.5/base64.py
    usr/lib/python3.5/cProfile.py
    usr/lib/python3.5/cgi.py
    usr/lib/python3.5/encodings/rot_13.py
    usr/lib/python3.5/idlelib/PyShell.py
    usr/lib/python3.5/keyword.py
    usr/lib/python3.5/lib2to3/pgen2/token.py
    usr/lib/python3.5/pdb.py
    usr/lib/python3.5/platform.py
    usr/lib/python3.5/profile.py
    usr/lib/python3.5/pydoc.py
    usr/lib/python3.5/quopri.py
    usr/lib/python3.5/smtpd.py
    usr/lib/python3.5/smtplib.py
    usr/lib/python3.5/symbol.py
    usr/lib/python3.5/tabnanny.py
    usr/lib/python3.5/tarfile.py
    usr/lib/python3.5/test/pystone.py
    usr/lib/python3.5/test/regrtest.py
    usr/lib/python3.5/timeit.py
    usr/lib/python3.5/trace.py
    usr/lib/python3.5/uu.py
    usr/lib/python3.5/webbrowser.py

[Stuart Bishop (stub)]

(the above failure was triggered by having one part using the python plugin, and another part listing 'vim' in its staging-packages. Switching to vim-tiny worked around the issue)

[Sergio Schvezov (sergiusens)]

El 06/10/16 a las 10:25, Stuart Bishop escribió:
> (the above failure was triggered by having one part using the python
> plugin, and another part listing 'vim' in its staging-packages.
> Switching to vim-tiny worked around the issue)
>

This is a tricky one, all the files in there are runnable, if we remove
the header change they will go to the wrong python interpreter when run.

[Stuart Bishop (stub)]

Technically, yes. In practice, its pretty unlikely anyone would execute them. But timeit.py or tabnanny may get hit sometimes. I think nowdays you are supposed to use python3 -m pdb.py.

If the munging is needed, shouldn't we be doing it for all parts rather than just the ones using the Python plugin? Some sort of post-processing done on the staging area before priming? Parts using the Python plugin would probably import the libraries directly, so the problem we are attempting to avoid more likely to be triggered by other parts pulling in the Python packages.

[Sergio Schvezov (sergiusens)]

El 06/10/16 a las 17:14, Stuart Bishop escribió:
> Technically, yes. In practice, its pretty unlikely anyone would execute
> them. But timeit.py or tabnanny may get hit sometimes. I think nowdays
> you are supposed to use python3 -m pdb.py.
>
> If the munging is needed, shouldn't we be doing it for all parts rather
> than just the ones using the Python plugin? Some sort of post-processing
> done on the staging area before priming? Parts using the Python plugin
> would probably import the libraries directly, so the problem we are
> attempting to avoid more likely to be triggered by other parts pulling
> in the Python packages.
>

 From memory, I thought we were doing this in our 'repo' module which
does stage-packages handling, I would need to check but it already does
"fixup" for many things coming from the (Ubuntu) archive

[Kyle Fazzari (kyrofa)]

I just hit this as well. Sergio, we rewrite the python shebangs in the `build` step of the python plugin. Perhaps we should indeed be handling this on a package basis in the repo instead, so any python pulled in (regardless of method: stage-packages or plugin) is runnable. Also, they'll be hacked the same way, so they don't conflict :P .