python(abi) autodetection needed for all .py[co] files, not just those beneath /usr/lib*/python*

[Jeff Johnson]

Tracker.

[Nils (nils-redhat-bugs)]

Created attachment 438224
Enhanced version of pythondeps.sh which detects the version of Python with which .py[co] files were built and adds the needed "python(abi) = ..." dependency

Description of problem:
Recently, all packages requiring "python(abi) = 2.6" were rebuilt for Python 2.7. Unfortunately, this left out all packages that don't have files in the Python standard module paths. As bug #621726 shows, python modules not in the standard paths will let the new python version attempt to rebuild them which fails due to SELinux policy, if run as root and policy is enforced (which is the right thing to do). If not run as root, startup time of the affected programs might be longer than necessary because python can't simply use the pre-built .py[co] files. In my opinion, the affected packages should simply have the "python(abi) = ..." dependency as well (since the shipped pre-built .py[co] files depend on it). I'll attach a version of /usr/lib/pythondeps.sh which generates such a dependency if .py[co] files are in a package.

Version-Release number of selected component (if applicable):
rpm-0:4.8.1-5.fc15.x86_64
rpm-build-0:4.8.1-5.fc15.x86_64

How reproducible:
Reproducible.

Steps to Reproduce:
A.1.) Install an affected version of e.g. system-config-firewall (1.2.25-1.fc14 from http://koji.fedoraproject.org/koji/buildinfo?buildID=168786 will do)
A.2.) Run it.

B) rpm -q --provides system-config-firewall | grep 'python(abi)'

Actual results:
A) SELinux alerts, as described in bug #621726
B) nothing

Expected results:
A) No SELinux alerts.
B) "python(abi) = ..."

[Nils (nils-redhat-bugs)]

To really work with Python 2.7, this needs file to be patched to recognize byte-compiled files generated by Python 2.7. See bug #623602.

[Orcan (orcan-redhat-bugs)]

But I don't think that is sufficient, as far as I can tell,
   /usr/lib/rpm/pythondeps.sh
does not use file to determine python(abi) type dependencies. Or is this caught by some other script?

[Panu (panu-redhat-bugs)]

The internal dependency generator which eventually calls pythondeps.sh and the like does uses libmagic (ie file) to figure file types, so fixing file does help.

[Jeff Johnson (n3npq)]

the python abi dep has always been crude

[Göran Uddeborg (goeran-uddeborg)]

Maybe this is somewhat debatable. At least if it is to be a hard requirement. If a package contains both a p.py and p.pyc file, and the pyc file doesn't match the python being used, everything will still work, AFAIK. Python will detect that the pyc is for the wrong version, and use the py file instead. Some efficiency is lost, but no functionality.

[Jeff Johnson (n3npq)]

Absolutely. The deeper issue is ensuring that SELinux file contexts are
attached to *.pyc reliably. The chosen path was through a packaging policy

[Jeff Johnson (n3npq)]

(continuing)
     All *.rpm packages should include *.pyc files.
so that rpm's means to attach SELinux's file contexts
could be used. But RPM could easily automate *.pyc
regeneration where needed. The issue would be
behavior with --verify but an additional
      changed because regenerated by rpm while installing
could be added (and filtered as desired) with --verify.

(aside)
Another reason for packaging *.pyc was to ensure that --erase
also removed side effects like generated/changed *.pyc.
The side effect could be handled in other ways during rpm --erase, like a glob on
*.pyc in same directory, and attaching the same file resolution handling to
*.rpmsave/*.rpmorig/*.rpmnew like resolutions other than just calling unlink(2).

[Jeff Johnson (n3npq)]

One final issue:

Tracking python module dependencies has been an ongoing and hotly debated
topic since forever with no implementation in sight. The automagic dependency
exraction that I enabled (from nalin, or at least based on nalin's other autodeps
frameworks) for python was _PURELY_ a place holder to show where better
could and should be done, so I could put a checkmark as "completed" for
an automated dependency extraction framework for python modules.

Short answer:
    Provides: python(abi) = 2.6
is utter smoke-and-mirrors. Better MUST be done for self assembling dependency
assertions used by depsolvers for python-* module tracking.

[Fedora (fedora-redhat-bugs)]

This message is a notice that Fedora 14 is now at end of life. Fedora
has stopped maintaining and issuing updates for Fedora 14. It is
Fedora's policy to close all bug reports from releases that are no
longer maintained. At this time, all open bugs with a Fedora 'version'
of '14' have been closed as WONTFIX.

(Please note: Our normal process is to give advanced warning of this
occurring, but we forgot to do that. A thousand apologies.)

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, feel free to reopen
this bug and simply change the 'version' to a later Fedora version.

Bug Reporter: Thank you for reporting this issue and we are sorry that
we were unable to fix it before Fedora 14 reached end of life. If you
would still like to see this bug fixed and are able to reproduce it
against a later version of Fedora, you are encouraged to click on
"Clone This Bug" (top right of this page) and open it against that
version of Fedora.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

The process we are following is described here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

[Jan (jan-redhat-bugs)]

(In reply to comment #3)
> The internal dependency generator which eventually calls pythondeps.sh and
> the like does uses libmagic (ie file) to figure file types, so fixing file
> does help.

Could you provide more information? The issue in file has been fixed however you reopened the bug. Is this issue somewhere else then?

[Panu (panu-redhat-bugs)]

File getting fixed was a pre-requisite for fixing this on rpm-side. It's the rpm-side of things that still hasn't been fixed (as I had pretty much forgotten the whole thing).

[Fedora (fedora-redhat-bugs)]

This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19

[Fedora (fedora-redhat-bugs)]

This message is a notice that Fedora 19 is now at end of life. Fedora
has stopped maintaining and issuing updates for Fedora 19. It is
Fedora's policy to close all bug reports from releases that are no
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 19 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

[Fedora (fedora-redhat-bugs)]

Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.