can't import lsb_release in python3 anymore

[Colin Keenan]

Ubuntu 12.10 (but grub2 says Debian Wheezy now?)
Python 3.2.3-6ubuntu3.1

Python 3.2.3 (default, Oct 19 2012, 19:53:16)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import lsb_release
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
EOFError: EOF read where not expected
>>>

This is a problem because it causes extlinux to fail. The following printed in reply to sudo aptitude full-upgrade:

The following partially installed packages will be configured:
  extlinux
No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B of archives. After unpacking 0 B will be used.
Setting up extlinux (2:4.05+dfsg-6) ...
P: Checking for EXTLINUX directory... found.
Traceback (most recent call last):
  File "/usr/bin/lsb_release", line 26, in <module>
    import lsb_release
EOFError: EOF read where not expected
dpkg: error processing extlinux (--configure):
 subprocess installed post-installation script returned error exit status 1
No apport report written because MaxReports is reached already
                                                              Errors were encountered while processing:
 extlinux
E: Sub-process /usr/bin/dpkg returned an error code (1)
A package failed to install. Trying to recover:
Setting up extlinux (2:4.05+dfsg-6) ...
P: Checking for EXTLINUX directory... found.
Traceback (most recent call last):
  File "/usr/bin/lsb_release", line 26, in <module>
    import lsb_release
EOFError: EOF read where not expected
dpkg: error processing extlinux (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 extlinux

[Colin Keenan (colinkeenan)]

I have solved this bug myself. These were the steps:

colin@colin:~$ find /usr/lib -name "lsb_release*"
/usr/lib/python2.7/dist-packages/lsb_release.py
/usr/lib/python2.7/dist-packages/lsb_release.pyc
/usr/lib/python3/dist-packages/lsb_release.py
/usr/lib/python3/dist-packages/__pycache__/lsb_release.cpython-32.pyc

colin@colin:~$ sudo rm /usr/lib/python3/dist-packages/__pycache__/lsb_release.cpython-32.pyc

And to verify it worked:

colin@colin:~$ python3
Python 3.2.3 (default, Oct 19 2012, 19:53:16)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import lsb_release
>>> quit()
colin@colin:~$

As you can see, no errors anymore on trying to import lsb_release. Running aptitude full-upgrade worked as well and finished configuring extlinux.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Steve Langasek (vorlon)]

Barry, this seems to be the first "unexpected EOF" bug I've seen where there someone was able to diagnose the problem. I'm seeing a *lot* of this kind of bug report against the lsb package; I'm not convinced that this is just filesystem corruption given the prevalence. Of course since the submitter here has already deleted the corrupt file we unfortunately can't analyze it now, but maybe some submitters of other recent bugs would be able to help?

[Barry Warsaw (barry)]

Actually, I think this bug is in line with many other related bugs Brian has identified in Python 2. All of them seem to come down to corrupted .pyc files, and all can be solved by essentially deleting the corrupted pyc and regenerating it, as is the effect of the steps in comment #1.

What's interesting about this bug is that it's the first so far identified for Python 3, but what's also interesting is that it's reported against Python 3.2 not 3.3. So far, we've seen this bug reported primarily against 2.7. The thing is that 3.2 and 2.7 share the same algorithm for writing pyc files, attempting to get an exclusive open on the file but *not* doing an atomic rename. In Python 3.3, with the switch to importlib, the atomic rename has been added. So far, I haven't seen this bug reported against Python 3.3 so I'm still going with my working hypothesis which is that *something* as yet unidentified is causing a race condition on writing the pyc files, thus truncating or corrupting them. I need to analyze write_compiled_module() in import.c in more detail to see if that's really possible. In parallel, more analysis is needed to try to identify whether there are indeed multiple writers to pyc files possible, e.g. during installation via ubiquity as Colin pointed out in IRC, or because of some later case. There has to be two processes that see the bare .py file and think they need to write a .pyc file at the same time, *and* that the protections against overwriting in import.c are not enough. The dual-writer process is as yet still unidentified.

[Barry Warsaw (barry)]

This (and all the other so far identified similar bugs) is probably this upstream bug:

http://bugs.python.org/issue13146