Race condition in py_compile corrupts pyc files

[Chris Weiss]

yeay for automatic bug reporting!

this is happening pretty regular, I have no idea what's triggering it, a cron job maybe?

if I run it manually, it get:

~$ sudo do-release-upgrade
Traceback (most recent call last):
  File "/usr/bin/do-release-upgrade", line 8, in <module>
    from DistUpgrade.DistUpgradeVersion import VERSION
EOFError: EOF read where not expected

ProblemType: Crash
DistroRelease: Ubuntu 12.10
Package: ubuntu-release-upgrader-core 1:0.181
ProcVersionSignature: Ubuntu 3.5.0-16.25-generic 3.5.4
Uname: Linux 3.5.0-16-generic x86_64
NonfreeKernelModules: fglrx
ApportVersion: 2.5.3-0ubuntu1
Architecture: amd64
Date: Sat Sep 29 15:11:46 2012
ExecutablePath: /usr/bin/do-release-upgrade
InterpreterPath: /usr/bin/python3.2mu
PackageArchitecture: all
ProcCmdline: /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q
ProcEnviron:
 TERM=linux
 PATH=(custom, no user)
PythonArgs: ['/usr/lib/ubuntu-release-upgrader/check-new-release', '-q']
SourcePackage: ubuntu-release-upgrader
Title: do-release-upgrade crashed with EOFError in /usr/lib/ubuntu-release-upgrader/check-new-release: EOF read where not expected
Traceback:
 Traceback (most recent call last):
   File "/usr/lib/ubuntu-release-upgrader/check-new-release", line 8, in <module>
     from DistUpgrade.DistUpgradeVersion import VERSION
 EOFError: EOF read where not expected
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

VarLogDistupgradeAptclonesystemstate.tar.gz: Error: command ['cat', '/var/log/dist-upgrade/apt-clone_system_state.tar.gz'] failed with exit code 1: cat: /var/log/dist-upgrade/apt-clone_system_state.tar.gz: No such file or directory

[Brian Murray (brian-murray)]

I wonder if something is wrong with /usr/lib/python3/dist-packages/DistUpgrade/DistUpgradeVersion.py on your system. The md5sum for it should be 5f632f7debefec8464862c418ee0f3ae. Could you verify the md5sum and if it is incorrect please reinstall the package python3-distupgrade? Thanks in advance.

[Chris Weiss (cweiss)]

I get 431cb3e648b99dde8665698b58109f9f but it is working now. I'll try re-installing when I get time.

[Chris Weiss (cweiss)]

issue has not returned, must have been fixed by some other commit

[Launchpad Janitor (janitor)]

[Expired for ubuntu-release-upgrader (Ubuntu) because there has been no activity for 60 days.]

[dmiranda (dmiranda)]

I had this bug on two machines.
Reinstalling python3-distupgrade corrects the bug on both.

[Brian Murray (brian-murray)]

I did a search for some other EOFErrrors and found the following:

bdmurray@bizarro:/mnt/storage/bug-attachments/update-manager$ grep -r 'EOFError' *
bug-1035192/Description.txt:Title: update-manager crashed with EOFError in /usr/lib/python3/dist-packages/UpdateManager/UpdatesAvailable.py: EOF read where not expected
bug-1035192/Traceback.txt:EOFError: EOF read where not expected
bug-1039783/Description.txt:Title: update-manager crashed with EOFError in /usr/lib/python3/dist-packages/UpdateManager/backend/InstallBackendAptdaemon.py: EOF read where not expected
bug-1039783/Traceback.txt:EOFError: EOF read where not expected
bug-1061108/Description.txt:Title: update-manager crashed with EOFError in /usr/lib/python3/dist-packages/UpdateManager/InstallProgress.py: EOF read where not expected
bug-1061108/Traceback.txt:EOFError: EOF read where not expected
bug-1093662/Description.txt:EOFError: EOF read where not expected
bug-853581/Description.txt:Title: update-manager crashed with EOFError in /usr/lib/python2.7/dist-packages/UpdateManager/GtkProgress.py: EOF read where object expected
bug-853581/Traceback.txt:EOFError: EOF read where object expected
bug-950006/Description.txt:EOFError: EOF read where not expected
bug-950006/Description.txt:EOFError: EOF read where not expected
bug-950006/Description.txt:EOFError: EOF read where not expected

So I wonder if this error is indicative of a more general problem somewhere.

[Barry Warsaw (barry)]

This message comes from Python's built-in 'marshal' module, and it happens when EOF occurs unexpectedly while unmarshalling a string object. Marshal is the technology that underlies .pyc files, so this is only going to happen if you have corrupt .pyc files.

I have no idea what could be causing your .pyc files to be corrupted, but it's unlikely to be a problem in Python or do-release-upgrade. Perhaps you have a bad disk or some other hardware problem, or perhaps you're running on a weird or corrupted file system. E.g. problems with an encrypted disk or RAID could cause this. It's possible that a full file system at the time of package installation or upgrade could also cause such problems.

There's probably nothing we can do about it except to recommend that you re-install the affected packages. This will cause Python to re-byte compile the py files into new pyc files, hopefully fixing the corruption.

I'm going to mark this as Incomplete, but if you find a way to reliably reproduce this, and can discount hardware or filesystem problems, feel free to re-open it. If you can identify a corrupt .pyc file, please attach it - it would be interesting to pick that apart.

[Chris Weiss (cweiss)]

I don't agree that it's incomplete. there exists possibility that python3-distupgrade will fail to properly install but not provide any error, despite the md5sum being wrong. re-installing corrects it.

I can say with absolute certainty that my hardware was not at fault. no other packages had an issue, and re-install worked for reasons I do not understand.

the question is, why does it allow a file to have the wrong md5sum and not at least complain about it?

[Barry Warsaw (barry)]

Note that .pyc files are not included in the packaging, they are produced at install time, so they don't contribute to any checksums. The .py files do, but these aren't what trigger the error message.

The reason for this is that at package build time, you don't know what versions of Python may be installed on the target system, and .pyc files are Python version specific. IOW, a .pyc file built for Python 3.2 is not guaranteed to work for 3.3.

AFAICT, what must have happened is that when a package was installed, and its pyc file was being written, *something* caused the pyc file to be truncated. The question is what though? Re-installing fixes the problem because the .pyc files are re-built at that time. Thus the corruption went away.

If it's not a file system problem or a hardware problem, I'm not sure what it could be. Another thing to note is that Python does its best to write a new .pyc file atomically, so I can't see how this could be a race condition (e.g. one process trying to read a .pyc file while the installation process is writing it). Plus, if the write fails you won't get a bogus .pyc file *and* you'd get an exception.

[Barry Warsaw (barry)]

Actually: http://bugs.python.org/issue16384

And I think I'm wrong. pyc files are written atomically in Python 3.3, but afaict that's not the case in Python 3.2. The import machinery was rewritten in Python for 3.3. So this could be a race condition for earlier Pythons.

[Barry Warsaw (barry)]

I still think this isn't the whole story. Even if pyc files aren't written atomically before Python 3.3, it still seems like the race condition window is exceedingly small. I don't have any brilliant ideas about what could be going wrong and haven't ever seen it myself or know how to reproduce it.

[Brian Murray (brian-murray)]

Only three of the similar bugs have dmesg information (which may show hardware errors) included and there was nothing denoting a hardware error in them. However, it makes me think we should be collecting something similar to currentdmesg with release upgrade bug reports.

[Brian Murray (brian-murray)]

This is also being tracked in bug 1093071.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python3.3 - 3.3.2-1ubuntu3

---------------
python3.3 (3.3.2-1ubuntu3) saucy; urgency=low

  * debian/patches/atomic-pyc-rename.diff: Add patch to fix possible race
    conditions when writing .pyc/.pyo files in py_compile.py.
    Issue #13146. LP: #1058884
 -- Barry Warsaw <email address hidden> Tue, 21 May 2013 15:13:37 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package python2.7 - 2.7.5-1ubuntu2

---------------
python2.7 (2.7.5-1ubuntu2) saucy; urgency=low

  * debian/patches/atomic-pyc-rename.diff: Add patch to fix possible race
    conditions when writing .pyc/.pyo files in py_compile.py.
    Issue #13146. LP: #1058884
 -- Barry Warsaw <email address hidden> Tue, 21 May 2013 11:08:08 -0400

[Colin Watson (cjwatson)]

Hello Chris, or anyone else affected,

Accepted python2.7 into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/python2.7/2.7.3-0ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Colin Watson (cjwatson)]

Hello Chris, or anyone else affected,

Accepted python2.7 into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/python2.7/2.7.3-5ubuntu4.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Colin Watson (cjwatson)]

Hello Chris, or anyone else affected,

Accepted python2.7 into raring-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/python2.7/2.7.4-2ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Colin Watson (cjwatson)]

Hello Chris, or anyone else affected,

Accepted python3.2 into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/python3.2/3.2.3-0ubuntu3.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Colin Watson (cjwatson)]

Hello Chris, or anyone else affected,

Accepted python3.2 into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/python3.2/3.2.3-6ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Colin Watson (cjwatson)]

Hello Chris, or anyone else affected,

Accepted python3.3 into raring-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/python3.3/3.3.1-1ubuntu5.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Ubuntu Foundations Team Bug Bot (crichton)]

As a part of the Stable Release Updates quality process a search for Launchpad bug reports using the version of python2.7 from raring-proposed was performed and bug 1198439 was found. Please investigate this bug report to ensure that a regression will not be created by this SRU. In the event that this is not a regression remove the "verification-failed" tag from this bug report and tag 1198439 "bot-stop-nagging". Thanks!

[Barry Warsaw (barry)]

More information requested in LP: #1198439

[Dick (oldcoot999)]

I don't have a clue what all the above chatter is all about.
What I do know is that Update Manager won't run -
I usually get several "internal system error" messages constantly popping up
so with all this happening 12.04 is almost useless.
I have had to switch to XP several times because 12.04 is so unstable.

[Chris Weiss (cweiss)]

I've done a 12.10, and now 13.04 update since, the issue has not repeated.

[Dick (oldcoot999)]

I would be glad to update to 12.10 and then 13.04 but my UPDATE MANAGER won't run. It can't find "dameon".

12.04 has been total trash!!!

[Brian Murray (brian-murray)]

I went through a few crashes with EOFErrors on them on errors.ubuntu.com today and did not find any with the python packages from -proposed in their dependencies. I did this by checking the individual instances of all the "web_link"s below.

lsb.json: "function": "/usr/bin/lsb_release:EOFError:/usr/bin/lsb_release@26",
lsb.json- "package": "lsb-release",
lsb.json- "web_link": "https://errors.ubuntu.com/problem/18b4f75d870cd066460324c37154f11285d298f5"
--
lsb.json: "function": "/usr/bin/lsb_release:EOFError:/usr/bin/lsb_release@20:/usr/lib/python2.7/optparse.py@90:/usr/lib/python2.7/gettext.py@49",
lsb.json- "package": "lsb-release",
lsb.json- "web_link": "https://errors.ubuntu.com/problem/16c54c3a70159920d5ab9ae1383f49eb631b76f2"
--
lsb.json: "function": "/usr/bin/lsb_release:EOFError:/usr/bin/lsb_release@22:/usr/lib/python3.3/subprocess.py@345:/usr/lib/python3.3/traceback.py@3:/usr/lib/python3.3/linecache.py@10:_find_and_load:_find_and_load_unlocked:_check_name_wrapper:load_module:load_module:module_for_loader_wrapper:_load_module:get_code",
lsb.json- "package": "lsb-release",
lsb.json- "web_link": "https://errors.ubuntu.com/problem/f6936f5fd60998c7f615667ec7989d4b159543ee"
--
ubuntuone-control-panel.json: "function": "/usr/bin/ubuntuone-control-panel-qt:EOFError:/usr/bin/ubuntuone-control-panel-qt@29:/usr/lib/python2.7/dist-packages/ubuntuone-control-panel/ubuntuone/controlpanel/gui/qt/main/__init__.py@24:/usr/lib/python2.7/dist-packages/ubuntuone-control-panel/ubuntuone/controlpanel/gui/qt/gui.py@27:/usr/lib/python2.7/dist-packages/ubuntuone-control-panel/ubuntuone/controlpanel/gui/qt/ui/mainwindow_ui.py@53:/usr/lib/python2.7/dist-packages/ubuntuone-control-panel/ubuntuone/controlpanel/gui/qt/controlpanel.py@46:/usr/lib/python2.7/dist-packages/ubuntuone-control-panel/ubuntuone/controlpanel/gui/qt/ui/controlpanel_ui.py@221:/usr/lib/python2.7/dist-packages/ubuntuone-control-panel/ubuntuone/controlpanel/gui/qt/devices.py@34:/usr/lib/python2.7/dist-packages/ubuntuone-control-panel/ubuntuone/controlpanel/gui/qt/ui/devices_ui.py@71:/usr/lib/python2.7/dist-packages/ubuntuone-control-panel/ubuntuone/controlpanel/gui/qt/device.py@37:/usr/lib/python2.7/dist-packages/ubuntuone-control-panel/ubuntuone/controlpanel/gui/qt/ui/device_ui.py@47",
ubuntuone-control-panel.json- "package": "ubuntuone-control-panel-qt",
ubuntuone-control-panel.json- "web_link": "https://errors.ubuntu.com/problem/0b65999c4a087ea0e6fd5c586a38e32a95a4a30d"
--
unattended-upgrades.json: "function": "/usr/bin/unattended-upgrade:EOFError:/usr/bin/unattended-upgrade@46",
unattended-upgrades.json- "package": "unattended-upgrades",
unattended-upgrades.json- "web_link": "https://errors.ubuntu.com/problem/819751911a6c78ec3743746b501da978004aa424"
--
unattended-upgrades.json: "function": "/usr/share/unattended-upgrades/unattended-upgrade-shutdown:EOFError:/usr/share/unattended-upgrades/unattended-upgrade-shutdown@40",
unattended-upgrades.json- "package": "unattended-upgrades",
unattended-upgrades.json- "web_link": "https://errors.ubuntu.com/problem/cc0ec656f55a79db561c64a36116cea90600547e"
--
unatten...

[hebein (gunther-hebein)]

Ich bin bis einschliesslich 25.9. ausser Landes. Ich werde mich um Ihr Anliegen nach meiner Rückkehr so schnell als möglich kümmern.

Mit freundlichen Grüßen,
Gunther Hebein

[Rolf Leggewie (r0lf)]

I'm willing to test this for precise but the bug lacks a test case!

[Dimitri John Ledkov (xnox)]

It's racy, thus no reliable way to reproduce.