Intermittent crashes copying files involving symbolic links

Bug #770566 reported by John S. Gruber on 2011-04-25
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Python
Fix Released
Unknown
python-distutils-extra
Medium
Martin Pitt
python-distutils-extra (Ubuntu)
Medium
Martin Pitt
python2.7 (Ubuntu)
Undecided
Unassigned

Bug Description

The approach used by python-distutils-extra to install icon (and perhaps other) symbolic links can proceed or crash depending upon the random order in which icon file copies are executed.

The crash happens while trying to create the debian/tmp/ file tree when apport's make processess calls python-distutils-extra through setup.py using the following command:

  python setup.py install \
  --root="/home/gruber/packages/uapport/debian/tmp/" \
  --install-purelib=/usr/lib/python2.7/site-packages/ \
  --no-compile --install-layout=deb

For this case I received the following message from pbuilder while trying to build the apport package for ubuntu:
copying data/icons/scalable/mimetypes/text-x-apport.svg -> /tmp/buildd/apport-1.20.1/debian/tmp/usr/share/icons/hicolor/scalable/mimetypes
error: /tmp/buildd/apport-1.20.1/debian/tmp/usr/share/icons/hicolor/scalable/mimetypes/text-x-apport.svg: No such file or directory
make: *** [install/apport] Error 1
dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2
-----------------
There are icon files in two different directories, one a symbolic link to the other.

In particular the source icon files are:

-rw-r--r-- 1 gruber gruber 16628 2011-04-25 12:52:10.493242000 -0400 data/icons/scalable/apps/apport.svg
lrwxrwxrwx 1 gruber gruber 18 2011-04-25 12:52:10.409823944 -0400 data/icons/scalable/mimetypes/text-x-apport.svg -> ../apps/apport.svg

these are to be copied by python-distutils to:
lrwxrwxrwx 1 gruber gruber 18 2011-04-25 17:44:10.579475687 -0400 debian/tmp/usr/share/icons/hicolor/scalable/mimetypes/text-x-apport.svg -> ../apps/apport.svg (1) and
lrwxrwxrwx 1 gruber gruber 18 2011-04-25 17:44:10.579475687 -0400 debian/tmp/usr/share/icons/hicolor/scalable/apps/apport.svg (2)

python-distutils-extra makes the symbolic link itself at the very end of auto.py, making destination directory for the symbolic link (1) in the process. python-distutils-extra command/build_icons.py add both source and destination files to distribution.data_files.

python-distutils then has the responsibility to finish the work on these icons. If the above entries were added by command/build_icons.py with the entry for the target icon file first, the build process completes. When file_util.py gets to the data_file entry for the target it builds its directory and then copies apport.svg to it. Next it gets to the entry for the mimetypes directory and checks to see if it is up-to-date and one way or the other proceeds.*

If the two data_file entries are produced in the opposite order file_util.py checks for the existence of destination directory (1) and then tries to find out whether the file is up-to-date. It isn't aware of the symbolic link and doesn't check for the existence of destination directory (2), which doesn't yet exist, so the newer function call fails when the attempt to stat the destination file is made. While python-distutils has functions for producing hard and symbolic links that code is not being accessed for this operation.

On my system the mimetypes directory has always come first and the build always fails. On launchpad the apps directory (target of the symbolic link) is usually listed first and normally the build proceeds without event.

The order of the entries is determined by glob.glob() calls in command/build_icons.py. The documentation for glob.glob() (http://docs.python.org/library/glob.html) says the python library uses the os.listdir() function to produce its results, and http://docs.python.org/library/os.html says that "The list is in arbitrary order." so whether the setup.py command and the build completes or not is also arbitrary.

* There is a second difference in execution between pbuilder and launchpad's build farm. The build farm keeps file modification time in seconds and python-distutils' dep_util.py uses subsecond floating point stat.st_mtime values around line 85 while copy times are preserved in file_util.py using stat[ST_MTIME], an integer value. This doesn't matter on an ext3 file system but causes the destination file to look older than the original on an ext4 file system, which tracks subsecond times (note the file times above). The symbolic-link-unaware copy process in this case erases the symbolic link from destination directory (1) and replaces it with the actual source file (replaces the symbolic link with the actual file). Using the comparison at the precision of 1 second is the better way to solve this to avoid floating point precision problems from continuing to confuse the "newer" function in dep_util.py.

description: updated
Changed in ubuntu:
assignee: nobody → John S. Gruber (jsjgruber)
Changed in python-distutils-extra:
assignee: nobody → John S. Gruber (jsjgruber)
affects: ubuntu → python-distutils-extra (Ubuntu)
Martin Pitt (pitti) on 2011-04-26
Changed in python-distutils-extra:
status: New → Triaged
Changed in python-distutils-extra (Ubuntu):
status: New → Triaged
Changed in python-distutils-extra:
importance: Undecided → Medium
Changed in python-distutils-extra (Ubuntu):
importance: Undecided → Medium
Martin Pitt (pitti) wrote :

Thanks for the analysis! I can reproduce this by fudging build_icons.run() to reverse() the glob result first.

This seems to be easier to reproduce by installing a broken symlink to a nonexisting directory. I added a test for this to test/auto.py.

Changed in python-distutils-extra:
assignee: John S. Gruber (jsjgruber) → Martin Pitt (pitti)
status: Triaged → In Progress
Martin Pitt (pitti) wrote :

I fixed this in trunk now. Took a bit more effort, but I'm glad that it's finally fixed for good:

http://bazaar.launchpad.net/~python-distutils-extra-hackers/python-distutils-extra/debian/revision/250

Changed in python-distutils-extra:
status: In Progress → Fix Released
Changed in python-distutils-extra (Ubuntu):
assignee: John S. Gruber (jsjgruber) → Martin Pitt (pitti)
status: Triaged → Fix Committed
Éric Araujo (merwok) wrote :

The st_mtime vs. ST_MTIME resolution problem has been reported upstream at http://bugs.python.org/issue11933 and we’re working on a fix.

Changed in python:
status: Unknown → New
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-distutils-extra - 2.27-1

---------------
python-distutils-extra (2.27-1) unstable; urgency=low

  * auto.py: Cleaner approach to determine whether a module is locally
    provided. This now catches all relative imports. (LP: #746565)
  * Fix installation of symlinks in data/ dir (LP: #770566):
    - test/auto.py: Add test for installing a symlink which points to a
      nonexisting target directory/file. This reproduces the gist of the
      problem.
    - test/auto.py: Preserve symlinks in copytree() calls, so that we can
      actually verify that symlinks are preserved properly.
    - test/auto.py: Drop requirement that diff throws no error messages, as it
      will complain about the broken symlink.
    - DistUtilsExtra/auto.py, install_auto: Use os.walk() instead of
      distutils.filelist.findall() to pick out symlinks, as the latter fails
      badly with broken symlinks.
    - DistUtilsExtra/command/build_icons.py: Ignore symbolic links. distutils
      breaks on them when they point to a nonexisting target, and we handle
      them in auto.py.
  * DistUtilsExtra/auto.py: Put back "except .. as" syntax, now that we
    dropped Python 2.5. Drop 2to3 from debian/rules for good now.
  * debian/control: Bump minimal Python version to >= 2.6.
 -- Ubuntu Archive Auto-Sync <email address hidden> Sun, 22 May 2011 13:55:08 +0000

Changed in python-distutils-extra (Ubuntu):
status: Fix Committed → Fix Released
Matthias Klose (doko) on 2011-05-26
Changed in python2.7 (Ubuntu):
status: New → Confirmed
Changed in python:
status: New → Fix Released
Matthias Klose (doko) wrote :

fixed in python 2.7.3 in precise

Changed in python2.7 (Ubuntu):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.