data loss on sort -u

[C de-Avillez]

[Impact] In some corner cases, 'sort -u' loses data.
[Test Case] http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9780
[Regression Potential] It's coreutils, and a change to a vital system utility. The coreutils test suite should save us from the worst mistakes, but some paranoia would be justified. Perhaps booting a system with this change and using it for a day or two would be wise.

Original report follows:

Last year a bug was reported to upstream coreutils [1] about problems on using 'sort -u'. Until recently, there was no known reproducing process; one such process was published last week [2].

Although triggering the bug can be said to be a non-completely trivial use case, data loss in 'sort' should be looked at as a serious issue.

Today Jim Meyering committed what looks like the final touches of the fix (incidentally also solving a different issue (a free memory read), also in 'sort'. There are 4 commits affected, and they are shown starting in [3].

Given this is a data loss potential, I am opening this bug for all affected releases (this bug affects Coreutils 8.6 onwards -- Precise and Quantal). I am still to run the tests on Lucid, Natty, and Oneiric.

[1] http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9780
[2] http://lists.gnu.org/archive/html/bug-coreutils/2012-08/msg00027.html
[3] http://lists.gnu.org/archive/html/bug-coreutils/2012-08/msg00057.html

[Colin Watson (cjwatson)]

I have run the test case from the bug report on all of hardy, lucid, natty, and oneiric (albeit in slightly stale chroots and only on i386) and have confirmed that none of them suffer from this bug.

[Launchpad Janitor (janitor)]

This bug was fixed in the package coreutils - 8.13-3.2ubuntu1

---------------
coreutils (8.13-3.2ubuntu1) quantal; urgency=low

  * Resynchronise with Debian. Remaining changes:
    - [80] Make 'uname -i -p' return the real processor/hardware, instead of
      unknown.
    - Build-depend on gettext:any instead of on gettext, so that apt-get can
      properly resolve build-dependencies on the tool when cross-building.
  * Backport upstream patches to fix data-loss and free-memory read bugs in
    'sort -u' (LP: #1038468).

coreutils (8.13-3.2) unstable; urgency=low

  * Non-maintainer upload.
  * Don't declare separate build-arch/build-indep targets when they just
    fall through to the same common build rule, since they don't have correct
    target dependencies themselves. Closes: #670481.
  * Enable hardening build flags. Thanks to Moritz Muehlenhoff for the
    patch. Closes: #653743.
  * debian/patches/99_Werror-format-string.dpatch: fix the gnulib test suite
    so that it doesn't fail to build with hardening flags on.
  * Mark coreutils Multi-Arch: foreign. Thanks to Colin Watson.
    Closes: #649397.
  * debian/patches/99_tests-misc-sort-continue-Port-to-Fedora-15.dpatch:
    cherry-pick from upstream to fix a build failure whenever the build
    system happens to be holding an fd open, as happens when running in
    certain environments (e.g., Lucas's test rebuild farm, or bzr-builddeb).
    Closes: #669555.
 -- Colin Watson <email address hidden> Mon, 10 Sep 2012 14:07:50 +0100

[Adam Conrad (adconrad)]

Hello C, or anyone else affected,

Accepted coreutils into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/coreutils/8.13-3ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

There were quite a few test cases in the gnu bug but I used this one:

perl -e 'printf "33\n"x2 ."7\n"x31 ."1\n"' | src/sort -S1 -u

and confirmed that the bug is fixed with coreutils version 8.13-3ubuntu3.1 from precise-proposed.

[Scott Kitterman (kitterman)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package coreutils - 8.13-3ubuntu3.1

---------------
coreutils (8.13-3ubuntu3.1) precise-proposed; urgency=low

  * Backport upstream patches to fix data-loss and free-memory read bugs in
    'sort -u' (LP: #1038468).
 -- Colin Watson <email address hidden> Mon, 10 Sep 2012 14:32:58 +0100