Regression: CRC error an i386

[Frederik Sdun]

Binary package hint: gzip

I tried to uncompress expat 2.0.1 from the sourceforge page on my Lucid testsystem and got this error:

gzip: expat-2.0.1.tar.gz: invalid compressed data--crc error

it works on my karmic and hardy machine. file tells me that it had been created on a FAT system:

expat-2.0.1.tar.gz: gzip compressed data, was "expat-2.0.1.tar", from FAT filesystem (MS-DOS, OS/2, NT), last modified: Thu Jun 7 04:34:43 2007

not all files are affected. if you need a file for testing download this one: http://heanet.dl.sourceforge.net/sourceforge/expat/expat-2.0.1.tar.gz

[Frederik Sdun (playya)]

This happens on a i386 system. amd64 works

[Qball Cow (qball-qballcow)]

I can confirm this.

[Jonathan Stone (jonathan-cs)]

Works now.

[Anders Rønningen (andersronningen)]

I have the same issue with the latest updates today.

[Anders Rønningen (andersronningen)]

I tried downloading the file mentioned in the bug description, but that one also fails to unpack.

[Stefan (stefan-redhat-bugs)]

Description of problem:
gzip fails to decompress valid compressed files.

Version-Release number of selected component (if applicable):
gzip-1.3.13-3.fc13.i686

How reproducible:
Always

Steps to Reproduce:
1. wget http://downloads.sourceforge.net/expat/expat-2.0.1.tar.gz
2. gzip -t expat-2.0.1.tar.gz

Actual results:
Fails with "invalid compressed data--crc error"

Expected results:
Test succeeds.

Additional info:
Same file runs fine on F12 box with all updates.

[Stefan (stefan-redhat-bugs)]

The build is broken by gzip-1.3.13-rsync.patch - rebuilding without solves the problem.

[sreedhar sambangi (sreesamb)]

I have the same issue even with updates till May 5

[sreedhar sambangi (sreesamb)]

Hi

I have more info,if the tar.gz file is created in MS-DOS ,its giving this error
I downloaded from the above link and when i gave
$file expat-2.0.1.tar.gz
expat-2.0.1.tar.gz: gzip compressed data, was "expat-2.0.1.tar", from FAT filesystem (MS-DOS, OS/2, NT)
but when i tried to extract it, its giving CRC error,

So i took the same file and copied to ubuntu 8.04 and was able to extract successfully .
Then i created expat-2.0.1.tar.gz on the 8.04 ubuntu ,copied to new ubuntu(lucid) and now the command for
$file expat-2.0.1.tar.gz
is
expat-2.0.1.tar.gz: gzip compressed data, was "expat-2.0.1.tar", from Unix,

I copied it to ubuntu 10.04 and its successfull.

So it means that , this problem is there for the files created in MS-DOS

regards
Sree

[Karel (karel-redhat-bugs)]

I cannot reproduce it on my machine (i686).

`gzip -t expat-2.0.1.tar.gz` succeeds both with F-12 and F-13.

Do you use x86_64?

[Karel (karel-redhat-bugs)]

Ah, you also have i686, sorry.

I also tested it on F-13 x86_64 and `gzip -t expat-2.0.1.tar.gz` works there as well.

Do we have the same expat-2.0.1.tar.gz?

$ md5sum expat-2.0.1.tar.gz
ee8b492592568805593f81f8cdf2a04c expat-2.0.1.tar.gz

[Stefan (stefan-redhat-bugs)]

Yes, same md5sum. Ubuntu seems to have the same bug: https://bugs.launchpad.net/bugs/524366

[Cliff Brake (cliff-brake)]

I'm seeing this as well.

[Cliff Brake (cliff-brake)]

copying a gzip binary built on a i386 system (fails on the 10.04 i386 system) to an older 9.10 system works fine, so it may be something in a library on i386 10.04 systems. It does not appear to be an issue in the gzip binary.

[Cliff Brake (cliff-brake)]

but gzip only depends on libc -- very strange

[Frederik Sdun (playya)]

Maybe it's a bug on the on the packageing side? Maybe we should file a bug in expat's bugtracker to figure out which version/tool they used to compress the files?

[Karel (karel-redhat-bugs)]

The preferred solution for me is to remove the rsync patch. It failed to find its way to the upstream release, and now it causes problems that cannot be easily reproduced.

I'll remove that patch from Rawhide if no fix is found in reasonable time.

[Derek A. Rhodes (physci)]

I'm finding the same symptoms with a SFML sdk, also from SourceForge. it extracts fine on amd64 8.04, but not i386 10.4

http://downloads.sourceforge.net/project/sfml/sfml/1.6/SFML-1.6-sdk-linux-32.tar.gz?use_mirror=internode
md5sum: acc678933c19558587aad8332ea6f459 SFML-1.6-sdk-linux-32.tar.gz

[Charles Manning (cdhmanning)]

Fedora has the same problem:
https://bugzilla.redhat.com/show_bug.cgi?id=588644

The suggestion there is that this is caused by a recent rsyc patch.

[Charles Manning (cdhmanning)]

Is there some way to go back to a previous version of gzip + dependencies apart from rolling back to 9.10?

[Charles Kearney (charkear)]

I need a fix for this as well. A way to go back?

[Charles Manning (cdhmanning)]

There is a manual workaround that works, but is sucky.

sudo apt-get install gzrt
gzrecover foo.tar.gz
That will give you foo.tar.recovered

[Daniel Li (lida-mail-163)]

gzrecover foo.tar.gz

no effect!

How to fix this problem in Lucid?

[Charles Manning (cdhmanning)]

gzrecover foo.tar.gz does not generate a fixed gz file. It will generate a file called foo.tar.recovered which is not a gz file.

If you want to make a new gz file you will have to do something like

gzrecover foo.tar.gz
mv foo.tar.recover new-foo.tar
gzip new-foo.tar

Now you have new-foo.tar.gz which should be OK.

[Badmotorfinger (e-launchpad-bohme-org)]

Same issue - only weirder. Have two machines both recently installed with 10.04. One can handle zlib.1.2.4.tar.gz (from zlib.org) and one cannot. Output sha1sum on the input file, /bin/gzip and its two dependent libraries are identical. Both systems claim to be up to date on packages. Output from strace on both machines compared, nothing notable.

This not an expat issue (we should probably stop bothering them over it ;-)

[Badmotorfinger (e-launchpad-bohme-org)]

Note that it is also - in my case - not related to files created on any particular OS. Output from file:

zlib-1.2.5.tar.gz: gzip compressed data, was "zlib-1.2.5.tar", from Unix, last modified: Tue Apr 20 00:16:41 2010, max compression

[Joost82 (j-rietdijk)]

I have the same problem. But when I used the previous version of expat-2.0.0.tar.gz I have no problems. So what is the difference between these to files?

[Jonathan Crockett (jcrockett)]

I also am seeing the same problem. I have two 10.04 32-bit installs, one on a MBP (2.2 Core2 /2Gb Ram) and the other a Dell XPS (2.6 CoreI7, 6Gb). The Dell has no problem extracting expat-2.0.1.tar.gz, but the MBP does. Both systems are up to date as of now. I downloaded gzip 1.4, built it and replaced /bin/gzip and the problem went away on the MBP.

[Jeff Katz (jeff-katzonline)]

I am seeing the same issue. Great test case: http://zlib.net/zlib-1.2.5.tar.gz

Installing gzip 1.4 from source fixed it. This is embarrassing.

[Thomas Senyk (thomas-senyk)]

Same problem here.

I got it working with:
http://packages.ubuntu.com/hardy/i386/gzip

[rashman00 (ryan-raasch)]

Same problem here on i386 platform. Just do

git clone git://git.savannah.gnu.org/gzip.git

bootstrap, configure, compile and install.

Why is the package causing the problem expat?

[Frederik Sdun (playya)]

I'll create a account to file the bug against upstream

[Marcel Stimberg (marcelstimberg)]

This bug is known and fixed upstream, the fix is in version 1.4.

http://git.savannah.gnu.org/cgit/gzip.git/commit/?id=b9e94c93df914bd1d9eec9f150b2e4e00702ae7b
Quoting from the commit:
gzip -d would fail with a CRC error...
...for some inputs, and some memcpy implementations. It is possible that an offending input has to be compressed "from FAT filesystem (MS-DOS, OS/2, NT)", since the sole reproducer no longer evokes a CRC error when uncompressed and recompressed on a GNU/Linux system. Also, using an unpatched reverse-memcpy-gzip on over 100,000 inputs on a GNU/Linux system did not turn up another reproducer.

Possibly the (one line) fix can be applied to the current Ubuntu version as well? I can't help with that as I'm myself unable to reproduce the problem (on i386 in Virtual Box).

[Marcel Stimberg (marcelstimberg)]

Testing on several 32bit and 64bit machines, I think I now know which combination is necessary to get the bug: A combination of 32bit (i386) with an ssse3 enabled processor. (To check: if "grep ssse3 /proc/cpuinfo" gives some output, the processor has sss3). Therefore I think the likely cause for the bug appearing now in lucid is the change in libc6 from March 21: "Backport 32bit SSSE3/SSE4 optimized memcmp and strcmp from trunk."
I'll test later today whether backporting the fix from the upstream 1.4 version works.

[Marcel Stimberg (marcelstimberg)]

I attached a bzr branch with the backported (even less than one line ;-) ) fix -- this fixes the issue for me. Testing welcome.

[Richard Andrews (bflatmaj7th)]

The patch works for me.
I had a problem with the same expat-2,0.1.tar.gz file of the OP.

I have built up the Ubuntu .deb for Lucid as follows:
 * unpack orig tarball [1]
 * applied ubuntu diff [1]
 * applied Marcel's diff on changelog and inflate.c
 * dpkg-buildpackage -us -uc -nc

[1] from http://packages.ubuntu.com/source/lucid/gzip

To test it I did
cd build
 ./gzip -d --t /path/to/expat-2.0.1.tar.gz

exit status 0

(Note it important to use./gzip as ./gunzip is a script which uses the installed gzip).

My cpuinfo flags are as follows
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts aperfmperf pni dtes64 monitor ds_cpl est tm2 ssse3 cx16 xtpr pdcm xsave lahf_lm

System was Ubuntu Lucid, fully up-to-date as of Jun 25 11:17:29 UTC 2010.

[Richard Andrews (bflatmaj7th)]

I think that gzip is only half the problem. The zlib library still has the fault, so any programs that use zlib compression via libz.so would still be broken.

[Richard Andrews (bflatmaj7th)]

Actually - I haven't tested libz as I don't have a test case, so it might be fine.

[Marcel Stimberg (marcelstimberg)]

Richard, thanks for testing. BTW: Instead of your steps 1-3, you can directly download the branch with
bzr branch lp:~marcelstimberg/ubuntu/lucid/gzip/gzip-fix-524366

I do not think zlib is affected, though. I had a quick glance at the code and it does not seem to use memcpy from libc6 but an unoptimized zmemcpy function. The problem in gzip only surfaced because optimziations in libc6 made use of the two memory ranges not overlapping which gzip did not properly check for.

[Badmotorfinger (e-launchpad-bohme-org)]

The notes about zlib above are *not* that zlib is also affected by this bug. It is that (humorously enough) the zlib tarball from the project page cannot be unpacked because of this bug. It is another sample to use for testing the fix.

[Karel (karel-redhat-bugs)]

Most likely the issue was not caused by the rsync patch.
Via launchpad:
http://git.savannah.gnu.org/cgit/gzip.git/commit/?id=b9e94c93df914bd1d9eec9f150b2e4e00702ae7b

[Fedora (fedora-redhat-bugs)]

gzip-1.3.13-4.fc13 has been submitted as an update for Fedora 13.
http://admin.fedoraproject.org/updates/gzip-1.3.13-4.fc13

[Fedora (fedora-redhat-bugs)]

gzip-1.3.13-4.fc13 has been pushed to the Fedora 13 testing repository. If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with
 su -c 'yum --enablerepo=updates-testing update gzip'. You can provide feedback for this update here: http://admin.fedoraproject.org/updates/gzip-1.3.13-4.fc13

[Stefan (stefan-redhat-bugs)]

gzip-1.3.13-4.fc13 fixes the problem for me.

[Fedora (fedora-redhat-bugs)]

gzip-1.3.13-4.fc13 has been pushed to the Fedora 13 stable repository. If problems still persist, please make note of it in this bug report.

[Martyn Welch (martyn-welch)]

A patch for this bug seems to have been available for over a month now, yet it's still not fixed in 10.04 and it seems the fix isn't currently in the 10.10 tree either (both seem to be at "1.3.12-9ubuntu1").

Has there been any progress with this - I can't move my development system to 10.04 until this is fixed, it's breaking the embedded Linux build system I use.

[MrPok (wiebenik)]

Please roll-out an update for Ubuntu 10.4 soon, as I know of several beginning Ubuntu users who are unable to resort to different methods of getting the contents out of the effected archives. This bug effects everyday basic use of the system;
why is the Importance of this bug still set to "Undecided" ?

[Martin Pitt (pitti)]

Merged and sponsored. Will be accepted into lucid-proposed after 10.04.1 freeze, and then copied to maverick. Thank you!

[Christian Iversen (chrivers)]

I just made a number of tests regarding this bug.

I can confirm that the bug exists, and that it does not appear on amd64.

Furthermore, I've tried using a vanilla gzip 1.3.12 and 1.3.13, both which fail with the same error.

However, both vanilla versions AND ubuntu version work if you set CFLAGS=-DNOMEMCPY, which causes gzip to use byte-by-byte transfers instead of calling memcpy()

In other words, this proves that it cannot have anything to do with the rsync patches. There must be a rarely-triggered bug in memcpy(), which only crops up on 32-bit.

It could be that gzip uses memcpy slightly incorrectly, but I am not going to guess exactly why the bug occurs.

[Marcel Stimberg (marcelstimberg)]

@Christian: I'm not sure what "rsync patches" you are referring to, but you are right about memcpy -- see comments #27 and #28 for an explanation. A fix should arrive in lucid-proposed soon.

[Martin Pitt (pitti)]

Accepted gzip into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Martyn Welch (martyn-welch)]

I pulled in this deb file without enabling all of proposed - I can confirm that this update does fix the issues I was having. Thanks.

[Martin Pitt (pitti)]

Copied from lucid-proposed to maverick.

[geraldo (geraldo-servus)]

The gzip package from proposed repository works fine for me and fixes the issues with gephi installation as described here: https://answers.launchpad.net/gephi/+question/113469. Thanks!

[Martin Pitt (pitti)]

Copied to lucid-updates.

[rgururaj (rgururaj)]

Not sure if this is related to this error, well this issue doesn't seems to be fixed.
I have installed 64 bit Ubuntu lucid, updated with the proposed updates.

Most of the time I get the below CRC error (29 times out of 30), ocassionaly (once out of 30) the unzip succeeds without any error! The unzip on the same archieve works with no issue on ubuntu 9.04 (30 out of 30 times). The archieve is built on windows machine.
     [java] java.util.zip.ZipException: invalid entry CRC (expected 0x2e52b734 but got 0x5cafda91)
     [java] at java.util.zip.ZipInputStream.readEnd(ZipInputStream.java:396)
     [java] at java.util.zip.ZipInputStream.read(ZipInputStream.java:156)
     [java] at java.util.zip.ZipInputStream.closeEntry(ZipInputStream.java:100)
     [java] at java.util.zip.ZipInputStream.getNextEntry(ZipInputStream.java:78)

[Frederik Sdun (playya)]

this bug is not affiliated to you bug. ZipInputStream is a zip stream, not a gzip