CRC error

[Benjamin Roberson]

A file gzip'd on VM #1 (Ubuntu 12.04.2 server x64, gzip 1.4-1ubuntu2) will give a CRC error while extracting on VM #2 (Ubuntu 12.04.2 server x64, gzip 1.4-1ubuntu2). Extracting the file on VM #1 works as does extracting the file on Ubuntu 10.04.4LTS. Both VM #1 and VM #2 are running ext4 for their root fs.

I can't upload the original file so I created a random text file by concatenating all linux headers. This file causes the same error:
https://s3.amazonaws.com/gzip-bug/kernel.h.gz

Both files have the same sha512sum:
$ sha512sum kernel.h.gz
0ea4226470f4da8f78b6272d65898c70de193b8a934e94ea81e4e86c8e477e7a44c7f45ef287f12db6eba6073c4726ab9373c883b2595011c9998dc10bff4db5 kernel.h.gz

The actual error looks like:
$ gunzip kernel.h.gz

gzip: kernel.h.gz: invalid compressed data--crc error

===================
==== VM #1 details ====
===================

### Ubuntu release:
$ lsb_release -rd
Description: Ubuntu 12.04.2 LTS
Release: 12.04

### Gzip version:
$ apt-cache policy gzip
gzip:
  Installed: 1.4-1ubuntu2
  Candidate: 1.4-1ubuntu2
  Version table:
 *** 1.4-1ubuntu2 0
        500 http://us.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
        100 /var/lib/dpkg/status

## CPUinfo
$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Core(TM)2 Duo CPU E7600 @ 3.06GHz
stepping : 10
microcode : 0xa07
cpu MHz : 3051.721
cache size : 3072 KB
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 sse4_1 x2apic xsave hypervisor lahf_lm dtherm
bogomips : 6103.44
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Core(TM)2 Duo CPU E7600 @ 3.06GHz
stepping : 10
microcode : 0xa07
cpu MHz : 3051.721
cache size : 3072 KB
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 sse4_1 x2apic xsave hypervisor lahf_lm dtherm
bogomips : 6103.44
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

## other VM details
VMware Fusion 4 guest

===================
==== VM #2 details ====
===================

## Ubuntu release:
$ lsb_release -rd
Description: Ubuntu 12.04.2 LTS
Release: 12.04

## Gzip version:
$ apt-cache policy gzip
gzip:
  Installed: 1.4-1ubuntu2
  Candidate: 1.4-1ubuntu2
  Version table:
 *** 1.4-1ubuntu2 0
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
        100 /var/lib/dpkg/status

$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5550 @ 2.67GHz
stepping : 5
microcode : 0x11
cpu MHz : 2666.760
cache size : 8192 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu tsc msr pae cx8 cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5333.52
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

## other VM details
AMI: ubuntu/images/ebs/ubuntu-precise-12.04-amd64-server-20130124 (ami-3fec7956)
Zone: us-east-1c
Type: m2.xlarge

[Bryan Fullerton (fehwalker)]

Can you confirm that the gunzip that's being run on both machines is /bin/gunzip, and that their checksum is the same?

There is also a /usr/lib/klibc/bin/gunzip that's part of klibc-utils, and it's quite an old version.

If that's all ok, is it possible to post a test file smaller than 344Mb, or does this issue only happen with large files?

[Benjamin Roberson (benjamin-roberson)]

I can confirm that both VMs are using /bin/gunzip and have the same checksum.

$ which gunzip
/bin/gunzip

$ sha512sum /bin/gunzip
483beae8cd211cff3bfa056bf573f0f0fa068eb0872924da9d01537232fa815d7bfd817bfb411fea65f20ae71e204256e0cc0255b06c1363fd4f80d918b2d76c /bin/gunzip

I tried with a smaller file 'kernel_headers.gz' and I cannot recreate the bug. The extraction works on VM #1 and VM #2, the file is 87M compressed and 383M uncompressed. The original file that causes the CRC error is 1531M uncompressed and 344M compressed.

I also started a new VM (#3) in the same EC2 zone, same AMI/Type, same checksum of /bin/gunzip, and tried expanding the original file ('kernel.h.gz' with the same sha512sum). It works. I was attempting to exclude the CPU as a factor and though it has the same CPU, the flags listed are different:

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5550 @ 2.67GHz
stepping : 5
microcode : 0x11
cpu MHz : 2666.760
cache size : 8192 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu de tsc msr pae cx8 sep cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl nonstop_tsc pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm dtherm
bogomips : 5333.52
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

[Benjamin Roberson (benjamin-roberson)]

The smaller file that will successfully expand on all systems: https://s3.amazonaws.com/gzip-bug/kernel_headers.gz