Apt and Soyuz generating release files with invalid SHA256 signatures

[Ryan Hass]

The following are just a two examples of the errors in the Release file.

http://archive.ubuntu.com/ubuntu/dists/hardy/Release:
 40d48bc44164c1da2a6dd1f503b55a3634d1b5ddee0d6ddd3ce16d7e7688c938 1178004 main/binary-i386/Packages.bz2
 baa89858c7e545390273530ba63c61b94c2e09d38c28b0a0311bfa7bde396181 1517892 main/binary-i386/Packages.gz

Actual:
$ sha256sum main/binary-i386/Packages.bz2
0d9685d8353341ebbd8de9370bb61c1630d4c0a46420bbebabae4197981523a4 main/binary-i386/Packages.bz2
5b906ae167349ecd6699f39ed22e9e98221f780f3a6b15b6443d11b8726270ff main/binary-i386/Packages.gz

[John Schofield (jschofield)]

I tested this out to confirm Ryan's findings:

I downloaded http://archive.ubuntu.com/ubuntu/dists/hardy/Release and http://archive.ubuntu.com/ubuntu/dists/hardy/main/binary-amd64/Packages.gz

"md5sum Packages.gz" matches the md5sum data in the Release file. "sha256sum Packages.gz" does NOT match. These are the correct files, but the sha256 hashes do not appear to be being generated correctly.

This is using versions 6.10 of both md5sum and sha256sum, and Release and Packages.gz downloaded from archive.ubuntu.com at 14:51 PDT today.

(Full disclosure: Ryan and I work for the same organization.)

[Kees Cook (kees)]

Thanks for the report. I'm poking at it myself now, and have forwarded it to the Soyuz folks.

[Celso Providelo (cprov)]

All SHA256 after 'feisty' are wrong, we are using apt_pkg.sha256sum() to generate them and it is broken (!)

{{{
>>> import apt_pkg
>>> apt_pkg.sha256sum(open('Packages.gz').read())
'baa89858c7e545390273530ba63c61b94c2e09d38c28b0a0311bfa7bde396181'
>>>
>>> from subprocess import call
>>> call(['sha256sum', 'Packages.gz'])
5b906ae167349ecd6699f39ed22e9e98221f780f3a6b15b6443d11b8726270ff Packages.gz
0
>>>
>>> from Crypto.Hash import SHA256
>>> print SHA256.new(open('Packages.gz').read()).hexdigest()
5b906ae167349ecd6699f39ed22e9e98221f780f3a6b15b6443d11b8726270ff
}}}

[Francois-Denis Gonthier (fdgonthier)]

Wrong checksum during receive of 'http://archive.ubuntu.com/ubuntu/dists/gutsy/main/binary-i386/Packages.gz':
sha256 expected: baa89858c7e545390273530ba63c61b94c2e09d38c28b0a0311bfa7bde396181, got: af96b1f3119c4ce4b0c6183750279bf7cbdfe62581289f03ad360787e79f968b
There have been errors!

Same problem in Gutsy.

[Michael Vogt (mvo)]

Thanks for your bugreport.

The usage of sha256sum() with file objects is generally prefered, so
 apt_pkg.sha256sum(open('Packages.gz')
should work.

However, there is a bug here somewhere because the string usage should work as well. It does in my tests for small strings, it might be a problem in the python<->c++ code with the string length, I'm investigating that now.

[Michael Vogt (mvo)]

Testing the string use-case shows that it goes wrong on 253 bytes:

$ dd if=/dev/urandom of=foo bs=1 count=252 ; python -c 'import apt_pkg; print apt_pkg.sha256sum(open("foo").read())' ; sha256sum foo
252+0 records in
252+0 records out
252 bytes (252 B) copied, 0,00294077 s, 85,7 kB/s
83c762165fbec99d6fd590ed2d3b291d40bfa8525c97b391d2cfb661c27e25fa
83c762165fbec99d6fd590ed2d3b291d40bfa8525c97b391d2cfb661c27e25fa foo

$ dd if=/dev/urandom of=foo bs=1 count=252 ; python -c 'import apt_pkg; print apt_pkg.sha256sum(open("foo").read())' ; sha256sum foo
253+0 records in
253+0 records out
253 bytes (253 B) copied, 0,00317775 s, 79,6 kB/s
976fc1a77523e602fd1fe36d13771d83bae61f8e5d5279ca97b158664ff8b8c8
d68a24e86b8037437a20a592a717c40c163127f4942b511b102f0b11e449794c foo

[Michael Vogt (mvo)]

It turns out that it is not as easy as this:

$ dd if=/dev/urandom of=foo bs=1 count=270 ; python -c 'import apt_pkg; print apt_pkg.sha256sum(open("foo").read())' ; sha256sum foo
270+0 records in
270+0 records out
270 bytes (270 B) copied, 0,00338191 s, 79,8 kB/s
002cba6fd9622137d286dcc428ed49f225d36d3b44b503db9ac816bf5b2a090e
002cba6fd9622137d286dcc428ed49f225d36d3b44b503db9ac816bf5b2a090e foo

So the content seems to be the key here. And indeed:
$ dd if=/dev/urandom of=foo bs=1k count=270 ; uuencode foo foo > foo2; python -c 'import apt_pkg; print apt_pkg.sha256sum(open("foo2").read())' ; sha256sum foo2
270+0 records in
270+0 records out
276480 bytes (276 kB) copied, 0,133452 s, 2,1 MB/s
ffcb3d53079b45720f3037d01b6adcda78c1df6a41963049d900d910c6e9d7a0
ffcb3d53079b45720f3037d01b6adcda78c1df6a41963049d900d910c6e9d7a0 foo2

is fine as are the uncompressed sha256 Package files (this is why it went unnoticed for so long).

[Michael Vogt (mvo)]

Apt does its sha256 checks on the uncompressed Packages file and that is correct in the Release file.

[Michael Vogt (mvo)]

It turned out that the python<->c++ glue code had issues with strings with \0 in them, this is fixed now.

[Dioktos (dioktos)]

So will the Relase files ever contain correct SHA256 checksums for the Packages.gz files?

[Michael Vogt (mvo)]

The fix is now in intrepid and I attach the diff for soyuz so that it can be fixed there too. Let me know if I can help further in any way.

[Celso Providelo (cprov)]

RF 7005