Apt and Soyuz generating release files with invalid SHA256 signatures

Bug #243630 reported by Ryan Hass on 2008-06-27
Affects Status Importance Assigned to Milestone
Launchpad itself
Celso Providelo
apt (Ubuntu)
Michael Vogt

Bug Description

The following are just a two examples of the errors in the Release file.

 40d48bc44164c1da2a6dd1f503b55a3634d1b5ddee0d6ddd3ce16d7e7688c938 1178004 main/binary-i386/Packages.bz2
 baa89858c7e545390273530ba63c61b94c2e09d38c28b0a0311bfa7bde396181 1517892 main/binary-i386/Packages.gz

$ sha256sum main/binary-i386/Packages.bz2
0d9685d8353341ebbd8de9370bb61c1630d4c0a46420bbebabae4197981523a4 main/binary-i386/Packages.bz2
5b906ae167349ecd6699f39ed22e9e98221f780f3a6b15b6443d11b8726270ff main/binary-i386/Packages.gz

Related branches

John Schofield (jschofield) wrote :

I tested this out to confirm Ryan's findings:

I downloaded http://archive.ubuntu.com/ubuntu/dists/hardy/Release and http://archive.ubuntu.com/ubuntu/dists/hardy/main/binary-amd64/Packages.gz

"md5sum Packages.gz" matches the md5sum data in the Release file. "sha256sum Packages.gz" does NOT match. These are the correct files, but the sha256 hashes do not appear to be being generated correctly.

This is using versions 6.10 of both md5sum and sha256sum, and Release and Packages.gz downloaded from archive.ubuntu.com at 14:51 PDT today.

(Full disclosure: Ryan and I work for the same organization.)

Kees Cook (kees) wrote :

Thanks for the report. I'm poking at it myself now, and have forwarded it to the Soyuz folks.

Celso Providelo (cprov) wrote :

All SHA256 after 'feisty' are wrong, we are using apt_pkg.sha256sum() to generate them and it is broken (!)

>>> import apt_pkg
>>> apt_pkg.sha256sum(open('Packages.gz').read())
>>> from subprocess import call
>>> call(['sha256sum', 'Packages.gz'])
5b906ae167349ecd6699f39ed22e9e98221f780f3a6b15b6443d11b8726270ff Packages.gz
>>> from Crypto.Hash import SHA256
>>> print SHA256.new(open('Packages.gz').read()).hexdigest()

Celso Providelo (cprov) on 2008-06-27
Changed in soyuz:
assignee: nobody → cprov
importance: Undecided → High
milestone: none → 1.99
status: New → Confirmed
Changed in soyuz:
milestone: 1.99 → none

Wrong checksum during receive of 'http://archive.ubuntu.com/ubuntu/dists/gutsy/main/binary-i386/Packages.gz':
sha256 expected: baa89858c7e545390273530ba63c61b94c2e09d38c28b0a0311bfa7bde396181, got: af96b1f3119c4ce4b0c6183750279bf7cbdfe62581289f03ad360787e79f968b
There have been errors!

Same problem in Gutsy.

Michael Vogt (mvo) wrote :

Thanks for your bugreport.

The usage of sha256sum() with file objects is generally prefered, so
should work.

However, there is a bug here somewhere because the string usage should work as well. It does in my tests for small strings, it might be a problem in the python<->c++ code with the string length, I'm investigating that now.

Changed in apt:
assignee: nobody → mvo
importance: Undecided → Critical
status: Confirmed → In Progress
Michael Vogt (mvo) wrote :

Testing the string use-case shows that it goes wrong on 253 bytes:

$ dd if=/dev/urandom of=foo bs=1 count=252 ; python -c 'import apt_pkg; print apt_pkg.sha256sum(open("foo").read())' ; sha256sum foo
252+0 records in
252+0 records out
252 bytes (252 B) copied, 0,00294077 s, 85,7 kB/s
83c762165fbec99d6fd590ed2d3b291d40bfa8525c97b391d2cfb661c27e25fa foo

$ dd if=/dev/urandom of=foo bs=1 count=252 ; python -c 'import apt_pkg; print apt_pkg.sha256sum(open("foo").read())' ; sha256sum foo
253+0 records in
253+0 records out
253 bytes (253 B) copied, 0,00317775 s, 79,6 kB/s
d68a24e86b8037437a20a592a717c40c163127f4942b511b102f0b11e449794c foo

Michael Vogt (mvo) wrote :

It turns out that it is not as easy as this:

$ dd if=/dev/urandom of=foo bs=1 count=270 ; python -c 'import apt_pkg; print apt_pkg.sha256sum(open("foo").read())' ; sha256sum foo
270+0 records in
270+0 records out
270 bytes (270 B) copied, 0,00338191 s, 79,8 kB/s
002cba6fd9622137d286dcc428ed49f225d36d3b44b503db9ac816bf5b2a090e foo

So the content seems to be the key here. And indeed:
$ dd if=/dev/urandom of=foo bs=1k count=270 ; uuencode foo foo > foo2; python -c 'import apt_pkg; print apt_pkg.sha256sum(open("foo2").read())' ; sha256sum foo2
270+0 records in
270+0 records out
276480 bytes (276 kB) copied, 0,133452 s, 2,1 MB/s
ffcb3d53079b45720f3037d01b6adcda78c1df6a41963049d900d910c6e9d7a0 foo2

is fine as are the uncompressed sha256 Package files (this is why it went unnoticed for so long).

Michael Vogt (mvo) wrote :

Apt does its sha256 checks on the uncompressed Packages file and that is correct in the Release file.

Michael Vogt (mvo) wrote :

It turned out that the python<->c++ glue code had issues with strings with \0 in them, this is fixed now.

Changed in apt:
status: In Progress → Fix Committed
Dioktos (dioktos) wrote :

So will the Relase files ever contain correct SHA256 checksums for the Packages.gz files?

Michael Vogt (mvo) wrote :

The fix is now in intrepid and I attach the diff for soyuz so that it can be fixed there too. Let me know if I can help further in any way.

Changed in apt:
status: Fix Committed → Fix Released
Kees Cook (kees) on 2008-09-11
Changed in soyuz:
status: Confirmed → In Progress
Celso Providelo (cprov) on 2008-09-12
Changed in soyuz:
milestone: none → 2.1.9
status: In Progress → Confirmed
Celso Providelo (cprov) on 2008-09-12
Changed in soyuz:
status: Confirmed → In Progress
Celso Providelo (cprov) wrote :

RF 7005

Changed in soyuz:
status: In Progress → Fix Committed
Celso Providelo (cprov) on 2008-09-18
Changed in soyuz:
status: Fix Committed → Fix Released
Curtis Hovey (sinzui) on 2009-10-09
tags: added: tech-debt
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers