apt-ftparchive on-disk cache format changed between lucid and precise, results in Packages files with silently corrupted checksums fields

[Steve Langasek]

[Impact]

The archive.ubuntu.com master server has just been upgraded from lucid to precise. As a result, the apt version went from 0.7.25.3 to 0.8.16~exp12, and apparently some time in that interval the on-disk format of apt-ftparchive's cache changed.

This wouldn't be a problem, except apt-ftparchive itself doesn't *notice* that the cache format has changed, and instead happily reads the existing files and parses them incorrectly, resulting in corrupted checksums output in the Packages file for each of the .debs.

apt-ftparchive should version its file formats so that it correctly notices incompatible cache files.

[Test Case]

* generate a cachedb with apt-ftparchive from lucid via
  apt-ftparchive --db old-db packages 2vcard_0.5-3_all.deb > Packages.1

* use apt-ftparchive from precise and run
  apt-ftparchive --db old-db packages 2vcard_0.5-3_all.deb > Packages.2
  and verify that it generates different hashes than Packages.1
  (diff -u Packages.1 Packages.2)

* use apt-ftparchive from precise-proposed and run:
  apt-ftparchive --db old-db packages 2vcard_0.5-3_all.deb > Packages.3
  and verify that the hashes are identical to the ones in Packages.1

[Michael Vogt (mvo)]

I will investigate this issue, but this smells more like a issue with libdb than with apt-ftparchive itself. But that might be wishful thinking, I will double check.

[Michael Vogt (mvo)]

Its not the fault of libdb, its the fault of commit 650faab01603caac04494d54cf6b10a65c00ea13

Here is the reason for the bug:
--- apt-0.7.25.3ubuntu7/ftparchive/cachedb.h
+++ apt-0.8.16~exp12ubuntu10/ftparchive/cachedb.h
...
    struct StatStore
    {
       uint32_t Flags;
       uint32_t mtime;
- uint32_t FileSize;
+ uint64_t FileSize;
       uint8_t MD5[16];
       uint8_t SHA1[20];
       uint8_t SHA256[32];
+ uint8_t SHA512[64];
    } CurStat;
...

Adding the sha512 at the end is fine as there is a flag for the hashes so even if it reads garbage it will not use the not-yet-generated sha512 hash.

But the change of FileSize made it incompatibel.

[Michael Vogt (mvo)]

Here is a fix for the bug. Its a bit heavy handed and will discard the record if the on-disk size and the struct size mismatch. This is not strictly needed if the struct StatStore is just appended and the FlagList is correctly used. It is in this case as the struct changed size right in the middle.

[Michael Vogt (mvo)]

This patch adds compatibility with the previous on-disk format. The code is slightly more complex but the already calculated hashes can be re-used and do not need to be calculated again.

[Michael Vogt (mvo)]

This will also affect debian squeeze to wheezy upgrades. Squeeze uses the old format with a 32bit FileSize and wheezy the new 64bit size.

[Michael Vogt (mvo)]

See also https://github.com/mvo5/apt/commits/bugfix/lp1274466-cache for a updated version with regression test and all.

[Michael Vogt (mvo)]

This is now in utopic

[Chris Halse Rogers (raof)]

Hello Steve, or anyone else affected,

Accepted apt into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/apt/0.8.16~exp12ubuntu10.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Colin Watson (cjwatson)]

Hello Steve, or anyone else affected,

Accepted apt into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/apt/1.0.1ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Mathew Hodson (mhodson)]

This SRU has been shadowed by a security update and needs to be re-merged.

[Steve Langasek (vorlon)]

The Precise Pangolin has reached end of life, so this bug will not be fixed for that release