apt-ftparchive on-disk cache format changed between lucid and precise, results in Packages files with silently corrupted checksums fields

Bug #1274466 reported by Steve Langasek on 2014-01-30
18
This bug affects 1 person
Affects Status Importance Assigned to Milestone
apt (Ubuntu)
Critical
Michael Vogt
Precise
High
Unassigned
Trusty
High
Unassigned

Bug Description

[Impact]

The archive.ubuntu.com master server has just been upgraded from lucid to precise. As a result, the apt version went from 0.7.25.3 to 0.8.16~exp12, and apparently some time in that interval the on-disk format of apt-ftparchive's cache changed.

This wouldn't be a problem, except apt-ftparchive itself doesn't *notice* that the cache format has changed, and instead happily reads the existing files and parses them incorrectly, resulting in corrupted checksums output in the Packages file for each of the .debs.

apt-ftparchive should version its file formats so that it correctly notices incompatible cache files.

[Test Case]

* generate a cachedb with apt-ftparchive from lucid via
  apt-ftparchive --db old-db packages 2vcard_0.5-3_all.deb > Packages.1

* use apt-ftparchive from precise and run
  apt-ftparchive --db old-db packages 2vcard_0.5-3_all.deb > Packages.2
  and verify that it generates different hashes than Packages.1
  (diff -u Packages.1 Packages.2)

* use apt-ftparchive from precise-proposed and run:
  apt-ftparchive --db old-db packages 2vcard_0.5-3_all.deb > Packages.3
  and verify that the hashes are identical to the ones in Packages.1

Michael Vogt (mvo) on 2014-01-30
Changed in apt (Ubuntu):
importance: Undecided → Critical
Michael Vogt (mvo) wrote :

I will investigate this issue, but this smells more like a issue with libdb than with apt-ftparchive itself. But that might be wishful thinking, I will double check.

Anders (eddiedog988) on 2014-03-13
Changed in apt (Ubuntu):
status: New → Confirmed
Steve Langasek (vorlon) on 2014-05-26
Changed in apt (Ubuntu):
assignee: nobody → Michael Vogt (mvo)
Michael Vogt (mvo) wrote :

Its not the fault of libdb, its the fault of commit 650faab01603caac04494d54cf6b10a65c00ea13

Here is the reason for the bug:
--- apt-0.7.25.3ubuntu7/ftparchive/cachedb.h
+++ apt-0.8.16~exp12ubuntu10/ftparchive/cachedb.h
...
    struct StatStore
    {
       uint32_t Flags;
       uint32_t mtime;
- uint32_t FileSize;
+ uint64_t FileSize;
       uint8_t MD5[16];
       uint8_t SHA1[20];
       uint8_t SHA256[32];
+ uint8_t SHA512[64];
    } CurStat;
...

Adding the sha512 at the end is fine as there is a flag for the hashes so even if it reads garbage it will not use the not-yet-generated sha512 hash.

But the change of FileSize made it incompatibel.

Michael Vogt (mvo) wrote :

Here is a fix for the bug. Its a bit heavy handed and will discard the record if the on-disk size and the struct size mismatch. This is not strictly needed if the struct StatStore is just appended and the FlagList is correctly used. It is in this case as the struct changed size right in the middle.

Michael Vogt (mvo) wrote :

This patch adds compatibility with the previous on-disk format. The code is slightly more complex but the already calculated hashes can be re-used and do not need to be calculated again.

Changed in apt (Ubuntu):
status: Confirmed → In Progress
Michael Vogt (mvo) wrote :

This will also affect debian squeeze to wheezy upgrades. Squeeze uses the old format with a 32bit FileSize and wheezy the new 64bit size.

tags: added: patch
Michael Vogt (mvo) wrote :

See also https://github.com/mvo5/apt/commits/bugfix/lp1274466-cache for a updated version with regression test and all.

Michael Vogt (mvo) wrote :

This is now in utopic

Changed in apt (Ubuntu Trusty):
status: New → In Progress
Changed in apt (Ubuntu):
status: In Progress → Fix Released
Changed in apt (Ubuntu Trusty):
importance: Undecided → High
Michael Vogt (mvo) on 2014-06-25
description: updated
Changed in apt (Ubuntu Precise):
status: New → In Progress

Hello Steve, or anyone else affected,

Accepted apt into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/apt/0.8.16~exp12ubuntu10.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in apt (Ubuntu Precise):
status: In Progress → Fix Committed
tags: added: verification-needed
Colin Watson (cjwatson) wrote :

Hello Steve, or anyone else affected,

Accepted apt into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/apt/1.0.1ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in apt (Ubuntu Trusty):
status: In Progress → Fix Committed
Michael Vogt (mvo) on 2014-09-09
description: updated
Michael Vogt (mvo) on 2014-09-09
description: updated
Mathew Hodson (mathew-hodson) wrote :

This SRU has been shadowed by a security update and needs to be re-merged.

Changed in apt (Ubuntu Trusty):
status: Fix Committed → In Progress
Changed in apt (Ubuntu Precise):
status: Fix Committed → In Progress
tags: added: verification-failed
removed: verification-needed
tags: removed: verification-failed
tags: added: precise
tags: added: trusty
Changed in apt (Ubuntu Precise):
status: In Progress → Confirmed
Changed in apt (Ubuntu Trusty):
status: In Progress → Confirmed
description: updated
Changed in apt (Ubuntu Precise):
importance: Undecided → High
status: Confirmed → Triaged
Changed in apt (Ubuntu Trusty):
status: Confirmed → Triaged
Changed in apt (Ubuntu Precise):
milestone: none → precise-updates
Changed in apt (Ubuntu Trusty):
milestone: none → trusty-updates
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers