apt-ftparchive on-disk cache format changed between lucid and precise, results in Packages files with silently corrupted checksums fields
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| apt (Ubuntu) |
Critical
|
Michael Vogt | ||
| Precise |
High
|
Unassigned | ||
| Trusty |
High
|
Unassigned |
Bug Description
[Impact]
The archive.ubuntu.com master server has just been upgraded from lucid to precise. As a result, the apt version went from 0.7.25.3 to 0.8.16~exp12, and apparently some time in that interval the on-disk format of apt-ftparchive's cache changed.
This wouldn't be a problem, except apt-ftparchive itself doesn't *notice* that the cache format has changed, and instead happily reads the existing files and parses them incorrectly, resulting in corrupted checksums output in the Packages file for each of the .debs.
apt-ftparchive should version its file formats so that it correctly notices incompatible cache files.
[Test Case]
* generate a cachedb with apt-ftparchive from lucid via
apt-ftparchive --db old-db packages 2vcard_
* use apt-ftparchive from precise and run
apt-ftparchive --db old-db packages 2vcard_
and verify that it generates different hashes than Packages.1
(diff -u Packages.1 Packages.2)
* use apt-ftparchive from precise-proposed and run:
apt-ftparchive --db old-db packages 2vcard_
and verify that the hashes are identical to the ones in Packages.1
Changed in apt (Ubuntu): | |
importance: | Undecided → Critical |
Michael Vogt (mvo) wrote : | #1 |
Changed in apt (Ubuntu): | |
status: | New → Confirmed |
Changed in apt (Ubuntu): | |
assignee: | nobody → Michael Vogt (mvo) |
Michael Vogt (mvo) wrote : | #2 |
Its not the fault of libdb, its the fault of commit 650faab01603caa
Here is the reason for the bug:
--- apt-0.7.
+++ apt-0.8.
...
struct StatStore
{
uint32_t Flags;
uint32_t mtime;
- uint32_t FileSize;
+ uint64_t FileSize;
uint8_t MD5[16];
uint8_t SHA1[20];
uint8_t SHA256[32];
+ uint8_t SHA512[64];
} CurStat;
...
Adding the sha512 at the end is fine as there is a flag for the hashes so even if it reads garbage it will not use the not-yet-generated sha512 hash.
But the change of FileSize made it incompatibel.
Michael Vogt (mvo) wrote : | #3 |
Here is a fix for the bug. Its a bit heavy handed and will discard the record if the on-disk size and the struct size mismatch. This is not strictly needed if the struct StatStore is just appended and the FlagList is correctly used. It is in this case as the struct changed size right in the middle.
Michael Vogt (mvo) wrote : | #4 |
This patch adds compatibility with the previous on-disk format. The code is slightly more complex but the already calculated hashes can be re-used and do not need to be calculated again.
Changed in apt (Ubuntu): | |
status: | Confirmed → In Progress |
Michael Vogt (mvo) wrote : | #5 |
This will also affect debian squeeze to wheezy upgrades. Squeeze uses the old format with a 32bit FileSize and wheezy the new 64bit size.
tags: | added: patch |
Michael Vogt (mvo) wrote : | #6 |
See also https:/
Michael Vogt (mvo) wrote : | #7 |
This is now in utopic
Changed in apt (Ubuntu Trusty): | |
status: | New → In Progress |
Changed in apt (Ubuntu): | |
status: | In Progress → Fix Released |
Changed in apt (Ubuntu Trusty): | |
importance: | Undecided → High |
description: | updated |
Changed in apt (Ubuntu Precise): | |
status: | New → In Progress |
Hello Steve, or anyone else affected,
Accepted apt into precise-proposed. The package will build now and be available at http://
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-
Further information regarding the verification process can be found at https:/
Changed in apt (Ubuntu Precise): | |
status: | In Progress → Fix Committed |
tags: | added: verification-needed |
Colin Watson (cjwatson) wrote : | #9 |
Hello Steve, or anyone else affected,
Accepted apt into trusty-proposed. The package will build now and be available at http://
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-
Further information regarding the verification process can be found at https:/
Changed in apt (Ubuntu Trusty): | |
status: | In Progress → Fix Committed |
description: | updated |
description: | updated |
Mathew Hodson (mhodson) wrote : | #10 |
This SRU has been shadowed by a security update and needs to be re-merged.
Changed in apt (Ubuntu Trusty): | |
status: | Fix Committed → In Progress |
Changed in apt (Ubuntu Precise): | |
status: | Fix Committed → In Progress |
tags: |
added: verification-failed removed: verification-needed |
tags: | removed: verification-failed |
tags: | added: precise |
tags: | added: trusty |
Changed in apt (Ubuntu Precise): | |
status: | In Progress → Confirmed |
Changed in apt (Ubuntu Trusty): | |
status: | In Progress → Confirmed |
description: | updated |
Changed in apt (Ubuntu Precise): | |
importance: | Undecided → High |
status: | Confirmed → Triaged |
Changed in apt (Ubuntu Trusty): | |
status: | Confirmed → Triaged |
Changed in apt (Ubuntu Precise): | |
milestone: | none → precise-updates |
Changed in apt (Ubuntu Trusty): | |
milestone: | none → trusty-updates |
I will investigate this issue, but this smells more like a issue with libdb than with apt-ftparchive itself. But that might be wishful thinking, I will double check.