mirror sync in progress, but the situation improves if `rm -fr /var/lib/apt/lists`

Bug #2107223 reported by Dan Bungert
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
apt (Ubuntu)
New
Undecided
Julian Andres Klode

Bug Description

A not-uncommon bug for Subiquity is one that looks like LP: #2105480, but we have seen many like it and that's just one example. In summary, some files will fail to download from the mirror with the message "Mirror sync in progress?"

For reasons I don't understand just yet, it seems /var/lib/apt/lists can get into a bad state. `apt-get update` just produces the "Mirror sync in progress?". The way to recover from it is to remove /var/lib/apt/lists entirely and start over. After removing /var/lib/apt/lists, `apt-get update` and subsequent operations perform as expected.

I have a live example for you, I don't know if this reproducer requires certain mirror state or not. Attached to this bug is a copy of /var/lib/apt/lists, and if you populate your test system /var/lib/apt/lists with the contents of this tarball and configure the system to hit mirror is.archive.ubuntu.com for series plucky, you may see errors like this:

Err:14 http://is.archive.ubuntu.com/ubuntu plucky/universe amd64 Packages
  404 Not Found [IP: 176.57.227.242 443]
  File has unexpected size (15764540 != 16133052). Mirror sync in progress? [IP: 176.57.227.242 443]

Revision history for this message
Dan Bungert (dbungert) wrote :
Dan Bungert (dbungert)
tags: added: rls-pp-incoming
Revision history for this message
Seth Arnold (seth-arnold) wrote :

That 3142 sure reminds me of apt-cacher-ng. I've seen this behavior from it before. (I'm aware some people swear by it but I've seen it myself and others have reported size and hash mismatches. I switched to squid-deb-proxy.)

If you've got apt-cacher-ng in your environment, try working around it for a while and see how it goes. I believe I had to clear *both* apt-cacher-ng and apt's downloads in order to work around this when it hit.

Thanks

Dan Bungert (dbungert)
description: updated
description: updated
Revision history for this message
Dan Bungert (dbungert) wrote :

Thanks Seth. No apt-cacher-ng in the original failing setup, but I do have squid-deb-proxy in the container I used to test the reproducer. I have re-confirmed, both in the container and the original installer VM, that no apt proxy is required to produce this bug.

tags: added: foundations-todo
removed: rls-pp-incoming
Changed in apt (Ubuntu):
assignee: nobody → Julian Andres Klode (juliank)
Revision history for this message
Julian Andres Klode (juliank) wrote :

I think we're making some progress on identifying issues similar to that, it turns out that the transaction handling is somewhat broken: If an `apt update` is cancelled, a partial transaction is committed, as we have no way to identify if a transaction is complete or not.

The confusing issue here is why we request Packages files that no longer exist. Fundamentally the problem should not happen because we request Packages files by their hash; but if we get an outdated hash and it no longer exists, we obviously are correct to fail.

The question is where we get the outdated hash from. If it's from the existing InRelease file, then it should not be fetching the file in the first place because the current copy should be up-to-date.

That being said, the files affected here are files not currently present: i.e. we have enabled new index types (cnf) as well as additional components (universe, multiverse, restricted).

I do not believe that the lists are actually in a bad state, rather what happens is that the mirror is in a bad state: It reports that our InRelease file is still up-to-date when we perform the If-Modified-Since request to update it.

That would explain the rest of the outcome: Given that the file is still up-to-date, a new copy did not appear in partial/ and now we try to fetch the new index files and fail because they don't exist. Some confusion there may be which error is reported, i.e. it's possible we are falling back to requesting by name rather than hash due to the 404 and then end up with the wrong file (hence the size mismatch).

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.