mirror sync in progress, but the situation improves if `rm -fr /var/lib/apt/lists`

[Dan Bungert]

A not-uncommon bug for Subiquity is one that looks like LP: #2105480, but we have seen many like it and that's just one example. In summary, some files will fail to download from the mirror with the message "Mirror sync in progress?"

For reasons I don't understand just yet, it seems /var/lib/apt/lists can get into a bad state. `apt-get update` just produces the "Mirror sync in progress?". The way to recover from it is to remove /var/lib/apt/lists entirely and start over. After removing /var/lib/apt/lists, `apt-get update` and subsequent operations perform as expected.

I have a live example for you, I don't know if this reproducer requires certain mirror state or not. Attached to this bug is a copy of /var/lib/apt/lists, and if you populate your test system /var/lib/apt/lists with the contents of this tarball and configure the system to hit mirror is.archive.ubuntu.com for series plucky, you may see errors like this:

Err:14 http://is.archive.ubuntu.com/ubuntu plucky/universe amd64 Packages
  404 Not Found [IP: 176.57.227.242 443]
  File has unexpected size (15764540 != 16133052). Mirror sync in progress? [IP: 176.57.227.242 443]

[Seth Arnold (seth-arnold)]

That 3142 sure reminds me of apt-cacher-ng. I've seen this behavior from it before. (I'm aware some people swear by it but I've seen it myself and others have reported size and hash mismatches. I switched to squid-deb-proxy.)

If you've got apt-cacher-ng in your environment, try working around it for a while and see how it goes. I believe I had to clear *both* apt-cacher-ng and apt's downloads in order to work around this when it hit.

Thanks

[Dan Bungert (dbungert)]

Thanks Seth. No apt-cacher-ng in the original failing setup, but I do have squid-deb-proxy in the container I used to test the reproducer. I have re-confirmed, both in the container and the original installer VM, that no apt proxy is required to produce this bug.

[Julian Andres Klode (juliank)]

I think we're making some progress on identifying issues similar to that, it turns out that the transaction handling is somewhat broken: If an `apt update` is cancelled, a partial transaction is committed, as we have no way to identify if a transaction is complete or not.

The confusing issue here is why we request Packages files that no longer exist. Fundamentally the problem should not happen because we request Packages files by their hash; but if we get an outdated hash and it no longer exists, we obviously are correct to fail.

The question is where we get the outdated hash from. If it's from the existing InRelease file, then it should not be fetching the file in the first place because the current copy should be up-to-date.

That being said, the files affected here are files not currently present: i.e. we have enabled new index types (cnf) as well as additional components (universe, multiverse, restricted).

I do not believe that the lists are actually in a bad state, rather what happens is that the mirror is in a bad state: It reports that our InRelease file is still up-to-date when we perform the If-Modified-Since request to update it.

That would explain the rest of the outcome: Given that the file is still up-to-date, a new copy did not appear in partial/ and now we try to fetch the new index files and fail because they don't exist. Some confusion there may be which error is reported, i.e. it's possible we are falling back to requesting by name rather than hash due to the 404 and then end up with the wrong file (hence the size mismatch).