[2.2] MAAS should delete cached apt lists before running apt update

[Mike Pontillo]

I have a local Ubuntu mirror that sometimes lags slightly behind the official mirror.

Thus, if I commission a MAAS node while the local mirror is lagging behind the images synced to MAAS (which are expecting the current packages), "apt-get update" does not overwrite the local apt lists, prints errors about hash mismatches, and later may fail to download packages which are more up to date in the main archive than the local mirror.

There is a way to prevent this issue: the first thing we should always do, before performing any actions with 'apt', is:

    rm -rf /var/cache/apt/lists/*
    apt-get update

[Andres Rodriguez (andreserl)]

Why does MAAS needs to deal with this? What's your environment looking like?

My guess is that if you are using a custom mirror, you are doing one of two things:

1. DNS poisoning
2. pointing MAAS to your mirror.

If you are doing 2, why would maas need to remove anything provided that maas should be replacing eveyrthing in /etc/apt and should be updating against that.

If you are doing 1, it could be a problem, but doing dns poisoning is bad....

That said, why is this a MAAS issue and not an APT issue? This seems to me like this is something APT needs to deal with.. ?

[Mike Pontillo (mpontillo)]

I'm doing (2). The MAAS images contain cached apt lists from when the images were built in /var/cache/apt. If the mirror is behind, the apt-get update and/or upgrade will fail because the image expects newer apt lists/hashes than are present on the mirror, and doesn't overwrite the newer lists with the older lists.

This might be something that should be fixed in cloud-init rather than MAAS; I just wanted to start the discussion here.

[Mike Pontillo (mpontillo)]

See attached screenshot when trying to commission a machine in MAAS using an out-of-date mirror.

[Mike Pontillo (mpontillo)]

I discussed this with mvo; here's the debug output he requested from a run of 'apt-get update -o debug::acquire::http=true':

...

GET /ubuntu/dists/xenial-updates/InRelease HTTP/1.1
Host: 192.168.122.218
Cache-Control: max-age=0
Accept: text/*
Range: bytes=102145-
If-Range: Fri, 21 Apr 2017 23:25:00 GMT
User-Agent: Debian APT-HTTP/1.3 (1.2.19)

Answer for: http://192.168.122.218/ubuntu/dists/xenial-updates/InRelease
HTTP/1.1 416 Requested Range Not Satisfiable
Server: nginx/1.10.0 (Ubuntu)
Date: Fri, 05 May 2017 14:19:34 GMT
Content-Type: text/html
Content-Length: 222
Connection: keep-alive
Content-Range: bytes */102145

...

[Scott Moser (smoser)]

I've talked about this with Robie Basak before.
For cloud-images building, it seems like it makes sense to clear out apt cache for updates and security. In the best case scenario, those files are up to date and apt wont re-download. That lasts only until the next SRU or security update, so it is not a long term win.

So it seems to make good sense to just clean those out, leaving only release files.
That will fix this problem for all stable releases where the release pocket is frozen. The problem will still exist during the development cycle of a release, but that seems much less impact.

[Julian Andres Klode (juliank)]

Nothing to see here from the apt side, so I'm getting rid of the tasks.