apt-get update issues invalid range requests for previously completed downloads

[Danny Thomas]

When an apt-get update fails, for instance due to a mismatch between the Release and Packages.gz due to a mirror sync/index update, it leaves files in the /var/lib/apt/lists/partial for the distribution:

15466565 16 -rw-r--r-- 1 root root 13860 Oct 22 12:50 /var/lib/apt/lists/partial/ourhost_internal_repo_dists_main_Release
 15466663 4 -rw-r--r-- 1 root root 475 Oct 22 12:50 /var/lib/apt/lists/partial/ourhost_internal_repo_dists_main_Release.gpg
 15471150 0 -rw-r--r-- 1 root root 0 Oct 22 12:52 /var/lib/apt/lists/partial/ourhost_internal_repo_dists_main_main_binary-amd64_Packages.gz.FAILED

In this case both Release and Release.gpg were complete, however when running apt-get update again it issues the revalidation requests for these resources with Range headers that are the length of the file resulting in '416 Requested Range Not Satisfiable':

GET .../Release HTTP/1.1
Cache-Control: max-age=0
Accept: text/*
Range: bytes=13860-
If-Range: Tue, 22 Oct 2019 12:50:05 GMT
User-Agent: Debian APT-HTTP/1.3 (1.6.12)

HTTP/1.1 416 Requested Range Not Satisfiable

It appears:

- There's no way for apt-get to know if the file is complete so this logic will always be hit if the file exists https://github.com/Debian/apt/blob/master/methods/http.cc#L956-L962
- The logic for determining whether a server supports range requests is only entered if an Accept-Range header exists, however an absence of this header means support is not available - https://github.com/Debian/apt/blob/master/methods/basehttp.cc#L251

[Danny Thomas (dmthomas78)]

Excuse the slightly bogus filenames in that listing - I manually redacted our host/distribution details.

[Julian Andres Klode (juliank)]

My understanding is that this is a server bug and apt is complying with the http specification.

[Julian Andres Klode (juliank)]

Ah the 416 should be fine if the content range is correct (*/complete size) then apt should realise that it has a complete file.

If there is an issue somewhere else this might need further investigation at a later point.

[Danny Thomas (dmthomas78)]

Looking deeper, I think we might also have a backend that's not correctly handling If-Range and returning the 416 before considering if the Release file is out of date, which is why we get stuck in a file size mismatch between Releases and Packages.gz.

Does look like there are maybe some issues around the handling of that error, and certainly the check for the host supporting range support.

[Julian Andres Klode (juliank)]

RFC 7233 allows clients to send range request without checking whether the server supports them.

The reason being that servers generally, if they do not support ranges, be ignoring any range values.

[Julian Andres Klode (juliank)]

That is, a server would usually respond with 200 ok with the complete file if it does not support ranges.