apt-get update issues invalid range requests for previously completed downloads

Bug #1849368 reported by Danny Thomas
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
apt (Ubuntu)
New
Undecided
Unassigned

Bug Description

When an apt-get update fails, for instance due to a mismatch between the Release and Packages.gz due to a mirror sync/index update, it leaves files in the /var/lib/apt/lists/partial for the distribution:

15466565 16 -rw-r--r-- 1 root root 13860 Oct 22 12:50 /var/lib/apt/lists/partial/ourhost_internal_repo_dists_main_Release
 15466663 4 -rw-r--r-- 1 root root 475 Oct 22 12:50 /var/lib/apt/lists/partial/ourhost_internal_repo_dists_main_Release.gpg
 15471150 0 -rw-r--r-- 1 root root 0 Oct 22 12:52 /var/lib/apt/lists/partial/ourhost_internal_repo_dists_main_main_binary-amd64_Packages.gz.FAILED

In this case both Release and Release.gpg were complete, however when running apt-get update again it issues the revalidation requests for these resources with Range headers that are the length of the file resulting in '416 Requested Range Not Satisfiable':

GET .../Release HTTP/1.1
Cache-Control: max-age=0
Accept: text/*
Range: bytes=13860-
If-Range: Tue, 22 Oct 2019 12:50:05 GMT
User-Agent: Debian APT-HTTP/1.3 (1.6.12)

HTTP/1.1 416 Requested Range Not Satisfiable

It appears:

- There's no way for apt-get to know if the file is complete so this logic will always be hit if the file exists https://github.com/Debian/apt/blob/master/methods/http.cc#L956-L962
- The logic for determining whether a server supports range requests is only entered if an Accept-Range header exists, however an absence of this header means support is not available - https://github.com/Debian/apt/blob/master/methods/basehttp.cc#L251

Revision history for this message
Danny Thomas (dmthomas78) wrote :

Excuse the slightly bogus filenames in that listing - I manually redacted our host/distribution details.

Revision history for this message
Julian Andres Klode (juliank) wrote :

My understanding is that this is a server bug and apt is complying with the http specification.

Revision history for this message
Julian Andres Klode (juliank) wrote :

Ah the 416 should be fine if the content range is correct (*/complete size) then apt should realise that it has a complete file.

If there is an issue somewhere else this might need further investigation at a later point.

Revision history for this message
Danny Thomas (dmthomas78) wrote :

Looking deeper, I think we might also have a backend that's not correctly handling If-Range and returning the 416 before considering if the Release file is out of date, which is why we get stuck in a file size mismatch between Releases and Packages.gz.

Does look like there are maybe some issues around the handling of that error, and certainly the check for the host supporting range support.

Revision history for this message
Julian Andres Klode (juliank) wrote :

RFC 7233 allows clients to send range request without checking whether the server supports them.

The reason being that servers generally, if they do not support ranges, be ignoring any range values.

Revision history for this message
Julian Andres Klode (juliank) wrote :

That is, a server would usually respond with 200 ok with the complete file if it does not support ranges.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.