mirror prober didn't detect that an official mirror was gone

[Laurent Sesquès]

The (now ex-)official mirror for Mongolia had been broken for some time but the prober apparently didn't notice it.
As of now it still thinks that all series are up to date.
https://launchpad.net/ubuntu/+mirror/archive.mnosi.org-archive
https://launchpad.net/ubuntu/+mirror/releases.mnosi.org-release

[Colin Watson (cjwatson)]

Something is weird:

$ HEAD http://ubuntu-archive.adsolux.com/ubuntu/dists/cosmic/main/binary-amd64/Packages.gz
200 OK
Connection: close
Date: Fri, 15 Jun 2018 14:30:18 GMT
Server: nginx
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8
Client-Date: Fri, 15 Jun 2018 14:30:18 GMT
Client-Peer: 185.53.179.8:80
Client-Response-Num: 1
X-Adblock-Key: MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALquDFETXRn0Hr05fUP7EJT77xYnPmRbpMy4vk8KYiHnkNpednjOANJcaXDXcKQJN0nXKZJL7TciJD8AoHXK158CAwEAAQ==_qPY4rlFewtdckgAsjtAuxnjrlauXNE9/sHWqo0dNlXqPhckUJMFH6zl97Ej9VVvKrv9mQv/WCCinH5j56sFrzg==
X-Buckets: bucket065
X-Check: 3c12dc4d54f8e22d666785b733b0052100c53444
X-Language: english
X-Template: tpl_CleanPeppermintBlack_twoclick

$ curl -iI http://ubuntu-archive.adsolux.com/ubuntu/dists/cosmic/main/binary-amd64/Packages.gz
HTTP/1.1 403 Forbidden
Server: nginx
Date: Fri, 15 Jun 2018 14:30:35 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive

I think it's doing User-Agent sniffing, and the prober uses something that causes it to return 200. Nasty.

[Laurent Sesquès (sajoupa)]

There are also many mirrors which the prober detected as lagging:
https://paste.ubuntu.com/p/Mg3hhjRZ99/
Looking at the logs (for instance https://launchpadlibrarian.net/374733940/mirror.squ.edu.om-archive-probe-logfile.txt), it does get 4xx responses.
However, the mirrors are not automatically maked Unofficial, which I believe is what shoud happen.

[Laurent Sesquès (sajoupa)]

There are also many mirrors which the prober detected as lagging:
https://paste.ubuntu.com/p/Mg3hhjRZ99/
Looking at the logs (for instance https://launchpadlibrarian.net/374733940/mirror.squ.edu.om-archive-probe-logfile.txt), it does get 4xx responses.
However, the mirrors are not automatically marked Unofficial, which I believe is what should happen.

[Colin Watson (cjwatson)]

As I understand it, the prober never sets the mirror status (Pending review, Unofficial, Official), and this is intentional: that's a field that's set by mirror admins, and we shouldn't generally have fields that are set both by admins and by automatic processes. For example, that could result in an admin setting the status to Official and then that work being lost because of a temporary network flap.

Instead, the idea is that things consuming the mirror list probably want to check both for mirror status (has this mirror been reviewed?) and mirror freshness (is this mirror up to date?), rather than assuming that mirror status is a proxy for mirror freshness.