mirror prober is fooled by mirrors which return content on not founds

[James Troup]

https://launchpad.net/ubuntu/+mirror/ubuntu.gds.tuwien.ac.at

This mirror returns a fluffy generic 'file not found' page instead of
a 404 which makes the mirror prober think the mirror is always
up-to-date (and it isn't, right now). e.g.

% wget -O /dev/null -S http://ubuntu.gds.tuwien.ac.at/cdimage/releases///hardy/ubuntu-8.04-beta-alternate-i386.iso
--00:43:42-- http://ubuntu.gds.tuwien.ac.at/cdimage/releases///hardy/ubuntu-8.04-beta-alternate-i386.iso
           => `/dev/null'
Resolving ubuntu.gds.tuwien.ac.at... 192.35.244.50
Connecting to ubuntu.gds.tuwien.ac.at|192.35.244.50|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 302 Found
  Date: Fri, 21 Mar 2008 00:42:18 GMT
  Server: Apache
  Location: http://gd.tuwien.ac.at/.admin/missing/server.php?name=UBUNTU.gds.tuwien.ac.at
  Connection: close
  Content-Type: text/html; charset=iso-8859-1
Location: http://gd.tuwien.ac.at/.admin/missing/server.php?name=UBUNTU.gds.tuwien.ac.at [following]
--00:43:42-- http://gd.tuwien.ac.at/.admin/missing/server.php?name=UBUNTU.gds.tuwien.ac.at
           => `/dev/null'
Resolving gd.tuwien.ac.at... 192.35.244.50
Connecting to gd.tuwien.ac.at|192.35.244.50|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Date: Fri, 21 Mar 2008 00:42:18 GMT
  Server: Apache
  X-Powered-By: PHP/5.0.4
  Connection: close
  Content-Type: text/html
Length: unspecified [text/html]

    [ <=> ] 8,395 --.--K/s

00:43:43 (120.76 KB/s) - `/dev/null' saved [8395]

%

I think it's reasonable to fail a mirror if it redirects you to some
URL which doesn't have the same filename as the original file you
asked for?

--
James

[Guilherme Salgado (salgado)]

It sounds reasonable to me.

[James Troup (elmo)]

This appears to be the cause of an increasing number of false positives; so much so that it's threatening the usefulness of the prober at release time. Is there any chance we could get this fixed ASAP? (Well, if it's not in the next week, it doesn't matter for another 6 months... so maybe it's too late for that :(

[Guilherme Salgado (salgado)]

I'll see what we can do about it, but just to make it clear, the problem is not that the mirrors "return content on not founds". The problem is that the mirrors don't respect the protocol and respond with a 302 (followed by a 200) when they should have responded with a 404.

I don't think we'd want to fail these mirrors only because they don't behave properly?

[Steve Langasek (vorlon)]

Guilherme,

They "don't behave properly" in a way that doesn't let us detect whether the mirror is usable or not - IMHO that absolutely is a reason to fail them.

It seems to me that James has identified a reasonable way to detect these mirrors without also causing false positives (i.e., checking that the target of the redirect matches the requested filename)?

[Guilherme Salgado (salgado)]

I don't know what I had in mind when I wrote my last comment, but I
certainly agree with you.

A fix for it is being cowboyed into production right now, btw.

[Guilherme Salgado (salgado)]

mainline r7237

[Curtis Hovey (sinzui)]

Fix released in Launchpad 2.1.11.