should delay and retry on http 503 from librarian

[Martin Pool]

If the Launchpad librarian returns an http error likely to indicate a transient problem, such as 503, udd should retry later.

For instance, <http://package-import.ubuntu.com/status/yoshimi.html#2011-02-13 00:18:09.092545>

Traceback (most recent call last):
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 1095, in <module>
    persistent_download_cache=options.persistent_download_cache))
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 1021, in main
    import_package(temp_dir, download_dir, importp, revid_db, bstore, possible_transports=possible_transports)
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 597, in import_package
    possible_transports=possible_transports)
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 367, in dget
    possible_transports=possible_transports)
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 320, in grab_file
    location_f = transport.do_catching_redirections(get_file, t, redirected)
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/__init__.py", line 1679, in do_catching_redirections
    return action(transport)
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 307, in get_file
    return transport.get(name)
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/http/__init__.py", line 126, in get
    code, response_file = self._get(relpath, None)
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/http/_pycurl.py", line 171, in _get
    return self._get_full(relpath)
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/http/_pycurl.py", line 215, in _get_full
    curl, 'expected 200 or 404 for full response.')
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/http/_pycurl.py", line 325, in _raise_curl_http_error
    % (code, msg, plaintext_body))
bzrlib.errors.InvalidHttpResponse: Invalid http response for https://launchpadlibrarian.net/51872524/yoshimi_0.058.1-1.dsc: Unable to handle http code 503: expected 200 or 404 for full response.:

[Martin Pool (mbp)]

Perhaps just requeue_package --auto is enough for now, but ideally this would go into the source code as well as the db state.

[Vincent Ladeuil (vila)]

spm mentioned 10+ minutes down times for librarian.

So waiting is not an option. many imports succeed in less than 10 minutes we shouldn't block them.

'retry' is the way to go but I'm pretty sure the p-i is already re-trying in this case.

It would be nice to make it clearer in the status page that these are transient and know errors though.

Not closing the bug until we better know what the p-i does and how we should report this kind of failure.

[Vincent Ladeuil (vila)]

The packages were requeued by poolie, so the p-i should indeed retry on its own in this case.

[Martin Pool (mbp)]

right, we shouldn't delay blocking the thread, but rather retry the package after a delay