checkwatches.py doesn't handle http -> https redirects

[Björn Tillenius]

If a bugzilla bug tracker automatically redirects http requests to https, and the bug tracker's base URL in Launchpad is http://something.com, checkwatches.py will POST to http://something.com/buglist.cgi and get redirected to http://something.com/buglist.cgi. When this happens, the form data gets discarded and a GET request is issued.

We should address this issue somehow, I think we can define our own redirect handler for urllib2, which will POST the form data to the new page, if it's only a http -> https redirect.

[Christian Reis (kiko)]

Working around this is pretty trivial; just update the bugtracker URL.

[Diogo Matsubara (matsubara)]

When such a redirection occurs we see OOPSes like OOPS-837CCW12
Exception type UnparseableBugData
Exception value Failed to parse XML description for http://bugs.maemo.org bugs [u'2080']: mismatched tag: line 26, column 4

[Diogo Matsubara (matsubara)]

https://pastebin.canonical.com/4308/ has a fix for it

[Gavin Panella (allenap)]

I do not think we should automatically cope with this situation. Instead, we should use Diogo's fix to improve reporting.

[Jeroen T. Vermeulen (jtv)]

There seem to have been two separate bugs here:

1. Due to the nonstandard redirect, Bugzilla was giving us an HTML page that wasn't also well-formed XML, where our bug-tracking code expected RDF, thus triggering this parse failure. This appears to have been fixed.

2. There's also a quiet failure. Since we were getting the wrong page, we also weren't picking up new bugs on the remote bug tracker. Once the BugZilla search page became well-formed XML, we were happily parsing the wrong page without failures.

With the attached branch, we start seeing bugs that were being passed by before.

[Launchpad QA Bot (lpqabot)]

Fixed in stable r12227 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/12227>.

[Jeroen T. Vermeulen (jtv)]

We tried the new code against 108 bugzilla instances. Sadly, most of them still failed. But Gavin tells me there's no clear indication that we've made anything worse. In 4 instances the check against getting HTML back instead of RDF fails.