OOPS trying to download livefs build logs (buildd images)

[Robert C Jennings]

While building buildd images in launchpad we've just started seeing a failure when trying to download build artifacts.

For a recent build like https://launchpad.net/~cloud-images/+livefs/ubuntu/groovy/cpc-buildd/+build/229667 our code will do the following using they python SDK:

1. Find a livefs by name via: `livefs = launchpadlib.launchpad.livefses.getByName(livefs_name)`
2. Request a build `build = livefs.requestBuild()`
3. Wait for the build to complete successfully
4. Get the build artifacts URLs with `urls = livefs.getFileUrls()`
5. For each artifact rewrite the file URLs from https://launchpad.net to http://api.launchpad.net/devel.
6. Make a request with redirections=0 and expect to catch httplib2.RedirectLimit with the 'location' in the response being the correct URL that includes a token.

This code takes advantage of the fact that the API endpoint (https://api.launchpad.net/devel) will happily serve up anything that https://launchpad.net will serve up, but will allow us to auth using OAuth instead of OpenID.

What we're seeing today is that step 6 fails to redirect and we're getting a 500 with an OOPS-id instead. Here's the oops for livefs build above: https://oops.canonical.com/?oopsid=OOPS-3d3ce87b80d9269a5ec87b6c3a5abefe

This is happening only for our public builds like buildd images.

[Robert C Jennings (rcj)]

This is affecting buildd image publication for all releases of Ubuntu

[Robert C Jennings (rcj)]

Here is what ended up in our log:

11:48:32 Downloading https://launchpad.net/~cloud-images/+livefs/ubuntu/groovy/cpc-buildd/+build/229667/+files/buildlog_ubuntu_groovy_s390x_s390x-buildd-all_cpc-buildd_BUILDING.txt.gz to out/private/build_log.s390x.all.txt...
11:48:32 Rewrote https://launchpad.net/~cloud-images/+livefs/ubuntu/groovy/cpc-buildd/+build/229667/+files/buildlog_ubuntu_groovy_s390x_s390x-buildd-all_cpc-buildd_BUILDING.txt.gz to https://api.launchpad.net/devel/~cloud-images/+livefs/ubuntu/groovy/cpc-buildd/+build/229667/+files/buildlog_ubuntu_groovy_s390x_s390x-buildd-all_cpc-buildd_BUILDING.txt.gz for OAuth access...
11:48:32 Using OAuth'd client to get launchpadlibrarian.net URL with token...
11:48:32 <class 'tuple'>
11:48:32 ({'date': 'Thu, 23 Jul 2020 16:48:32 GMT', 'server': 'zope.server.http (HTTP)', 'x-powered-by': 'Zope (www.zope.org), Python (www.python.org)', 'content-length': '10', 'content-type': 'text/plain', 'x-lazr-oopsid': 'OOPS-3d3ce87b80d9269a5ec87b6c3a5abefe', 'connection': 'close', 'status': '500'}, b'ValueError')