linaro-fetch-image unusable since latest linaro 12.08 release

[Fathi Boudra]

l-f-i allows to download release up to 12.03 release.
In addition, latest changes to snapshots.linaro.org (django app) broke fetch image/crawler.
You can't get the latest images/hwpack anymore.

Gesha suggested to fix all the code broken by the new snapshots.linaro.org or just fix django template.

Note: it breaks also the pre-built images as we re-use fetch image/crawler code in the jenkins job.

[Данило Шеган (danilo)]

Gesha has said he'll fix the templates in lp:linaro-license-protection.

We should also create a separate bug to provide a stable API for fetching the directory contents in lp:linaro-license-protection. Actually, that probably makes more sense as a blueprint.

For now, let's resolve this as cleanly as possible (and that means keeping even older fetch-image variants working using the similar HTML structure in the template), and then think about the API-approach.

[Georgy Redkozubov (gesha)]

Template was fixed.

[Georgy Redkozubov (gesha)]

Patterns were fixed in daily-prebuilt-images/crawler.py and linaro-ci/find_latest.py.
The only issue left is to auto accept licenses by crawler.py, which means that we need to use LicenseProtectedFileFetcher there as well.

[James Tunnicliffe (dooferlad)]

Linaro fetch image doesn't care about what web pages look like - it
has a crawler that runs on the server to index all the files. As long
as the index hasn't moved, it should still work as long as the
download part follows redirects and can cope with the x-sendfile
stuff. That hasn't been tested.

As for an API the new implementation provides nearly everything you
need to interact with the server without scraping pages. To get a
download you now need a cookie named license_accepted_<md5sum of
license>. Currently our download script scrapes this license out of
the license page (which is quite easy), but it would be better to have
a ?format=txt option to add to license pages to avoid having to do
this and it should be very simple to add.