http generator should support processing "raw" ubuntu tarballs

[Daniel Manrique]

Right now, the http generator downloads the pointed-to URL, blindly renames it to $something.tar.xz and puts it in the pool.

We tried pointing system-image-server to http://cdimage.ubuntu.com/ubuntu-touch/daily-preinstalled/current/vivid-preinstalled-touch-armhf.tar.gz, and disaster ensued because of this blind rename: when trying to rerun the import-images process, it was unable to xz-decompress the .gz (disguised as .xz) and failed to generate the delta, crashing with a loud traceback in the process.

This also limits system-image-server in that it has to live in a system that hosts copies of the tarballs locally, so it can use the cdimage_ generators which *do* handle files properly.

The http generator could be forked into http_ubuntu, http_device and http_custom generators (perhaps?) which know how to "postprocess" http-downloaded files, essentially applying the same operation as the cdimage_ generators, this would ensure that the end result tarballs are correct, and would enable s-i-s to live in a server completely separate from cdimage.ubuntu.com.

In talks with Steve Langasek, he mentioned that a key feature for this to be feasible is for system-image-server to be able to verify gpg signatures for the images it downloads. So clearly that would be part of this implementation.

As a "companion" feature, I wonder if the http generator should be a bit more distrustful of stuff it downloaded, actively verify that at least the file format is correct (regardless of filename, it should be an xzipped tarfile) and refuse to process it if the format is bogus. Let me know if I should file this last mini-feature as a separate bug.

[Steve Langasek (vorlon)]

Discussed this in realtime today. Rather than modifying the http generator to import "raw" rootfs tarballs from cdimage.ubuntu.com, what I would like to see implemented in system-image server is support for a new "remote_si" (bikeshed the name to taste) generator which talks to an existing system-image server remotely, and walks the tree to find the latest image part (device tarball, etc) on a specified channel - using the full channels.json/channels.json.gpg/index.json validation chain that already exists. This lets us cryptographically ensure integrity of the download using client and server code that already exists, rather than requiring deployment of a new server feature on system-image.ubuntu.com that would only be used by the capomastro server and not shared with other clients of system-image.ubuntu.com.

And as a stopgap measure until this is implemented, we've recommended that PES simply mirror the content from cdimage.ubuntu.com via rsync and run the cdimage generator locally for imports.

[Steve Langasek (vorlon)]

Oops, it's been pointed out that I put this comment on the wrong bug.

We do need support for importing from a /remote/ cdimage tree. This is necessary to enable us to split cdimage.ubuntu.com and system-image.ubuntu.com master hosts. So this bug as described is still valid, but is not the first priority (which is bug #1447633).

Any implementation of an http raw importer needs to be accompanied by fixes to check gpg signatures on the retrieved objects.