It should skip image importing instead of raising exception and aborting if files are empty/unrecognized

[Caio Begotti]

Sometimes it may happen that SIS will try to fetch a file (via the http generator) and it won't verify it for real so it's possible the file is just an Apache error in HTML in disguise or whatever (because of ${reasons}). SIS will try to import it anyway and once it tries to use that file again to generate deltas it explodes similar to this:

2015-03-24 12:48:36,428 DEBUG Unxzipping file: /tmp/tmpGcbK6A/source.tar
xz: /srv/sis.capomastro.staging.canonical.com/pool/custom-be77fe799af0a6371d595c2af79340da3f1968f519b0e1b6899be9bc62c64e17.tar.xz: No such file or directory
2015-03-24 12:48:36,431 DEBUG Unxzipping file: /tmp/tmpGcbK6A/target.tar
xz: /srv/sis.capomastro.staging.canonical.com/pool/custom-a84a9ce7ddd6e73b1ab6b4ade7b43d9715555028f8b4c9995a453a778486c8a2.tar.xz: File format not recognized
Traceback (most recent call last):
  File "/srv/system_image_server/bin/import-images", line 253, in <module>
    abspath)
  File "/srv/system_image_server/bin/../lib/systemimage/generators.py", line 113, in generate_delta
    os.path.join(tempdir, "target.tar"))
  File "/srv/system_image_server/bin/../lib/systemimage/diff.py", line 82, in __init__
    self.source_file = tarfile.open(source, 'r:')
  File "/usr/lib/python2.7/tarfile.py", line 1678, in open
    return func(name, filemode, fileobj, **kwargs)
  File "/usr/lib/python2.7/tarfile.py", line 1705, in taropen
    return cls(name, mode, fileobj, **kwargs)
  File "/usr/lib/python2.7/tarfile.py", line 1574, in __init__
    self.firstmember = self.next()
  File "/usr/lib/python2.7/tarfile.py", line 2338, in next
    raise ReadError("empty file")
tarfile.ReadError: empty file

I have caught this when testing SIS with Capomastro but I hadn't had the chance to write a patch yet, so I'm just reporting it here to have some history of the problem. IMHO SIS should simply log or mark, then skip this image and not abort with an exception, which will block processing for all others images configured in the server.

[Daniel Manrique (roadmr)]

One additional thing we noticed is that the http generator is very naive in that it assumes *anything* it downloaded (be it a 404 error page, or perhaps a gz-format file, or anything else you point it to) is actually a .xz in disguise. It blindly trusts the user and will simply rename (to $name-$checksum.tar.xz) and place in the pool the given file.

This leads to the above error or similar when generating deltas, because the generate_delta method also assumes that it will only ever deal with xz files (does tools.xz_uncompress without catching exceptions or checking actual magic or mime types).

I think the http generator could check the file to ensure it's .xz actually, and if it isn't, output a message and discard the file. Otherwise the pool gets quite messed up. It's worth looking at our expectations for files in the pool (i.e. are they always xz'd tars?) so maybe http generator could further validate them.

[Caio Begotti (caio1982)]

All those vivid-preinstalled-touch-armhf.tar.gz images we pull from cdimage right now, for instance, are all gzipped tarballs. Not xzed. I just checked the official image server and Capomastro on prodstack and beside these (pulled using the http generator indeed) the rest of the pool files are all xzed. The official system-image.ubuntu.com has about 20k files and all are xzed too, as the files they pull with the http generator are already xzed anyway. So, if cdimage could deliver xzed files we wouldn't have this problem when using the http generator it seems, but for now s-i-s needs to repack them as it does with the other generators.