Image server always exits after wrongly processing a non-image file

[Caio Begotti]

If you point image server etc/config to a file that is 404 it will try to import the error page (an Apache HTML page) and be fine with it, but the next time it runs if the file is finally present (for whatever reason, say, its build process finally finished) then the image server will always exit with the following error because it can't generate deltas for non-image files.

Importing an error 404 page (the hash is always the same unless you customize the error page):

2015-03-11 15:28:38,911 INFO Calling 'http' generator for a new file
2015-03-11 15:28:38,932 DEBUG Path generated: /srv/sis.capomastro.staging.canonical.com/pool/device-55bea2e50b35102ac2ea4c9888283520966e49cf8ff24f05228d74da29d23b97.tar.xz
2015-03-11 15:28:39,005 DEBUG Signing file: /srv/sis.capomastro.staging.canonical.com/pool/device-55bea2e50b35102ac2ea4c9888283520966e49cf8ff24f05228d74da29d23b97.tar.xz.asc
2015-03-11 15:28:39,032 DEBUG Signing file: /srv/sis.capomastro.staging.canonical.com/pool/device-55bea2e50b35102ac2ea4c9888283520966e49cf8ff24f05228d74da29d23b97.json.asc
2015-03-11 15:28:39,049 INFO New file from 'http': pool/device-55bea2e50b35102ac2ea4c9888283520966e49cf8ff24f05228d74da29d23b97.tar.xz

Delta generation:

2015-03-11 15:50:09,271 INFO Generating delta from '1' for 'http'
2015-03-11 15:50:09,272 DEBUG Path generated: /srv/sis.capomastro.staging.canonical.com/pool/device-7b303c2368c4b1c3e57b3fb293c3c046c5269bd58df26e3de06b8d9539324da1.delta-device-55bea2e50b35102ac2ea4c9888283520966e49cf8ff24f05228d74da29d23b97.tar.xz
2015-03-11 15:50:09,272 DEBUG Unxzipping file: /tmp/tmp0REe1i/source.tar
xz: /srv/sis.capomastro.staging.canonical.com/pool/device-55bea2e50b35102ac2ea4c9888283520966e49cf8ff24f05228d74da29d23b97.tar.xz: File format not recognized
2015-03-11 15:50:09,300 DEBUG Unxzipping file: /tmp/tmp0REe1i/target.tar
xz: /srv/sis.capomastro.staging.canonical.com/pool/device-7b303c2368c4b1c3e57b3fb293c3c046c5269bd58df26e3de06b8d9539324da1.tar.xz: File format not recognized
Traceback (most recent call last):
  File "bin/import-images", line 253, in <module>
    abspath)
  File "/srv/system_image_server/bin/../lib/systemimage/generators.py", line 113, in generate_delta
    os.path.join(tempdir, "target.tar"))
  File "/srv/system_image_server/bin/../lib/systemimage/diff.py", line 82, in __init__
    self.source_file = tarfile.open(source, 'r:')
  File "/usr/lib/python2.7/tarfile.py", line 1678, in open
    return func(name, filemode, fileobj, **kwargs)
  File "/usr/lib/python2.7/tarfile.py", line 1705, in taropen
    return cls(name, mode, fileobj, **kwargs)
  File "/usr/lib/python2.7/tarfile.py", line 1574, in __init__
    self.firstmember = self.next()
  File "/usr/lib/python2.7/tarfile.py", line 2338, in next
    raise ReadError("empty file")
tarfile.ReadError: empty file

The only way to recover from this is to manually log into the machine and clear everything up myself.

[Caio Begotti (caio1982)]

The related branch is just an example on what's expected from image server. Of course that was more like a hack, but it solved the problem for us temporarily. Ideally all urlopen and urlretrieve should handle HTTP errors and only download/open stuff if they are really there.