sstream-mirror traceback on bad image download

[Jeff Lane ]

I want to use sstream-mirror to mirror the stream and images at cloud-images.ubuntu.com. However, when an image is either interrupted, or the downloaded image, for whatever reason, fails the sha256sum check, sstream-mirror aborts with a traceback and there is no way to prevent this behaviour.

.............................................................Traceback (most recent call last):
  File "/usr/bin/sstream-mirror", line 157, in <module>
    main()
  File "/usr/bin/sstream-mirror", line 153, in main
    tmirror.sync(smirror, initial_path)
  File "/usr/lib/python3/dist-packages/simplestreams/mirrors/__init__.py", line 91, in sync
    return self.sync_index(reader, path, data, content)
  File "/usr/lib/python3/dist-packages/simplestreams/mirrors/__init__.py", line 254, in sync_index
    self.sync(reader, path=epath)
  File "/usr/lib/python3/dist-packages/simplestreams/mirrors/__init__.py", line 89, in sync
    return self.sync_products(reader, path, data, content)
  File "/usr/lib/python3/dist-packages/simplestreams/mirrors/__init__.py", line 341, in sync_products
    self.insert_item(item, src, target, pgree, ipath_cs)
  File "/usr/lib/python3/dist-packages/simplestreams/mirrors/__init__.py", line 478, in insert_item
    mutable=False, size=data.get('size'))
  File "/usr/lib/python3/dist-packages/simplestreams/objectstores/__init__.py", line 179, in insert
    raise checksum_util.invalid_checksum_for_reader(reader)
simplestreams.checksum_util.InvalidChecksum: Invalid sha256 Checksum at http://cloud-images.ubuntu.com/daily/server/xenial/20170620/xenial-server-cloudimg-powerpc-disk1.img. Found 98266186e7069ac66c96cfdbc2858c734ca78dda733dcd7d51094e89353d6d74. Expected 70c495b4e1bd6fc542e9bffb4fe6bca1d6b0df2e00b2f6cd122950fb3e8c06de. read 142643200 bytes expected 476512256 bytes. (size 142643200 expected 476512256)

The above could happen a number of ways, incomplete download, lost packets during transit, etc. I was able to reproduce the original failures by starting a session of sstream-mirror and then in a separate terminal truncating one of the inprogress files. Truncating the downloading file caused the checksum to fail, and the traceback to occur. The first encounter was just internet weirdness causing a bad image download.

Anyway, the problem here, beyond the traceback which is unhelpful, since we KNOW the issue, is that the whole thing aborts. So, if I am in the process of doing a FULL mirror of multiple arches and releases, I have to start all over again (yes, I realise it will pick up where it left off) rather than sstream-mirror attempting a retry and skipping on failure.

Observed Behaviour: sstream-mirror throws a traceback rather than a handled exception and aborts the entire attempt.

Preferred Behaviour: sstream-mirror, on failure of the sha256sum check retries the failed file 1 or 2 times. on subsequent failure, sstream-mirror outputs a suitable error message and proceeds to the next item.