Cluster image download gives up and logs an IOError too soon

[Mike Pontillo]

This is error #1 in bug #1457788; original description:

When restarting the two main MAAS services (maas-regiond and maas-clusterd), a bunch of stacktraces are printed in the logs. Ultimately, the services start okay.

service maas-regiond restart && service maas-clusterd restart

Most of these stacktraces seems to be caused by races in how the various services are started and although it is not critical, it's really painful to see that MAAS doesn't deal more gracefully with this.

This is the combination of both regiond.log and clusterd.log: http://paste.ubuntu.com/11280700/

2015-05-22 08:39:28+0000 [-] Downloading images failed.
 Traceback (most recent call last):
   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 423, in errback
     self._startRunCallbacks(fail)
   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks
     self._runCallbacks()
   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
     current.result = callback(current.result, *args, **kw)
   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1155, in gotResult
     _inlineCallbacks(r, g, deferred)
 --- <exception caught here> ---
   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
     result = result.throwExceptionIntoGenerator(g)
   File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
     return g.throw(self.type, self.value, self.tb)
   File "/usr/lib/python2.7/dist-packages/provisioningserver/pserv_services/image_download_service.py", line 135, in maybe_start_download
     yield self._start_download()
   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
     result = result.throwExceptionIntoGenerator(g)
   File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
     return g.throw(self.type, self.value, self.tb)
   File "/usr/lib/python2.7/dist-packages/provisioningserver/pserv_services/image_download_service.py", line 121, in _start_download
     get_proxy_url("https"))
   File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker
     result = context.call(ctx, function, *args, **kwargs)
   File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
     return self.currentContext().callWithContext(ctx, func, *args, **kw)
   File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
     return func(*args,**kw)
   File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 154, in wrapper
     return func(*args, **kwargs)
   File "/usr/lib/python2.7/dist-packages/provisioningserver/rpc/boot_images.py", line 87, in _run_import
     boot_resources.import_images(sources)
   File "/usr/lib/python2.7/dist-packages/provisioningserver/import_images/boot_resources.py", line 246, in import_images
     image_descriptions = download_all_image_descriptions(sources)
   File "/usr/lib/python2.7/dist-packages/provisioningserver/import_images/download_descriptions.py", line 204, in download_all_image_descriptions
     source['url'], keyring=source.get('keyring', None))
   File "/usr/lib/python2.7/dist-packages/provisioningserver/import_images/download_descriptions.py", line 195, in download_image_descriptions
     dumper.sync(reader, rpath)
   File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 82, in sync
     content, payload = reader.read_json(path)
   File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 39, in read_json
     raw = self.source(path).read().decode('utf-8')
   File "/usr/lib/python2.7/dist-packages/simplestreams/contentsource.py", line 143, in read
     self.open()
   File "/usr/lib/python2.7/dist-packages/simplestreams/contentsource.py", line 139, in open
     self.fd = self._open()
   File "/usr/lib/python2.7/dist-packages/simplestreams/contentsource.py", line 135, in _open
     raise myerr
 exceptions.IOError: Unable to open http://192.168.10.11/images-stream/streams/v1/index.json. mirrors=[]

This is the cluster trying to sync from the region before the region is up.

[Raphaël Badin (rvb)]

Raising this to critical since this is a crash and we shouldn't tolerate them even if they seem inconsequential.