Autopilot failed while configuring AZs

[Mark Shuttleworth]

Logs attached. As an aside, it seemed stuck indefinitely till Andreas made it fail properly, so it might be that we need to also detect such failures and fail properly :)

[Andreas Hasenack (ahasenack)]

nova-api-os-compute.log on the nova cloud controller was full of these:

2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack Traceback (most recent call last):
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 125, in __call__
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack return req.get_response(self.application)
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack application, catch_exc_info=False)
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack return resp(environ, start_response)
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 634, in __call__
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack return self._call_app(env, start_response)
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 554, in _call_app
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack return self._app(env, _fake_start_response)
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack return resp(environ, start_response)
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack return resp(environ, start_response)
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack response = self.app(environ, start_response)
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack return resp(environ, start_response)
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2015-07-17 19:59:47.499 108159 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/open...

[Andreas Hasenack (ahasenack)]

I did a first pass on the logs. Some facts:
- at 18:56:11 landscape calls NCC to add a node to a host aggregate. That fails with a 404: NCC doesn't know about the compute node maas-1-15
- from 18:56:54 onwards, NCC responds with a 500 to the landscape calls, and logs the IndexError backtrace. Eventually landscape gives up and fails the deploy.
- nova conductor is pretty upset basically all the time the deployment is ongoing, complaining about exchanges that don't exist
- at the same time, nova-compute is complaining that nova-conductor went away
- landscape decides to start post-deploy jobs just at 18:55:51. Bootstrap was at 17:26:25.

[Andreas Hasenack (ahasenack)]

Logs from the cloud nodes. Warning: this is 2.4Gb uncompressed

[Adam Collard (adam-collard)]

For nova-c-c task, we need more information: what version of N-C-C was used? If with Juju, which charm version, which Juju, which OpenStack version?

[Adam Collard (adam-collard)]

For Landscape task, we're fairly confident this has been fixed as of Landscape 16.03 (deploying Trusty/Liberty using Juju 1.25)

[Launchpad Janitor (janitor)]

[Expired for nova-cloud-controller (Ubuntu) because there has been no activity for 60 days.]