on R3.1 build38 mitaka HA setup, the mysql comes up after restarting it, but VM creation fails with following stack trace in nova-api.log Setup details: Config Nodes : [u'nodec7', u'nodec8', u'nodeg36'] Control Nodes : [u'nodec7', u'nodec8', u'nodeg36'] Compute Nodes : [u'nodei1', u'nodei2', u'nodei3'] Openstack Node : [u'nodec7', u'nodec8', u'nodeg36'] WebUI Node : nodec8 Analytics Nodes : [u'nodec7', u'nodec8', u'nodeg36'] Database Nodes : [u'nodec7', u'nodec8', u'nodeg36'] Physical Devices : [u'blr-mx1', u'hooper', u"'hooper'"] nova-api.log: 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions [req-71adf19d-cc99-41dc-a4e2-6500a4cc70fc 0fb41841bee2429a8389f30a34d9f29f e337e7852def47dc8bc604edb90f93f1 - - -] Unexpected exception in API method 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/images.py", line 145, in detail 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions **page_params) 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/image/api.py", line 68, in get_all 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions return session.detail(context, **kwargs) 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 300, in detail 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions for image in images: 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 254, in list 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions for image in paginate(params, return_request_id): 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 238, in paginate 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions images, resp = self._list(url, "images") 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 63, in _list 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions resp, body = self.client.get(url) 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 275, in get 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions return self._request('GET', url, **kwargs) 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 256, in _request 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions raise exc.CommunicationError(message=message) 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions CommunicationError: Error finding address for http://10.204.217.76:9292/v1/images/detail?is_public=none&limit=20: ('Connection aborted.', BadStatusLine("''",)) 2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions 2016-11-07 17:48:04.100 12887 INFO nova.api.openstack.wsgi [req-71adf19d-cc99-41dc-a4e2-6500a4cc70fc 0fb41841bee2429a8389f30a34d9f29f e337e7852def47dc8bc604edb90f93f1 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. 2016-11-07 17:48:04.106 12887 INFO nova.osapi_compute.wsgi.server [req-71adf19d-cc99-41dc-a4e2-6500a4cc70fc 0fb41841bee2429a8389f30a34d9f29f e337e7852def47dc8bc604edb90f93f1 - - -] 192.168.192.6 "GET /v2.1/e337e7852def47dc8bc604edb90f93f1/images/detail HTTP/1.1" status: 500 len: 487 time: 6.0429599 2016-11-07 17:48:09.348 12931 INFO nova.osapi_compute.wsgi.server [req-e60d73b4-19a8-463c-95ce-159ae45c59de f3f0fe6232df495aa36d93be2f82e497 a00763e126364e3290de5eeb6d39040b - - -] 192.168.192.6 "GET /v2.1/a00763e126364e3290de5eeb6d39040b/flavors HTTP/1.1" status: 200 len: 2251 time: 0.0290070 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions [req-7b7b0386-c86f-4f26-a705-dab78f25406c 0751816c5a024734beea8b1118a9d52c 76e456e8a22a47c58ce9dde81c8df069 - - -] Unexpected exception in API method 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 629, in create 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions **create_kwargs) 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 154, in inner 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs) 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1556, in create 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota) 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1139, in _create_instance 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions reservation_id, max_count) 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 866, in _validate_and_build_base_options 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions pci_request_info, requested_networks) 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1062, in create_pci_requests_for_sriov_ports 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions neutron = get_client(context, admin=True) 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 139, in get_client 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions _ADMIN_AUTH = _load_auth_plugin(CONF) 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in _load_auth_plugin 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions auth_plugin = ks_loading.load_auth_from_conf_options(conf, NEUTRON_GROUP) 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/keystoneauth1/loading/conf.py", line 135, in load_from_conf_options 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions return plugin.load_from_options_getter(_getter, **kwargs) 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/keystoneauth1/loading/base.py", line 148, in load_from_options_getter 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions return self.load_from_options(**kwargs) 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/keystoneauth1/loading/base.py", line 123, in load_from_options 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions raise exceptions.MissingRequiredOptions(missing_required) 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url 2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions 2016-11-07 17:48:42.170 12898 INFO nova.api.openstack.wsgi [req-7b7b0386-c86f-4f26-a705-dab78f25406c 0751816c5a024734beea8b1118a9d52c 76e456e8a22a47c58ce9dde81c8df069 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.