Juniper Openstack

Comment 2 for bug 1611566

Revision history for this message

Sudheendra Rao (sudheendra-k) wrote on 2016-11-07:

on R3.1 build38 mitaka HA setup, the mysql comes up after restarting it, but VM creation fails with following stack trace in nova-api.log

Setup details:
Config Nodes : [u'nodec7', u'nodec8', u'nodeg36']
Control Nodes : [u'nodec7', u'nodec8', u'nodeg36']
Compute Nodes : [u'nodei1', u'nodei2', u'nodei3']
Openstack Node : [u'nodec7', u'nodec8', u'nodeg36']
WebUI Node : nodec8
Analytics Nodes : [u'nodec7', u'nodec8', u'nodeg36']
Database Nodes : [u'nodec7', u'nodec8', u'nodeg36']
Physical Devices : [u'blr-mx1', u'hooper', u"'hooper'"]

nova-api.log:

2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions [req-71adf19d-cc99-41dc-a4e2-6500a4cc70fc 0fb41841bee2429a8389f30a34d9f29f e337e7852def47dc8bc604edb90f93f1 - - -] Unexpected exception in API method
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/images.py", line 145, in detail
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions **page_params)
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/image/api.py", line 68, in get_all
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions return session.detail(context, **kwargs)
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 300, in detail
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions for image in images:
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 254, in list
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions for image in paginate(params, return_request_id):
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 238, in paginate
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions images, resp = self._list(url, "images")
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 63, in _list
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions resp, body = self.client.get(url)
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 275, in get
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions return self._request('GET', url, **kwargs)
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 256, in _request
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions raise exc.CommunicationError(message=message)
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions CommunicationError: Error finding address for http://10.204.217.76:9292/v1/images/detail?is_public=none&limit=20: ('Connection aborted.', BadStatusLine("''",))
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions
2016-11-07 17:48:04.100 12887 INFO nova.api.openstack.wsgi [req-71adf19d-cc99-41dc-a4e2-6500a4cc70fc 0fb41841bee2429a8389f30a34d9f29f e337e7852def47dc8bc604edb90f93f1 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'glanceclient.exc.CommunicationError'>
2016-11-07 17:48:04.106 12887 INFO nova.osapi_compute.wsgi.server [req-71adf19d-cc99-41dc-a4e2-6500a4cc70fc 0fb41841bee2429a8389f30a34d9f29f e337e7852def47dc8bc604edb90f93f1 - - -] 192.168.192.6 "GET /v2.1/e337e7852def47dc8bc604edb90f93f1/images/detail HTTP/1.1" status: 500 len: 487 time: 6.0429599
2016-11-07 17:48:09.348 12931 INFO nova.osapi_compute.wsgi.server [req-e60d73b4-19a8-463c-95ce-159ae45c59de f3f0fe6232df495aa36d93be2f82e497 a00763e126364e3290de5eeb6d39040b - - -] 192.168.192.6 "GET /v2.1/a00763e126364e3290de5eeb6d39040b/flavors HTTP/1.1" status: 200 len: 2251 time: 0.0290070
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions [req-7b7b0386-c86f-4f26-a705-dab78f25406c 0751816c5a024734beea8b1118a9d52c 76e456e8a22a47c58ce9dde81c8df069 - - -] Unexpected exception in API method
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 629, in create
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions **create_kwargs)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 154, in inner
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1556, in create
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1139, in _create_instance
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions reservation_id, max_count)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 866, in _validate_and_build_base_options
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions pci_request_info, requested_networks)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1062, in create_pci_requests_for_sriov_ports
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions neutron = get_client(context, admin=True)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 139, in get_client
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions _ADMIN_AUTH = _load_auth_plugin(CONF)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in _load_auth_plugin
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions auth_plugin = ks_loading.load_auth_from_conf_options(conf, NEUTRON_GROUP)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/keystoneauth1/loading/conf.py", line 135, in load_from_conf_options
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions return plugin.load_from_options_getter(_getter, **kwargs)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/keystoneauth1/loading/base.py", line 148, in load_from_options_getter
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions return self.load_from_options(**kwargs)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/keystoneauth1/loading/base.py", line 123, in load_from_options
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions raise exceptions.MissingRequiredOptions(missing_required)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions
2016-11-07 17:48:42.170 12898 INFO nova.api.openstack.wsgi [req-7b7b0386-c86f-4f26-a705-dab78f25406c 0751816c5a024734beea8b1118a9d52c 76e456e8a22a47c58ce9dde81c8df069 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.

on R3.1 build38 mitaka HA setup, the mysql comes up after restarting it, but VM creation fails with following stack trace in nova-api.log

Setup details:
Config Nodes : [u'nodec7', u'nodec8', u'nodeg36'] 
Control Nodes : [u'nodec7', u'nodec8', u'nodeg36'] 
Compute Nodes : [u'nodei1', u'nodei2', u'nodei3'] 
Openstack Node : [u'nodec7', u'nodec8', u'nodeg36'] 
WebUI Node : nodec8 
Analytics Nodes : [u'nodec7', u'nodec8', u'nodeg36'] 
Database Nodes : [u'nodec7', u'nodec8', u'nodeg36'] 
Physical Devices : [u'blr-mx1', u'hooper', u"'hooper'"]

nova-api.log:

2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions [req-71adf19d-cc99-41dc-a4e2-6500a4cc70fc 0fb41841bee2429a8389f30a34d9f29f e337e7852def47dc8bc604edb90f93f1 - - -] Unexpected exception in API method
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/images.py", line 145, in detail
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions     **page_params)
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/image/api.py", line 68, in get_all
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions     return session.detail(context, **kwargs)
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 300, in detail
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions     for image in images:
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 254, in list
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions     for image in paginate(params, return_request_id):
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 238, in paginate
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions     images, resp = self._list(url, "images")
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 63, in _list
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions     resp, body = self.client.get(url)
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 275, in get
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions     return self._request('GET', url, **kwargs)
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 256, in _request
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions     raise exc.CommunicationError(message=message)
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions CommunicationError: Error finding address for http://10.204.217.76:9292/v1/images/detail?is_public=none&limit=20: ('Connection aborted.', BadStatusLine("''",))
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions 
2016-11-07 17:48:04.100 12887 INFO nova.api.openstack.wsgi [req-71adf19d-cc99-41dc-a4e2-6500a4cc70fc 0fb41841bee2429a8389f30a34d9f29f e337e7852def47dc8bc604edb90f93f1 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'glanceclient.exc.CommunicationError'>
2016-11-07 17:48:04.106 12887 INFO nova.osapi_compute.wsgi.server [req-71adf19d-cc99-41dc-a4e2-6500a4cc70fc 0fb41841bee2429a8389f30a34d9f29f e337e7852def47dc8bc604edb90f93f1 - - -] 192.168.192.6 "GET /v2.1/e337e7852def47dc8bc604edb90f93f1/images/detail HTTP/1.1" status: 500 len: 487 time: 6.0429599
2016-11-07 17:48:09.348 12931 INFO nova.osapi_compute.wsgi.server [req-e60d73b4-19a8-463c-95ce-159ae45c59de f3f0fe6232df495aa36d93be2f82e497 a00763e126364e3290de5eeb6d39040b - - -] 192.168.192.6 "GET /v2.1/a00763e126364e3290de5eeb6d39040b/flavors HTTP/1.1" status: 200 len: 2251 time: 0.0290070
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions [req-7b7b0386-c86f-4f26-a705-dab78f25406c 0751816c5a024734beea8b1118a9d52c 76e456e8a22a47c58ce9dde81c8df069 - - -] Unexpected exception in API method
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 629, in create
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions     **create_kwargs)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 154, in inner
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1556, in create
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1139, in _create_instance
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions     reservation_id, max_count)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 866, in _validate_and_build_base_options
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions     pci_request_info, requested_networks)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1062, in create_pci_requests_for_sriov_ports
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions     neutron = get_client(context, admin=True)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 139, in get_client
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions     _ADMIN_AUTH = _load_auth_plugin(CONF)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in _load_auth_plugin
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions     auth_plugin = ks_loading.load_auth_from_conf_options(conf, NEUTRON_GROUP)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/keystoneauth1/loading/conf.py", line 135, in load_from_conf_options
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions     return plugin.load_from_options_getter(_getter, **kwargs)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/keystoneauth1/loading/base.py", line 148, in load_from_options_getter
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions     return self.load_from_options(**kwargs)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/keystoneauth1/loading/base.py", line 123, in load_from_options
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions     raise exceptions.MissingRequiredOptions(missing_required)
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url
2016-11-07 17:48:42.169 12898 ERROR nova.api.openstack.extensions 
2016-11-07 17:48:42.170 12898 INFO nova.api.openstack.wsgi [req-7b7b0386-c86f-4f26-a705-dab78f25406c 0751816c5a024734beea8b1118a9d52c 76e456e8a22a47c58ce9dde81c8df069 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.