Activity log for bug #1549074

Date Who What changed Old value New value Message
2016-02-24 03:10:47 Thiyagu bug added bug
2016-02-24 03:10:47 Thiyagu attachment added Config_and_logs.txt https://bugs.launchpad.net/bugs/1549074/+attachment/4579555/+files/Config_and_logs.txt
2016-02-24 03:14:09 Thiyagu description This is 3 Node Liberty setup (controller, Network and Compute) configured with neutron Network option 2 : Self servicing network and is with openvswitch. I found everything is working fine except while launching instance I'm getting error as follows , By tracing keystone.log and nova-api.log and based on time stamp I see Keystone fails the authentication followed by nova-api started throwing the exception errors. Attached the config files and logs for you reference ... Request you to help me ASAP. Thank You ****************************************************** 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions [req-61818a91-ddff-4d25-8855-0dcde336e6b0 1cd7f867708b46e0a795abad8a7a66c3 6e6b9259a526447fa4274442b81c3b55 - - -] Unexpected exception in API method 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 611, in create 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions **create_kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 149, in inner 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1581, in create 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1181, in _create_instance 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions auto_disk_config, reservation_id, max_count) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 955, in _validate_and_build_base_options 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions pci_request_info, requested_networks) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1059, in create_pci_requests_for_sriov_ports 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions neutron = get_client(context, admin=True) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 237, in get_client 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions auth_token = _ADMIN_AUTH.get_token(_SESSION) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return self.get_access(session).auth_token 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions self.auth_ref = self.get_auth_ref(session) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/generic/base.py", line 186, in get_auth_ref 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return self._plugin.get_auth_ref(session, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py", line 190, in get_auth_ref 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions authenticated=False, log=False, **rkwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return self.request(url, 'POST', **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions raise exceptions.from_response(resp, method, url) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-d2ea540d-cdbc-44b9-bbed-ab2fc92ebbe4) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions 2016-02-23 18:46:50.048 17814 INFO nova.api.openstack.wsgi [req-61818a91-ddff-4d25-8855-0dcde336e6b0 1cd7f867708b46e0a795abad8a7a66c3 6e6b9259a526447fa4274442b81c3b55 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'keystoneclient.exceptions.Unauthorized'> ****************************************************** This is 3 Node Liberty setup (controller, Network and Compute) configured with neutron Network option 2 : Self servicing network and is with openvswitch. I found everything is working fine except while launching instance I'm getting error as follows , By tracing keystone.log and nova-api.log and based on time stamp I see Keystone fails the authentication followed by nova-api started throwing the exception errors. Attached the config files and logs for you reference ... Request you to help me ASAP. Controller: 10.182.131.93 Network: 10.182.131.84 Compute: 10.182.131.87 Thank You ****************************************************** 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions [req-61818a91-ddff-4d25-8855-0dcde336e6b0 1cd7f867708b46e0a795abad8a7a66c3 6e6b9259a526447fa4274442b81c3b55 - - -] Unexpected exception in API method 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 611, in create 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions **create_kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 149, in inner 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1581, in create 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1181, in _create_instance 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions auto_disk_config, reservation_id, max_count) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 955, in _validate_and_build_base_options 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions pci_request_info, requested_networks) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1059, in create_pci_requests_for_sriov_ports 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions neutron = get_client(context, admin=True) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 237, in get_client 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions auth_token = _ADMIN_AUTH.get_token(_SESSION) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return self.get_access(session).auth_token 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions self.auth_ref = self.get_auth_ref(session) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/generic/base.py", line 186, in get_auth_ref 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return self._plugin.get_auth_ref(session, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py", line 190, in get_auth_ref 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions authenticated=False, log=False, **rkwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return self.request(url, 'POST', **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions raise exceptions.from_response(resp, method, url) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-d2ea540d-cdbc-44b9-bbed-ab2fc92ebbe4) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions 2016-02-23 18:46:50.048 17814 INFO nova.api.openstack.wsgi [req-61818a91-ddff-4d25-8855-0dcde336e6b0 1cd7f867708b46e0a795abad8a7a66c3 6e6b9259a526447fa4274442b81c3b55 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'keystoneclient.exceptions.Unauthorized'> ******************************************************
2016-02-26 08:55:07 jichenjc bug added subscriber jichenjc
2016-03-01 10:14:06 Markus Zoeller (markus_z) nova: status New Incomplete
2016-04-07 09:57:05 Eyal Traitel attachment added n-api.log https://bugs.launchpad.net/nova/+bug/1549074/+attachment/4627326/+files/n-api.log
2016-04-07 09:57:27 Eyal Traitel attachment added key.log https://bugs.launchpad.net/nova/+bug/1549074/+attachment/4627327/+files/key.log
2016-04-07 09:57:52 Eyal Traitel attachment added nova.conf https://bugs.launchpad.net/nova/+bug/1549074/+attachment/4627328/+files/nova.conf
2016-04-07 09:58:05 Eyal Traitel attachment added neutron.conf https://bugs.launchpad.net/nova/+bug/1549074/+attachment/4627329/+files/neutron.conf
2016-04-09 07:04:59 Eyal Traitel nova: status Incomplete New
2016-04-18 18:09:19 Sean Dague nova: status New Incomplete
2016-06-18 04:18:24 Launchpad Janitor nova: status Incomplete Expired