Activity log for bug #1663866

Date Who What changed Old value New value Message
2017-02-11 10:39:33 sundarkh bug added bug
2017-02-11 10:39:40 sundarkh nominated for series juniperopenstack/r3.2
2017-02-11 10:39:40 sundarkh bug task added juniperopenstack/r3.2
2017-02-11 10:39:46 sundarkh juniperopenstack/r3.2: milestone r3.2.1.0
2017-02-11 10:39:52 sundarkh juniperopenstack/r3.2: assignee Abhay Joshi (abhayj)
2017-02-11 10:39:55 sundarkh juniperopenstack/r3.2: importance Undecided High
2017-02-11 10:40:56 sundarkh summary SM:R3.2build23:centos: Keystone authentication failure SM:R3.2build23:centos: nova error at keystoneclient.exceptions.NotFound
2017-02-11 10:41:12 sundarkh summary SM:R3.2build23:centos: nova error at keystoneclient.exceptions.NotFound SM:R3.2build23:centos: vm creation failure due to nova error at keystoneclient.exceptions.NotFound
2017-02-11 10:44:28 sundarkh description SM:R3.2build23:centos: Keystone authentication failure 1) Install SM 3.2 build 23 ; reimage the target with centos 72; 2) Provision gets completed in single node setup 3) vm creation fails with keystone authentication failure traceback Traceback (most recent call last): File "tcutils/wrappers.py", line 73, in wrapper result = function(self, *args, **kwargs) File "scripts/heat/test_heat.py", line 233, in test_ecmp_svc_creation_with_heat vms = self.config_vms(end_vn_list) File "common/heat/base.py", line 185, in config_vms vm_fix = self.verify_vms(stack, vn_list, env, stack_name) File "common/heat/base.py", line 200, in verify_vms vn_obj=vn_list[0].obj, vm_name=vm1_name, connections=self.connections)) File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 679, in useFixture reraise(*exc_info) File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 666, in useFixture fixture.setUp() File "/contrail-test/fixtures/vm_test.py", line 177, in setUp self.create() File "/contrail-test/fixtures/vm_test.py", line 214, in create fixed_ips=self.fixed_ips) File "/contrail-test/fixtures/openstack.py", line 101, in create_vm return self.nova_h.create_vm(vm_name=vm_name, image_name=image_name, vn_ids=vn_ids, **kwargs) File "/contrail-test/fixtures/nova_test.py", line 521, in create_vm min_count=count, max_count=count, userdata=userdata) File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 942, in create **boot_kwargs) File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 540, in _boot return_raw=return_raw, **kwargs) File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 169, in _create _resp, body = self.api.client.post(url, body=body) File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 449, in post return self._cs_request(url, 'POST', **kwargs) File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 424, in _cs_request resp, body = self._time_request(url, method, **kwargs) File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 397, in _time_request resp, body = self.request(url, method, **kwargs) File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 391, in request raise exceptions.from_response(resp, body, url, method) ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'keystoneclient.exceptions.NotFound'> (HTTP 500) (Request-ID: req-8db274e2-284e-42c2-935a-19926fbdda4c) Keystone log during vm creation ges/keystone/middleware/core.py:314 2017-02-11 02:13:38.580 4411 INFO keystone.common.wsgi [req-7231bfb6-59e7-4db5-bd7d-2eb0e73424b5 - - - - -] GET http://10.204.217.112:35357/v2.0/tenants 2017-02-11 02:13:38.580 4411 DEBUG keystone.policy.backends.rules [req-7231bfb6-59e7-4db5-bd7d-2eb0e73424b5 - - - - -] enforce admin_required: {'user_id': u'0684da2d42f54fe29b2360f1ef8cfbc0', u'is_admin': 0, u'roles': [u'admin'], 'tenant_id': u'aedd0a5017924a72ae166d2e4c2e5bc9'} enforce /usr/lib/python2.7/site-packages/keystone/policy/backends/rules.py:76 2017-02-11 02:13:38.597 4411 INFO eventlet.wsgi.server [req-7231bfb6-59e7-4db5-bd7d-2eb0e73424b5 - - - - -] 172.17.0.1 - - [11/Feb/2017 02:13:38] "GET /v2.0/tenants HTTP/1.1" 200 2010 0.038864 2017-02-11 02:13:38.803 4410 DEBUG keystone.middleware.core [req-f76c3fc9-135b-4299-9dcf-535e8b94bdb0 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. process_request /usr/lib/python2.7/site-packages/keystone/middleware/core.py:310 2017-02-11 02:13:38.805 4410 INFO keystone.common.wsgi [req-f76c3fc9-135b-4299-9dcf-535e8b94bdb0 - - - - -] POST http://10.204.217.112:35357/v2.0/tokens 2017-02-11 02:13:38.937 4410 INFO eventlet.wsgi.server [req-f76c3fc9-135b-4299-9dcf-535e8b94bdb0 - - - - -] 172.17.0.1 - - [11/Feb/2017 02:13:38] "POST /v2.0/tokens HTTP/1.1" 200 4335 0.134523 SM:R3.2build23:centos: Keystone authentication failure 1) Install SM 3.2 build 23 ; reimage the target with centos 72; 2) Provision gets completed in single node setup 3) vm creation fails with keystone authentication failure traceback Traceback (most recent call last):   File "tcutils/wrappers.py", line 73, in wrapper     result = function(self, *args, **kwargs)   File "scripts/heat/test_heat.py", line 233, in test_ecmp_svc_creation_with_heat     vms = self.config_vms(end_vn_list)   File "common/heat/base.py", line 185, in config_vms     vm_fix = self.verify_vms(stack, vn_list, env, stack_name)   File "common/heat/base.py", line 200, in verify_vms     vn_obj=vn_list[0].obj, vm_name=vm1_name, connections=self.connections))   File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 679, in useFixture     reraise(*exc_info)   File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 666, in useFixture     fixture.setUp()   File "/contrail-test/fixtures/vm_test.py", line 177, in setUp     self.create()   File "/contrail-test/fixtures/vm_test.py", line 214, in create     fixed_ips=self.fixed_ips)   File "/contrail-test/fixtures/openstack.py", line 101, in create_vm     return self.nova_h.create_vm(vm_name=vm_name, image_name=image_name, vn_ids=vn_ids, **kwargs)   File "/contrail-test/fixtures/nova_test.py", line 521, in create_vm     min_count=count, max_count=count, userdata=userdata)   File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 942, in create     **boot_kwargs)   File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 540, in _boot     return_raw=return_raw, **kwargs)   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 169, in _create     _resp, body = self.api.client.post(url, body=body)   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 449, in post     return self._cs_request(url, 'POST', **kwargs)   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 424, in _cs_request     resp, body = self._time_request(url, method, **kwargs)   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 397, in _time_request     resp, body = self.request(url, method, **kwargs)   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 391, in request     raise exceptions.from_response(resp, body, url, method) ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'keystoneclient.exceptions.NotFound'> (HTTP 500) (Request-ID: req-8db274e2-284e-42c2-935a-19926fbdda4c) Keystone log during vm creation ges/keystone/middleware/core.py:314 2017-02-11 02:13:38.580 4411 INFO keystone.common.wsgi [req-7231bfb6-59e7-4db5-bd7d-2eb0e73424b5 - - - - -] GET http://10.204.217.112:35357/v2.0/tenants 2017-02-11 02:13:38.580 4411 DEBUG keystone.policy.backends.rules [req-7231bfb6-59e7-4db5-bd7d-2eb0e73424b5 - - - - -] enforce admin_required: {'user_id': u'0684da2d42f54fe29b2360f1ef8cfbc0', u'is_admin': 0, u'roles': [u'admin'], 'tenant_id': u'aedd0a5017924a72ae166d2e4c2e5bc9'} enforce /usr/lib/python2.7/site-packages/keystone/policy/backends/rules.py:76 2017-02-11 02:13:38.597 4411 INFO eventlet.wsgi.server [req-7231bfb6-59e7-4db5-bd7d-2eb0e73424b5 - - - - -] 172.17.0.1 - - [11/Feb/2017 02:13:38] "GET /v2.0/tenants HTTP/1.1" 200 2010 0.038864 2017-02-11 02:13:38.803 4410 DEBUG keystone.middleware.core [req-f76c3fc9-135b-4299-9dcf-535e8b94bdb0 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. process_request /usr/lib/python2.7/site-packages/keystone/middleware/core.py:310 2017-02-11 02:13:38.805 4410 INFO keystone.common.wsgi [req-f76c3fc9-135b-4299-9dcf-535e8b94bdb0 - - - - -] POST http://10.204.217.112:35357/v2.0/tokens 2017-02-11 02:13:38.937 4410 INFO eventlet.wsgi.server [req-f76c3fc9-135b-4299-9dcf-535e8b94bdb0 - - - - -] 172.17.0.1 - - [11/Feb/2017 02:13:38] "POST /v2.0/tokens HTTP/1.1" 200 4335 0.134523 nova-api log 2017-02-11 02:40:05.493 8933 INFO nova.osapi_compute.wsgi.server [req-6e8d5b8f-3e8e-4c2c-983f-2a7bbb715d13 deb4e9145db54bd9b0edc9dba6a69107 56f97773a5f8446ba62a7def30424def - - -] 172.17.0.1 "GET /v2/56f97773a5f8446ba62a7def30424def/os-availability-zone/detail HTTP/1.1" status: 200 len: 781 time: 0.2021399 2017-02-11 02:40:05.511 8932 INFO nova.osapi_compute.wsgi.server [req-b9c23cc6-0d34-45e3-9cb9-f4f902c51659 deb4e9145db54bd9b0edc9dba6a69107 56f97773a5f8446ba62a7def30424def - - -] 172.17.0.1 "GET /v2/56f97773a5f8446ba62a7def30424def/os-services?binary=nova-compute HTTP/1.1" status: 200 len: 387 time: 0.0281222 2017-02-11 02:40:23.367 8932 DEBUG nova.osapi_compute.wsgi.server [-] (8932) accepted ('172.17.0.1', 56187) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826 2017-02-11 02:40:23.370 8932 DEBUG keystoneclient.session [-] REQ: curl -g -i -X GET http://10.204.217.112:35357/v3/auth/tokens -H "X-Subject-Token: {SHA1}7e0e3f69f1d3ba6529de87615118703dd267abad" -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}bead95805037ce9cb78470b70b464e5685e3cc2d" _http_log_request /usr/lib/python2.7/site-packages/keystoneclient/session.py:198 2017-02-11 02:40:23.479 8932 DEBUG keystoneclient.session [-] RESP: [200] content-length: 7142 x-subject-token: {SHA1}7e0e3f69f1d3ba6529de87615118703dd267abad vary: X-Auth-Token connection: keep-alive date: Sat, 11 Feb 2017 10:40:23 GMT content-type: application/json x-openstack-request-id: req-af261384-53a9-4fc3-a47d-df45577cef50 RESP BODY: {"token": {"methods": ["password", "token"], "roles": [{"id": "35e74ec26c984b6ca3fef38d6fa94aba", "name": "admin"}], "expires_at": "2017-02-11T11:40:23.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "aedd0a5017924a72ae166d2e4c2e5bc9", "name": "admin"}, "catalog": "<removed>", "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "0684da2d42f54fe29b2360f1ef8cfbc0", "name": "admin"}, "audit_ids": ["FZUYTTrlTA6x7VrdjuW38g"], "issued_at": "2017-02-11T10:40:23.354166"}} _http_log_response /usr/lib/python2.7/site-packages/keystoneclient/session.py:215 2017-02-11 02:40:23.482 8932 DEBUG nova.api.openstack.wsgi [req-e8ad12be-ddf9-43e9-9362-957cee23a4f8 0684da2d42f54fe29b2360f1ef8cfbc0 aedd0a5017924a72ae166d2e4c2e5bc9 - - -] Calling method '<bound method AvailabilityZoneController.detail of <nova.api.openstack.compute.availability_zone.AvailabilityZoneController object at 0x7226210>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:792 2017-02-11 02:40:23.523 8932 INFO nova.osapi_compute.wsgi.server [req-e8ad12be-ddf9-43e9-9362-957cee23a4f8 0684da2d42f54fe29b2360f1ef8cfbc0 aedd0a5017924a72ae166d2e4c2e5bc9 - - -] 172.17.0.1 "GET /v2/aedd0a5017924a72ae166d2e4c2e5bc9/os-availability-zone/detail HTTP/1.1" status: 200 len: 781 time: 0.1546571 2017-02-11 02:40:23.525 8932 DEBUG nova.osapi_compute.wsgi.server [-] (8932) accepted ('172.17.0.1', 56191) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826 2017-02-11 02:40:23.531 8932 DEBUG nova.api.openstack.wsgi [req-4ba507b2-ec58-4733-9e49-4526869b5bf7 0684da2d42f54fe29b2360f1ef8cfbc0 aedd0a5017924a72ae166d2e4c2e5bc9 - - -] Calling method '<bound method ServiceController.index of <nova.api.openstack.compute.services.ServiceController object at 0x75afa50>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:792 2017-02-11 02:40:23.545 8932 INFO nova.osapi_compute.wsgi.server [req-4ba507b2-ec58-4733-9e49-4526869b5bf7 0684da2d42f54fe29b2360f1ef8cfbc0 aedd0a5017924a72ae166d2e4c2e5bc9 - - -] 172.17.0.1 "GET /v2/aedd0a5017924a72ae166d2e4c2e5bc9/os-services?binary=nova-compute HTTP/1.1" status: 200 len: 387 time: 0.0195000 ~
2017-02-13 09:54:49 sundarkh summary SM:R3.2build23:centos: vm creation failure due to nova error at keystoneclient.exceptions.NotFound SM:R3.2build23: vm creation failure due to nova error at keystoneclient.exceptions.NotFound
2017-02-13 09:55:12 sundarkh description SM:R3.2build23:centos: Keystone authentication failure 1) Install SM 3.2 build 23 ; reimage the target with centos 72; 2) Provision gets completed in single node setup 3) vm creation fails with keystone authentication failure traceback Traceback (most recent call last):   File "tcutils/wrappers.py", line 73, in wrapper     result = function(self, *args, **kwargs)   File "scripts/heat/test_heat.py", line 233, in test_ecmp_svc_creation_with_heat     vms = self.config_vms(end_vn_list)   File "common/heat/base.py", line 185, in config_vms     vm_fix = self.verify_vms(stack, vn_list, env, stack_name)   File "common/heat/base.py", line 200, in verify_vms     vn_obj=vn_list[0].obj, vm_name=vm1_name, connections=self.connections))   File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 679, in useFixture     reraise(*exc_info)   File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 666, in useFixture     fixture.setUp()   File "/contrail-test/fixtures/vm_test.py", line 177, in setUp     self.create()   File "/contrail-test/fixtures/vm_test.py", line 214, in create     fixed_ips=self.fixed_ips)   File "/contrail-test/fixtures/openstack.py", line 101, in create_vm     return self.nova_h.create_vm(vm_name=vm_name, image_name=image_name, vn_ids=vn_ids, **kwargs)   File "/contrail-test/fixtures/nova_test.py", line 521, in create_vm     min_count=count, max_count=count, userdata=userdata)   File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 942, in create     **boot_kwargs)   File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 540, in _boot     return_raw=return_raw, **kwargs)   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 169, in _create     _resp, body = self.api.client.post(url, body=body)   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 449, in post     return self._cs_request(url, 'POST', **kwargs)   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 424, in _cs_request     resp, body = self._time_request(url, method, **kwargs)   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 397, in _time_request     resp, body = self.request(url, method, **kwargs)   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 391, in request     raise exceptions.from_response(resp, body, url, method) ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'keystoneclient.exceptions.NotFound'> (HTTP 500) (Request-ID: req-8db274e2-284e-42c2-935a-19926fbdda4c) Keystone log during vm creation ges/keystone/middleware/core.py:314 2017-02-11 02:13:38.580 4411 INFO keystone.common.wsgi [req-7231bfb6-59e7-4db5-bd7d-2eb0e73424b5 - - - - -] GET http://10.204.217.112:35357/v2.0/tenants 2017-02-11 02:13:38.580 4411 DEBUG keystone.policy.backends.rules [req-7231bfb6-59e7-4db5-bd7d-2eb0e73424b5 - - - - -] enforce admin_required: {'user_id': u'0684da2d42f54fe29b2360f1ef8cfbc0', u'is_admin': 0, u'roles': [u'admin'], 'tenant_id': u'aedd0a5017924a72ae166d2e4c2e5bc9'} enforce /usr/lib/python2.7/site-packages/keystone/policy/backends/rules.py:76 2017-02-11 02:13:38.597 4411 INFO eventlet.wsgi.server [req-7231bfb6-59e7-4db5-bd7d-2eb0e73424b5 - - - - -] 172.17.0.1 - - [11/Feb/2017 02:13:38] "GET /v2.0/tenants HTTP/1.1" 200 2010 0.038864 2017-02-11 02:13:38.803 4410 DEBUG keystone.middleware.core [req-f76c3fc9-135b-4299-9dcf-535e8b94bdb0 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. process_request /usr/lib/python2.7/site-packages/keystone/middleware/core.py:310 2017-02-11 02:13:38.805 4410 INFO keystone.common.wsgi [req-f76c3fc9-135b-4299-9dcf-535e8b94bdb0 - - - - -] POST http://10.204.217.112:35357/v2.0/tokens 2017-02-11 02:13:38.937 4410 INFO eventlet.wsgi.server [req-f76c3fc9-135b-4299-9dcf-535e8b94bdb0 - - - - -] 172.17.0.1 - - [11/Feb/2017 02:13:38] "POST /v2.0/tokens HTTP/1.1" 200 4335 0.134523 nova-api log 2017-02-11 02:40:05.493 8933 INFO nova.osapi_compute.wsgi.server [req-6e8d5b8f-3e8e-4c2c-983f-2a7bbb715d13 deb4e9145db54bd9b0edc9dba6a69107 56f97773a5f8446ba62a7def30424def - - -] 172.17.0.1 "GET /v2/56f97773a5f8446ba62a7def30424def/os-availability-zone/detail HTTP/1.1" status: 200 len: 781 time: 0.2021399 2017-02-11 02:40:05.511 8932 INFO nova.osapi_compute.wsgi.server [req-b9c23cc6-0d34-45e3-9cb9-f4f902c51659 deb4e9145db54bd9b0edc9dba6a69107 56f97773a5f8446ba62a7def30424def - - -] 172.17.0.1 "GET /v2/56f97773a5f8446ba62a7def30424def/os-services?binary=nova-compute HTTP/1.1" status: 200 len: 387 time: 0.0281222 2017-02-11 02:40:23.367 8932 DEBUG nova.osapi_compute.wsgi.server [-] (8932) accepted ('172.17.0.1', 56187) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826 2017-02-11 02:40:23.370 8932 DEBUG keystoneclient.session [-] REQ: curl -g -i -X GET http://10.204.217.112:35357/v3/auth/tokens -H "X-Subject-Token: {SHA1}7e0e3f69f1d3ba6529de87615118703dd267abad" -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}bead95805037ce9cb78470b70b464e5685e3cc2d" _http_log_request /usr/lib/python2.7/site-packages/keystoneclient/session.py:198 2017-02-11 02:40:23.479 8932 DEBUG keystoneclient.session [-] RESP: [200] content-length: 7142 x-subject-token: {SHA1}7e0e3f69f1d3ba6529de87615118703dd267abad vary: X-Auth-Token connection: keep-alive date: Sat, 11 Feb 2017 10:40:23 GMT content-type: application/json x-openstack-request-id: req-af261384-53a9-4fc3-a47d-df45577cef50 RESP BODY: {"token": {"methods": ["password", "token"], "roles": [{"id": "35e74ec26c984b6ca3fef38d6fa94aba", "name": "admin"}], "expires_at": "2017-02-11T11:40:23.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "aedd0a5017924a72ae166d2e4c2e5bc9", "name": "admin"}, "catalog": "<removed>", "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "0684da2d42f54fe29b2360f1ef8cfbc0", "name": "admin"}, "audit_ids": ["FZUYTTrlTA6x7VrdjuW38g"], "issued_at": "2017-02-11T10:40:23.354166"}} _http_log_response /usr/lib/python2.7/site-packages/keystoneclient/session.py:215 2017-02-11 02:40:23.482 8932 DEBUG nova.api.openstack.wsgi [req-e8ad12be-ddf9-43e9-9362-957cee23a4f8 0684da2d42f54fe29b2360f1ef8cfbc0 aedd0a5017924a72ae166d2e4c2e5bc9 - - -] Calling method '<bound method AvailabilityZoneController.detail of <nova.api.openstack.compute.availability_zone.AvailabilityZoneController object at 0x7226210>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:792 2017-02-11 02:40:23.523 8932 INFO nova.osapi_compute.wsgi.server [req-e8ad12be-ddf9-43e9-9362-957cee23a4f8 0684da2d42f54fe29b2360f1ef8cfbc0 aedd0a5017924a72ae166d2e4c2e5bc9 - - -] 172.17.0.1 "GET /v2/aedd0a5017924a72ae166d2e4c2e5bc9/os-availability-zone/detail HTTP/1.1" status: 200 len: 781 time: 0.1546571 2017-02-11 02:40:23.525 8932 DEBUG nova.osapi_compute.wsgi.server [-] (8932) accepted ('172.17.0.1', 56191) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826 2017-02-11 02:40:23.531 8932 DEBUG nova.api.openstack.wsgi [req-4ba507b2-ec58-4733-9e49-4526869b5bf7 0684da2d42f54fe29b2360f1ef8cfbc0 aedd0a5017924a72ae166d2e4c2e5bc9 - - -] Calling method '<bound method ServiceController.index of <nova.api.openstack.compute.services.ServiceController object at 0x75afa50>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:792 2017-02-11 02:40:23.545 8932 INFO nova.osapi_compute.wsgi.server [req-4ba507b2-ec58-4733-9e49-4526869b5bf7 0684da2d42f54fe29b2360f1ef8cfbc0 aedd0a5017924a72ae166d2e4c2e5bc9 - - -] 172.17.0.1 "GET /v2/aedd0a5017924a72ae166d2e4c2e5bc9/os-services?binary=nova-compute HTTP/1.1" status: 200 len: 387 time: 0.0195000 ~ SM:R3.2build23:centos: Keystone authentication failure 1) Install SM 3.2 build 23 ; reimage the target with centos 72; 2) Provision gets completed in single node setup 3) vm creation fails with keystone authentication failure traceback Traceback (most recent call last):   File "tcutils/wrappers.py", line 73, in wrapper     result = function(self, *args, **kwargs)   File "scripts/heat/test_heat.py", line 233, in test_ecmp_svc_creation_with_heat     vms = self.config_vms(end_vn_list)   File "common/heat/base.py", line 185, in config_vms     vm_fix = self.verify_vms(stack, vn_list, env, stack_name)   File "common/heat/base.py", line 200, in verify_vms     vn_obj=vn_list[0].obj, vm_name=vm1_name, connections=self.connections))   File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 679, in useFixture     reraise(*exc_info)   File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 666, in useFixture     fixture.setUp()   File "/contrail-test/fixtures/vm_test.py", line 177, in setUp     self.create()   File "/contrail-test/fixtures/vm_test.py", line 214, in create     fixed_ips=self.fixed_ips)   File "/contrail-test/fixtures/openstack.py", line 101, in create_vm     return self.nova_h.create_vm(vm_name=vm_name, image_name=image_name, vn_ids=vn_ids, **kwargs)   File "/contrail-test/fixtures/nova_test.py", line 521, in create_vm     min_count=count, max_count=count, userdata=userdata)   File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 942, in create     **boot_kwargs)   File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 540, in _boot     return_raw=return_raw, **kwargs)   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 169, in _create     _resp, body = self.api.client.post(url, body=body)   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 449, in post     return self._cs_request(url, 'POST', **kwargs)   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 424, in _cs_request     resp, body = self._time_request(url, method, **kwargs)   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 397, in _time_request     resp, body = self.request(url, method, **kwargs)   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 391, in request     raise exceptions.from_response(resp, body, url, method) ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'keystoneclient.exceptions.NotFound'> (HTTP 500) (Request-ID: req-8db274e2-284e-42c2-935a-19926fbdda4c) Keystone log during vm creation ges/keystone/middleware/core.py:314 2017-02-11 02:13:38.580 4411 INFO keystone.common.wsgi [req-7231bfb6-59e7-4db5-bd7d-2eb0e73424b5 - - - - -] GET http://10.204.217.112:35357/v2.0/tenants 2017-02-11 02:13:38.580 4411 DEBUG keystone.policy.backends.rules [req-7231bfb6-59e7-4db5-bd7d-2eb0e73424b5 - - - - -] enforce admin_required: {'user_id': u'0684da2d42f54fe29b2360f1ef8cfbc0', u'is_admin': 0, u'roles': [u'admin'], 'tenant_id': u'aedd0a5017924a72ae166d2e4c2e5bc9'} enforce /usr/lib/python2.7/site-packages/keystone/policy/backends/rules.py:76 2017-02-11 02:13:38.597 4411 INFO eventlet.wsgi.server [req-7231bfb6-59e7-4db5-bd7d-2eb0e73424b5 - - - - -] 172.17.0.1 - - [11/Feb/2017 02:13:38] "GET /v2.0/tenants HTTP/1.1" 200 2010 0.038864 2017-02-11 02:13:38.803 4410 DEBUG keystone.middleware.core [req-f76c3fc9-135b-4299-9dcf-535e8b94bdb0 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. process_request /usr/lib/python2.7/site-packages/keystone/middleware/core.py:310 2017-02-11 02:13:38.805 4410 INFO keystone.common.wsgi [req-f76c3fc9-135b-4299-9dcf-535e8b94bdb0 - - - - -] POST http://10.204.217.112:35357/v2.0/tokens 2017-02-11 02:13:38.937 4410 INFO eventlet.wsgi.server [req-f76c3fc9-135b-4299-9dcf-535e8b94bdb0 - - - - -] 172.17.0.1 - - [11/Feb/2017 02:13:38] "POST /v2.0/tokens HTTP/1.1" 200 4335 0.134523 nova-api log 2017-02-11 02:40:05.493 8933 INFO nova.osapi_compute.wsgi.server [req-6e8d5b8f-3e8e-4c2c-983f-2a7bbb715d13 deb4e9145db54bd9b0edc9dba6a69107 56f97773a5f8446ba62a7def30424def - - -] 172.17.0.1 "GET /v2/56f97773a5f8446ba62a7def30424def/os-availability-zone/detail HTTP/1.1" status: 200 len: 781 time: 0.2021399 2017-02-11 02:40:05.511 8932 INFO nova.osapi_compute.wsgi.server [req-b9c23cc6-0d34-45e3-9cb9-f4f902c51659 deb4e9145db54bd9b0edc9dba6a69107 56f97773a5f8446ba62a7def30424def - - -] 172.17.0.1 "GET /v2/56f97773a5f8446ba62a7def30424def/os-services?binary=nova-compute HTTP/1.1" status: 200 len: 387 time: 0.0281222 2017-02-11 02:40:23.367 8932 DEBUG nova.osapi_compute.wsgi.server [-] (8932) accepted ('172.17.0.1', 56187) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826 2017-02-11 02:40:23.370 8932 DEBUG keystoneclient.session [-] REQ: curl -g -i -X GET http://10.204.217.112:35357/v3/auth/tokens -H "X-Subject-Token: {SHA1}7e0e3f69f1d3ba6529de87615118703dd267abad" -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}bead95805037ce9cb78470b70b464e5685e3cc2d" _http_log_request /usr/lib/python2.7/site-packages/keystoneclient/session.py:198 2017-02-11 02:40:23.479 8932 DEBUG keystoneclient.session [-] RESP: [200] content-length: 7142 x-subject-token: {SHA1}7e0e3f69f1d3ba6529de87615118703dd267abad vary: X-Auth-Token connection: keep-alive date: Sat, 11 Feb 2017 10:40:23 GMT content-type: application/json x-openstack-request-id: req-af261384-53a9-4fc3-a47d-df45577cef50 RESP BODY: {"token": {"methods": ["password", "token"], "roles": [{"id": "35e74ec26c984b6ca3fef38d6fa94aba", "name": "admin"}], "expires_at": "2017-02-11T11:40:23.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "aedd0a5017924a72ae166d2e4c2e5bc9", "name": "admin"}, "catalog": "<removed>", "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "0684da2d42f54fe29b2360f1ef8cfbc0", "name": "admin"}, "audit_ids": ["FZUYTTrlTA6x7VrdjuW38g"], "issued_at": "2017-02-11T10:40:23.354166"}}  _http_log_response /usr/lib/python2.7/site-packages/keystoneclient/session.py:215 2017-02-11 02:40:23.482 8932 DEBUG nova.api.openstack.wsgi [req-e8ad12be-ddf9-43e9-9362-957cee23a4f8 0684da2d42f54fe29b2360f1ef8cfbc0 aedd0a5017924a72ae166d2e4c2e5bc9 - - -] Calling method '<bound method AvailabilityZoneController.detail of <nova.api.openstack.compute.availability_zone.AvailabilityZoneController object at 0x7226210>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:792 2017-02-11 02:40:23.523 8932 INFO nova.osapi_compute.wsgi.server [req-e8ad12be-ddf9-43e9-9362-957cee23a4f8 0684da2d42f54fe29b2360f1ef8cfbc0 aedd0a5017924a72ae166d2e4c2e5bc9 - - -] 172.17.0.1 "GET /v2/aedd0a5017924a72ae166d2e4c2e5bc9/os-availability-zone/detail HTTP/1.1" status: 200 len: 781 time: 0.1546571 2017-02-11 02:40:23.525 8932 DEBUG nova.osapi_compute.wsgi.server [-] (8932) accepted ('172.17.0.1', 56191) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826 2017-02-11 02:40:23.531 8932 DEBUG nova.api.openstack.wsgi [req-4ba507b2-ec58-4733-9e49-4526869b5bf7 0684da2d42f54fe29b2360f1ef8cfbc0 aedd0a5017924a72ae166d2e4c2e5bc9 - - -] Calling method '<bound method ServiceController.index of <nova.api.openstack.compute.services.ServiceController object at 0x75afa50>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:792 2017-02-11 02:40:23.545 8932 INFO nova.osapi_compute.wsgi.server [req-4ba507b2-ec58-4733-9e49-4526869b5bf7 0684da2d42f54fe29b2360f1ef8cfbc0 aedd0a5017924a72ae166d2e4c2e5bc9 - - -] 172.17.0.1 "GET /v2/aedd0a5017924a72ae166d2e4c2e5bc9/os-services?binary=nova-compute HTTP/1.1" status: 200 len: 387 time: 0.0195000 ~ Seen with centos,ubuntu Distros
2017-02-13 09:57:07 sundarkh tags server-manager blok server-manager
2017-02-13 09:57:13 sundarkh tags blok server-manager blocker server-manager
2017-02-13 18:33:06 Abhay Joshi juniperopenstack/r3.2: assignee Abhay Joshi (abhayj) Dheeraj Gautam (dgautam)
2017-02-13 18:33:22 Abhay Joshi juniperopenstack: assignee Dheeraj Gautam (dgautam)
2017-02-13 23:16:18 Jeba Paulaiyan nominated for series juniperopenstack/trunk
2017-02-13 23:16:18 Jeba Paulaiyan bug task added juniperopenstack/trunk
2017-02-13 23:16:26 Jeba Paulaiyan juniperopenstack/trunk: importance Undecided High
2017-02-13 23:16:29 Jeba Paulaiyan juniperopenstack/trunk: milestone r4.0
2017-02-17 06:30:44 OpenContrail Admin juniperopenstack/r3.2: status New In Progress
2017-02-17 06:45:39 OpenContrail Admin juniperopenstack/trunk: status New In Progress
2017-02-17 15:35:01 OpenContrail Admin juniperopenstack/r3.2: status In Progress Fix Committed
2017-02-17 15:35:22 OpenContrail Admin juniperopenstack/trunk: status In Progress Fix Committed
2017-03-10 06:58:30 sundarkh juniperopenstack/r3.2: status Fix Committed New
2017-03-16 23:09:27 OpenContrail Admin juniperopenstack/r3.2: status New In Progress
2017-03-17 17:12:06 OpenContrail Admin juniperopenstack/r3.2: status In Progress Fix Committed
2017-03-22 05:52:32 Jeba Paulaiyan juniperopenstack/r3.2: milestone r3.2.1.0 r3.2.2.0