VMs failing to launch with kestone_V3

[aswani kumar]

BUILD 9 3.2 mitaka single node

Tested in single node with keystoneV3
not able to create vms in the default domain itself

2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager Traceback (most recent call last):
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1573, in _allocate_network_async
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager bind_host_id=bind_host_id)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 666, in allocate_for_instance
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager self._delete_ports(neutron, instance, created_port_ids)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager self.force_reraise()
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager six.reraise(self.type_, self.value, self.tb)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 655, in allocate_for_instance
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager security_group_ids, available_macs, dhcp_opts)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 254, in _create_port
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager port = port_client.create_port(port_req_body)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 97, in with_params
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager ret = self.function(instance, *args, **kwargs)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 662, in create_port
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager return self.post(self.ports_path, body=body)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 363, in post
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager headers=headers, params=params)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 286, in do_request
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager resp, replybody = self.httpclient.do_request(action, method, body=body)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 306, in do_request
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager return self.request(url, method, **kwargs)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 294, in request
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager resp = super(SessionClient, self).request(*args, **kwargs)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager return self.session.request(url, method, **kwargs)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager return func(*args, **kwargs)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 370, in request
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager auth_headers = self.get_auth_headers(auth)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 624, in get_auth_headers
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager return auth.get_headers(self, **kwargs)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 84, in get_headers
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager token = self.get_token(session)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 90, in get_token
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager return self.get_access(session).auth_token
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager self.auth_ref = self.get_auth_ref(session)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 181, in get_auth_ref
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager return self._plugin.get_auth_ref(session, **kwargs)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 167, in get_auth_ref
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager authenticated=False, log=False, **rkwargs)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 572, in post
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager return self.request(url, 'POST', **kwargs)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager return func(*args, **kwargs)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 467, in request
raise exceptions.from_response(resp, method, url)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager BadRequest: Expecting to find domain in project - the server could not comply with the request since it is either malforme
d or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-a72bf5c6-423a-4d05-b4bf-071e1628ebce)
2016-12-08 12:46:41.807 6475 ERROR nova.compute.manager
2016-12-08 12:46:42.139 6475 INFO nova.virt.libvirt.driver [req-a02a79cb-65fc-4133-8817-60214daeef31 3008fa85d9214795855a65c93fcd3518 34be44c330384b7ea210c38f146eaff6 - - -] [ins
tance: 78863ba3-608d-4856-9580-9f2511987534] Creating image
2016-12-08 12:46:42.661 6475 ERROR nova.compute.manager [req-a02a79cb-65fc-4133-8817-60214daeef31 3008fa85d9214795855a65c93fcd3518 34be44c330384b7ea210c38f146eaff6 - - -] [instan
ce: 78863ba3-608d-4856-9580-9f2511987534] Instance failed to spawn

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27050
Submitter: Deepinder Setia (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27062
Submitter: Deepinder Setia (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27062
Committed: http://github.org/Juniper/contrail-provisioning/commit/fd4d1dfd341fc4b77c5ef74fa8ee434e003c9fc2
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit fd4d1dfd341fc4b77c5ef74fa8ee434e003c9fc2
Author: Deepinder Setia <email address hidden>
Date: Thu Dec 8 16:55:44 2016 -0800

Nova compute needs user and project domain names for operations
in service tenant if keystone V3 is used. Otherwise domain information
isn't available and the token is rejected by keystone.

Change-Id: I2e679442db6514db74bff416dc4b6522979b070c
Closes-Bug: #1648434

[aswani kumar (aswanikumar90)]

Build 3.2 12
seeing this same bug on multinode setup not able to launch VM’s
I verified your fix https://review.opencontrail.org/27062 was there and nova.conf file got updated.
I am not seeing this issue in single node fix is working

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27347
Submitter: Deepinder Setia (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27347
Committed: http://github.org/Juniper/contrail-provisioning/commit/c2c27588bcb27782a90608504ad3f61ba0dc005d
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit c2c27588bcb27782a90608504ad3f61ba0dc005d
Author: Deepinder Setia <email address hidden>
Date: Thu Dec 15 10:30:07 2016 -0800

Nova compute needs user and project domain names for operations
in service tenant if keystone V3 is used. Otherwise domain information
isn't available and the token is rejected by keystone.

Change-Id: Ie89542a0a4a91af8dab616adb970ed65c845ab3e
Closes-Bug: #1648434

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27050
Committed: http://github.org/Juniper/contrail-provisioning/commit/10f2bd88bd6ed18d956d19480b6c7601bdb7a957
Submitter: Zuul (<email address hidden>)
Branch: master

commit 10f2bd88bd6ed18d956d19480b6c7601bdb7a957
Author: Deepinder Setia <email address hidden>
Date: Thu Dec 8 16:55:44 2016 -0800

Nova compute needs user and project domain names for operations
in service tenant if keystone V3 is used. Otherwise domain information
isn't available and the token is rejected by keystone.

Change-Id: I2e679442db6514db74bff416dc4b6522979b070c
Closes-Bug: #1648434

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27762
Submitter: Deepinder Setia (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27762
Committed: http://github.org/Juniper/contrail-provisioning/commit/03887ea0b6275373996145b0468315c6b68eb27a
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit 03887ea0b6275373996145b0468315c6b68eb27a
Author: Deepinder Setia <email address hidden>
Date: Thu Dec 8 16:55:44 2016 -0800

Nova compute needs user and project domain names for operations
in service tenant if keystone V3 is used. Otherwise domain information
isn't available and the token is rejected by keystone.

Change-Id: I2e679442db6514db74bff416dc4b6522979b070c
Closes-Bug: #1648434
(cherry picked from commit fd4d1dfd341fc4b77c5ef74fa8ee434e003c9fc2)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28077
Submitter: Deepinder Setia (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28078
Submitter: Deepinder Setia (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/28077
Committed: http://github.org/Juniper/contrail-provisioning/commit/9e72e23f56d36cbe0f2f92f124e71f31997e40c3
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit 9e72e23f56d36cbe0f2f92f124e71f31997e40c3
Author: Deepinder Setia <email address hidden>
Date: Thu Dec 15 10:30:07 2016 -0800

Nova compute needs user and project domain names for operations
in service tenant if keystone V3 is used. Otherwise domain information
isn't available and the token is rejected by keystone.

Change-Id: Ie89542a0a4a91af8dab616adb970ed65c845ab3e
Closes-Bug: #1648434
(cherry picked from commit c2c27588bcb27782a90608504ad3f61ba0dc005d)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/28078
Committed: http://github.org/Juniper/contrail-provisioning/commit/1772999b5b7a47d32c226617d6ac730a87d0f01e
Submitter: Zuul (<email address hidden>)
Branch: master

commit 1772999b5b7a47d32c226617d6ac730a87d0f01e
Author: Deepinder Setia <email address hidden>
Date: Thu Dec 15 10:30:07 2016 -0800

Nova compute needs user and project domain names for operations
in service tenant if keystone V3 is used. Otherwise domain information
isn't available and the token is rejected by keystone.

Change-Id: Ie89542a0a4a91af8dab616adb970ed65c845ab3e
Closes-Bug: #1648434
(cherry picked from commit c2c27588bcb27782a90608504ad3f61ba0dc005d)