Nova boot fails with Neutron Client Exception for SSL enabled cluster

[musharani]

While creating VM the below neutron client exception is thrown with SSL enabled setup.

ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'neutronclient.common.exceptions.InternalServerError'> (HTTP 500) (Request-ID: req-25aea402-9780-4f5a-bd6c-1fd12576b840)

If you check in the neutron server log it is throwing permission denied error message for /etc/contrail/ssl/private/contrail.key.

still setup in the same state:
node: nodel10
username/password: root/c0ntrail123

From neutron server log:
------------------------
2016-12-12 15:51:49.869 32468 ERROR neutron.api.v2.resource [req-31720e32-3979-43fe-a3ce-a87b8d8f86c3 796d6de4733b4dd8b728caa4ea426fe9 3765ed7b37be49c7b35d1339ae0f398e - - -] show failed
2016-12-12 15:51:49.869 32468 ERROR neutron.api.v2.resource Traceback (most recent call last):
2016-12-12 15:51:49.869 32468 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/resource.py", line 84, in resource
2016-12-12 15:51:49.869 32468 ERROR neutron.api.v2.resource result = method(request=request, **args)
2016-12-12 15:51:49.869 32468 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/extensions/quotasv2.py", line 92, in show
2016-12-12 15:51:49.869 32468 ERROR neutron.api.v2.resource return {self._resource_name: self._get_quotas(request, id)}
2016-12-12 15:51:49.869 32468 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/extensions/quotasv2.py", line 67, in _get_quotas
2016-12-12 15:51:49.869 32468 ERROR neutron.api.v2.resource tenant_id)
2016-12-12 15:51:49.869 32468 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/quota/driver.py", line 134, in get_tenant_quotas
2016-12-12 15:51:49.869 32468 ERROR neutron.api.v2.resource default_project = cls._get_vnc_conn().project_read(
2016-12-12 15:51:49.869 32468 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/quota/driver.py", line 87, in _get_vnc_conn
2016-12-12 15:51:49.869 32468 ERROR neutron.api.v2.resource auth_token_url=auth_token_url)
2016-12-12 15:51:49.869 32468 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 246, in __init__
2016-12-12 15:51:49.869 32468 ERROR neutron.api.v2.resource certs)
2016-12-12 15:51:49.869 32468 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/cfgm_common/utils.py", line 146, in getCertKeyCaBundle
2016-12-12 15:51:49.869 32468 ERROR neutron.api.v2.resource if os.path.getmtime(cert) > bundle_mod_time:
2016-12-12 15:51:49.869 32468 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/genericpath.py", line 54, in getmtime
2016-12-12 15:51:49.869 32468 ERROR neutron.api.v2.resource return os.stat(filename).st_mtime
2016-12-12 15:51:49.869 32468 ERROR neutron.api.v2.resource OSError: [Errno 13] Permission denied: '/etc/contrail/ssl/private/contrail.key'
2016-12-12 15:51:49.869 32468 ERROR neutron.api.v2.resource

From nova-api.log:
------------------
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions [req-ec5aa0d3-3cc7-4bf2-9e24-2791ff225419 796d6de4733b4dd8b728caa4ea426fe9 3765ed7b37be49c7b35d1339ae0f398e - - -] Unexpected exception in API method
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 629, in create
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions **create_kwargs)
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 154, in inner
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs)
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1556, in create
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota)
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1139, in _create_instance
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions reservation_id, max_count)
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 834, in _validate_and_build_base_options
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions requested_networks, max_count)
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 448, in _check_requested_networks
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions max_count)
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1181, in validate_networks
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions quotas = neutron.show_quota(tenant_id=context.project_id)['quota']
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 97, in with_params
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions ret = self.function(instance, *args, **kwargs)
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 625, in show_quota
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions return self.get(self.quota_path % (tenant_id), params=_params)
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 358, in get
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions headers=headers, params=params)
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 335, in retry_request
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions headers=headers, params=params)
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 298, in do_request
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions self._handle_fault_response(status_code, replybody, resp)
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 273, in _handle_fault_response
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions exception_handler_v20(status_code, error_body)
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 84, in exception_handler_v20
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions request_ids=request_ids)
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions InternalServerError: Request Failed: internal server error while processing your request.
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions Neutron server returns request_ids: ['req-31720e32-3979-43fe-a3ce-a87b8d8f86c3']
2016-12-12 15:51:49.874 5220 ERROR nova.api.openstack.extensions
2016-12-12 15:51:49.876 5220 INFO nova.api.openstack.wsgi [req-ec5aa0d3-3cc7-4bf2-9e24-2791ff225419 796d6de4733b4dd8b728caa4ea426fe9 3765ed7b37be49c7b35d1339ae0f398e - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'neutronclient.common.exceptions.InternalServerError'>

[Ignatious Johnson Christopher (ijohnson-x)]

I see two issue here,

1. Neutron client not honoring insecure flag, We need this(https://review.openstack.org/#/c/357803/1/neutronclient/shell.py ) upstream fix in our neutron client package,
2. As I said earlier the self-signed certificates are created with first node IP or VIP as the commonName, In Mutli-node setup(no VIP) we update the endpoint of neutron with last node IP, in ContrailPlugin.ini we set individual config node ip. This causes the cert verification to fail. In case of HA setup this won’t happen because the endpoint will be pointing to VIP and ContrailPlugin.ini also will be populated with VIP.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27166
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27167
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27168
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27167
Committed: http://github.org/Juniper/contrail-provisioning/commit/4214f36f220c1c12d625d2bb0ef62b57c6243f13
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 4214f36f220c1c12d625d2bb0ef62b57c6243f13
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon Dec 12 22:05:01 2016 -0800

Adding new argument first_cfgm_ip to setup-vnc-config entrypoint script.
Which will be used to populate the ContrailPluginIni and create neutron
endpoint in a SSL enabled cluster non HA setup(no VIP).

Change-Id: I2f55802ae1eb9dfd281e5a30de993b76885f4d1d
Partial-Bug: 1649239

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27206
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27207
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27208
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27207
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/9bf7645e001a2511ad112bb9198c4fe8b424c2d8
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 9bf7645e001a2511ad112bb9198c4fe8b424c2d8
Author: Ignatious Johnson Christopher <email address hidden>
Date: Tue Dec 13 09:42:22 2016 -0800

Passing new argument first_cfgm_ip to setup-vnc-config entrypoint script.
Which will be used to populate the ContrailPluginIni and create neutron
endpoint in a SSL enabled cluster non HA setup(no VIP).

Change-Id: I3a2ca5c07cd38c8573d1275654dcb53d30cb0059
Partial-Bug: 1649239

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27166
Committed: http://github.org/Juniper/contrail-provisioning/commit/c8cb358982fcd320faa3ffa8ed03fe62d4649776
Submitter: Zuul (<email address hidden>)
Branch: master

commit c8cb358982fcd320faa3ffa8ed03fe62d4649776
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon Dec 12 22:05:01 2016 -0800

Adding new argument first_cfgm_ip to setup-vnc-config entrypoint script.
Which will be used to populate the ContrailPluginIni and create neutron
endpoint in a SSL enabled cluster non HA setup(no VIP).

Change-Id: I2f55802ae1eb9dfd281e5a30de993b76885f4d1d
Partial-Bug: 1649239

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27168
Committed: http://github.org/Juniper/contrail-provisioning/commit/259e4632e32b09a96c706cfd96c4da163ffc1087
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit 259e4632e32b09a96c706cfd96c4da163ffc1087
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon Dec 12 22:05:01 2016 -0800

Adding new argument first_cfgm_ip to setup-vnc-config entrypoint script.
Which will be used to populate the ContrailPluginIni and create neutron
endpoint in a SSL enabled cluster non HA setup(no VIP).

Change-Id: I2f55802ae1eb9dfd281e5a30de993b76885f4d1d
Partial-Bug: 1649239

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27208
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/a8d340231e5316a0c24f71e9fdd03409316046fc
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit a8d340231e5316a0c24f71e9fdd03409316046fc
Author: Ignatious Johnson Christopher <email address hidden>
Date: Tue Dec 13 09:42:22 2016 -0800

Passing new argument first_cfgm_ip to setup-vnc-config entrypoint script.
Which will be used to populate the ContrailPluginIni and create neutron
endpoint in a SSL enabled cluster non HA setup(no VIP).

Change-Id: I3a2ca5c07cd38c8573d1275654dcb53d30cb0059
Partial-Bug: 1649239

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27206
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/e5c750fed90ccce445c9c5384775f7883f6ce4f1
Submitter: Zuul (<email address hidden>)
Branch: master

commit e5c750fed90ccce445c9c5384775f7883f6ce4f1
Author: Ignatious Johnson Christopher <email address hidden>
Date: Tue Dec 13 09:42:22 2016 -0800

Passing new argument first_cfgm_ip to setup-vnc-config entrypoint script.
Which will be used to populate the ContrailPluginIni and create neutron
endpoint in a SSL enabled cluster non HA setup(no VIP).

Change-Id: I3a2ca5c07cd38c8573d1275654dcb53d30cb0059
Partial-Bug: 1649239

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27899
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27899
Committed: http://github.org/Juniper/contrail-provisioning/commit/58e3f4b314cd177231f0772e54c89db70a260d9f
Submitter: Zuul (<email address hidden>)
Branch: R3.1.1.x

commit 58e3f4b314cd177231f0772e54c89db70a260d9f
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon Dec 5 14:16:03 2016 -0800

Disable unncecessary sending resource operation notification to DHCP agent

Change-Id: I732ac3359286151210778dd2acd43e53ad9dd397
Closes-Bug: 1639014
(cherry picked from commit d4a4b002ad97ad87090e5da87736b01ce6d2e527)

keystone_ssl_enabled is not initialized in the code path, when orchestrator
is set to 'none', setup contrail analytics comomnents pass none as
orchestrator, so initializing it.

Change-Id: I0b95c83a067f004f17a46cab84b95842b3a76037
Closes-Bug: 1647512
(cherry picked from commit 7b9b70a296a6e77389940e8a2094fcba1c45057b)

Adding new argument first_cfgm_ip to setup-vnc-config entrypoint script.
Which will be used to populate the ContrailPluginIni and create neutron
endpoint in a SSL enabled cluster non HA setup(no VIP).

Change-Id: I2f55802ae1eb9dfd281e5a30de993b76885f4d1d
Partial-Bug: 1649239
(cherry picked from commit 259e4632e32b09a96c706cfd96c4da163ffc1087)

Making sure that the ssl cert directories are with read
permission for the group users as well.

Change-Id: Iaad4670faf2ccf6ffe323fd7fb3580bf43f3f7ba
Closes-Bug: 1651275
(cherry picked from commit 405d8afd3ca75b999312708fe3e6f0f8e998e32d)