When setting `keystone_service_internaluri_insecure: true` in user_variables.yml, the generated nova.conf file is setting `insecure = True` in the [keystone_auth] section, but not in the [neutron] section. In an environment where publicURL is using https with a self-signed cert, you get the following trace in nova-compute.log:
2016-04-06 03:07:57.640 28418 ERROR oslo_messaging.rpc.dispatcher [req-96f7de0a-fe8e-420f-8d7c-411da4a7f2de fa05ffd2ddc440bb88ca8c96424d6f2c 351448c1c0af460a81cf9c579d4dd44d - - -] Exception during message handling: SSL exception connecting to https://72.32.112.255:5000/v3/auth/tokens: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 6936, in start_instance
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher return self.manager.start_instance(ctxt, instance)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher payload)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 333, in decorated_function
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance_uuid=instance_uuid)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 304, in decorated_function
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 383, in decorated_function
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 361, in decorated_function
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 349, in decorated_function
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 2852, in start_instance
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher self._power_on(context, instance)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 2820, in _power_on
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher network_info = self._get_instance_nw_info(context, instance)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 1383, in _get_instance_nw_info
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher return self.network_api.get_instance_nw_info(context, instance)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 747, in get_instance_nw_info
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher preexisting_port_ids)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 763, in _get_instance_nw_info
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher preexisting_port_ids)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1515, in _build_network_info_model
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher client = get_client(context, admin=True)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 205, in get_client
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher auth_token = _ADMIN_AUTH.get_token(_SESSION)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 104, in get_token
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher return self.get_access(session).auth_token
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 144, in get_access
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher self.auth_ref = self.get_auth_ref(session)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/generic/base.py", line 176, in get_auth_ref
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher return self._plugin.get_auth_ref(session, **kwargs)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/v3/base.py", line 173, in get_auth_ref
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher authenticated=False, log=False, **rkwargs)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 499, in post
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher return self.request(url, 'POST', **kwargs)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/keystoneclient/utils.py", line 318, in inner
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher return func(*args, **kwargs)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 384, in request
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher resp = send(**kwargs)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 422, in _send_request
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher raise exceptions.SSLError(msg)
2016-04-06 03:07:57.640 28418 TRACE oslo_messaging.rpc.dispatcher SSLError: SSL exception connecting to https://72.32.112.255:5000/v3/auth/tokens: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
Adding `insecure = True` to the [neutron] section resolves this issue. This can be set via an override:
nova_nova_conf_overrides:
neutron:
insecure: true
However, given that we are setting the option in the [keystone_auth] section, we should probably be adding it to the [neutron] section as well.
Hi Jordan can you pass me your nova.conf configurations under the two sections [keystone_ authtoken] and [neutron].