I'm using tacker (stable/queens; devstack; all-in-one) on an Ubuntu 16.04 server to register an RHOSP13 VIM on an RHEL server. Despite authentication on the side of RHOSP13 is done successfully, my VIM register attempt fails. What is interesting is that although the authentication req/rsp is using the public ip address of the RHOSP13 controller (i.e., http://10.35.42.103:5000/v3), tacker or keystone somehow appears to be using of the private IP address of the RHOSP13 controller (i.e., http://192.168.24.7:35357) during post-authentication handling of the VIM register request. Any thoughts on what may be causing this misbehavior? I'm attaching relevant portion of the tacker log files. Thanks a lot -- Gurol
---
2019-02-04 03:34:11.014 DEBUG tacker.alarm_receiver [req-294e911d-b2e0-4122-8105-d8b7e2ec2bcd admin admin] Process request: POST /v1.0/vims HTTP/1.0
Accept: */*
Accept-Encoding: gzip, deflate
Cache-Control: no-cache
Connection: keep-alive
Content-Length: 543
Content-Type: application/json
Cookie: csrftoken=vWQIopLkGbOdqDksoydDbGiCka2PfVIcDPgQVONJJvH7M8MFtCpB1HuDwx6gZOfr
Host: 10.34.24.8:9890
Postman-Token: 9bf1e798-94e5-4b24-ae2a-7042ccd0bdc8
User-Agent: PostmanRuntime/7.6.0
X-Auth-Token: gAAAAABcV4hH3QU33PwAnVto3esD-MaUff4uNbtiTnioIO7tBx-B2GXPimp11ADL7Ci2oKKpt2X4eQQgzHn9eP-hRu9UPsHZBXjqpaGgTNjDzFv6m8MI59zsno6rG6Z_83qhSX7584FWwjKYTz-EfAJ0ZqeuMeZ72PpWB_bFk-IbWHKTH0cP1N4
{
"vim": {
"type": "openstack",
"auth_url": "http://10.35.42.103:5000/v3",
"tenant_id": "701080dc026542c783c73da2fa995593",
"auth_cred": {
"username": "nfvuser",
"user_domain_name": "Default",
"password": "nfvuser",
"cert_verify": "True"
},
"vim_project": {
"project_name": "nfv",
"project_domain_name": "Default"
},
"name": "openstack-xyz",
"description": "openstack-xyz",
"is_default": false
}
} from (pid=1164) process_request /opt/stack/tacker/tacker/alarm_receiver.py:43
2019-02-04 03:34:11.213 DEBUG tacker.api.v1.base [req-f4347dd3-7f0e-44c6-a751-23d33d901232 nfv nfv_user] Request body: {u'vim': {u'vim_project': {u'project_name': u'nfv', u'project_domain_name': u'Default'}, u'tenant_id': u'701080dc026542c783c73da2fa995593', u'description': u'openstack-xyz', u'is_default': False, u'auth_cred': {u'username': u'nfvuser', u'user_domain_name': u'Default', u'password': u'***', u'cert_verify': u'True'}, u'auth_url': u'http://10.35.42.103:5000/v3', u'type': u'openstack', u'name': u'openstack-xyz'}} from (pid=1164) prepare_request_body /opt/stack/tacker/tacker/api/v1/base.py:510
2019-02-04 03:34:11.214 DEBUG tacker.common.log [req-f4347dd3-7f0e-44c6-a751-23d33d901232 nfv nfv_user] tacker.nfvo.nfvo_plugin.NfvoPlugin method create_vim called with arguments (<tacker.context.Context object at 0x7f2ad7fd8790>,) {'vim': {u'vim': {u'vim_project': {u'project_name': u'nfv', u'project_domain_name': u'Default'}, u'tenant_id': u'701080dc026542c783c73da2fa995593', u'description': u'openstack-xyz', u'is_default': False, u'auth_cred': {u'username': u'nfvuser', u'user_domain_name': u'Default', u'password': u'***', u'cert_verify': u'True'}, u'auth_url': u'http://10.35.42.103:5000/v3', u'type': u'openstack', u'name': u'openstack-xyz'}}} from (pid=1164) wrapper /opt/stack/tacker/tacker/common/log.py:34
2019-02-04 03:34:11.215 DEBUG tacker.nfvo.nfvo_plugin [req-f4347dd3-7f0e-44c6-a751-23d33d901232 nfv nfv_user] Create vim called with parameters {u'vim': {u'vim_project': {u'project_name': u'nfv', u'project_domain_name': u'Default'}, u'tenant_id': u'701080dc026542c783c73da2fa995593', u'description': u'openstack-xyz', u'is_default': False, u'auth_cred': {u'username': u'nfvuser', u'user_domain_name': u'Default', u'password': u'***', u'cert_verify': u'True'}, u'auth_url': u'http://10.35.42.103:5000/v3', u'type': u'openstack', u'name': u'openstack-xyz'}} from (pid=1164) create_vim /opt/stack/tacker/tacker/nfvo/nfvo_plugin.py:104
2019-02-04 03:34:11.216 DEBUG tacker.common.log [req-f4347dd3-7f0e-44c6-a751-23d33d901232 nfv nfv_user] tacker.nfvo.drivers.vim.openstack_driver.OpenStack_Driver method register_vim called with arguments () {'vim_obj': {'status': 'PENDING', u'vim_project': {u'project_name': u'nfv', u'project_domain_name': u'Default'}, u'tenant_id': u'701080dc026542c783c73da2fa995593', u'description': u'openstack-xyz', u'is_default': False, u'auth_cred': {u'username': u'nfvuser', u'user_domain_name': u'Default', u'password': u'***', u'cert_verify': u'True'}, u'auth_url': u'http://10.35.42.103:5000/v3', u'type': u'openstack', 'id': 'c28770a7-1662-4254-860d-a170e17ca40d', u'name': u'openstack-xyz'}, 'context': <tacker.context.Context object at 0x7f2ad7fd8790>} from (pid=1164) wrapper /opt/stack/tacker/tacker/common/log.py:34
2019-02-04 03:34:11.217 DEBUG keystoneclient.session [req-f4347dd3-7f0e-44c6-a751-23d33d901232 nfv nfv_user] REQ: curl -g -i -X GET http://10.35.42.103:5000/v3 -H "Accept: application/json" -H "User-Agent: python-keystoneclient" from (pid=1164) _http_log_request /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:215
2019-02-04 03:34:11.279 DEBUG keystoneclient.session [req-f4347dd3-7f0e-44c6-a751-23d33d901232 nfv nfv_user] RESP: [200] Date: Mon, 04 Feb 2019 00:34:11 GMT Server: Apache Vary: X-Auth-Token,Accept-Encoding x-openstack-request-id: req-ad5c75d7-ce6c-4ce8-86aa-0e604e825e4d Content-Encoding: gzip Content-Length: 195 Content-Type: application/json
RESP BODY: {"version": {"status": "stable", "updated": "2018-02-28T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.10", "links": [{"href": "http://10.35.42.103:5000/v3/", "rel": "self"}]}}
from (pid=1164) _http_log_response /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:255
2019-02-04 03:35:15.014 WARNING keystoneauth.discover [req-f4347dd3-7f0e-44c6-a751-23d33d901232 nfv nfv_user] Failed to contact the endpoint at http://192.168.24.7:35357 for discovery. Fallback to using that endpoint as the base url.: ConnectFailure: Unable to establish connection to http://192.168.24.7:35357: HTTPConnectionPool(host='192.168.24.7', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2adc055d90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-02-04 03:36:17.828 DEBUG tacker.common.log [req-f4347dd3-7f0e-44c6-a751-23d33d901232 nfv nfv_user] tacker.nfvo.drivers.vim.openstack_driver.OpenStack_Driver method delete_vim_auth called with arguments () {'vim_id': 'c28770a7-1662-4254-860d-a170e17ca40d', 'context': <tacker.context.Context object at 0x7f2ad7fd8790>, 'auth': {u'username': u'nfvuser', u'password': u'***', u'cert_verify': u'True', u'user_domain_name': u'Default', 'project_name': None, 'auth_url': u'http://10.35.42.103:5000/v3', 'project_id': None, 'project_domain_name': u'Default'}} from (pid=1164) wrapper /opt/stack/tacker/tacker/common/log.py:34
2019-02-04 03:36:17.829 DEBUG tacker.nfvo.drivers.vim.openstack_driver [req-f4347dd3-7f0e-44c6-a751-23d33d901232 nfv nfv_user] Attempting to delete key for vim id c28770a7-1662-4254-860d-a170e17ca40d from (pid=1164) delete_vim_auth /opt/stack/tacker/tacker/nfvo/drivers/vim/openstack_driver.py:225
2019-02-04 03:36:17.830 WARNING tacker.nfvo.drivers.vim.openstack_driver [req-f4347dd3-7f0e-44c6-a751-23d33d901232 nfv nfv_user] VIM key deletion failed for vim c28770a7-1662-4254-860d-a170e17ca40d: OSError: [Errno 2] No such file or directory: '/etc/tacker/vim/fernet_keys/c28770a7-1662-4254-860d-a170e17ca40d'
2019-02-04 03:36:17.831 ERROR tacker.api.v1.resource [req-f4347dd3-7f0e-44c6-a751-23d33d901232 nfv nfv_user] create failed: No details.: ConnectFailure: Unable to establish connection to http://192.168.24.7:35357/regions?: HTTPConnectionPool(host='192.168.24.7', port=35357): Max retries exceeded with url: /regions (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2adc055d90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource Traceback (most recent call last):
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/opt/stack/tacker/tacker/api/v1/resource.py", line 77, in resource
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource result = method(request=request, **args)
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/opt/stack/tacker/tacker/api/v1/base.py", line 393, in create
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource obj = obj_creator(request.context, **kwargs)
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/opt/stack/tacker/tacker/common/log.py", line 35, in wrapper
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource return method(*args, **kwargs)
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/opt/stack/tacker/tacker/nfvo/nfvo_plugin.py", line 121, in create_vim
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource auth=vim_obj['auth_cred'])
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource self.force_reraise()
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource six.reraise(self.type_, self.value, self.tb)
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/opt/stack/tacker/tacker/nfvo/nfvo_plugin.py", line 113, in create_vim
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource vim_obj=vim_obj)
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/opt/stack/tacker/tacker/common/driver_manager.py", line 70, in invoke
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource return getattr(driver, method_name)(**kwargs)
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/opt/stack/tacker/tacker/common/log.py", line 35, in wrapper
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource return method(*args, **kwargs)
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/opt/stack/tacker/tacker/nfvo/drivers/vim/openstack_driver.py", line 207, in register_vim
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource self.discover_placement_attr(vim_obj, ks_client)
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/opt/stack/tacker/tacker/nfvo/drivers/vim/openstack_driver.py", line 190, in discover_placement_attr
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource regions_list = self._find_regions(ks_client)
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/opt/stack/tacker/tacker/nfvo/drivers/vim/openstack_driver.py", line 180, in _find_regions
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource region_info = ks_client.regions.list()
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/keystoneclient/v3/regions.py", line 82, in list
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource **kwargs)
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py", line 75, in func
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource return f(*args, **new_kwargs)
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py", line 397, in list
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource self.collection_key)
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py", line 125, in _list
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource resp, body = self.client.get(url, **kwargs)
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 304, in get
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource return self.request(url, 'GET', **kwargs)
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 463, in request
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource return self.session.request(url, method, **kwargs)
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 698, in request
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource resp = send(**kwargs)
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 772, in _send_request
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource raise exceptions.ConnectFailure(msg)
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource ConnectFailure: Unable to establish connection to http://192.168.24.7:35357/regions?: HTTPConnectionPool(host='192.168.24.7', port=35357): Max retries exceeded with url: /regions (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2adc055d90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-02-04 03:36:17.831 TRACE tacker.api.v1.resource
2019-02-04 03:36:17.834 INFO tacker.wsgi [req-f4347dd3-7f0e-44c6-a751-23d33d901232 nfv nfv_user] 10.34.34.34 - - [04/Feb/2019 03:36:17] "POST /v1.0/vims HTTP/1.1" 500 367 126.820522
---
