Although I can register plain OpenStack VIM accounts (e.g., stable/queens, stable/rocky) under tacker, I fail to register the RHOSP13 the same way. Root cause of the problem appears to be in the way auth_url address is used or not used. Stated in other words, while I explicitly indicate the "public" keystone URL for RHOSP13 (i.e., 10.35.42.102:5000) to be used, somewhere along the way, tacker appears to use the "admin" keystone URL (i.e., 192.168.24.7:35357) and fails to communicate since this URL is on a private network to which tacker has no access. I'm attaching the relevant outputs. Any thoughts on this misbehavior? Thanks much!
$ $ openstack endpoint list | grep keystone
| 58a35c427dab41febf68cb872b8a6629 | regionOne | keystone | identity | True | internal | http://192.168.46.27:5000 |
| 8b5b94c2a7a14b6d80ca96a1f8a92c31 | regionOne | keystone | identity | True | admin | http://192.168.24.7:35357 |
| c99110bc515f4eeb97b85e9664644a5f | regionOne | keystone | identity | True | public | http://10.35.42.102:5000 |
$ cat vim-openstack-rhosp13.yaml
auth_url: 'http://10.35.42.102:5000/v3'
user_domain_name: 'Default'
username: 'nfvuser'
password: 'nfvuser'
cert_verify: 'False'
project_domain_name: 'Default'
project_name: 'nfv'
$ openstack vim register --config-file vim-openstack-rhosp13.yaml --description 'vim-openstack-rhosp13' vim-openstack-rhosp13
Request Failed: internal server error while processing your request.
$ sudo journalctl -f -u devstack@t*
Feb 16 16:55:07 tc-nfv-mano tacker-server[955]: 2019-02-16 16:55:07.910 DEBUG tacker.wsgi [req-19661e80-7975-4a24-bc84-915bd2c577a0 nfv nfv_user] (955) accepted ('10.34.24.8', 50940) from (pid=955) server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:883
Feb 16 16:55:07 tc-nfv-mano tacker-server[955]: 2019-02-16 16:55:07.911 DEBUG tacker.alarm_receiver [req-19661e80-7975-4a24-bc84-915bd2c577a0 nfv nfv_user] Process request: POST /v1.0/vims.json HTTP/1.0
Feb 16 16:55:07 tc-nfv-mano tacker-server[955]: Accept: application/json
Feb 16 16:55:07 tc-nfv-mano tacker-server[955]: Accept-Encoding: gzip, deflate
Feb 16 16:55:07 tc-nfv-mano tacker-server[955]: Connection: keep-alive
Feb 16 16:55:07 tc-nfv-mano tacker-server[955]: Content-Length: 349
Feb 16 16:55:07 tc-nfv-mano tacker-server[955]: Content-Type: application/json
Feb 16 16:55:07 tc-nfv-mano tacker-server[955]: Host: 10.34.24.8:9890
Feb 16 16:55:07 tc-nfv-mano tacker-server[955]: User-Agent: python-tackerclient
Feb 16 16:55:07 tc-nfv-mano tacker-server[955]: X-Auth-Token: gAAAAABcaBY7QY7JENMMPLGmtn7nAwPC317tVxHbVdk8QxTw_gHJ1oiDLbKkxR-hpqvX6cX3WFKpNNoC4Aaxy25fdpU4noVDJlgwUmCjUYhOFsVzDCUAz7DaTcbvDr5ACp_2N5qq9toaiWmMHI53WC1E7QkPJnWfCv9dQ2xSQZqpusA98Hs8szQ
Feb 16 16:55:07 tc-nfv-mano tacker-server[955]: [1B blob data]
Feb 16 16:55:07 tc-nfv-mano tacker-server[955]: {"vim": {"description": "vim-openstack-rhosp13", "name": "vim-openstack-rhosp13", "is_default": false, "auth_cred": {"username": "nfvuser", "user_domain_name": "Default", "password": "nfvuser", "cert_verify": "False"}, "auth_url": "http://10.35.42.102:5000/v3", "type": "openstack", "vim_project": {"name": "nfv", "project_domain_name": "Default"}}} from (pid=955) process_request /opt/stack/tacker/tacker/alarm_receiver.py:43
Feb 16 16:55:08 tc-nfv-mano tacker-server[955]: 2019-02-16 16:55:08.146 DEBUG tacker.api.v1.base [req-e1326b9c-7d8f-4160-a7d5-f0ba14e0fe18 admin admin] Request body: {u'vim': {u'description': u'vim-openstack-rhosp13', u'vim_project': {u'name': u'nfv', u'project_domain_name': u'Default'}, u'is_default': False, u'auth_url': u'http://10.35.42.102:5000/v3', u'auth_cred': {u'username': u'nfvuser', u'user_domain_name': u'Default', u'password': u'***', u'cert_verify': u'False'}, u'type': u'openstack', u'name': u'vim-openstack-rhosp13'}} from (pid=955) prepare_request_body /opt/stack/tacker/tacker/api/v1/base.py:510
Feb 16 16:55:08 tc-nfv-mano tacker-server[955]: 2019-02-16 16:55:08.148 DEBUG tacker.common.log [req-e1326b9c-7d8f-4160-a7d5-f0ba14e0fe18 admin admin] tacker.nfvo.nfvo_plugin.NfvoPlugin method create_vim called with arguments (<tacker.context.Context object at 0x7f1c4bdcab50>,) {'vim': {u'vim': {u'description': u'vim-openstack-rhosp13', u'vim_project': {u'name': u'nfv', u'project_domain_name': u'Default'}, u'is_default': False, u'auth_url': u'http://10.35.42.102:5000/v3', 'tenant_id': u'885d1646353647fd989f7274f788dbb3', u'auth_cred': {u'username': u'nfvuser', u'user_domain_name': u'Default', u'password': u'***', u'cert_verify': u'False'}, u'type': u'openstack', u'name': u'vim-openstack-rhosp13'}}} from (pid=955) wrapper /opt/stack/tacker/tacker/common/log.py:34
Feb 16 16:55:08 tc-nfv-mano tacker-server[955]: 2019-02-16 16:55:08.149 DEBUG tacker.nfvo.nfvo_plugin [req-e1326b9c-7d8f-4160-a7d5-f0ba14e0fe18 admin admin] Create vim called with parameters {u'vim': {u'description': u'vim-openstack-rhosp13', u'vim_project': {u'name': u'nfv', u'project_domain_name': u'Default'}, u'is_default': False, u'auth_url': u'http://10.35.42.102:5000/v3', 'tenant_id': u'885d1646353647fd989f7274f788dbb3', u'auth_cred': {u'username': u'nfvuser', u'user_domain_name': u'Default', u'password': u'***', u'cert_verify': u'False'}, u'type': u'openstack', u'name': u'vim-openstack-rhosp13'}} from (pid=955) create_vim /opt/stack/tacker/tacker/nfvo/nfvo_plugin.py:104
Feb 16 16:55:08 tc-nfv-mano tacker-server[955]: 2019-02-16 16:55:08.150 DEBUG tacker.common.log [req-e1326b9c-7d8f-4160-a7d5-f0ba14e0fe18 admin admin] tacker.nfvo.drivers.vim.openstack_driver.OpenStack_Driver method register_vim called with arguments () {'vim_obj': {'status': 'PENDING', u'description': u'vim-openstack-rhosp13', u'vim_project': {u'name': u'nfv', u'project_domain_name': u'Default'}, u'is_default': False, u'auth_url': u'http://10.35.42.102:5000/v3', 'tenant_id': u'885d1646353647fd989f7274f788dbb3', u'auth_cred': {u'username': u'nfvuser', u'user_domain_name': u'Default', u'password': u'***', u'cert_verify': u'False'}, u'type': u'openstack', 'id': '5e204046-96d7-406c-8cbe-a0d272f3be15', u'name': u'vim-openstack-rhosp13'}, 'context': <tacker.context.Context object at 0x7f1c4bdcab50>} from (pid=955) wrapper /opt/stack/tacker/tacker/common/log.py:34
Feb 16 16:55:08 tc-nfv-mano tacker-server[955]: 2019-02-16 16:55:08.151 DEBUG keystoneclient.session [req-e1326b9c-7d8f-4160-a7d5-f0ba14e0fe18 admin admin] REQ: curl -g -i --insecure -X GET http://10.35.42.102:5000/v3 -H "Accept: application/json" -H "User-Agent: python-keystoneclient" from (pid=955) _http_log_request /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:215
Feb 16 16:55:08 tc-nfv-mano tacker-server[955]: 2019-02-16 16:55:08.205 DEBUG keystoneclient.session [req-e1326b9c-7d8f-4160-a7d5-f0ba14e0fe18 admin admin] RESP: [200] Date: Sat, 16 Feb 2019 13:55:08 GMT Server: Apache Vary: X-Auth-Token,Accept-Encoding x-openstack-request-id: req-cc72caac-3b23-4d23-951e-8476dccc46e8 Content-Encoding: gzip Content-Length: 195 Content-Type: application/json
Feb 16 16:55:08 tc-nfv-mano tacker-server[955]: RESP BODY: {"version": {"status": "stable", "updated": "2018-02-28T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.10", "links": [{"href": "http://10.35.42.102:5000/v3/", "rel": "self"}]}}
Feb 16 16:55:08 tc-nfv-mano tacker-server[955]: from (pid=955) _http_log_response /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:255
Feb 16 16:56:11 tc-nfv-mano tacker-server[955]: 2019-02-16 16:56:11.780 WARNING keystoneauth.discover [req-e1326b9c-7d8f-4160-a7d5-f0ba14e0fe18 admin admin] Failed to contact the endpoint at http://192.168.24.7:35357 for discovery. Fallback to using that endpoint as the base url.: ConnectFailure: Unable to establish connection to http://192.168.24.7:35357: HTTPConnectionPool(host='192.168.24.7', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c4bc53110>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.593 DEBUG tacker.common.log [req-e1326b9c-7d8f-4160-a7d5-f0ba14e0fe18 admin admin] tacker.nfvo.drivers.vim.openstack_driver.OpenStack_Driver method delete_vim_auth called with arguments () {'vim_id': '5e204046-96d7-406c-8cbe-a0d272f3be15', 'context': <tacker.context.Context object at 0x7f1c4bdcab50>, 'auth': {u'username': u'nfvuser', u'password': u'***', u'cert_verify': u'False', u'user_domain_name': u'Default', 'project_name': u'nfv', 'auth_url': u'http://10.35.42.102:5000/v3', 'project_id': None, 'project_domain_name': u'Default'}} from (pid=955) wrapper /opt/stack/tacker/tacker/common/log.py:34
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.594 DEBUG tacker.nfvo.drivers.vim.openstack_driver [req-e1326b9c-7d8f-4160-a7d5-f0ba14e0fe18 admin admin] Attempting to delete key for vim id 5e204046-96d7-406c-8cbe-a0d272f3be15 from (pid=955) delete_vim_auth /opt/stack/tacker/tacker/nfvo/drivers/vim/openstack_driver.py:225
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.595 WARNING tacker.nfvo.drivers.vim.openstack_driver [req-e1326b9c-7d8f-4160-a7d5-f0ba14e0fe18 admin admin] VIM key deletion failed for vim 5e204046-96d7-406c-8cbe-a0d272f3be15: OSError: [Errno 2] No such file or directory: '/etc/tacker/vim/fernet_keys/5e204046-96d7-406c-8cbe-a0d272f3be15'
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 ERROR tacker.api.v1.resource [req-e1326b9c-7d8f-4160-a7d5-f0ba14e0fe18 admin admin] create failed: No details.: ConnectFailure: Unable to establish connection to http://192.168.24.7:35357/regions?: HTTPConnectionPool(host='192.168.24.7', port=35357): Max retries exceeded with url: /regions (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c4be2bb90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource Traceback (most recent call last):
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/opt/stack/tacker/tacker/api/v1/resource.py", line 77, in resource
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource result = method(request=request, **args)
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/opt/stack/tacker/tacker/api/v1/base.py", line 393, in create
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource obj = obj_creator(request.context, **kwargs)
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/opt/stack/tacker/tacker/common/log.py", line 35, in wrapper
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource return method(*args, **kwargs)
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/opt/stack/tacker/tacker/nfvo/nfvo_plugin.py", line 121, in create_vim
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource auth=vim_obj['auth_cred'])
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource self.force_reraise()
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource six.reraise(self.type_, self.value, self.tb)
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/opt/stack/tacker/tacker/nfvo/nfvo_plugin.py", line 113, in create_vim
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource vim_obj=vim_obj)
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/opt/stack/tacker/tacker/common/driver_manager.py", line 70, in invoke
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource return getattr(driver, method_name)(**kwargs)
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/opt/stack/tacker/tacker/common/log.py", line 35, in wrapper
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource return method(*args, **kwargs)
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/opt/stack/tacker/tacker/nfvo/drivers/vim/openstack_driver.py", line 207, in register_vim
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource self.discover_placement_attr(vim_obj, ks_client)
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/opt/stack/tacker/tacker/nfvo/drivers/vim/openstack_driver.py", line 190, in discover_placement_attr
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource regions_list = self._find_regions(ks_client)
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/opt/stack/tacker/tacker/nfvo/drivers/vim/openstack_driver.py", line 180, in _find_regions
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource region_info = ks_client.regions.list()
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/keystoneclient/v3/regions.py", line 82, in list
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource **kwargs)
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py", line 75, in func
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource return f(*args, **new_kwargs)
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py", line 397, in list
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource self.collection_key)
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py", line 125, in _list
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource resp, body = self.client.get(url, **kwargs)
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 304, in get
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource return self.request(url, 'GET', **kwargs)
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 463, in request
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource return self.session.request(url, method, **kwargs)
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 698, in request
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource resp = send(**kwargs)
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 772, in _send_request
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource raise exceptions.ConnectFailure(msg)
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource ConnectFailure: Unable to establish connection to http://192.168.24.7:35357/regions?: HTTPConnectionPool(host='192.168.24.7', port=35357): Max retries exceeded with url: /regions (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c4be2bb90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.597 TRACE tacker.api.v1.resource
Feb 16 16:57:14 tc-nfv-mano tacker-server[955]: 2019-02-16 16:57:14.602 INFO tacker.wsgi [req-e1326b9c-7d8f-4160-a7d5-f0ba14e0fe18 admin admin] 10.34.24.8 - - [16/Feb/2019 16:57:14] "POST /v1.0/vims.json HTTP/1.1" 500 367 126.690577
