tacker

vim-register fails under certain VIM environments

Bug #1920088 reported by Hiroo Kitamura on 2021-03-19

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tacker	Fix Released	Medium	Hiroo Kitamura

Bug Description

Under certain VIM environments, there is a issue with vim-register failing.

As an operation, public-endpoint of keystone is specified in vim-config specified at the time of vim-register.
However, in the internal processing of vim-register, there is a process to get region-info, but the keystone when executing the getting region-info API changes to admin-endpoint instead of public-endpoint.
Therefore, vim-register will fail in a VIM environment where public-endpoint is open but admin-endpoint is close.

If you check the belowd log, you can see that the API is executed to public-endpoint at the time of #1, but the API is executed to admin-endpoint at the time of #2 on the below source code.

* register-vim
$ cat /opt/stack/data/tacker/vim_config.yaml
auth_url: 'http://127.0.0.1/identity'
username: 'nfv_user'
password: 'devstack'
project_name: 'nfv'
project_domain_name: 'Default'
user_domain_name: 'Default'
cert_verify: 'False'
$ openstack vim register --is-default --description "Default VIM" --config-file /opt/stack/data/tacker/vim_config.yaml VIM0
Request Failed: internal server error while processing your request.

* source code
tacker/nfvo/drivers/vim/openstack_driver.py
---
    @log.log
    def register_vim(self, vim_obj):
        """Validate and set VIM placements."""

        if 'key_type' in vim_obj['auth_cred']:
            vim_obj['auth_cred'].pop(u'key_type')
        if 'secret_uuid' in vim_obj['auth_cred']:
            vim_obj['auth_cred'].pop(u'secret_uuid')

        ks_client = self.authenticate_vim(vim_obj) #1
        self.discover_placement_attr(vim_obj, ks_client) #2
        self.encode_vim_auth(vim_obj['id'], vim_obj['auth_cred'])
        LOG.debug('VIM registration completed for %s', vim_obj)

* log
tacker-server[1494654]: {"vim": {"auth_url": "http://127.0.0.1/identity", "type": "openstack", "vim_project": {"name": "nfv", "project_domain_name": "Default"}, "auth_cred": {"username": "nfv_user", "password": "***", "user_domain_name": "Default", "cert_verify": "False"}, "name": "VIM0", "description": "Default VIM", "is_default": true}} process_request /opt/stack/tacker/tacker/alarm_receiver.py:52
tacker-server[1494654]: 2021-03-18 17:16:44.970 DEBUG tacker.api.v1.base [req-e955caf9-5795-4526-bd61-6aaa57f7fdc5 nfv_user nfv] Request body: {'vim': {'auth_url': 'http://127.0.0.1/identity', 'type': 'openstack', 'vim_project': {'name': 'nfv', 'project_domain_name': 'Default'}, 'auth_cred': {'username': 'nfv_user', 'password': '***', 'user_domain_name': 'Default', 'cert_verify': 'False'}, 'name': 'VIM0', 'description': 'Default VIM', 'is_default': True}} prepare_request_body /opt/stack/tacker/tacker/api/v1/base.py:512
...
tacker-server[1494654]: 2021-03-18 17:16:44.972 DEBUG keystoneclient.session [req-e955caf9-5795-4526-bd61-6aaa57f7fdc5 nfv_user nfv] REQ: curl -g -i --insecure -X GET http://127.0.0.1/identity/v3 -H "Accept: application/json" -H "User-Agent: python-keystoneclient" _http_log_request /usr/local/lib/python3.8/dist-packages/keystoneclient/session.py:215
tacker-server[1494654]: 2021-03-18 17:16:44.977 DEBUG keystoneclient.session [req-e955caf9-5795-4526-bd61-6aaa57f7fdc5 nfv_user nfv] RESP: [200] Date: Thu, 18 Mar 2021 08:16:44 GMT Server: Apache/2.4.41 (Ubuntu) Content-Type: application/json Content-Length: 253 Vary: X-Auth-Token x-openstack-request-id: req-5226ffa6-c377-4059-b781-a6b5df7eb094 Connection: close
...
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource Traceback (most recent call last):
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/opt/stack/tacker/tacker/api/v1/resource.py", line 77, in resource
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource result = method(request=request, **args)
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/opt/stack/tacker/tacker/api/v1/base.py", line 394, in create
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource obj = obj_creator(request.context, **kwargs)
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/opt/stack/tacker/tacker/common/log.py", line 35, in wrapper
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource return method(*args, **kwargs)
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/opt/stack/tacker/tacker/nfvo/nfvo_plugin.py", line 133, in create_vim
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource self._vim_drivers.invoke(vim_type,
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource self.force_reraise()
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource six.reraise(self.type_, self.value, self.tb)
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource raise value
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/opt/stack/tacker/tacker/nfvo/nfvo_plugin.py", line 127, in create_vim
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource self._vim_drivers.invoke(vim_type,
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/opt/stack/tacker/tacker/common/driver_manager.py", line 71, in invoke
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource return getattr(driver, method_name)(**kwargs)
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/opt/stack/tacker/tacker/common/log.py", line 35, in wrapper
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource return method(*args, **kwargs)
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/opt/stack/tacker/tacker/nfvo/drivers/vim/openstack_driver.py", line 179, in register_vim
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource self.discover_placement_attr(vim_obj, ks_client)
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/opt/stack/tacker/tacker/nfvo/drivers/vim/openstack_driver.py", line 162, in discover_placement_attr
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource regions_list = self._find_regions(ks_client)
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/opt/stack/tacker/tacker/nfvo/drivers/vim/openstack_driver.py", line 152, in _find_regions
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource region_info = ks_client.regions.list()
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/usr/local/lib/python3.8/dist-packages/keystoneclient/v3/regions.py", line 87, in list
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource return super(RegionManager, self).list(
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/usr/local/lib/python3.8/dist-packages/keystoneclient/base.py", line 86, in func
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource return f(*args, **new_kwargs)
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/usr/local/lib/python3.8/dist-packages/keystoneclient/base.py", line 448, in list
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource list_resp = self._list(url_query, self.collection_key)
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/usr/local/lib/python3.8/dist-packages/keystoneclient/base.py", line 141, in _list
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource resp, body = self.client.get(url, **kwargs)
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/usr/local/lib/python3.8/dist-packages/keystoneauth1/adapter.py", line 395, in get
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource return self.request(url, 'GET', **kwargs)
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/usr/local/lib/python3.8/dist-packages/keystoneauth1/adapter.py", line 554, in request
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/usr/local/lib/python3.8/dist-packages/keystoneauth1/adapter.py", line 257, in request
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource return self.session.request(url, method, **kwargs)
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/usr/local/lib/python3.8/dist-packages/keystoneauth1/session.py", line 921, in request
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource resp = send(**kwargs)
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource File "/usr/local/lib/python3.8/dist-packages/keystoneauth1/session.py", line 1028, in _send_request
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource raise exceptions.ConnectFailure(msg)
tacker-server[1494654]: 2021-03-18 17:16:45.032 1494654 ERROR tacker.api.v1.resource keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://dummy-admin/identity/regions?: HTTPConnectionPool(host='dummy-admin', port=80): Max retries exceeded with url: /identity/regions (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc305923a60>: Failed to establish a new connection: [Errno -2] Name or service not known'))

Yasufumi Ogawa (yasufum) on 2021-03-31

Changed in tacker:
importance:	Undecided → Medium

Revision history for this message

Hiroo Kitamura (h-kitamura) wrote on 2021-05-31:

As a result of additional investigation, it is assumed that there is a suspicion of keystoneauth, so I issued a bugreport below.
https://bugs.launchpad.net/keystoneauth/+bug/1930194

Changed in tacker:
status:	New → In Progress
assignee:	nobody → Hiroo Kitamura (h-kitamura)

Hiroo Kitamura (h-kitamura) on 2021-08-02

Changed in tacker:
status:	In Progress → Opinion

OpenStack Infra (hudson-openstack) on 2021-11-01

Changed in tacker:
status:	Opinion → In Progress

Revision history for this message

Hiromu Asahina (h-asahina) wrote on 2021-11-01 (last edit on 2021-11-01):

I found that it's a matter of Tacker.

When we create a keystone-client instance, the client obtains an endpoint for API requests from service catalogs [1].
In this procedure, the endpoint is determined mainly by ``interface`` attribute, which means ``auth_url`` in the VIM configuration doesn't affect the endpoint selection.
The reason that the keystone-client accesses to ``admin`` endpoint is simply because ``admin`` is the default value of the ``interface`` attirbute [2].

Therefore, I think it's not Keystone's bug, but Tacker's bug and, we can fix it by serving appropriate attributes to the keystone-client.

[1] https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/identity/base.py#L275-L279
[2] https://github.com/openstack/python-keystoneclient/blob/d5cb761763988217cdd00a7981a62e2083fd381f/keystoneclient/httpclient.py#L251

Revision history for this message

Masaki Oyama (oyamamasaki) wrote on 2021-12-09:

We have same problem.
This patch works in our environment just for your information.

```
diff --git a/tacker/vnfm/keystone.py b/tacker/vnfm/keystone.py
--- a/tacker/vnfm/keystone.py
+++ b/tacker/vnfm/keystone.py
@@ -59,7 +59,8 @@ class Keystone(object):
         else:
             auth_plugin = identity.v3.Password(**kwargs)
         ses = self.get_session(auth_plugin=auth_plugin, verify=verify)
- cli = client.Client(DEFAULT_IDENTITY_VERSION, session=ses)
+ cli = client.Client(DEFAULT_IDENTITY_VERSION, session=ses,
+ endpoint_override=auth_plugin.auth_url)
         return cli
```

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-01-07: Fix merged to tacker (master)

Reviewed: https://review.opendev.org/c/openstack/tacker/+/816107
Committed: https://opendev.org/openstack/tacker/commit/fff6cec4f2b63034cbf2a55a36cebca563b7a082
Submitter: "Zuul (22348)"
Branch: master

commit fff6cec4f2b63034cbf2a55a36cebca563b7a082
Author: Hiromu Asahina <email address hidden>
Date: Sat Oct 30 02:25:47 2021 +0900

Fix VIM registration error

    The VIM registration fails in an environment where an admin-endpoint is
    close even if a reachable public-endpoint is used in the VIM
    configuration.

    The cause of this problem is that the endpoint for the keystone API and
    the endpoint for the authentication are different inside the keystone
    client. The keystone client implicitly selects admin-endpoint for API
    requests from service catalogs unless we specify an endpoint URL or
    interface [1], i.e., the auth_url in the VIM configuration isn't used
    for API requests.

This patch solves the above problem by specifying an endpoint URL when
creating a keystone client instance.

    Note that specifying an interface can be a better solution, as the
    keystone can provide different interfaces with the same endpoint URL.
    However, to achieve this, we have to add a new field to the VIM
    configuration for the interface, which makes a huge modification.
    Thus, this patch doesn't take this solution.

Also, this patch includes a tiny refactoring on the OpenStack driver.

[1] https://github.com/openstack/python-keystoneclient/blob/d5cb761763988217cdd00a7981a62e2083fd381f/keystoneclient/httpclient.py#L251

    Change-Id: Ic743fadbd1ddd59ca6755949e03b77c8715be459
    Closes-bug: #1920088
    Signed-off-by: Hiromu Asahina <email address hidden>