stale api name is not updated after connection failure

Bug #1922163 reported by norman shen
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
keystoneauth
New
Undecided
Unassigned

Bug Description

We have a use case that has to change endpoint url for a specific interface and do not want to restart service. but we fount that service seemly continued using stale endpoint info although it had been updated.

error logs are appended

```log
ConnectFailure: Unable to establish connection to http://neutron-server.openstack.svc.cn-south-1.myinspurcloud.com:9696/v2.0/ports?tenant_id=6fdf046e23d14d0a9ce52bd5b71f0afa&device_id=281925a1-ff2b-43ea-aee4-9930c3fa9df7: HTTPConnectionPool(host='neutron-server.openstack.svc.cn-south-1.myinspurcloud.com', port=9696): Max retries exceeded with url: /v2.0/ports?tenant_id=6fdf046e23d14d0a9ce52bd5b71f0afa&device_id=281925a1-ff2b-43ea-aee4-9930c3fa9df7 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3f345214d0>: Failed to establish a new connection: [Errno -2] No address found',))
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] Traceback (most recent call last):
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6843, in _heal_instance_info_cache
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] self.network_api.get_instance_nw_info(context, instance)
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] File "/usr/lib/python2.7/dist-packages/nova/network/base_api.py", line 249, in get_instance_nw_info
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] result = self._get_instance_nw_info(context, instance, **kwargs)
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1481, in _get_instance_nw_info
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] refresh_vif_id)
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 2481, in _build_network_info_model
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] data = client.list_ports(**search_opts)
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 122, in wrapper
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] ret = obj(*args, **kwargs)
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 794, in list_ports
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] **_params)
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 122, in wrapper
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] ret = obj(*args, **kwargs)
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 369, in list
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] for r in self._pagination(collection, path, **params):
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] res = self.get(path, params=params)
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 122, in wrapper
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] ret = obj(*args, **kwargs)
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 354, in get
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] headers=headers, params=params)
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 122, in wrapper
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] ret = obj(*args, **kwargs)
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] headers=headers, params=params)
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 122, in wrapper
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] ret = obj(*args, **kwargs)
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 282, in do_request
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] headers=headers)
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 343, in do_request
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] return self.request(url, method, **kwargs)
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 331, in request
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] resp = super(SessionClient, self).request(*args, **kwargs)
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] return self.session.request(url, method, **kwargs)
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 698, in request
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] resp = send(**kwargs)
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 772, in _send_request
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] raise exceptions.ConnectFailure(msg)
2021-04-01 06:28:10,344.344 47430 ERROR nova.compute.manager [instance: 281925a1-ff2b-43ea-aee4-9930c3fa9df7] ConnectFailure: Unable to establish connection to http://neutron-server.openstack.svc.cn-south-1.myinspurcloud.com:9696/v2.0/ports?tenant_id=6fdf046e23d14d0a9ce52bd5b71f0afa&device_id=281925a1-ff2b-43ea-aee4-9930c3fa9df7: HTTPConnectionPool(host='neutron-server.openstack.svc.cn-south-1.myinspurcloud.com', port=9696): Max retries exceeded with url: /v2.0/ports?tenant_id=6fdf046e23d14d0a9ce52bd5b71f0afa&device_id=281925a1-ff2b-43ea-aee4-9930c3fa9df7 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3f345214d0>: Failed to establish a new connection: [Errno -2] No address found',))
```

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.