| 2020-06-11 10:44:43 |
LIU Yulong |
bug |
|
|
added bug |
| 2020-06-11 10:45:27 |
LIU Yulong |
neutron: assignee |
|
LIU Yulong (dragon889) |
|
| 2020-06-13 15:37:54 |
OpenStack Infra |
neutron: status |
Confirmed |
In Progress |
|
| 2020-06-15 07:52:43 |
Jakub Libosvar |
tags |
|
l3-dvr-backlog |
|
| 2020-07-02 13:16:28 |
OpenStack Infra |
neutron: status |
In Progress |
Fix Released |
|
| 2021-04-14 13:26:42 |
Slawek Kaplonski |
neutron: assignee |
LIU Yulong (dragon889) |
|
|
| 2021-04-14 13:26:45 |
Slawek Kaplonski |
tags |
l3-dvr-backlog |
l3-dvr-backlog timeout-abandon |
|
| 2021-04-27 00:12:55 |
OpenStack Infra |
tags |
l3-dvr-backlog timeout-abandon |
in-stable-ussuri l3-dvr-backlog timeout-abandon |
|
| 2021-04-27 03:39:32 |
Hemanth Nakkina |
bug task added |
|
neutron (Ubuntu) |
|
| 2021-04-27 03:40:16 |
Hemanth Nakkina |
nominated for series |
|
Ubuntu Focal |
|
| 2021-04-27 03:40:16 |
Hemanth Nakkina |
bug task added |
|
neutron (Ubuntu Focal) |
|
| 2021-04-27 03:50:56 |
Hemanth Nakkina |
nominated for series |
|
Ubuntu Impish |
|
| 2021-04-27 03:50:56 |
Hemanth Nakkina |
bug task added |
|
neutron (Ubuntu Impish) |
|
| 2021-04-27 03:50:56 |
Hemanth Nakkina |
nominated for series |
|
Ubuntu Hirsute |
|
| 2021-04-27 03:50:56 |
Hemanth Nakkina |
bug task added |
|
neutron (Ubuntu Hirsute) |
|
| 2021-04-27 03:50:56 |
Hemanth Nakkina |
nominated for series |
|
Ubuntu Groovy |
|
| 2021-04-27 03:50:56 |
Hemanth Nakkina |
bug task added |
|
neutron (Ubuntu Groovy) |
|
| 2021-04-27 03:51:06 |
Hemanth Nakkina |
neutron (Ubuntu Impish): status |
New |
Fix Released |
|
| 2021-04-27 03:51:13 |
Hemanth Nakkina |
neutron (Ubuntu Hirsute): status |
New |
Fix Released |
|
| 2021-04-27 03:51:20 |
Hemanth Nakkina |
neutron (Ubuntu Groovy): status |
New |
Fix Released |
|
| 2021-04-27 03:54:54 |
Hemanth Nakkina |
tags |
in-stable-ussuri l3-dvr-backlog timeout-abandon |
in-stable-ussuri l3-dvr-backlog sts timeout-abandon |
|
| 2021-04-27 04:42:36 |
Hemanth Nakkina |
description |
In patch [1] it introduced a binding of DB uniq constraint for L3
agent gateway. In some extreme case the DvrFipGatewayPortAgentBinding
is in DB while the gateway port not. The current code path only checks
the binding existence which will pass a "None" port to the following
code path that results an AttributeError.
[1] https://review.opendev.org/#/c/702547/
Exception log:
2020-06-11 15:39:28.361 1285214 INFO neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 does not exist on host host-compute-1. Creating one.
2020-06-11 15:39:28.370 1285214 DEBUG neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 already exists on host host-compute-1. Probably it was just created by other worker. create_fip_agent_gw_port_if_not_exists /usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:927
2020-06-11 15:39:28.390 1285214 DEBUG neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port None found for the destination host: host-compute-1 create_fip_agent_gw_port_if_not_exists /usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:933
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Exception during message handling: AttributeError: 'NoneType' object has no attribute 'get'
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 170, in _process_incoming
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 348, in get_agent_gateway_port
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server admin_ctx, network_id, host)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py", line 953, in create_fip_agent_gw_port_if_not_exists
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self._populate_mtu_and_subnets_for_ports(context, [agent_port])
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1978, in _populate_mtu_and_subnets_for_ports
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server for p in self._each_port_having_fixed_ips(ports)]
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1925, in _each_port_having_fixed_ips
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server fixed_ips = port.get('fixed_ips', [])
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server AttributeError: 'NoneType' object has no attribute 'get'
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server |
In patch [1] it introduced a binding of DB uniq constraint for L3
agent gateway. In some extreme case the DvrFipGatewayPortAgentBinding
is in DB while the gateway port not. The current code path only checks
the binding existence which will pass a "None" port to the following
code path that results an AttributeError.
[1] https://review.opendev.org/#/c/702547/
Exception log:
2020-06-11 15:39:28.361 1285214 INFO neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 does not exist on host host-compute-1. Creating one.
2020-06-11 15:39:28.370 1285214 DEBUG neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 already exists on host host-compute-1. Probably it was just created by other worker. create_fip_agent_gw_port_if_not_exists /usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:927
2020-06-11 15:39:28.390 1285214 DEBUG neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port None found for the destination host: host-compute-1 create_fip_agent_gw_port_if_not_exists /usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:933
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Exception during message handling: AttributeError: 'NoneType' object has no attribute 'get'
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 170, in _process_incoming
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 348, in get_agent_gateway_port
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server admin_ctx, network_id, host)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py", line 953, in create_fip_agent_gw_port_if_not_exists
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self._populate_mtu_and_subnets_for_ports(context, [agent_port])
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1978, in _populate_mtu_and_subnets_for_ports
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server for p in self._each_port_having_fixed_ips(ports)]
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1925, in _each_port_having_fixed_ips
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server fixed_ips = port.get('fixed_ips', [])
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server AttributeError: 'NoneType' object has no attribute 'get'
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
-------------------------------------------------------------------------
[Impact]
In some cases the DvrFipGatewayPortAgentBinding is in DB but the gateway port does not exist.
This resulted in connectivity issues to FIP for the new VMs launched on that compute node.
The fix creates the gateway port if it does not exist.
[Test Case]
This is a race condition and difficult to reproduce. The test case simulated the error condition to verify the fix.
* Deploy openstack with dvr l3ha and centralised snat on neutron nodes
* Deploy instances and delete them. This step is to ensure FIP Agent gateway's are created on compute nodes
Check the following command to see FIP Agent gateway information
openstack port list --network ext_net -c id -c device_id -c binding_host_id -c device_owner -c fixed_ips | grep floatingip_agent_gateway
* Pick one of the compute node that has no instances and delete the FIP Agent gateway port (port id can be determined from above command)
openstack port delete <port id>
* Launch an instance on the compute node
openstack server create --wait --image cirros --flavor m1.cirros --nic net-id=<network id> --availability-zone nova:<hostname> cirros-test1
* Verify neutron-server logs for error
ERROR oslo_messaging.rpc.server AttributeError: 'NoneType' object has no attribute 'get'
* Assign floating ip and tried to ping fip and the ping fails
[Where problems could occur]
The fix itself adds an extra check to determine the cases when the gateway port needs to be created.
And hence it is not expected to cause any regression. |
|
| 2021-04-27 04:43:30 |
Hemanth Nakkina |
description |
In patch [1] it introduced a binding of DB uniq constraint for L3
agent gateway. In some extreme case the DvrFipGatewayPortAgentBinding
is in DB while the gateway port not. The current code path only checks
the binding existence which will pass a "None" port to the following
code path that results an AttributeError.
[1] https://review.opendev.org/#/c/702547/
Exception log:
2020-06-11 15:39:28.361 1285214 INFO neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 does not exist on host host-compute-1. Creating one.
2020-06-11 15:39:28.370 1285214 DEBUG neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 already exists on host host-compute-1. Probably it was just created by other worker. create_fip_agent_gw_port_if_not_exists /usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:927
2020-06-11 15:39:28.390 1285214 DEBUG neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port None found for the destination host: host-compute-1 create_fip_agent_gw_port_if_not_exists /usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:933
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Exception during message handling: AttributeError: 'NoneType' object has no attribute 'get'
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 170, in _process_incoming
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 348, in get_agent_gateway_port
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server admin_ctx, network_id, host)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py", line 953, in create_fip_agent_gw_port_if_not_exists
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self._populate_mtu_and_subnets_for_ports(context, [agent_port])
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1978, in _populate_mtu_and_subnets_for_ports
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server for p in self._each_port_having_fixed_ips(ports)]
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1925, in _each_port_having_fixed_ips
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server fixed_ips = port.get('fixed_ips', [])
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server AttributeError: 'NoneType' object has no attribute 'get'
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
-------------------------------------------------------------------------
[Impact]
In some cases the DvrFipGatewayPortAgentBinding is in DB but the gateway port does not exist.
This resulted in connectivity issues to FIP for the new VMs launched on that compute node.
The fix creates the gateway port if it does not exist.
[Test Case]
This is a race condition and difficult to reproduce. The test case simulated the error condition to verify the fix.
* Deploy openstack with dvr l3ha and centralised snat on neutron nodes
* Deploy instances and delete them. This step is to ensure FIP Agent gateway's are created on compute nodes
Check the following command to see FIP Agent gateway information
openstack port list --network ext_net -c id -c device_id -c binding_host_id -c device_owner -c fixed_ips | grep floatingip_agent_gateway
* Pick one of the compute node that has no instances and delete the FIP Agent gateway port (port id can be determined from above command)
openstack port delete <port id>
* Launch an instance on the compute node
openstack server create --wait --image cirros --flavor m1.cirros --nic net-id=<network id> --availability-zone nova:<hostname> cirros-test1
* Verify neutron-server logs for error
ERROR oslo_messaging.rpc.server AttributeError: 'NoneType' object has no attribute 'get'
* Assign floating ip and tried to ping fip and the ping fails
[Where problems could occur]
The fix itself adds an extra check to determine the cases when the gateway port needs to be created.
And hence it is not expected to cause any regression. |
In patch [1] it introduced a binding of DB uniq constraint for L3
agent gateway. In some extreme case the DvrFipGatewayPortAgentBinding
is in DB while the gateway port not. The current code path only checks
the binding existence which will pass a "None" port to the following
code path that results an AttributeError.
[1] https://review.opendev.org/#/c/702547/
Exception log:
2020-06-11 15:39:28.361 1285214 INFO neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 does not exist on host host-compute-1. Creating one.
2020-06-11 15:39:28.370 1285214 DEBUG neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 already exists on host host-compute-1. Probably it was just created by other worker. create_fip_agent_gw_port_if_not_exists /usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:927
2020-06-11 15:39:28.390 1285214 DEBUG neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port None found for the destination host: host-compute-1 create_fip_agent_gw_port_if_not_exists /usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:933
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Exception during message handling: AttributeError: 'NoneType' object has no attribute 'get'
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 170, in _process_incoming
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 348, in get_agent_gateway_port
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server admin_ctx, network_id, host)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py", line 953, in create_fip_agent_gw_port_if_not_exists
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self._populate_mtu_and_subnets_for_ports(context, [agent_port])
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1978, in _populate_mtu_and_subnets_for_ports
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server for p in self._each_port_having_fixed_ips(ports)]
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1925, in _each_port_having_fixed_ips
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server fixed_ips = port.get('fixed_ips', [])
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server AttributeError: 'NoneType' object has no attribute 'get'
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
-------------------------------------------------------------------------
[SRU]
[Impact]
In some cases the DvrFipGatewayPortAgentBinding is in DB but the gateway port does not exist.
This resulted in connectivity issues to FIP for the new VMs launched on that compute node.
The fix creates the gateway port if it does not exist.
[Test Case]
This is a race condition and difficult to reproduce. The test case simulated the error condition to verify the fix.
* Deploy openstack with dvr l3ha and centralised snat on neutron nodes
* Deploy instances and delete them. This step is to ensure FIP Agent gateway's are created on compute nodes
Check the following command to see FIP Agent gateway information
openstack port list --network ext_net -c id -c device_id -c binding_host_id -c device_owner -c fixed_ips | grep floatingip_agent_gateway
* Pick one of the compute node that has no instances and delete the FIP Agent gateway port (port id can be determined from above command)
openstack port delete <port id>
* Launch an instance on the compute node
openstack server create --wait --image cirros --flavor m1.cirros --nic net-id=<network id> --availability-zone nova:<hostname> cirros-test1
* Verify neutron-server logs for error
ERROR oslo_messaging.rpc.server AttributeError: 'NoneType' object has no attribute 'get'
* Assign floating ip and tried to ping fip and the ping fails
[Where problems could occur]
The fix itself adds an extra check to determine the cases when the gateway port needs to be created.
And hence it is not expected to cause any regression. |
|
| 2021-04-27 04:44:34 |
Hemanth Nakkina |
tags |
in-stable-ussuri l3-dvr-backlog sts timeout-abandon |
in-stable-ussuri l3-dvr-backlog sts sts-sru-needed timeout-abandon |
|
| 2021-04-27 04:45:30 |
Hemanth Nakkina |
attachment added |
|
Debdiff for focal https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1883089/+attachment/5492639/+files/lp1883089_focal.debdiff |
|
| 2021-04-27 04:49:08 |
Hemanth Nakkina |
neutron (Ubuntu Focal): assignee |
|
Hemanth Nakkina (hemanth-n) |
|
| 2021-04-27 08:25:49 |
Edward Hope-Morley |
bug task added |
|
cloud-archive |
|
| 2021-04-27 08:26:09 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/victoria |
|
| 2021-04-27 08:26:09 |
Edward Hope-Morley |
bug task added |
|
cloud-archive/victoria |
|
| 2021-04-27 08:26:09 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/ussuri |
|
| 2021-04-27 08:26:09 |
Edward Hope-Morley |
bug task added |
|
cloud-archive/ussuri |
|
| 2021-04-27 08:26:17 |
Edward Hope-Morley |
cloud-archive/victoria: status |
New |
Fix Released |
|
| 2021-04-27 10:09:08 |
Edward Hope-Morley |
description |
In patch [1] it introduced a binding of DB uniq constraint for L3
agent gateway. In some extreme case the DvrFipGatewayPortAgentBinding
is in DB while the gateway port not. The current code path only checks
the binding existence which will pass a "None" port to the following
code path that results an AttributeError.
[1] https://review.opendev.org/#/c/702547/
Exception log:
2020-06-11 15:39:28.361 1285214 INFO neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 does not exist on host host-compute-1. Creating one.
2020-06-11 15:39:28.370 1285214 DEBUG neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 already exists on host host-compute-1. Probably it was just created by other worker. create_fip_agent_gw_port_if_not_exists /usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:927
2020-06-11 15:39:28.390 1285214 DEBUG neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port None found for the destination host: host-compute-1 create_fip_agent_gw_port_if_not_exists /usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:933
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Exception during message handling: AttributeError: 'NoneType' object has no attribute 'get'
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 170, in _process_incoming
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 348, in get_agent_gateway_port
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server admin_ctx, network_id, host)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py", line 953, in create_fip_agent_gw_port_if_not_exists
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self._populate_mtu_and_subnets_for_ports(context, [agent_port])
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1978, in _populate_mtu_and_subnets_for_ports
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server for p in self._each_port_having_fixed_ips(ports)]
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1925, in _each_port_having_fixed_ips
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server fixed_ips = port.get('fixed_ips', [])
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server AttributeError: 'NoneType' object has no attribute 'get'
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
-------------------------------------------------------------------------
[SRU]
[Impact]
In some cases the DvrFipGatewayPortAgentBinding is in DB but the gateway port does not exist.
This resulted in connectivity issues to FIP for the new VMs launched on that compute node.
The fix creates the gateway port if it does not exist.
[Test Case]
This is a race condition and difficult to reproduce. The test case simulated the error condition to verify the fix.
* Deploy openstack with dvr l3ha and centralised snat on neutron nodes
* Deploy instances and delete them. This step is to ensure FIP Agent gateway's are created on compute nodes
Check the following command to see FIP Agent gateway information
openstack port list --network ext_net -c id -c device_id -c binding_host_id -c device_owner -c fixed_ips | grep floatingip_agent_gateway
* Pick one of the compute node that has no instances and delete the FIP Agent gateway port (port id can be determined from above command)
openstack port delete <port id>
* Launch an instance on the compute node
openstack server create --wait --image cirros --flavor m1.cirros --nic net-id=<network id> --availability-zone nova:<hostname> cirros-test1
* Verify neutron-server logs for error
ERROR oslo_messaging.rpc.server AttributeError: 'NoneType' object has no attribute 'get'
* Assign floating ip and tried to ping fip and the ping fails
[Where problems could occur]
The fix itself adds an extra check to determine the cases when the gateway port needs to be created.
And hence it is not expected to cause any regression. |
In patch [1] it introduced a binding of DB uniq constraint for L3
agent gateway. In some extreme case the DvrFipGatewayPortAgentBinding
is in DB while the gateway port not. The current code path only checks
the binding existence which will pass a "None" port to the following
code path that results an AttributeError.
[1] https://review.opendev.org/#/c/702547/
Exception log:
2020-06-11 15:39:28.361 1285214 INFO neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 does not exist on host host-compute-1. Creating one.
2020-06-11 15:39:28.370 1285214 DEBUG neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 already exists on host host-compute-1. Probably it was just created by other worker. create_fip_agent_gw_port_if_not_exists /usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:927
2020-06-11 15:39:28.390 1285214 DEBUG neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port None found for the destination host: host-compute-1 create_fip_agent_gw_port_if_not_exists /usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:933
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Exception during message handling: AttributeError: 'NoneType' object has no attribute 'get'
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 170, in _process_incoming
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 348, in get_agent_gateway_port
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server admin_ctx, network_id, host)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py", line 953, in create_fip_agent_gw_port_if_not_exists
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server self._populate_mtu_and_subnets_for_ports(context, [agent_port])
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1978, in _populate_mtu_and_subnets_for_ports
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server for p in self._each_port_having_fixed_ips(ports)]
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1925, in _each_port_having_fixed_ips
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server fixed_ips = port.get('fixed_ips', [])
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server AttributeError: 'NoneType' object has no attribute 'get'
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
-------------------------------------------------------------------------
[SRU]
[Impact]
In some cases the DvrFipGatewayPortAgentBinding is in DB but the gateway port does not exist.
This resulted in connectivity issues to FIP for the new VMs launched on that compute node.
The fix creates the gateway port if it does not exist.
[Test Plan]
This is a race condition and difficult to reproduce. The test case simulated the error condition to verify the fix.
* Deploy openstack with dvr l3ha and centralised snat on neutron nodes
* Deploy instances and delete them. This step is to ensure FIP Agent gateway's are created on compute nodes
Check the following command to see FIP Agent gateway information
openstack port list --network ext_net -c id -c device_id -c binding_host_id -c device_owner -c fixed_ips | grep floatingip_agent_gateway
* Pick one of the compute node that has no instances and delete the FIP Agent gateway port (port id can be determined from above command)
openstack port delete <port id>
* Launch an instance on the compute node
openstack server create --wait --image cirros --flavor m1.cirros --nic net-id=<network id> --availability-zone nova:<hostname> cirros-test1
* Verify neutron-server logs for error
ERROR oslo_messaging.rpc.server AttributeError: 'NoneType' object has no attribute 'get'
* Assign floating ip and tried to ping fip and the ping fails
[Where problems could occur]
The fix itself adds an extra check to determine the cases when the gateway port needs to be created.
And hence it is not expected to cause any regression. |
|
| 2021-05-14 11:25:38 |
Timo Aaltonen |
neutron (Ubuntu Focal): status |
New |
Fix Committed |
|
| 2021-05-14 11:25:43 |
Timo Aaltonen |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2021-05-14 11:25:47 |
Timo Aaltonen |
bug |
|
|
added subscriber SRU Verification |
| 2021-05-14 11:25:52 |
Timo Aaltonen |
tags |
in-stable-ussuri l3-dvr-backlog sts sts-sru-needed timeout-abandon |
in-stable-ussuri l3-dvr-backlog sts sts-sru-needed timeout-abandon verification-needed verification-needed-focal |
|
| 2021-05-17 06:04:15 |
Hemanth Nakkina |
tags |
in-stable-ussuri l3-dvr-backlog sts sts-sru-needed timeout-abandon verification-needed verification-needed-focal |
in-stable-ussuri l3-dvr-backlog sts sts-sru-needed timeout-abandon verification-done-focal verification-needed |
|
| 2021-05-24 10:20:51 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2021-05-24 10:23:42 |
Launchpad Janitor |
neutron (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2021-05-24 18:13:50 |
Corey Bryant |
cloud-archive/ussuri: status |
New |
Fix Committed |
|
| 2021-05-24 18:13:55 |
Corey Bryant |
tags |
in-stable-ussuri l3-dvr-backlog sts sts-sru-needed timeout-abandon verification-done-focal verification-needed |
in-stable-ussuri l3-dvr-backlog sts sts-sru-needed timeout-abandon verification-done-focal verification-needed verification-ussuri-needed |
|
| 2021-05-26 04:33:38 |
Hemanth Nakkina |
tags |
in-stable-ussuri l3-dvr-backlog sts sts-sru-needed timeout-abandon verification-done-focal verification-needed verification-ussuri-needed |
in-stable-ussuri l3-dvr-backlog sts sts-sru-needed timeout-abandon verification-done verification-done-focal verification-ussuri-done |
|
| 2021-06-21 03:43:40 |
Hemanth Nakkina |
cloud-archive/ussuri: status |
Fix Committed |
Fix Released |
|
| 2021-06-21 03:52:22 |
Hemanth Nakkina |
cloud-archive: status |
New |
Fix Released |
|
| 2021-08-17 06:11:25 |
Brett Milford |
bug |
|
|
added subscriber Brett Milford |
