| 2019-10-21 11:33:23 |
Oleg Bondarev |
bug |
|
|
added bug |
| 2019-10-21 11:43:37 |
OpenStack Infra |
neutron: status |
New |
In Progress |
|
| 2019-10-22 07:29:53 |
Lajos Katona |
tags |
|
ovs-fw |
|
| 2019-10-27 20:44:10 |
OpenStack Infra |
neutron: status |
In Progress |
Fix Released |
|
| 2019-10-29 23:19:21 |
OpenStack Infra |
tags |
ovs-fw |
in-stable-train ovs-fw |
|
| 2019-10-30 19:28:19 |
OpenStack Infra |
tags |
in-stable-train ovs-fw |
in-stable-stein in-stable-train ovs-fw |
|
| 2019-11-15 06:24:53 |
OpenStack Infra |
tags |
in-stable-stein in-stable-train ovs-fw |
in-stable-rocky in-stable-stein in-stable-train ovs-fw |
|
| 2019-11-15 14:17:43 |
OpenStack Infra |
tags |
in-stable-rocky in-stable-stein in-stable-train ovs-fw |
in-stable-queens in-stable-rocky in-stable-stein in-stable-train ovs-fw |
|
| 2019-11-28 10:45:42 |
Bernard Cafarelli |
tags |
in-stable-queens in-stable-rocky in-stable-stein in-stable-train ovs-fw |
in-stable-queens in-stable-rocky in-stable-stein in-stable-train neutron-proactive-backport-potential ovs-fw |
|
| 2021-05-03 08:18:54 |
Seyeong Kim |
bug task added |
|
neutron (Ubuntu) |
|
| 2021-05-03 08:19:12 |
Seyeong Kim |
nominated for series |
|
Ubuntu Bionic |
|
| 2021-05-03 08:19:12 |
Seyeong Kim |
bug task added |
|
neutron (Ubuntu Bionic) |
|
| 2021-05-03 08:19:22 |
Seyeong Kim |
neutron (Ubuntu): status |
New |
Fix Released |
|
| 2021-05-03 08:19:26 |
Seyeong Kim |
neutron (Ubuntu Bionic): assignee |
|
Seyeong Kim (seyeongkim) |
|
| 2021-05-03 08:19:35 |
Seyeong Kim |
neutron (Ubuntu Bionic): status |
New |
In Progress |
|
| 2021-05-03 08:19:50 |
Seyeong Kim |
attachment added |
|
lp1849098_bionic.debdiff https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1849098/+attachment/5494377/+files/lp1849098_bionic.debdiff |
|
| 2021-05-03 08:24:29 |
Seyeong Kim |
description |
neutron-openvswitch-agent meets unbound port:
2019-10-17 11:32:21.868 135 WARNING neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Device ef34215f-e099-4fd0-935f-c9a42951d166 not defined on plugin or binding failed
Later when applying firewall rules:
2019-10-17 11:32:21.901 135 INFO neutron.agent.securitygroups_rpc [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Preparing filters for devices {'ef34215f-e099-4fd0-935f-c9a42951d166', 'e9c97cf0-1a5e-4d77-b57b-0ba474d12e29', 'fff1bb24-6423-4486-87c4-1fe17c552cca', '2e20f9ee-bcb5-445c-b31f-d70d276d45c9', '03a60047-cb07-42a4-8b49-619d5982a9bd', 'a452cea2-deaf-4411-bbae-ce83870cbad4', '79b03e5c-9be0-4808-9784-cb4878c3dbd5', '9b971e75-3c1b-463d-88cf-3f298105fa6e'}
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Error while processing VIF ports: neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port o-hm0 from its other_config: {}
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 530, in get_or_create_ofport
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port = self.sg_port_map.ports[port_id]
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'ef34215f-e099-4fd0-935f-c9a42951d166'
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 81, in get_tag_from_other_config
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return int(other_config['tag'])
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'tag'
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2280, in rpc_loop
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, provisioning_needed)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1847, in process_network_ports
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info.get('updated', set()))
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 258, in setup_port_filters
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.prepare_devices_filter(new_devices)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 125, in decorated_function
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent *args, **kwargs)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 133, in prepare_devices_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._apply_port_filter(device_ids)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 164, in _apply_port_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.firewall.prepare_port_filter(device)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 555, in prepare_port_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port = self.get_or_create_ofport(port)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 532, in get_or_create_ofport
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_vlan_id = self._get_port_vlan_tag(ovs_port.port_name)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 516, in _get_port_vlan_tag
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return get_tag_from_other_config(self.int_br.br, port_name)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 84, in get_tag_from_other_config
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_name=port_name, other_config=other_config)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port o-hm0 from its other_config: {}
2019-10-17 11:32:21.909 135 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Agent out of sync with plugin!
this happens in each agent cycle so agent can't do anything.
Need to handle OVSFWTagNotFound in prepare_port_filter() like was done for update_port_filter in https://review.opendev.org/#/c/630910/ |
[Impact]
somehow port is unbounded, then neutron-openvswitch-agent raise
OVSFWTagNotFound, then creating new instance will be failed.
[Test Plan]
1. deploy bionic openstack env
2. launch one instance
3. modify neutron-openvswitch-agent code inside nova-compute
- https://pastebin.ubuntu.com/p/nBRKkXmjx8/
4. restart neutron-openvswitch-agent
5. check if there are a lot of cannot get tag for port ..
6. launch another instance.
7. It fails after vif_plugging_timeout, with "virtual interface creation failed"
[Where problems could occur]
You need to restart service. and as patch, Basically it will be ok as it adds only exceptions. but getting or creating vif port part can have issue.
[Others]
Original description.
neutron-openvswitch-agent meets unbound port:
2019-10-17 11:32:21.868 135 WARNING neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Device ef34215f-e099-4fd0-935f-c9a42951d166 not defined on plugin or binding failed
Later when applying firewall rules:
2019-10-17 11:32:21.901 135 INFO neutron.agent.securitygroups_rpc [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Preparing filters for devices {'ef34215f-e099-4fd0-935f-c9a42951d166', 'e9c97cf0-1a5e-4d77-b57b-0ba474d12e29', 'fff1bb24-6423-4486-87c4-1fe17c552cca', '2e20f9ee-bcb5-445c-b31f-d70d276d45c9', '03a60047-cb07-42a4-8b49-619d5982a9bd', 'a452cea2-deaf-4411-bbae-ce83870cbad4', '79b03e5c-9be0-4808-9784-cb4878c3dbd5', '9b971e75-3c1b-463d-88cf-3f298105fa6e'}
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Error while processing VIF ports: neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port o-hm0 from its other_config: {}
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 530, in get_or_create_ofport
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port = self.sg_port_map.ports[port_id]
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'ef34215f-e099-4fd0-935f-c9a42951d166'
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 81, in get_tag_from_other_config
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return int(other_config['tag'])
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'tag'
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2280, in rpc_loop
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, provisioning_needed)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1847, in process_network_ports
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info.get('updated', set()))
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 258, in setup_port_filters
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.prepare_devices_filter(new_devices)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 125, in decorated_function
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent *args, **kwargs)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 133, in prepare_devices_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._apply_port_filter(device_ids)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 164, in _apply_port_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.firewall.prepare_port_filter(device)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 555, in prepare_port_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port = self.get_or_create_ofport(port)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 532, in get_or_create_ofport
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_vlan_id = self._get_port_vlan_tag(ovs_port.port_name)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 516, in _get_port_vlan_tag
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return get_tag_from_other_config(self.int_br.br, port_name)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 84, in get_tag_from_other_config
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_name=port_name, other_config=other_config)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port o-hm0 from its other_config: {}
2019-10-17 11:32:21.909 135 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Agent out of sync with plugin!
this happens in each agent cycle so agent can't do anything.
Need to handle OVSFWTagNotFound in prepare_port_filter() like was done for update_port_filter in https://review.opendev.org/#/c/630910/ |
|
| 2021-05-03 08:24:41 |
Seyeong Kim |
tags |
in-stable-queens in-stable-rocky in-stable-stein in-stable-train neutron-proactive-backport-potential ovs-fw |
in-stable-queens in-stable-rocky in-stable-stein in-stable-train neutron-proactive-backport-potential ovs-fw sts |
|
| 2021-05-03 08:24:54 |
Seyeong Kim |
bug |
|
|
added subscriber SRU Developers |
| 2021-05-03 08:41:34 |
Dominique Poulain |
bug |
|
|
added subscriber Dominique Poulain |
| 2021-05-04 13:45:50 |
Eric Desrochers |
bug |
|
|
added subscriber STS Sponsors |
| 2021-05-04 15:39:20 |
Corey Bryant |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2021-05-04 15:49:43 |
Dan Streetman |
description |
[Impact]
somehow port is unbounded, then neutron-openvswitch-agent raise
OVSFWTagNotFound, then creating new instance will be failed.
[Test Plan]
1. deploy bionic openstack env
2. launch one instance
3. modify neutron-openvswitch-agent code inside nova-compute
- https://pastebin.ubuntu.com/p/nBRKkXmjx8/
4. restart neutron-openvswitch-agent
5. check if there are a lot of cannot get tag for port ..
6. launch another instance.
7. It fails after vif_plugging_timeout, with "virtual interface creation failed"
[Where problems could occur]
You need to restart service. and as patch, Basically it will be ok as it adds only exceptions. but getting or creating vif port part can have issue.
[Others]
Original description.
neutron-openvswitch-agent meets unbound port:
2019-10-17 11:32:21.868 135 WARNING neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Device ef34215f-e099-4fd0-935f-c9a42951d166 not defined on plugin or binding failed
Later when applying firewall rules:
2019-10-17 11:32:21.901 135 INFO neutron.agent.securitygroups_rpc [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Preparing filters for devices {'ef34215f-e099-4fd0-935f-c9a42951d166', 'e9c97cf0-1a5e-4d77-b57b-0ba474d12e29', 'fff1bb24-6423-4486-87c4-1fe17c552cca', '2e20f9ee-bcb5-445c-b31f-d70d276d45c9', '03a60047-cb07-42a4-8b49-619d5982a9bd', 'a452cea2-deaf-4411-bbae-ce83870cbad4', '79b03e5c-9be0-4808-9784-cb4878c3dbd5', '9b971e75-3c1b-463d-88cf-3f298105fa6e'}
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Error while processing VIF ports: neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port o-hm0 from its other_config: {}
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 530, in get_or_create_ofport
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port = self.sg_port_map.ports[port_id]
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'ef34215f-e099-4fd0-935f-c9a42951d166'
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 81, in get_tag_from_other_config
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return int(other_config['tag'])
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'tag'
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2280, in rpc_loop
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, provisioning_needed)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1847, in process_network_ports
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info.get('updated', set()))
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 258, in setup_port_filters
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.prepare_devices_filter(new_devices)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 125, in decorated_function
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent *args, **kwargs)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 133, in prepare_devices_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._apply_port_filter(device_ids)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 164, in _apply_port_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.firewall.prepare_port_filter(device)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 555, in prepare_port_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port = self.get_or_create_ofport(port)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 532, in get_or_create_ofport
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_vlan_id = self._get_port_vlan_tag(ovs_port.port_name)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 516, in _get_port_vlan_tag
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return get_tag_from_other_config(self.int_br.br, port_name)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 84, in get_tag_from_other_config
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_name=port_name, other_config=other_config)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port o-hm0 from its other_config: {}
2019-10-17 11:32:21.909 135 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Agent out of sync with plugin!
this happens in each agent cycle so agent can't do anything.
Need to handle OVSFWTagNotFound in prepare_port_filter() like was done for update_port_filter in https://review.opendev.org/#/c/630910/ |
[Impact]
somehow port is unbounded, then neutron-openvswitch-agent raise
OVSFWTagNotFound, then creating new instance will be failed.
[Test Plan]
1. deploy bionic openstack env
2. launch one instance
3. modify neutron-openvswitch-agent code inside nova-compute
- https://pastebin.ubuntu.com/p/nBRKkXmjx8/
4. restart neutron-openvswitch-agent
5. check if there are a lot of cannot get tag for port ..
6. launch another instance.
7. It fails after vif_plugging_timeout, with "virtual interface creation failed"
[Where problems could occur]
issues could occur getting or creating vif port
[Others]
Original description.
neutron-openvswitch-agent meets unbound port:
2019-10-17 11:32:21.868 135 WARNING neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Device ef34215f-e099-4fd0-935f-c9a42951d166 not defined on plugin or binding failed
Later when applying firewall rules:
2019-10-17 11:32:21.901 135 INFO neutron.agent.securitygroups_rpc [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Preparing filters for devices {'ef34215f-e099-4fd0-935f-c9a42951d166', 'e9c97cf0-1a5e-4d77-b57b-0ba474d12e29', 'fff1bb24-6423-4486-87c4-1fe17c552cca', '2e20f9ee-bcb5-445c-b31f-d70d276d45c9', '03a60047-cb07-42a4-8b49-619d5982a9bd', 'a452cea2-deaf-4411-bbae-ce83870cbad4', '79b03e5c-9be0-4808-9784-cb4878c3dbd5', '9b971e75-3c1b-463d-88cf-3f298105fa6e'}
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Error while processing VIF ports: neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port o-hm0 from its other_config: {}
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 530, in get_or_create_ofport
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port = self.sg_port_map.ports[port_id]
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'ef34215f-e099-4fd0-935f-c9a42951d166'
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 81, in get_tag_from_other_config
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return int(other_config['tag'])
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'tag'
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2280, in rpc_loop
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, provisioning_needed)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1847, in process_network_ports
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info.get('updated', set()))
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 258, in setup_port_filters
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.prepare_devices_filter(new_devices)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 125, in decorated_function
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent *args, **kwargs)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 133, in prepare_devices_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._apply_port_filter(device_ids)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 164, in _apply_port_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.firewall.prepare_port_filter(device)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 555, in prepare_port_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port = self.get_or_create_ofport(port)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 532, in get_or_create_ofport
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_vlan_id = self._get_port_vlan_tag(ovs_port.port_name)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 516, in _get_port_vlan_tag
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return get_tag_from_other_config(self.int_br.br, port_name)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 84, in get_tag_from_other_config
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_name=port_name, other_config=other_config)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port o-hm0 from its other_config: {}
2019-10-17 11:32:21.909 135 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Agent out of sync with plugin!
this happens in each agent cycle so agent can't do anything.
Need to handle OVSFWTagNotFound in prepare_port_filter() like was done for update_port_filter in https://review.opendev.org/#/c/630910/ |
|
| 2021-05-04 15:52:01 |
Dan Streetman |
description |
[Impact]
somehow port is unbounded, then neutron-openvswitch-agent raise
OVSFWTagNotFound, then creating new instance will be failed.
[Test Plan]
1. deploy bionic openstack env
2. launch one instance
3. modify neutron-openvswitch-agent code inside nova-compute
- https://pastebin.ubuntu.com/p/nBRKkXmjx8/
4. restart neutron-openvswitch-agent
5. check if there are a lot of cannot get tag for port ..
6. launch another instance.
7. It fails after vif_plugging_timeout, with "virtual interface creation failed"
[Where problems could occur]
issues could occur getting or creating vif port
[Others]
Original description.
neutron-openvswitch-agent meets unbound port:
2019-10-17 11:32:21.868 135 WARNING neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Device ef34215f-e099-4fd0-935f-c9a42951d166 not defined on plugin or binding failed
Later when applying firewall rules:
2019-10-17 11:32:21.901 135 INFO neutron.agent.securitygroups_rpc [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Preparing filters for devices {'ef34215f-e099-4fd0-935f-c9a42951d166', 'e9c97cf0-1a5e-4d77-b57b-0ba474d12e29', 'fff1bb24-6423-4486-87c4-1fe17c552cca', '2e20f9ee-bcb5-445c-b31f-d70d276d45c9', '03a60047-cb07-42a4-8b49-619d5982a9bd', 'a452cea2-deaf-4411-bbae-ce83870cbad4', '79b03e5c-9be0-4808-9784-cb4878c3dbd5', '9b971e75-3c1b-463d-88cf-3f298105fa6e'}
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Error while processing VIF ports: neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port o-hm0 from its other_config: {}
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 530, in get_or_create_ofport
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port = self.sg_port_map.ports[port_id]
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'ef34215f-e099-4fd0-935f-c9a42951d166'
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 81, in get_tag_from_other_config
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return int(other_config['tag'])
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'tag'
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2280, in rpc_loop
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, provisioning_needed)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1847, in process_network_ports
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info.get('updated', set()))
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 258, in setup_port_filters
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.prepare_devices_filter(new_devices)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 125, in decorated_function
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent *args, **kwargs)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 133, in prepare_devices_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._apply_port_filter(device_ids)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 164, in _apply_port_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.firewall.prepare_port_filter(device)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 555, in prepare_port_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port = self.get_or_create_ofport(port)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 532, in get_or_create_ofport
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_vlan_id = self._get_port_vlan_tag(ovs_port.port_name)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 516, in _get_port_vlan_tag
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return get_tag_from_other_config(self.int_br.br, port_name)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 84, in get_tag_from_other_config
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_name=port_name, other_config=other_config)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port o-hm0 from its other_config: {}
2019-10-17 11:32:21.909 135 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Agent out of sync with plugin!
this happens in each agent cycle so agent can't do anything.
Need to handle OVSFWTagNotFound in prepare_port_filter() like was done for update_port_filter in https://review.opendev.org/#/c/630910/ |
[Impact]
somehow port is unbounded, then neutron-openvswitch-agent raise
OVSFWTagNotFound, then creating new instance will be failed.
[Test Plan]
1. deploy bionic openstack env
2. launch one instance
3. modify neutron-openvswitch-agent code inside nova-compute
- https://pastebin.ubuntu.com/p/nBRKkXmjx8/
4. restart neutron-openvswitch-agent
5. check if there are a lot of cannot get tag for port ..
6. launch another instance.
7. It fails after vif_plugging_timeout, with "virtual interface creation failed"
[Where problems could occur]
while no regressions are expected, if they do occur it would be when getting or creating vif port
[Others]
Original description.
neutron-openvswitch-agent meets unbound port:
2019-10-17 11:32:21.868 135 WARNING neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Device ef34215f-e099-4fd0-935f-c9a42951d166 not defined on plugin or binding failed
Later when applying firewall rules:
2019-10-17 11:32:21.901 135 INFO neutron.agent.securitygroups_rpc [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Preparing filters for devices {'ef34215f-e099-4fd0-935f-c9a42951d166', 'e9c97cf0-1a5e-4d77-b57b-0ba474d12e29', 'fff1bb24-6423-4486-87c4-1fe17c552cca', '2e20f9ee-bcb5-445c-b31f-d70d276d45c9', '03a60047-cb07-42a4-8b49-619d5982a9bd', 'a452cea2-deaf-4411-bbae-ce83870cbad4', '79b03e5c-9be0-4808-9784-cb4878c3dbd5', '9b971e75-3c1b-463d-88cf-3f298105fa6e'}
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Error while processing VIF ports: neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port o-hm0 from its other_config: {}
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 530, in get_or_create_ofport
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port = self.sg_port_map.ports[port_id]
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'ef34215f-e099-4fd0-935f-c9a42951d166'
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 81, in get_tag_from_other_config
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return int(other_config['tag'])
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'tag'
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2280, in rpc_loop
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, provisioning_needed)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1847, in process_network_ports
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info.get('updated', set()))
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 258, in setup_port_filters
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.prepare_devices_filter(new_devices)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 125, in decorated_function
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent *args, **kwargs)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 133, in prepare_devices_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._apply_port_filter(device_ids)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 164, in _apply_port_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.firewall.prepare_port_filter(device)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 555, in prepare_port_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port = self.get_or_create_ofport(port)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 532, in get_or_create_ofport
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_vlan_id = self._get_port_vlan_tag(ovs_port.port_name)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 516, in _get_port_vlan_tag
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return get_tag_from_other_config(self.int_br.br, port_name)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 84, in get_tag_from_other_config
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_name=port_name, other_config=other_config)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port o-hm0 from its other_config: {}
2019-10-17 11:32:21.909 135 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Agent out of sync with plugin!
this happens in each agent cycle so agent can't do anything.
Need to handle OVSFWTagNotFound in prepare_port_filter() like was done for update_port_filter in https://review.opendev.org/#/c/630910/ |
|
| 2021-05-05 09:44:42 |
Robie Basak |
neutron (Ubuntu Bionic): status |
In Progress |
Fix Committed |
|
| 2021-05-05 09:44:46 |
Robie Basak |
bug |
|
|
added subscriber SRU Verification |
| 2021-05-05 09:44:51 |
Robie Basak |
tags |
in-stable-queens in-stable-rocky in-stable-stein in-stable-train neutron-proactive-backport-potential ovs-fw sts |
in-stable-queens in-stable-rocky in-stable-stein in-stable-train neutron-proactive-backport-potential ovs-fw sts verification-needed verification-needed-bionic |
|
| 2021-05-05 12:01:47 |
Hemanth Nakkina |
tags |
in-stable-queens in-stable-rocky in-stable-stein in-stable-train neutron-proactive-backport-potential ovs-fw sts verification-needed verification-needed-bionic |
in-stable-queens in-stable-rocky in-stable-stein in-stable-train neutron-proactive-backport-potential ovs-fw sts verification-done verification-done-bionic |
|
| 2021-05-11 08:23:00 |
Edward Hope-Morley |
bug task added |
|
cloud-archive |
|
| 2021-05-11 08:23:12 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/queens |
|
| 2021-05-11 08:23:12 |
Edward Hope-Morley |
bug task added |
|
cloud-archive/queens |
|
| 2021-05-13 08:01:12 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2021-05-13 08:03:40 |
Launchpad Janitor |
neutron (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2021-05-13 15:04:04 |
Corey Bryant |
cloud-archive/queens: status |
New |
Fix Released |
|
| 2021-05-21 02:40:29 |
Hemanth Nakkina |
cloud-archive: status |
New |
Fix Released |
|
| 2021-10-12 12:00:52 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~hopem/ubuntu/+source/neutron/+git/neutron/+merge/410050 |
|
