OVSFWTagNotFound raised many times in neutron-openvswitch-agent logs

Bug #1811405 reported by Slawek Kaplonski
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Lajos Katona

Bug Description

I found out that recently exception like:

Jan 11 08:56:11.074296 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-207e9202-dfc1-4b0d-b9d4-85eaec6174ee None None] Error while processing VIF ports: neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port tap20acc597-23 from its other_config: {}
Jan 11 08:56:11.074665 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
Jan 11 08:56:11.074999 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 526, in get_or_create_ofport
Jan 11 08:56:11.075349 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port = self.sg_port_map.ports[port_id]
Jan 11 08:56:11.075713 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: '20acc597-23f0-4c18-980e-061aa6494ccf'
Jan 11 08:56:11.076036 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
Jan 11 08:56:11.077854 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
Jan 11 08:56:11.078269 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
Jan 11 08:56:11.078611 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
Jan 11 08:56:11.079337 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 79, in get_tag_from_other_config
Jan 11 08:56:11.079670 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return int(other_config['tag'])
Jan 11 08:56:11.080000 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'tag'
Jan 11 08:56:11.080351 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
Jan 11 08:56:11.080677 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
Jan 11 08:56:11.081027 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
Jan 11 08:56:11.081354 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
Jan 11 08:56:11.081681 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2196, in rpc_loop
Jan 11 08:56:11.082077 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, provisioning_needed)
Jan 11 08:56:11.082475 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python3.5/dist-packages/osprofiler/profiler.py", line 160, in wrapper
Jan 11 08:56:11.082907 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs)
Jan 11 08:56:11.083233 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1770, in process_network_ports
Jan 11 08:56:11.083560 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info.get('updated', set()))
Jan 11 08:56:11.083903 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 268, in setup_port_filters
Jan 11 08:56:11.084232 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.refresh_firewall(updated_devices)
Jan 11 08:56:11.084563 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 122, in decorated_function
Jan 11 08:56:11.084901 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent *args, **kwargs)
Jan 11 08:56:11.085230 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 221, in refresh_firewall
Jan 11 08:56:11.085575 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._apply_port_filter(device_ids, update_filter=True)
Jan 11 08:56:11.085961 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 153, in _apply_port_filter
Jan 11 08:56:11.086287 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.firewall.update_port_filter(device)
Jan 11 08:56:11.086680 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 581, in update_port_filter
Jan 11 08:56:11.086999 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port = self.get_or_create_ofport(port)
Jan 11 08:56:11.087321 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 528, in get_or_create_ofport
Jan 11 08:56:11.087936 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_vlan_id = self._get_port_vlan_tag(ovs_port.port_name)
Jan 11 08:56:11.088270 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 512, in _get_port_vlan_tag
Jan 11 08:56:11.089768 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return get_tag_from_other_config(self.int_br.br, port_name)
Jan 11 08:56:11.090081 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 82, in get_tag_from_other_config
Jan 11 08:56:11.090328 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_name=port_name, other_config=other_config)
Jan 11 08:56:11.090619 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port tap20acc597-23 from its other_config: {}

Is appearing in q-agt logs in CI jobs quite often.
Example: http://logs.openstack.org/41/626641/17/check/grenade-py3/50d0a7b/logs/screen-q-agt.txt.gz?level=ERROR

I'm not sure if this is direct reason of test failures but we should avoid such exceptions in logs IMHO if it is "normal" case.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/630910

Changed in neutron:
assignee: nobody → Lajos Katona (lajos-katona)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/631584

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/631584
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f8e0a497ad6a36fa0d9510cda9e7344c2ed73763
Submitter: Zuul
Branch: master

commit f8e0a497ad6a36fa0d9510cda9e7344c2ed73763
Author: Slawek Kaplonski <email address hidden>
Date: Thu Jan 17 18:18:49 2019 +0100

    Skip ovsdb_monitor events for ports with ofport=-1

    There is no need to trigger neutron-openvswitch-agent to
    start processing ports in case if invalid port (with
    ofport=-1) will be added to bridge.
    Such port will be processed later when ofport will be
    changed.

    Change-Id: I3465daf4809d5d56565f59b177b5f8870352cc9d
    Related-Bug: #1808171
    Related-Bug: #1811405

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/630910
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=22f55822aab17c758f5f723c304b1acaf3179ec1
Submitter: Zuul
Branch: master

commit 22f55822aab17c758f5f723c304b1acaf3179ec1
Author: Lajos Katona <email address hidden>
Date: Tue Jan 15 09:20:54 2019 +0100

    Catch OVSFWTagNotFound in update_port_filter

    In neutron.agent.linux.openvswitch_firewall.firewall make the method
    update_port_filter catch OVSFWTagNotFound and log it to avoid
    traceback in log files.

    Change-Id: I584d867f0e1c47447cb8790fd715fa01ec902438
    Closes-Bug: #1811405

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 14.0.0.0rc1

This issue was fixed in the openstack/neutron 14.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/707712

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky)

Reviewed: https://review.opendev.org/707712
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6d17829aa664e7cf6a92291963ff622c8b763b53
Submitter: Zuul
Branch: stable/rocky

commit 6d17829aa664e7cf6a92291963ff622c8b763b53
Author: Lajos Katona <email address hidden>
Date: Tue Jan 15 09:20:54 2019 +0100

    Catch OVSFWTagNotFound in update_port_filter

    In neutron.agent.linux.openvswitch_firewall.firewall make the method
    update_port_filter catch OVSFWTagNotFound and log it to avoid
    traceback in log files.

    Conflicts:
      neutron/agent/linux/openvswitch_firewall/firewall.py

    Change-Id: I584d867f0e1c47447cb8790fd715fa01ec902438
    Closes-Bug: #1811405
    (cherry picked from commit 22f55822aab17c758f5f723c304b1acaf3179ec1)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 13.0.7

This issue was fixed in the openstack/neutron 13.0.7 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/738649

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/queens)

Reviewed: https://review.opendev.org/738649
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8fd0f912caf2e7db9ccf701354d7a7a9b89adf5c
Submitter: Zuul
Branch: stable/queens

commit 8fd0f912caf2e7db9ccf701354d7a7a9b89adf5c
Author: Lajos Katona <email address hidden>
Date: Tue Jan 15 09:20:54 2019 +0100

    Catch OVSFWTagNotFound in update_port_filter

    In neutron.agent.linux.openvswitch_firewall.firewall make the method
    update_port_filter catch OVSFWTagNotFound and log it to avoid
    traceback in log files.

    Conflicts:
      neutron/agent/linux/openvswitch_firewall/firewall.py

    Change-Id: I584d867f0e1c47447cb8790fd715fa01ec902438
    Closes-Bug: #1811405
    (cherry picked from commit 22f55822aab17c758f5f723c304b1acaf3179ec1)
    (cherry picked from commit 6d17829aa664e7cf6a92291963ff622c8b763b53)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron queens-eol

This issue was fixed in the openstack/neutron queens-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.