OVSFWTagNotFound raised many times in neutron-openvswitch-agent logs

[Slawek Kaplonski]

I found out that recently exception like:

Jan 11 08:56:11.074296 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-207e9202-dfc1-4b0d-b9d4-85eaec6174ee None None] Error while processing VIF ports: neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port tap20acc597-23 from its other_config: {}
Jan 11 08:56:11.074665 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
Jan 11 08:56:11.074999 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 526, in get_or_create_ofport
Jan 11 08:56:11.075349 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port = self.sg_port_map.ports[port_id]
Jan 11 08:56:11.075713 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: '20acc597-23f0-4c18-980e-061aa6494ccf'
Jan 11 08:56:11.076036 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
Jan 11 08:56:11.077854 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
Jan 11 08:56:11.078269 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
Jan 11 08:56:11.078611 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
Jan 11 08:56:11.079337 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 79, in get_tag_from_other_config
Jan 11 08:56:11.079670 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return int(other_config['tag'])
Jan 11 08:56:11.080000 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'tag'
Jan 11 08:56:11.080351 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
Jan 11 08:56:11.080677 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
Jan 11 08:56:11.081027 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
Jan 11 08:56:11.081354 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
Jan 11 08:56:11.081681 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2196, in rpc_loop
Jan 11 08:56:11.082077 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, provisioning_needed)
Jan 11 08:56:11.082475 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python3.5/dist-packages/osprofiler/profiler.py", line 160, in wrapper
Jan 11 08:56:11.082907 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs)
Jan 11 08:56:11.083233 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1770, in process_network_ports
Jan 11 08:56:11.083560 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info.get('updated', set()))
Jan 11 08:56:11.083903 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 268, in setup_port_filters
Jan 11 08:56:11.084232 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.refresh_firewall(updated_devices)
Jan 11 08:56:11.084563 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 122, in decorated_function
Jan 11 08:56:11.084901 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent *args, **kwargs)
Jan 11 08:56:11.085230 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 221, in refresh_firewall
Jan 11 08:56:11.085575 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._apply_port_filter(device_ids, update_filter=True)
Jan 11 08:56:11.085961 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 153, in _apply_port_filter
Jan 11 08:56:11.086287 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.firewall.update_port_filter(device)
Jan 11 08:56:11.086680 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 581, in update_port_filter
Jan 11 08:56:11.086999 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port = self.get_or_create_ofport(port)
Jan 11 08:56:11.087321 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 528, in get_or_create_ofport
Jan 11 08:56:11.087936 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_vlan_id = self._get_port_vlan_tag(ovs_port.port_name)
Jan 11 08:56:11.088270 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 512, in _get_port_vlan_tag
Jan 11 08:56:11.089768 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return get_tag_from_other_config(self.int_br.br, port_name)
Jan 11 08:56:11.090081 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 82, in get_tag_from_other_config
Jan 11 08:56:11.090328 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_name=port_name, other_config=other_config)
Jan 11 08:56:11.090619 ubuntu-xenial-rax-dfw-0001639322 neutron-openvswitch-agent[13907]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port tap20acc597-23 from its other_config: {}

Is appearing in q-agt logs in CI jobs quite often.
Example: http://logs.openstack.org/41/626641/17/check/grenade-py3/50d0a7b/logs/screen-q-agt.txt.gz?level=ERROR

I'm not sure if this is direct reason of test failures but we should avoid such exceptions in logs IMHO if it is "normal" case.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/630910

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/631584

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/631584
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f8e0a497ad6a36fa0d9510cda9e7344c2ed73763
Submitter: Zuul
Branch: master

commit f8e0a497ad6a36fa0d9510cda9e7344c2ed73763
Author: Slawek Kaplonski <email address hidden>
Date: Thu Jan 17 18:18:49 2019 +0100

    Skip ovsdb_monitor events for ports with ofport=-1

    There is no need to trigger neutron-openvswitch-agent to
    start processing ports in case if invalid port (with
    ofport=-1) will be added to bridge.
    Such port will be processed later when ofport will be
    changed.

    Change-Id: I3465daf4809d5d56565f59b177b5f8870352cc9d
    Related-Bug: #1808171
    Related-Bug: #1811405

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/630910
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=22f55822aab17c758f5f723c304b1acaf3179ec1
Submitter: Zuul
Branch: master

commit 22f55822aab17c758f5f723c304b1acaf3179ec1
Author: Lajos Katona <email address hidden>
Date: Tue Jan 15 09:20:54 2019 +0100

    Catch OVSFWTagNotFound in update_port_filter

    In neutron.agent.linux.openvswitch_firewall.firewall make the method
    update_port_filter catch OVSFWTagNotFound and log it to avoid
    traceback in log files.

    Change-Id: I584d867f0e1c47447cb8790fd715fa01ec902438
    Closes-Bug: #1811405

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/707712

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/707712
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6d17829aa664e7cf6a92291963ff622c8b763b53
Submitter: Zuul
Branch: stable/rocky

commit 6d17829aa664e7cf6a92291963ff622c8b763b53
Author: Lajos Katona <email address hidden>
Date: Tue Jan 15 09:20:54 2019 +0100

    Catch OVSFWTagNotFound in update_port_filter

    In neutron.agent.linux.openvswitch_firewall.firewall make the method
    update_port_filter catch OVSFWTagNotFound and log it to avoid
    traceback in log files.

    Conflicts:
      neutron/agent/linux/openvswitch_firewall/firewall.py

    Change-Id: I584d867f0e1c47447cb8790fd715fa01ec902438
    Closes-Bug: #1811405
    (cherry picked from commit 22f55822aab17c758f5f723c304b1acaf3179ec1)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.7 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/738649

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738649
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8fd0f912caf2e7db9ccf701354d7a7a9b89adf5c
Submitter: Zuul
Branch: stable/queens

commit 8fd0f912caf2e7db9ccf701354d7a7a9b89adf5c
Author: Lajos Katona <email address hidden>
Date: Tue Jan 15 09:20:54 2019 +0100

    Catch OVSFWTagNotFound in update_port_filter

    In neutron.agent.linux.openvswitch_firewall.firewall make the method
    update_port_filter catch OVSFWTagNotFound and log it to avoid
    traceback in log files.

    Conflicts:
      neutron/agent/linux/openvswitch_firewall/firewall.py

    Change-Id: I584d867f0e1c47447cb8790fd715fa01ec902438
    Closes-Bug: #1811405
    (cherry picked from commit 22f55822aab17c758f5f723c304b1acaf3179ec1)
    (cherry picked from commit 6d17829aa664e7cf6a92291963ff622c8b763b53)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron queens-eol release.