Security group updates fail when port hasn't been initialized yet

[Jakub Libosvar]

It happens that tpi patch ports between trunk bridge and integration bridge don't have internal vlan tag in other_config row in ovsdb. It looks like a race between trunk handler and ovs agent.

Example of failure: http://logs.openstack.org/92/527992/2/check/neutron-tempest-plugin-dvr-multinode-scenario/166eee3/logs/subnode-2/screen-q-agt.txt.gz#_Dec_14_18_31_25_801432

Trace example:
Dec 14 18:31:25.801432 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-412981e2-ac8a-4fe8-8ec4-288bbb63e2a7 None None] Error while processing VIF ports: OVSFWTagNotFound: Cannot get tag for port tpi-6457d45d-b6 from its other_config: {}
Dec 14 18:31:25.801580 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
Dec 14 18:31:25.801708 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2080, in rpc_loop
Dec 14 18:31:25.801838 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, ovs_restarted)
Dec 14 18:31:25.801965 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 157, in wrapper
Dec 14 18:31:25.802089 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs)
Dec 14 18:31:25.802214 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1676, in process_network_ports
Dec 14 18:31:25.802345 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info.get('updated', set()))
Dec 14 18:31:25.802476 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 256, in setup_port_filters
Dec 14 18:31:25.802600 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.refresh_firewall(updated_devices)
Dec 14 18:31:25.802725 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 110, in decorated_function
Dec 14 18:31:25.802850 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent *args, **kwargs)
Dec 14 18:31:25.802983 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 209, in refresh_firewall
Dec 14 18:31:25.803103 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._apply_port_filter(device_ids, update_filter=True)
Dec 14 18:31:25.803237 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 141, in _apply_port_filter
Dec 14 18:31:25.803366 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.firewall.update_port_filter(device)
Dec 14 18:31:25.803492 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 509, in update_port_filter
Dec 14 18:31:25.803612 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.prepare_port_filter(port)
Dec 14 18:31:25.803763 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 492, in prepare_port_filter
Dec 14 18:31:25.803894 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port = self.get_or_create_ofport(port)
Dec 14 18:31:25.804021 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 464, in get_or_create_ofport
Dec 14 18:31:25.804152 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_vlan_id = self._get_port_vlan_tag(ovs_port.port_name)
Dec 14 18:31:25.804275 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 448, in _get_port_vlan_tag
Dec 14 18:31:25.804414 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return get_tag_from_other_config(self.int_br.br, port_name)
Dec 14 18:31:25.804548 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/openvswitch_firewall/firewall.py", line 74, in get_tag_from_other_config
Dec 14 18:31:25.804670 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_name=port_name, other_config=other_config)
Dec 14 18:31:25.804794 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent OVSFWTagNotFound: Cannot get tag for port tpi-6457d45d-b6 from its other_config: {}
Dec 14 18:31:25.804923 ubuntu-xenial-rax-ord-0001444835 neutron-openvswitch-agent[17015]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent

[Jakub Libosvar (libosvar)]

After digging more into this bug, I saw it's not related to trunk. The error occurs when update_security_group is called via RPC before port security was initialized by ovs agent. The update is called but ovsdb doesn't have the information about port's network yet.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/531414

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/531414
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ed57c3de4241f94ead2bc7c7761bab99dd61894e
Submitter: Zuul
Branch: master

commit ed57c3de4241f94ead2bc7c7761bab99dd61894e
Author: Jakub Libosvar <email address hidden>
Date: Fri Jan 5 16:04:44 2018 +0100

    ovsfw: Don't create rules if updated port doesn't exist

    The firewall won't attempt on update to initialize port in case
    port hasn't been initialized by sg_agent yet. This fixes a race where update
    rpc call arrives between wiring tap device with integration bridge and
    firewall initialization.

    Change-Id: Ice0667df606ae23061acebceea23ab6e49dadbcf
    Closes-bug: #1740885

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/638372

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/638372
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6d375dcced03c083faa172057fb1966f48a7da7e
Submitter: Zuul
Branch: stable/pike

commit 6d375dcced03c083faa172057fb1966f48a7da7e
Author: Jakub Libosvar <email address hidden>
Date: Fri Jan 5 16:04:44 2018 +0100

    ovsfw: Don't create rules if updated port doesn't exist

    The firewall won't attempt on update to initialize port in case
    port hasn't been initialized by sg_agent yet. This fixes a race where update
    rpc call arrives between wiring tap device with integration bridge and
    firewall initialization.

    Change-Id: Ice0667df606ae23061acebceea23ab6e49dadbcf
    Closes-bug: #1740885
    (cherry picked from commit ed57c3de4241f94ead2bc7c7761bab99dd61894e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.7 release.