[Neutron] Duplicate tags lead to openvswitch-agent failure

[Alexander Rubtsov]

MOS: 9.2 (build 604)

The issue is observed when several OVS ports have the same tag.
For example:
_uuid : fd288042-c080-4204-b098-378fefed5e4d
bond_active_slave : []
bond_downdelay : 0
bond_fake_iface : false
bond_mode : []
bond_updelay : 0
external_ids : {}
fake_bridge : false
interfaces : [aff46edd-aff1-42d9-b3e8-d4821dd9231f]
lacp : []
mac : []
name : "vhuefaf8b3f-ef"
other_config : {net_uuid="22a06ed0-f7d8-48f2-b789-3d5f0d6c7e07", network_type=vlan, physical_network=default, segmentation_id="78", tag="2"}
qos : []
rstp_statistics : {}
rstp_status : {}
statistics : {}
status : {}
tag : 2
trunks : []
vlan_mode : []

_uuid : 2e3194b5-a207-409a-af5f-a0c7dd73480b
bond_active_slave : []
bond_downdelay : 0
bond_fake_iface : false
bond_mode : []
bond_updelay : 0
external_ids : {}
fake_bridge : false
interfaces : [002dfdbe-c785-4632-9352-9a62179b33cd]
lacp : []
mac : []
name : "vhu4159bd5d-38"
other_config : {net_uuid="81ae691a-ebf7-44eb-9151-43b01f385bdf", network_type=vlan, physical_network=default, segmentation_id="148", tag="2"}
qos : []
rstp_statistics : {}
rstp_status : {}
statistics : {}
status : {}
tag : 2
trunks : []
vlan_mode : []

In this case, openvswitch-agent is unable to start and throws the following traceback:
2018-03-15T04:17:03.106379+01:00 compute-0-2.domain.tld neutron-openvswitch-agent[25638]: 2018-03-15 04:17:03.105 25638 CRITICAL neutron [req-2dfb8bf8-57e7-46b4-8a2a-cced2462f63a - - - - -] KeyError: 2
2018-03-15 04:17:03.105 25638 ERROR neutron Traceback (most recent call last):
2018-03-15 04:17:03.105 25638 ERROR neutron File "/usr/bin/neutron-openvswitch-agent", line 10, in <module>
2018-03-15 04:17:03.105 25638 ERROR neutron sys.exit(main())
2018-03-15 04:17:03.105 25638 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main
2018-03-15 04:17:03.105 25638 ERROR neutron agent_main.main()
2018-03-15 04:17:03.105 25638 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 49, in main
2018-03-15 04:17:03.105 25638 ERROR neutron mod.main()
2018-03-15 04:17:03.105 25638 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/main.py", line 37, in main
2018-03-15 04:17:03.105 25638 ERROR neutron ovs_neutron_agent.main(bridge_classes)
2018-03-15 04:17:03.105 25638 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2421, in main
2018-03-15 04:17:03.105 25638 ERROR neutron agent = OVSNeutronAgent(bridge_classes, cfg.CONF)
2018-03-15 04:17:03.105 25638 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 384, in __init__
2018-03-15 04:17:03.105 25638 ERROR neutron self._restore_local_vlan_map()
2018-03-15 04:17:03.105 25638 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 499, in _restore_local_vlan_map
2018-03-15 04:17:03.105 25638 ERROR neutron self.available_local_vlans.remove(local_vlan)
2018-03-15 04:17:03.105 25638 ERROR neutron KeyError: 2

It might be related to these upstream bugs:
https://bugs.launchpad.net/neutron/+bug/1625305
https://bugs.launchpad.net/neutron/+bug/1526974

I'm not allowed to attach the log files here.
Please contact me directly if you need to review the entire output of "ovs-vsctl list Port" and openvswitch-agent.log

[Alexander Rubtsov (arubtsov)]

sla1 for 9.0-updates

[Denis Meltsaykin (dmeltsaykin)]

Please provide steps to reproduce.

[Sasikiran (sasikiran-vaddi)]

The issue is not observed frequently, I would like to add some more information.

Say for example there is one vm, vm1 which is hosting on compute-A which belongs to network net1 and another vm, vm2 from net2 is hosting on compute-A. If the tap devices associated to these two vm holds same tag value then neutron-openswitch-agent will fall into repeated restart due to failure in below validation[1].

From the above example, when a port which is looped for vm1 from net1 then it pass this validation[2] and removes vlan from available_local_vlans and updates _local_vlan_hints with this network uuid and tag. On the next iternation for the other network net2 of vm2 will pass this validation[2] and executes self.available_local_vlans.remove(local_vlan) which will fail because this vlan/tag is already deleted when vm1 is iterated. So, this has triggered a ERROR if incase of two vms from different networks holds same tag value.

[1]: https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L354-L358
[2]: https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L354-L355