ci-overcloud job failing "Error while processing VIF ports"

[Derek Higgins]

ci overcloud jobs started failing between 5 and 8 AM GMT

Error from http://logs.openstack.org/73/79873/5/check-tripleo/check-tripleo-overcloud-precise/859d4d4/

var/log/upstart/neutron-openvswitch-agent.log ( on contoller and 1 compute)

[-] Error while processing VIF ports
Traceback (most recent call last):
  File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1230, in rpc_loop
    sync = self.process_network_ports(port_info)
  File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1084, in process_network_ports
    devices_added_updated)
  File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 984, in treat_devices_added_or_updated
    details['admin_state_up'])
  File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 893, in treat_vif_port
    physical_network, segmentation_id)
  File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 593, in port_bound
    physical_network, segmentation_id)
  File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 459, in provision_local_vlan
    (segmentation_id, ofports))
  File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/agent/linux/ovs_lib.py", line 190, in mod_flow
    flow_str = _build_flow_expr_str(kwargs, 'mod')
  File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/agent/linux/ovs_lib.py", line 546, in _build_flow_expr_str
    raise exceptions.InvalidInput(error_message=msg)
InvalidInput: Invalid input for operation: Cannot match priority on flow deletion or modification.
2014-03-21 05:20:56.329 7601 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent

merge times and traceback details seem to match up with
https://review.openstack.org/#/c/58533/19

currently I'm testing a revert to see if it fixes things

[Kyle Mestery (mestery)]

As near as I can tell, this Neutron commit [1] modified the flow building behavior to reject modifying flows with a priority set [2]. However, there was no UT added to detect this case. Further, the agent code is clearly modifying flows with a priority set [3]. Thus, this error is being triggered.

I have a patch which removes priority from flow modification in the agent and all unit tests, and also adds a negative UT case to make sure we catch this in the future.

[1] https://review.openstack.org/#/c/58533/
[2] https://github.com/openstack/neutron/blob/master/neutron/agent/linux/ovs_lib.py#L544
[3] https://github.com/openstack/neutron/blob/master/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py#L373

[Derek Higgins (derekh)]

Also just to confirm, the commit mentioned does seem to have caused the problem the ci-overcloud job passes with it reverted
https://review.openstack.org/#/c/82098/1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/82141

[Kyle Mestery (mestery)]

The patch I posted has made it past a jenkins run on triple-o:

https://jenkins07.openstack.org/job/check-tripleo-overcloud-precise/247/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/82141
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=252a83890abfb825497f308147aeafc6ee1e0731
Submitter: Jenkins
Branch: master

commit 252a83890abfb825497f308147aeafc6ee1e0731
Author: Kyle Mestery <email address hidden>
Date: Fri Mar 21 09:41:31 2014 +0000

    Don't set priority when calling mod_flow

    This commit [1] changed the behavior of modifying flows to not allow priority
    to be set on flow modification. The agent code which was calling mod_flow was
    not updated to respect this behavior. The unit tests were also not updated in
    this regard. I've corrected this behavior now, and added a negative UT case to
    catch this in the future.

    [1] https://review.openstack.org/#/c/58533/

    Closes-Bug: #1295703

    Change-Id: I20737637cc84567a5ec19efaad18e7f334007f18