ovs-agent fails to set up OVS flows for security groups

[Ilya Shakhat]

User impact: no connectivity to VM, VM fails to get IP address via DHCP

Steps to reproduce:
1. Install MOS 9.2, with "Open vSwitch Firewall Driver" enabled, no DVR, no L2pop, no L3HA. Reproduced on 1 ctrl + 3 cmp, QEMU-based env.
2. Boot Cirros-based VM in default internal network.
3. Observe VM's console, no IP address is received.

Issue: ovs-agent log on compute node contains the following error:
2016-12-06 08:23:19.876 28465 ERROR neutron.agent.common.ovs_lib [req-7d98adc9-5b7c-45ab-8882-12f2f58f82c5 - - - - -] Unable to execute ['ovs-ofctl', 'add-flows', 'br-int', '-']. Exception: Exit code: 1;
Stdin: hard_timeout=0,idle_timeout=0,priority=100,table=0,cookie=9483114818310017266,in_port=11,actions=set_field:11->reg5,set_field:9->reg6,resubmit(,71)
hard_timeout=0,idle_timeout=0,priority=90,table=0,dl_dst=fa:16:3e:9c:8a:17,cookie=9483114818310017266,actions=set_field:11->reg5,set_field:9->reg6,resubmit(,81)
hard_timeout=0,idle_timeout=0,priority=95,dl_type=0x86dd,reg5=11,nw_proto=58,cookie=9483114818310017266,table=71,icmp_type=130,in_port=11,actions=normal
hard_timeout=0,idle_timeout=0,priority=95,dl_type=0x86dd,reg5=11,nw_proto=58,cookie=9483114818310017266,table=71,icmp_type=131,in_port=11,actions=normal
hard_timeout=0,idle_timeout=0,priority=95,dl_type=0x86dd,reg5=11,nw_proto=58,cookie=9483114818310017266,table=71,icmp_type=132,in_port=11,actions=normal
hard_timeout=0,idle_timeout=0,priority=95,dl_type=0x86dd,reg5=11,nw_proto=58,cookie=9483114818310017266,table=71,icmp_type=135,in_port=11,actions=normal
hard_timeout=0,idle_timeout=0,priority=95,dl_type=0x86dd,reg5=11,nw_proto=58,cookie=9483114818310017266,table=71,icmp_type=136,in_port=11,actions=normal
hard_timeout=0,idle_timeout=0,priority=95,dl_type=0x0806,reg5=11,dl_src=fa:16:3e:9c:8a:17,arp_spa=192.168.111.7,cookie=9483114818310017266,table=71,in_port=11,actions=normal
hard_timeout=0,idle_timeout=0,priority=65,dl_type=0x0800,ct_state=-trk,reg5=11,dl_src=fa:16:3e:9c:8a:17,cookie=9483114818310017266,table=71,nw_src=192.168.111.7,in_port=11,actions=ct(table=72,zone=NXM_NX_
REG6[0..15])
hard_timeout=0,idle_timeout=0,priority=65,dl_type=0x86dd,ct_state=-trk,reg5=11,dl_src=fa:16:3e:9c:8a:17,cookie=9483114818310017266,ipv6_src=fe80::f816:3eff:fe9c:8a17,table=71,in_port=11,actions=ct(table=7
2,zone=NXM_NX_REG6[0..15])
hard_timeout=0,idle_timeout=0,priority=80,dl_type=0x0800,reg5=11,nw_proto=17,cookie=9483114818310017266,tp_dst=67,table=71,tp_src=68,in_port=11,actions=resubmit(,73)
hard_timeout=0,idle_timeout=0,priority=80,dl_type=0x86dd,reg5=11,nw_proto=17,cookie=9483114818310017266,tp_dst=547,table=71,tp_src=546,in_port=11,actions=resubmit(,73)
hard_timeout=0,idle_timeout=0,priority=70,dl_type=0x0800,reg5=11,nw_proto=17,cookie=9483114818310017266,tp_dst=68,table=71,tp_src=67,in_port=11,actions=drop
hard_timeout=0,idle_timeout=0,priority=70,dl_type=0x86dd,reg5=11,nw_proto=17,cookie=9483114818310017266,tp_dst=546,table=71,tp_src=547,in_port=11,actions=drop
hard_timeout=0,idle_timeout=0,priority=10,ct_state=-trk,reg5=11,cookie=9483114818310017266,table=71,in_port=11,actions=drop
hard_timeout=0,idle_timeout=0,priority=100,table=73,dl_dst=fa:16:3e:9c:8a:17,cookie=9483114818310017266,actions=set_field:11->reg5,resubmit(,81)
hard_timeout=0,idle_timeout=0,priority=90,dl_type=0x0800,ct_state=+new-est,reg5=11,cookie=9483114818310017266,table=73,actions=ct(commit,zone=NXM_NX_REG6[0..15]),normal
hard_timeout=0,idle_timeout=0,priority=90,dl_type=0x86dd,ct_state=+new-est,reg5=11,cookie=9483114818310017266,table=73,actions=ct(commit,zone=NXM_NX_REG6[0..15]),normal
hard_timeout=0,idle_timeout=0,priority=80,table=73,reg5=11,cookie=9483114818310017266,actions=normal
hard_timeout=0,idle_timeout=0,priority=100,dl_type=0x0806,reg5=11,cookie=9483114818310017266,table=81,dl_dst=fa:16:3e:9c:8a:17,actions=strip_vlan,output:11
hard_timeout=0,idle_timeout=0,priority=100,dl_type=0x86dd,reg5=11,nw_proto=58,cookie=9483114818310017266,table=81,dl_dst=fa:16:3e:9c:8a:17,icmp_type=130,actions=strip_vlan,output:11
hard_timeout=0,idle_timeout=0,priority=100,dl_type=0x86dd,reg5=11,nw_proto=58,cookie=9483114818310017266,table=81,dl_dst=fa:16:3e:9c:8a:17,icmp_type=131,actions=strip_vlan,output:11
hard_timeout=0,idle_timeout=0,priority=100,dl_type=0x86dd,reg5=11,nw_proto=58,cookie=9483114818310017266,table=81,dl_dst=fa:16:3e:9c:8a:17,icmp_type=132,actions=strip_vlan,output:11
hard_timeout=0,idle_timeout=0,priority=100,dl_type=0x86dd,reg5=11,nw_proto=58,cookie=9483114818310017266,table=81,dl_dst=fa:16:3e:9c:8a:17,icmp_type=135,actions=strip_vlan,output:11
hard_timeout=0,idle_timeout=0,priority=100,dl_type=0x86dd,reg5=11,nw_proto=58,cookie=9483114818310017266,table=81,dl_dst=fa:16:3e:9c:8a:17,icmp_type=136,actions=strip_vlan,output:11
hard_timeout=0,idle_timeout=0,priority=95,dl_type=0x0800,reg5=11,nw_proto=17,cookie=9483114818310017266,tp_dst=68,table=81,tp_src=67,actions=strip_vlan,output:11
hard_timeout=0,idle_timeout=0,priority=95,dl_type=0x86dd,reg5=11,nw_proto=17,cookie=9483114818310017266,tp_dst=546,table=81,tp_src=547,actions=strip_vlan,output:11
hard_timeout=0,idle_timeout=0,priority=90,dl_type=0x0800,ct_state=-trk,reg5=11,cookie=9483114818310017266,table=81,actions=ct(table=82,zone=NXM_NX_REG6[0..15])
hard_timeout=0,idle_timeout=0,priority=90,dl_type=0x86dd,ct_state=-trk,reg5=11,cookie=9483114818310017266,table=81,actions=ct(table=82,zone=NXM_NX_REG6[0..15])
hard_timeout=0,idle_timeout=0,priority=80,ct_state=+trk,reg5=11,cookie=9483114818310017266,table=81,dl_dst=fa:16:3e:9c:8a:17,actions=resubmit(,82)
hard_timeout=0,idle_timeout=0,priority=50,table=82,cookie=9483114818310017266,ct_state=+trk+inv,actions=drop
hard_timeout=0,idle_timeout=0,priority=50,reg5=11,ct_mark=0x1,cookie=9483114818310017266,table=82,actions=drop
hard_timeout=0,idle_timeout=0,priority=50,cookie=9483114818310017266,ct_state=+est-rel+rpl,reg5=11,ct_mark=0x0,table=82,dl_dst=fa:16:3e:9c:8a:17,ct_zone=9,actions=strip_vlan,output:11
hard_timeout=0,idle_timeout=0,priority=50,cookie=9483114818310017266,ct_state=-new-est+rel-inv,reg5=11,ct_mark=0x0,table=82,dl_dst=fa:16:3e:9c:8a:17,ct_zone=9,actions=strip_vlan,output:11
hard_timeout=0,idle_timeout=0,priority=40,ct_state=-est,reg5=11,cookie=9483114818310017266,table=82,actions=drop
hard_timeout=0,idle_timeout=0,priority=40,dl_type=0x0800,ct_state=+est,reg5=11,cookie=9483114818310017266,table=82,actions=ct(commit,zone=NXM_NX_REG6[0..15],exec(set_field:0x1->ct_mark))
hard_timeout=0,idle_timeout=0,priority=40,dl_type=0x86dd,ct_state=+est,reg5=11,cookie=9483114818310017266,table=82,actions=ct(commit,zone=NXM_NX_REG6[0..15],exec(set_field:0x1->ct_mark))
hard_timeout=0,idle_timeout=0,priority=50,table=72,cookie=9483114818310017266,ct_state=+trk+inv,actions=drop
hard_timeout=0,idle_timeout=0,priority=50,reg5=11,ct_mark=0x1,cookie=9483114818310017266,table=72,actions=drop
hard_timeout=0,idle_timeout=0,priority=50,cookie=9483114818310017266,ct_state=+est-rel+rpl,reg5=11,ct_mark=0x0,table=72,ct_zone=9,actions=normal
hard_timeout=0,idle_timeout=0,priority=50,cookie=9483114818310017266,ct_state=-new-est+rel-inv,reg5=11,ct_mark=0x0,table=72,ct_zone=9,actions=normal
hard_timeout=0,idle_timeout=0,priority=40,ct_state=-est,reg5=11,cookie=9483114818310017266,table=72,actions=drop
hard_timeout=0,idle_timeout=0,priority=40,dl_type=0x0800,ct_state=+est,reg5=11,cookie=9483114818310017266,table=72,actions=ct(commit,zone=NXM_NX_REG6[0..15],exec(set_field:0x1->ct_mark))
hard_timeout=0,idle_timeout=0,priority=40,dl_type=0x86dd,ct_state=+est,reg5=11,cookie=9483114818310017266,table=72,actions=ct(commit,zone=NXM_NX_REG6[0..15],exec(set_field:0x1->ct_mark))
hard_timeout=0,idle_timeout=0,priority=70,dl_type=0x86dd,ct_state=+est-rel-rpl,reg5=11,dl_src=fa:16:3e:9c:8a:17,cookie=9483114818310017266,table=72,actions=resubmit(,73)
hard_timeout=0,idle_timeout=0,priority=70,dl_type=0x86dd,ct_state=+new-est,reg5=11,dl_src=fa:16:3e:9c:8a:17,cookie=9483114818310017266,table=72,actions=resubmit(,73)
hard_timeout=0,idle_timeout=0,priority=70,dl_type=0x0800,ct_state=+est-rel-rpl,reg5=11,dl_src=fa:16:3e:9c:8a:17,cookie=9483114818310017266,table=72,actions=resubmit(,73)
hard_timeout=0,idle_timeout=0,priority=70,dl_type=0x0800,ct_state=+new-est,reg5=11,dl_src=fa:16:3e:9c:8a:17,cookie=9483114818310017266,table=72,actions=resubmit(,73); Stdout: ; Stderr: OFPT_ERROR (xid=0x1
6): OFPBMC_BAD_MASK
NXT_FLOW_MOD (xid=0x16):
(***truncated to 64 bytes from 128***)
00000000 01 04 00 80 00 00 00 16-00 00 23 20 00 00 00 0d |..........# ....|
00000010 83 9a ca ae fd 9c 90 f2-47 00 00 00 00 00 00 41 |........G......A|
00000020 ff ff ff ff ff ff 00 00-00 32 00 00 00 00 00 00 |.........2......|
00000030 00 00 00 02 00 0b 00 00-04 06 fa 16 3e 9c 8a 17 |............>...|

ovs-vswitchd.log:
2016-12-06T08:23:19.873Z|00153|connmgr|INFO|br-int<->unix: sending OFPBMC_BAD_MASK error reply to NXT_FLOW_MOD message
2016-12-06T08:23:19.874Z|00154|connmgr|INFO|br-int<->unix: 8 flow_mods in the last 0 s (8 adds)

existing OVS flows:
root@node-3:~# ovs-ofctl dump-ports br-int
OFPST_PORT reply (xid=0x2): 3 ports
  port 11: rx pkts=11, bytes=1674, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=14, bytes=900, drop=0, errs=0, coll=0
  port LOCAL: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=16, bytes=1084, drop=0, errs=0, coll=0
  port 1: rx pkts=6, bytes=252, drop=?, errs=?, frame=?, over=?, crc=?
           tx pkts=10, bytes=832, drop=?, errs=?, coll=?
root@node-3:~#
root@node-3:~# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
 cookie=0x839acaaefd9c90f2, duration=203.974s, table=0, n_packets=11, n_bytes=1674, idle_age=62, priority=100,in_port=11 actions=load:0xb->NXM_NX_REG5[],load:0x9->NXM_NX_REG6[],resubmit(,71)
 cookie=0x839acaaefd9c90f2, duration=203.973s, table=0, n_packets=0, n_bytes=0, idle_age=203, priority=90,dl_dst=fa:16:3e:9c:8a:17 actions=load:0xb->NXM_NX_REG5[],load:0x9->NXM_NX_REG6[],resubmit(,81)
 cookie=0x839acaaefd9c90f2, duration=42497.418s, table=0, n_packets=8, n_bytes=428, idle_age=56, priority=0 actions=NORMAL
 cookie=0x839acaaefd9c90f2, duration=42497.378s, table=23, n_packets=0, n_bytes=0, idle_age=42497, priority=0 actions=drop
 cookie=0x839acaaefd9c90f2, duration=42497.322s, table=24, n_packets=0, n_bytes=0, idle_age=42497, priority=0 actions=drop
 cookie=0x839acaaefd9c90f2, duration=203.973s, table=71, n_packets=0, n_bytes=0, idle_age=203, priority=95,icmp6,reg5=0xb,in_port=11,icmp_type=130 actions=NORMAL
 cookie=0x839acaaefd9c90f2, duration=203.973s, table=71, n_packets=0, n_bytes=0, idle_age=203, priority=95,icmp6,reg5=0xb,in_port=11,icmp_type=131 actions=NORMAL
 cookie=0x839acaaefd9c90f2, duration=203.972s, table=71, n_packets=0, n_bytes=0, idle_age=203, priority=95,icmp6,reg5=0xb,in_port=11,icmp_type=132 actions=NORMAL
 cookie=0x839acaaefd9c90f2, duration=203.972s, table=71, n_packets=1, n_bytes=78, idle_age=181, priority=95,icmp6,reg5=0xb,in_port=11,icmp_type=135 actions=NORMAL
 cookie=0x839acaaefd9c90f2, duration=203.971s, table=71, n_packets=0, n_bytes=0, idle_age=203, priority=95,icmp6,reg5=0xb,in_port=11,icmp_type=136 actions=NORMAL
 cookie=0x839acaaefd9c90f2, duration=203.971s, table=71, n_packets=0, n_bytes=0, idle_age=203, priority=95,arp,reg5=0xb,in_port=11,dl_src=fa:16:3e:9c:8a:17,arp_spa=192.168.111.7 actions=NORMAL
 cookie=0x839acaaefd9c90f2, duration=42496.350s, table=71, n_packets=54, n_bytes=11433, idle_age=62, priority=0 actions=drop
 cookie=0x839acaaefd9c90f2, duration=42496.320s, table=72, n_packets=0, n_bytes=0, idle_age=42496, priority=0 actions=drop
 cookie=0x839acaaefd9c90f2, duration=42496.269s, table=73, n_packets=0, n_bytes=0, idle_age=42496, priority=0 actions=drop
 cookie=0x839acaaefd9c90f2, duration=42496.236s, table=81, n_packets=0, n_bytes=0, idle_age=42496, priority=0 actions=drop
 cookie=0x839acaaefd9c90f2, duration=42496.206s, table=82, n_packets=0, n_bytes=0, idle_age=42496, priority=0 actions=drop

[Ilya Shakhat (shakhat)]

It appears that the command fails on setting up flow with conntrack dependency (ovs-vswitchd applied 8 flows and failed on #9 which has ct_state=-trk).
The reason is in old kernel:
root@node-3:~# uname -a
Linux node-3.domain.tld 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
(see https://mail.openvswitch.org/pipermail/ovs-discuss/2016-June/041483.html)

We should rather always install new kernel, or prohibit selection of OVS-based security groups in UI

[Alexander Ignatov (aignatov)]

ovs firewall driver doesn't work with kernels < 4.3, moving this bug to invalid

[Jun Park (jun-park-earth)]

well, I am experiencing the same problem even with kernel = 4.4 with MOS9.2

# uname -a
Linux node-1 4.4.0-62-generic #83~14.04.1-Ubuntu SMP Wed Jan 18 18:10:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

<163>Feb 10 23:22:17 node-1 neutron-openvswitch-agent: 2017-02-10 23:22:17.494 75429 ERROR neutron.agent.common.ovs_li
b [req-bbd054b2-32ec-4c99-b701-3dab9005018d - - - - -] Unable to execute ['ovs-ofctl', 'add-flows', 'br-int', '-']. Except
ion: Exit code: 1; Stdin: hard_timeout=0,idle_timeout=0,priority=100,table=0,cookie=11020666099743680420,in_port=8,actions
=set_field:8->reg5,set_field:2->reg6,resubmit(,71)

We build out this new env with MOS9.2, by choosing OVS for firewalls, DVR on & L2pop on, with ml2.

[Eugene Nikanorov (enikanorov)]

What is your OVS version?

[Jun Park (jun-park-earth)]

# ovs-vsctl --version
ovs-vsctl (Open vSwitch) 2.4.1
Compiled Jun 2 2016 15:21:16
DB Schema 7.12.1

[Inessa Vasilevskaya (ivasilevskaya)]

@jun-park-earth, the ovsfw feature requires ovs 2.5. You should upgrade ovs packages (openvswitch-common and openvswitch-switch) to version 2.5

[Dmitry Sutyagin (dsutyagin)]

Our mirror contains both 2.4.0 and 2.6.1, however only 2.4.0 is included in mos9.0 repo (http://mirror.fuel-infra.org/mos-repos/ubuntu/9.2/dists/mos9.0/main/binary-amd64/Packages)
Therefore this looks like a bug on Fuel/MOS side since the packages required by this feature should have been installed automatically during deployment. Moving back to "Confirmed".

[Dmitry Sutyagin (dsutyagin)]

Please ignore my prev. comment. openvswitch 2.6.1 is in mos9.0-updates. This repo should be enabled in Fuel UI (or a local mirror of it). Looks like this repo was not used/available during deployment. I will contact Jun to further diagnose the issue.