ovs-agent fails to set up OVS flows for security groups

Bug #1647593 reported by Ilya Shakhat
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Invalid
Undecided
MOS Neutron
9.x
Invalid
Undecided
MOS Neutron

Bug Description

User impact: no connectivity to VM, VM fails to get IP address via DHCP

Steps to reproduce:
1. Install MOS 9.2, with "Open vSwitch Firewall Driver" enabled, no DVR, no L2pop, no L3HA. Reproduced on 1 ctrl + 3 cmp, QEMU-based env.
2. Boot Cirros-based VM in default internal network.
3. Observe VM's console, no IP address is received.

Issue: ovs-agent log on compute node contains the following error:
2016-12-06 08:23:19.876 28465 ERROR neutron.agent.common.ovs_lib [req-7d98adc9-5b7c-45ab-8882-12f2f58f82c5 - - - - -] Unable to execute ['ovs-ofctl', 'add-flows', 'br-int', '-']. Exception: Exit code: 1;
Stdin: hard_timeout=0,idle_timeout=0,priority=100,table=0,cookie=9483114818310017266,in_port=11,actions=set_field:11->reg5,set_field:9->reg6,resubmit(,71)
hard_timeout=0,idle_timeout=0,priority=90,table=0,dl_dst=fa:16:3e:9c:8a:17,cookie=9483114818310017266,actions=set_field:11->reg5,set_field:9->reg6,resubmit(,81)
hard_timeout=0,idle_timeout=0,priority=95,dl_type=0x86dd,reg5=11,nw_proto=58,cookie=9483114818310017266,table=71,icmp_type=130,in_port=11,actions=normal
hard_timeout=0,idle_timeout=0,priority=95,dl_type=0x86dd,reg5=11,nw_proto=58,cookie=9483114818310017266,table=71,icmp_type=131,in_port=11,actions=normal
hard_timeout=0,idle_timeout=0,priority=95,dl_type=0x86dd,reg5=11,nw_proto=58,cookie=9483114818310017266,table=71,icmp_type=132,in_port=11,actions=normal
hard_timeout=0,idle_timeout=0,priority=95,dl_type=0x86dd,reg5=11,nw_proto=58,cookie=9483114818310017266,table=71,icmp_type=135,in_port=11,actions=normal
hard_timeout=0,idle_timeout=0,priority=95,dl_type=0x86dd,reg5=11,nw_proto=58,cookie=9483114818310017266,table=71,icmp_type=136,in_port=11,actions=normal
hard_timeout=0,idle_timeout=0,priority=95,dl_type=0x0806,reg5=11,dl_src=fa:16:3e:9c:8a:17,arp_spa=192.168.111.7,cookie=9483114818310017266,table=71,in_port=11,actions=normal
hard_timeout=0,idle_timeout=0,priority=65,dl_type=0x0800,ct_state=-trk,reg5=11,dl_src=fa:16:3e:9c:8a:17,cookie=9483114818310017266,table=71,nw_src=192.168.111.7,in_port=11,actions=ct(table=72,zone=NXM_NX_
REG6[0..15])
hard_timeout=0,idle_timeout=0,priority=65,dl_type=0x86dd,ct_state=-trk,reg5=11,dl_src=fa:16:3e:9c:8a:17,cookie=9483114818310017266,ipv6_src=fe80::f816:3eff:fe9c:8a17,table=71,in_port=11,actions=ct(table=7
2,zone=NXM_NX_REG6[0..15])
hard_timeout=0,idle_timeout=0,priority=80,dl_type=0x0800,reg5=11,nw_proto=17,cookie=9483114818310017266,tp_dst=67,table=71,tp_src=68,in_port=11,actions=resubmit(,73)
hard_timeout=0,idle_timeout=0,priority=80,dl_type=0x86dd,reg5=11,nw_proto=17,cookie=9483114818310017266,tp_dst=547,table=71,tp_src=546,in_port=11,actions=resubmit(,73)
hard_timeout=0,idle_timeout=0,priority=70,dl_type=0x0800,reg5=11,nw_proto=17,cookie=9483114818310017266,tp_dst=68,table=71,tp_src=67,in_port=11,actions=drop
hard_timeout=0,idle_timeout=0,priority=70,dl_type=0x86dd,reg5=11,nw_proto=17,cookie=9483114818310017266,tp_dst=546,table=71,tp_src=547,in_port=11,actions=drop
hard_timeout=0,idle_timeout=0,priority=10,ct_state=-trk,reg5=11,cookie=9483114818310017266,table=71,in_port=11,actions=drop
hard_timeout=0,idle_timeout=0,priority=100,table=73,dl_dst=fa:16:3e:9c:8a:17,cookie=9483114818310017266,actions=set_field:11->reg5,resubmit(,81)
hard_timeout=0,idle_timeout=0,priority=90,dl_type=0x0800,ct_state=+new-est,reg5=11,cookie=9483114818310017266,table=73,actions=ct(commit,zone=NXM_NX_REG6[0..15]),normal
hard_timeout=0,idle_timeout=0,priority=90,dl_type=0x86dd,ct_state=+new-est,reg5=11,cookie=9483114818310017266,table=73,actions=ct(commit,zone=NXM_NX_REG6[0..15]),normal
hard_timeout=0,idle_timeout=0,priority=80,table=73,reg5=11,cookie=9483114818310017266,actions=normal
hard_timeout=0,idle_timeout=0,priority=100,dl_type=0x0806,reg5=11,cookie=9483114818310017266,table=81,dl_dst=fa:16:3e:9c:8a:17,actions=strip_vlan,output:11
hard_timeout=0,idle_timeout=0,priority=100,dl_type=0x86dd,reg5=11,nw_proto=58,cookie=9483114818310017266,table=81,dl_dst=fa:16:3e:9c:8a:17,icmp_type=130,actions=strip_vlan,output:11
hard_timeout=0,idle_timeout=0,priority=100,dl_type=0x86dd,reg5=11,nw_proto=58,cookie=9483114818310017266,table=81,dl_dst=fa:16:3e:9c:8a:17,icmp_type=131,actions=strip_vlan,output:11
hard_timeout=0,idle_timeout=0,priority=100,dl_type=0x86dd,reg5=11,nw_proto=58,cookie=9483114818310017266,table=81,dl_dst=fa:16:3e:9c:8a:17,icmp_type=132,actions=strip_vlan,output:11
hard_timeout=0,idle_timeout=0,priority=100,dl_type=0x86dd,reg5=11,nw_proto=58,cookie=9483114818310017266,table=81,dl_dst=fa:16:3e:9c:8a:17,icmp_type=135,actions=strip_vlan,output:11
hard_timeout=0,idle_timeout=0,priority=100,dl_type=0x86dd,reg5=11,nw_proto=58,cookie=9483114818310017266,table=81,dl_dst=fa:16:3e:9c:8a:17,icmp_type=136,actions=strip_vlan,output:11
hard_timeout=0,idle_timeout=0,priority=95,dl_type=0x0800,reg5=11,nw_proto=17,cookie=9483114818310017266,tp_dst=68,table=81,tp_src=67,actions=strip_vlan,output:11
hard_timeout=0,idle_timeout=0,priority=95,dl_type=0x86dd,reg5=11,nw_proto=17,cookie=9483114818310017266,tp_dst=546,table=81,tp_src=547,actions=strip_vlan,output:11
hard_timeout=0,idle_timeout=0,priority=90,dl_type=0x0800,ct_state=-trk,reg5=11,cookie=9483114818310017266,table=81,actions=ct(table=82,zone=NXM_NX_REG6[0..15])
hard_timeout=0,idle_timeout=0,priority=90,dl_type=0x86dd,ct_state=-trk,reg5=11,cookie=9483114818310017266,table=81,actions=ct(table=82,zone=NXM_NX_REG6[0..15])
hard_timeout=0,idle_timeout=0,priority=80,ct_state=+trk,reg5=11,cookie=9483114818310017266,table=81,dl_dst=fa:16:3e:9c:8a:17,actions=resubmit(,82)
hard_timeout=0,idle_timeout=0,priority=50,table=82,cookie=9483114818310017266,ct_state=+trk+inv,actions=drop
hard_timeout=0,idle_timeout=0,priority=50,reg5=11,ct_mark=0x1,cookie=9483114818310017266,table=82,actions=drop
hard_timeout=0,idle_timeout=0,priority=50,cookie=9483114818310017266,ct_state=+est-rel+rpl,reg5=11,ct_mark=0x0,table=82,dl_dst=fa:16:3e:9c:8a:17,ct_zone=9,actions=strip_vlan,output:11
hard_timeout=0,idle_timeout=0,priority=50,cookie=9483114818310017266,ct_state=-new-est+rel-inv,reg5=11,ct_mark=0x0,table=82,dl_dst=fa:16:3e:9c:8a:17,ct_zone=9,actions=strip_vlan,output:11
hard_timeout=0,idle_timeout=0,priority=40,ct_state=-est,reg5=11,cookie=9483114818310017266,table=82,actions=drop
hard_timeout=0,idle_timeout=0,priority=40,dl_type=0x0800,ct_state=+est,reg5=11,cookie=9483114818310017266,table=82,actions=ct(commit,zone=NXM_NX_REG6[0..15],exec(set_field:0x1->ct_mark))
hard_timeout=0,idle_timeout=0,priority=40,dl_type=0x86dd,ct_state=+est,reg5=11,cookie=9483114818310017266,table=82,actions=ct(commit,zone=NXM_NX_REG6[0..15],exec(set_field:0x1->ct_mark))
hard_timeout=0,idle_timeout=0,priority=50,table=72,cookie=9483114818310017266,ct_state=+trk+inv,actions=drop
hard_timeout=0,idle_timeout=0,priority=50,reg5=11,ct_mark=0x1,cookie=9483114818310017266,table=72,actions=drop
hard_timeout=0,idle_timeout=0,priority=50,cookie=9483114818310017266,ct_state=+est-rel+rpl,reg5=11,ct_mark=0x0,table=72,ct_zone=9,actions=normal
hard_timeout=0,idle_timeout=0,priority=50,cookie=9483114818310017266,ct_state=-new-est+rel-inv,reg5=11,ct_mark=0x0,table=72,ct_zone=9,actions=normal
hard_timeout=0,idle_timeout=0,priority=40,ct_state=-est,reg5=11,cookie=9483114818310017266,table=72,actions=drop
hard_timeout=0,idle_timeout=0,priority=40,dl_type=0x0800,ct_state=+est,reg5=11,cookie=9483114818310017266,table=72,actions=ct(commit,zone=NXM_NX_REG6[0..15],exec(set_field:0x1->ct_mark))
hard_timeout=0,idle_timeout=0,priority=40,dl_type=0x86dd,ct_state=+est,reg5=11,cookie=9483114818310017266,table=72,actions=ct(commit,zone=NXM_NX_REG6[0..15],exec(set_field:0x1->ct_mark))
hard_timeout=0,idle_timeout=0,priority=70,dl_type=0x86dd,ct_state=+est-rel-rpl,reg5=11,dl_src=fa:16:3e:9c:8a:17,cookie=9483114818310017266,table=72,actions=resubmit(,73)
hard_timeout=0,idle_timeout=0,priority=70,dl_type=0x86dd,ct_state=+new-est,reg5=11,dl_src=fa:16:3e:9c:8a:17,cookie=9483114818310017266,table=72,actions=resubmit(,73)
hard_timeout=0,idle_timeout=0,priority=70,dl_type=0x0800,ct_state=+est-rel-rpl,reg5=11,dl_src=fa:16:3e:9c:8a:17,cookie=9483114818310017266,table=72,actions=resubmit(,73)
hard_timeout=0,idle_timeout=0,priority=70,dl_type=0x0800,ct_state=+new-est,reg5=11,dl_src=fa:16:3e:9c:8a:17,cookie=9483114818310017266,table=72,actions=resubmit(,73); Stdout: ; Stderr: OFPT_ERROR (xid=0x1
6): OFPBMC_BAD_MASK
NXT_FLOW_MOD (xid=0x16):
(***truncated to 64 bytes from 128***)
00000000 01 04 00 80 00 00 00 16-00 00 23 20 00 00 00 0d |..........# ....|
00000010 83 9a ca ae fd 9c 90 f2-47 00 00 00 00 00 00 41 |........G......A|
00000020 ff ff ff ff ff ff 00 00-00 32 00 00 00 00 00 00 |.........2......|
00000030 00 00 00 02 00 0b 00 00-04 06 fa 16 3e 9c 8a 17 |............>...|

ovs-vswitchd.log:
2016-12-06T08:23:19.873Z|00153|connmgr|INFO|br-int<->unix: sending OFPBMC_BAD_MASK error reply to NXT_FLOW_MOD message
2016-12-06T08:23:19.874Z|00154|connmgr|INFO|br-int<->unix: 8 flow_mods in the last 0 s (8 adds)

existing OVS flows:
root@node-3:~# ovs-ofctl dump-ports br-int
OFPST_PORT reply (xid=0x2): 3 ports
  port 11: rx pkts=11, bytes=1674, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=14, bytes=900, drop=0, errs=0, coll=0
  port LOCAL: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=16, bytes=1084, drop=0, errs=0, coll=0
  port 1: rx pkts=6, bytes=252, drop=?, errs=?, frame=?, over=?, crc=?
           tx pkts=10, bytes=832, drop=?, errs=?, coll=?
root@node-3:~#
root@node-3:~# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
 cookie=0x839acaaefd9c90f2, duration=203.974s, table=0, n_packets=11, n_bytes=1674, idle_age=62, priority=100,in_port=11 actions=load:0xb->NXM_NX_REG5[],load:0x9->NXM_NX_REG6[],resubmit(,71)
 cookie=0x839acaaefd9c90f2, duration=203.973s, table=0, n_packets=0, n_bytes=0, idle_age=203, priority=90,dl_dst=fa:16:3e:9c:8a:17 actions=load:0xb->NXM_NX_REG5[],load:0x9->NXM_NX_REG6[],resubmit(,81)
 cookie=0x839acaaefd9c90f2, duration=42497.418s, table=0, n_packets=8, n_bytes=428, idle_age=56, priority=0 actions=NORMAL
 cookie=0x839acaaefd9c90f2, duration=42497.378s, table=23, n_packets=0, n_bytes=0, idle_age=42497, priority=0 actions=drop
 cookie=0x839acaaefd9c90f2, duration=42497.322s, table=24, n_packets=0, n_bytes=0, idle_age=42497, priority=0 actions=drop
 cookie=0x839acaaefd9c90f2, duration=203.973s, table=71, n_packets=0, n_bytes=0, idle_age=203, priority=95,icmp6,reg5=0xb,in_port=11,icmp_type=130 actions=NORMAL
 cookie=0x839acaaefd9c90f2, duration=203.973s, table=71, n_packets=0, n_bytes=0, idle_age=203, priority=95,icmp6,reg5=0xb,in_port=11,icmp_type=131 actions=NORMAL
 cookie=0x839acaaefd9c90f2, duration=203.972s, table=71, n_packets=0, n_bytes=0, idle_age=203, priority=95,icmp6,reg5=0xb,in_port=11,icmp_type=132 actions=NORMAL
 cookie=0x839acaaefd9c90f2, duration=203.972s, table=71, n_packets=1, n_bytes=78, idle_age=181, priority=95,icmp6,reg5=0xb,in_port=11,icmp_type=135 actions=NORMAL
 cookie=0x839acaaefd9c90f2, duration=203.971s, table=71, n_packets=0, n_bytes=0, idle_age=203, priority=95,icmp6,reg5=0xb,in_port=11,icmp_type=136 actions=NORMAL
 cookie=0x839acaaefd9c90f2, duration=203.971s, table=71, n_packets=0, n_bytes=0, idle_age=203, priority=95,arp,reg5=0xb,in_port=11,dl_src=fa:16:3e:9c:8a:17,arp_spa=192.168.111.7 actions=NORMAL
 cookie=0x839acaaefd9c90f2, duration=42496.350s, table=71, n_packets=54, n_bytes=11433, idle_age=62, priority=0 actions=drop
 cookie=0x839acaaefd9c90f2, duration=42496.320s, table=72, n_packets=0, n_bytes=0, idle_age=42496, priority=0 actions=drop
 cookie=0x839acaaefd9c90f2, duration=42496.269s, table=73, n_packets=0, n_bytes=0, idle_age=42496, priority=0 actions=drop
 cookie=0x839acaaefd9c90f2, duration=42496.236s, table=81, n_packets=0, n_bytes=0, idle_age=42496, priority=0 actions=drop
 cookie=0x839acaaefd9c90f2, duration=42496.206s, table=82, n_packets=0, n_bytes=0, idle_age=42496, priority=0 actions=drop

Tags: area-neutron
Ilya Shakhat (shakhat)
tags: added: area-neutron
Revision history for this message
Ilya Shakhat (shakhat) wrote :

It appears that the command fails on setting up flow with conntrack dependency (ovs-vswitchd applied 8 flows and failed on #9 which has ct_state=-trk).
The reason is in old kernel:
root@node-3:~# uname -a
Linux node-3.domain.tld 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
(see https://mail.openvswitch.org/pipermail/ovs-discuss/2016-June/041483.html)

We should rather always install new kernel, or prohibit selection of OVS-based security groups in UI

Revision history for this message
Alexander Ignatov (aignatov) wrote :

ovs firewall driver doesn't work with kernels < 4.3, moving this bug to invalid

Revision history for this message
Jun Park (jun-park-earth) wrote :

well, I am experiencing the same problem even with kernel = 4.4 with MOS9.2

# uname -a
Linux node-1 4.4.0-62-generic #83~14.04.1-Ubuntu SMP Wed Jan 18 18:10:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

<163>Feb 10 23:22:17 node-1 neutron-openvswitch-agent: 2017-02-10 23:22:17.494 75429 ERROR neutron.agent.common.ovs_li
b [req-bbd054b2-32ec-4c99-b701-3dab9005018d - - - - -] Unable to execute ['ovs-ofctl', 'add-flows', 'br-int', '-']. Except
ion: Exit code: 1; Stdin: hard_timeout=0,idle_timeout=0,priority=100,table=0,cookie=11020666099743680420,in_port=8,actions
=set_field:8->reg5,set_field:2->reg6,resubmit(,71)

We build out this new env with MOS9.2, by choosing OVS for firewalls, DVR on & L2pop on, with ml2.

Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

What is your OVS version?

Revision history for this message
Jun Park (jun-park-earth) wrote :

# ovs-vsctl --version
ovs-vsctl (Open vSwitch) 2.4.1
Compiled Jun 2 2016 15:21:16
DB Schema 7.12.1

Revision history for this message
Inessa Vasilevskaya (ivasilevskaya) wrote :

@jun-park-earth, the ovsfw feature requires ovs 2.5. You should upgrade ovs packages (openvswitch-common and openvswitch-switch) to version 2.5

Revision history for this message
Dmitry Sutyagin (dsutyagin) wrote :

Our mirror contains both 2.4.0 and 2.6.1, however only 2.4.0 is included in mos9.0 repo (http://mirror.fuel-infra.org/mos-repos/ubuntu/9.2/dists/mos9.0/main/binary-amd64/Packages)
Therefore this looks like a bug on Fuel/MOS side since the packages required by this feature should have been installed automatically during deployment. Moving back to "Confirmed".

Revision history for this message
Dmitry Sutyagin (dsutyagin) wrote :

Please ignore my prev. comment. openvswitch 2.6.1 is in mos9.0-updates. This repo should be enabled in Fuel UI (or a local mirror of it). Looks like this repo was not used/available during deployment. I will contact Jun to further diagnose the issue.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.