Invalid entries in OVS flow table on a 3 node install using GRE Tunnels

[Sergio]

Hi,

I'm running Ubuntu 12.04 and all packages were installed from the repositories and the configuration was created per the following: http://docs.openstack.org/folsom/basic-install/content/basic-install_intro.html. The Controller and Network nodes are in a VM each and the Compute node is installed on the KVM host. I could create VMs but couldn't ping them from the Network node using their ip address or a floating IP address. I researched the and after several hours with wireshark I tracked it down to invalid flow entries on the ovs on the Network node. The MAC addresses in the flows are wrong and don't match the ones displayed by ifconfig or the ones shown in Wireshark. This persists after restarts of the Network, Controller, and Compute nodes. The dynamically created interfaces (tap... etc) have new MACs and the flow tables and OVS database contain bogus entries. I'm including only the relevant information but if you need other information on my configuration, please let me know: The relevant info is:

On the network node:

sergio@OsNetwork:~$ sudo ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=201351.547s, table=0, n_packets=9670, n_bytes=1183368, priority=3,tun_id=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=mod_vlan_vid:1,output:1
 cookie=0x0, duration=201351.577s, table=0, n_packets=23209, n_bytes=1833512, priority=4,in_port=1,dl_vlan=1 actions=set_tunnel:0x1,NORMAL
 cookie=0x0, duration=201351.509s, table=0, n_packets=0, n_bytes=0, priority=3,tun_id=0x1,dl_dst=fa:16:3e:71:73:40 actions=mod_vlan_vid:1,NORMAL
 cookie=0x0, duration=201351.306s, table=0, n_packets=0, n_bytes=0, priority=3,tun_id=0x1,dl_dst=fa:16:3e:0a:e6:73 actions=mod_vlan_vid:1,NORMAL
 cookie=0x0, duration=201352.023s, table=0, n_packets=40526, n_bytes=5471439, priority=1 actions=drop

sergio@OsNetwork:~$ sudo brctl show
bridge name bridge id STP enabled interfaces
br-ex 0000.00163e270a2b no eth3
       qg-4aab25ab-d7
br-int 0000.06a39d5d8c4d no qr-0307f22d-f5
       tap093d67fc-15
br-tun 0000.72e8904f5843 no

sergio@OsNetwork:~$ sudo ifconfig tap093d67fc-15
tap093d67fc-15 Link encap:Ethernet HWaddr ae:5c:7e:86:83:f8
          inet addr:10.5.5.2 Bcast:10.5.5.255 Mask:255.255.255.0
          inet6 addr: fe80::ac5c:7eff:fe86:83f8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:11265 errors:0 dropped:0 overruns:0 frame:0
          TX packets:19455 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1296638 (1.2 MB) TX bytes:1565814 (1.5 MB)

 sergio@OsNetwork:~$ sudo ifconfig qr-0307f22d-f5
qr-0307f22d-f5 Link encap:Ethernet HWaddr ba:72:d2:95:82:69
          inet addr:10.5.5.1 Bcast:10.5.5.255 Mask:255.255.255.0
          inet6 addr: fe80::b872:d2ff:fe95:8269/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:17872 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3762 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1565032 (1.5 MB) TX bytes:219644 (219.6 KB)

Note the MAC addresses referred to in the flows and the MACs on the interfaces are different, now for the ovs database entries:

sergio@OsNetwork:~$ sudo ovs-vsctl list Interface
_uuid : af8746e0-4366-40bf-8c50-09b32cab436b
admin_state : up
cfm_fault : []
cfm_mpid : []
cfm_remote_mpids : []
duplex : []
external_ids : {attached-mac="fa:16:3e:71:73:40", iface-id="093d67fc-15ad-4f01-be08-bb5ee527d2ff", iface-status=active}
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current : []
link_resets : 1
link_speed : []
link_state : up
mac : []
mtu : 1500
name : "tap093d67fc-15"
ofport : 1
options : {}
other_config : {}
statistics : {collisions=0, rx_bytes=1566704, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=19460, tx_bytes=1297710, tx_dropped=0, tx_errors=0, tx_packets=11273}
status : {driver_name=openvswitch, driver_version="", firmware_version=""}
type : internal

_uuid : 1ab4d371-4630-4508-bb1c-2bc28e7e8a6b
admin_state : up
cfm_fault : []
cfm_mpid : []
cfm_remote_mpids : []
duplex : []
external_ids : {attached-mac="fa:16:3e:0a:e6:73", iface-id="0307f22d-f55f-49f0-b70a-c8ffb08ac177", iface-status=active}
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current : []
link_resets : 1
link_speed : []
link_state : up
mac : []
mtu : 1500
name : "qr-0307f22d-f5"
ofport : 2
options : {}
other_config : {}
statistics : {collisions=0, rx_bytes=219730, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=3763, tx_bytes=1565336, tx_dropped=0, tx_errors=0, tx_packets=17876}
status : {driver_name=openvswitch, driver_version="", firmware_version=""}
type : internal

-- snip --

To fix the problem, I issue the following commands and connectivity is immediately established and pings flow back and forth as do arps and all other traffic.
sergio@OsNetwork:~$ sudo ovs-ofctl add-flow br-tun "priority=3 tun_id=0x1 dl_dst=ba:72:d2:95:82:69 actions=mod_vlan_vid:1 normal"
sergio@OsNetwork:~$ sudo ovs-ofctl add-flow br-tun "priority=3 tun_id=0x1 dl_dst=ae:5c:7e:86:83:f8 actions=mod_vlan_vid:1 normal"
sergio@OsNetwork:~$ ovs-ofctl del-flows br-tun "tun_id=0x1 dl_dst=fa:16:3e:71:73:40"
sergio@OsNetwork:~$ ovs-ofctl del-flows br-tun "tun_id=0x1 dl_dst=fa:16:3e:0a:e6:73"

Hopefully this sets you on the path for a fix. If I've filed this but in the wrong place, please let me know and I'll file it elsewhere.

[Mark McClain (markmcclain)]

The report is against Folsom. This should be confirmed against master or Grizzly?

[david (davidgramsay)]

confirmed against grizzly

[Maciej Gałkiewicz (maciej-galkiewicz)]

I am probably affected by this bug us well. In my setup flows are not created at all. If I understand correctly they should have been added to virtual switch during tap interface creation. Is it possible that I misconfigured sth or my openvswitch (1.4.2+git20120612-9 from debian) does not support implemented flows?

[Hua Zhang (zhhuabj)]

this is review link, https://review.openstack.org/35984

[Hua Zhang (zhhuabj)]

can someone help me review the code, many thanks.

[Hua Zhang (zhhuabj)]

hi Sergio, I can't reproduce your phenomenon, I change the vif driver to LibvirtOpenVswitchVirtualPortDriver and deploy one VM,
the mac address of tap is fa:16:3e:34:c7:23, and the flow rule of ovs is the same with this
the mac address of VM is fe:16:3e:34:c7:23

they are different, but the network function is ok