Bug #1626010 “OVS Firewall cannot handle non unique MACs” : Bugs : neutron

Revision history for this message

Bence Romsics (bence-romsics) wrote on 2016-09-21:

#1

Trying to find the openflow rule dropping the test traffic generated by ping:

$ openstack port show port0 | grep -w id # the parent port
| id | a8560361-7af7-4503-b478-4e9b95db2628
$ sudo ovs-ofctl show br-int
...
60(tpi-a8560361-7a): addr:82:46:63:ff:1a:c6
     config: 0
     state: 0
     speed: 0 Mbps now, 0 Mbps max
...

$ sudo ip netns exec "qdhcp-$( openstack network show net0 | awk '/ id / { print $4 }' )" ping 10.0.4.6
# concurrent with the ping
$ sudo ovs-ofctl dump-flows br-int | grep -v n_packets=0 | grep actions=drop | column -t
...
cookie=0xa462c2e0451a9866, duration=8266.222s, table=71, n_packets=2, n_bytes=723, idle_age=8259, priority=70,udp,reg5=0x3c,in_port=60,tp_src=67,tp_dst=68 actions=drop
cookie=0xa462c2e0451a9866, duration=8266.222s, table=71, n_packets=318, n_bytes=28972, idle_age=64, priority=10,ct_state=-trk,reg5=0x3c,in_port=60 actions=drop
cookie=0xa462c2e0451a9866, duration=8266.201s, table=72, n_packets=4, n_bytes=300, idle_age=8249, priority=50,ct_state=+inv+trk actions=drop

Then the only rule with an increasing packet count (n_packets) is this:
priority=10,ct_state=-trk,reg5=0x3c,in_port=60 actions=drop

Revision history for this message

Armando Migliaccio (armando-migliaccio) wrote on 2016-09-21:

#2

Thanks for reporting this; Kuba is in the process of adding more test coverage to confirm gaps.

@Kuba: any chance you can attempt a triage?

Changed in neutron:
assignee:	nobody → Jakub Libosvar (libosvar)
tags:	added: newton-rc-potential

Revision history for this message

Armando Migliaccio (armando-migliaccio) wrote on 2016-09-21:

#3

I would not call this interference, as from your description it seems that firewall and trunk work at least in cases where you don't choose the parent MAC for your subports. This is a new use case for the OVS firewall code and it might be a unplanned limitation in the OVS firewall code.

We need to establish if this is an issue with OVS versus other technologies like Linux Bridge though. How critical is for you using the same MAC for both subports and trunk port?

Armando Migliaccio (armando-migliaccio) on 2016-09-21

tags:

added: ovs sg-fw trunk

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-21: Related fix proposed to neutron (master)

#4

Related fix proposed to branch: master
Review: https://review.openstack.org/374388

Revision history for this message

Kevin Benton (kevinbenton) wrote on 2016-09-22: Re: Connectivity problem on trunk parent with MAC reuse and openvswitch firewall driver

#5

Definitely sounds like a bug in the firewall. Can two VMs have the same mac with OVS firewall on different networks?

Revision history for this message

Bence Romsics (bence-romsics) wrote on 2016-09-22:

#6

First a missing piece from the original bug report that helps to reproduce the bug. For the 'ip netns exec qdhcp-... ping' hack to work the default security group must be edited. I have deleted the default ingress rules having a remote_group_id and re-created them without specifying one. In short I changed the security group rules to ALL ALLOW.

Based on Kevin's idea I managed to reproduce the problem without using trunks at all. So this is not an interference between trunk and the openvswitch firewall driver. But seems like a bug of the openvswitch firewall driver in itself.

I booted multiple VMs, each with a single regular port, all ports having the same MAC. Then ping fails for all except the last one. I tried it first with 2 VMs: The 1st fails and the 2nd succeeds. Then I also tried with 3 VMs: The 1st and 2nd fails and the 3rd succeeds. So it looks as if we had an openflow rule match collision for ports having the same MAC address.

About the original use case leading to the bug report. Reusing the parent's MAC for the child ports is needed:
* if one wants to keep the bring-up logic of VLAN interfaces static (if you reuse the MAC you can have prefab images), and/or
* if one does not have access to the neutron API from inside the instance with the trunk.

The exact commands to reproduce the error without trunks:

openstack network create net0
openstack network create net1
openstack network create net2
openstack subnet create --network net0 --subnet-range 10.0.4.0/24 subnet0
openstack subnet create --network net1 --subnet-range 10.0.5.0/24 subnet1
openstack subnet create --network net2 --subnet-range 10.0.6.0/24 subnet2
openstack port create --network net0 port0
mac="$( openstack port show port0 | awk '/ mac_address / { print $4 }' )"
openstack port create --network net1 --mac-address "$mac" port1
openstack port create --network net2 --mac-address "$mac" port2
openstack server create --flavor cirros256 --image cirros-0.3.4-x86_64-uec --nic port-id=port0 --key-name key0 --wait vm0
openstack server create --flavor cirros256 --image cirros-0.3.4-x86_64-uec --nic port-id=port1 --key-name key0 --wait vm1
openstack server create --flavor cirros256 --image cirros-0.3.4-x86_64-uec --nic port-id=port2 --key-name key0 --wait vm2
openstack server list
# packets lost
sudo ip netns exec "qdhcp-$( openstack network show net0 | awk '/ id / { print $4 }' )" ping VM0-IP
# packets lost
sudo ip netns exec "qdhcp-$( openstack network show net1 | awk '/ id / { print $4 }' )" ping VM1-IP
# works
sudo ip netns exec "qdhcp-$( openstack network show net2 | awk '/ id / { print $4 }' )" ping VM2-IP

First a missing piece from the original bug report that helps to reproduce the bug. For the 'ip netns exec qdhcp-... ping' hack to work the default security group must be edited. I have deleted the default ingress rules having a remote_group_id and re-created them without specifying one. In short I changed the security group rules to ALL ALLOW.

Based on Kevin's idea I managed to reproduce the problem without using trunks at all. So this is not an interference between trunk and the openvswitch firewall driver. But seems like a bug of the openvswitch firewall driver in itself.

I booted multiple VMs, each with a single regular port, all ports having the same MAC. Then ping fails for all except the last one. I tried it first with 2 VMs: The 1st fails and the 2nd succeeds. Then I also tried with 3 VMs: The 1st and 2nd fails and the 3rd succeeds. So it looks as if we had an openflow rule match collision for ports having the same MAC address.

About the original use case leading to the bug report. Reusing the parent's MAC for the child ports is needed:
* if one wants to keep the bring-up logic of VLAN interfaces static (if you reuse the MAC you can have prefab images), and/or
* if one does not have access to the neutron API from inside the instance with the trunk.

The exact commands to reproduce the error without trunks:

openstack network create net0
openstack network create net1
openstack network create net2
openstack subnet create --network net0 --subnet-range 10.0.4.0/24 subnet0
openstack subnet create --network net1 --subnet-range 10.0.5.0/24 subnet1
openstack subnet create --network net2 --subnet-range 10.0.6.0/24 subnet2
openstack port create --network net0 port0
mac="$( openstack port show port0 | awk '/ mac_address / { print $4 }' )"
openstack port create --network net1 --mac-address "$mac" port1
openstack port create --network net2 --mac-address "$mac" port2
openstack server create --flavor cirros256 --image cirros-0.3.4-x86_64-uec --nic port-id=port0 --key-name key0 --wait vm0
openstack server create --flavor cirros256 --image cirros-0.3.4-x86_64-uec --nic port-id=port1 --key-name key0 --wait vm1
openstack server create --flavor cirros256 --image cirros-0.3.4-x86_64-uec --nic port-id=port2 --key-name key0 --wait vm2
openstack server list
# packets lost
sudo ip netns exec "qdhcp-$( openstack network show net0 | awk '/ id / { print $4 }' )" ping VM0-IP
# packets lost
sudo ip netns exec "qdhcp-$( openstack network show net1 | awk '/ id / { print $4 }' )" ping VM1-IP
# works
sudo ip netns exec "qdhcp-$( openstack network show net2 | awk '/ id / { print $4 }' )" ping VM2-IP

Jakub Libosvar (libosvar) on 2016-09-22

Changed in neutron:
importance:	Undecided → High

Revision history for this message

Bence Romsics (bence-romsics) wrote on 2016-09-22:

#7

Download full text (3.5 KiB)

May have found the colliding openflow rule. I was looking for it by creating a good and a known bad state. I was expecting the bad case to have (at least) one less openflow rule, assuming the rules collide we likely overwrite an existing rule instead of creating a new rule.

First booted two VMs, one port each (port0 and port1), with different MACs. Dumped the flows on br-int. (State 1)
Then deleted the 2nd VM, created a 3rd, reusing the MAC of the 1st (port2). Dumped the flows on br-int. (State 2)
Cleaned up the variable fields (n_packets, n_bytes, etc.) in the output of ovs-ofctl dump-flows.
Diffed the flow rules in state 1 and 2. In the diff hunk below, the lines starting with minus correspond to State 1. The plus lines correspond to State 2. The hunk below was the only asymmetric hunk, ie. the only one where the number of rules deleted and added are different. Therefore I think this (priority=90 below) is the colliding openflow rule.

-table=0, priority=100,in_port=76 actions=load:0x4c->NXM_NX_REG5[],load:0xfff->NXM_NX_REG6[],resubmit(,71)
-table=0, priority=90,dl_dst=fa:16:3e:8a:48:37 actions=load:0x4b->NXM_NX_REG5[],load:0xfff->NXM_NX_REG6[],resubmit(,81)
-table=0, priority=90,dl_dst=fa:16:3e:0d:7d:a1 actions=load:0x4c->NXM_NX_REG5[],load:0xfff->NXM_NX_REG6[],resubmit(,81)
+table=0, priority=100,in_port=77 actions=load:0x4d->NXM_NX_REG5[],load:0xfff->NXM_NX_REG6[],resubmit(,71)
+table=0, priority=90,dl_dst=fa:16:3e:8a:48:37 actions=load:0x4d->NXM_NX_REG5[],load:0xfff->NXM_NX_REG6[],resubmit(,81)

port0 fa:16:3e:8a:48:37 c4572c11-1dfb-43c3-8c70-54c92d03658e in_port=75
port1 fa:16:3e:0d:7d:a1 d1c23106-3154-441d-9b82-b6dae2636cba
port2 fa:16:3e:8a:48:37 517e2264-ddf7-4e01-858f-b9b303bc5038 in_port=77

All the details:

openstack network create net1
openstack network create net2
openstack subnet create --network net0 --subnet-range 10.0.4.0/24 subnet0
openstack subnet create --network net1 --subnet-range 10.0.5.0/24 subnet1
openstack subnet create --network net2 --subnet-range 10.0.5.0/24 subnet2
openstack port create --network net0 port0
mac="$( openstack port show port0 | awk '/ mac_address / { print $4 }' )"
openstack port create --network net1 port1
openstack port create --network net2 --mac-address "$mac" port2
openstack port list > openstack-port-list.0
sudo ovs-ofctl dump-flows br-int > ovs-ofctl-dump-flows-br-int.0
sudo ovs-ofctl show br-int > ovs-ofctl-show-br-int.0
openstack server create --flavor cirros256 --image cirros-0.3.4-x86_64-uec --nic port-id=port0 --key-name key0 --wait vm0
sudo ovs-ofctl dump-flows br-int > ovs-ofctl-dump-flows-br-int.1
sudo ovs-ofctl show br-int > ovs-ofctl-show-br-int.1
openstack server create --flavor cirros256 --image cirros-0.3.4-x86_64-uec --nic port-id=port1 --key-name key0 --wait vm1
sudo ovs-ofctl dump-flows br-int > ovs-ofctl-dump-flows-br-int.2
sudo ovs-ofctl show br-int > ovs-ofctl-show-br-int.2
openstack server delete vm1
sudo ovs-ofctl dump-flows br-int > ovs-ofctl-dump-flows-br-int.3
sudo ovs-ofctl show br-int > ovs-ofctl-show-br-int.3
openstack server create --flavor cirros256 --image cirros-0.3.4-x86_64-uec --nic port-id=port2 --key-name key0 --wait vm2
sudo ovs-ofctl dump-flows br-in...

May have found the colliding openflow rule. I was looking for it by creating a good and a known bad state. I was expecting the bad case to have (at least) one less openflow rule, assuming the rules collide we likely overwrite an existing rule instead of creating a new rule.

First booted two VMs, one port each (port0 and port1), with different MACs. Dumped the flows on br-int. (State 1)
Then deleted the 2nd VM, created a 3rd, reusing the MAC of the 1st (port2). Dumped the flows on br-int. (State 2)
Cleaned up the variable fields (n_packets, n_bytes, etc.) in the output of ovs-ofctl dump-flows.
Diffed the flow rules in state 1 and 2. In the diff hunk below, the lines starting with minus correspond to State 1. The plus lines correspond to State 2. The hunk below was the only asymmetric hunk, ie. the only one where the number of rules deleted and added are different. Therefore I think this (priority=90 below) is the colliding openflow rule.

-table=0, priority=100,in_port=76 actions=load:0x4c->NXM_NX_REG5[],load:0xfff->NXM_NX_REG6[],resubmit(,71)
-table=0, priority=90,dl_dst=fa:16:3e:8a:48:37 actions=load:0x4b->NXM_NX_REG5[],load:0xfff->NXM_NX_REG6[],resubmit(,81)
-table=0, priority=90,dl_dst=fa:16:3e:0d:7d:a1 actions=load:0x4c->NXM_NX_REG5[],load:0xfff->NXM_NX_REG6[],resubmit(,81)
+table=0, priority=100,in_port=77 actions=load:0x4d->NXM_NX_REG5[],load:0xfff->NXM_NX_REG6[],resubmit(,71)
+table=0, priority=90,dl_dst=fa:16:3e:8a:48:37 actions=load:0x4d->NXM_NX_REG5[],load:0xfff->NXM_NX_REG6[],resubmit(,81)

port0 fa:16:3e:8a:48:37 c4572c11-1dfb-43c3-8c70-54c92d03658e in_port=75
port1 fa:16:3e:0d:7d:a1 d1c23106-3154-441d-9b82-b6dae2636cba 
port2 fa:16:3e:8a:48:37 517e2264-ddf7-4e01-858f-b9b303bc5038 in_port=77

All the details:

openstack network create net1
openstack network create net2
openstack subnet create --network net0 --subnet-range 10.0.4.0/24 subnet0
openstack subnet create --network net1 --subnet-range 10.0.5.0/24 subnet1
openstack subnet create --network net2 --subnet-range 10.0.5.0/24 subnet2
openstack port create --network net0 port0
mac="$( openstack port show port0 | awk '/ mac_address / { print $4 }' )"
openstack port create --network net1 port1
openstack port create --network net2 --mac-address "$mac" port2
openstack port list > openstack-port-list.0
sudo ovs-ofctl dump-flows br-int > ovs-ofctl-dump-flows-br-int.0
sudo ovs-ofctl show br-int > ovs-ofctl-show-br-int.0
openstack server create --flavor cirros256 --image cirros-0.3.4-x86_64-uec --nic port-id=port0 --key-name key0 --wait vm0
sudo ovs-ofctl dump-flows br-int > ovs-ofctl-dump-flows-br-int.1
sudo ovs-ofctl show br-int > ovs-ofctl-show-br-int.1
openstack server create --flavor cirros256 --image cirros-0.3.4-x86_64-uec --nic port-id=port1 --key-name key0 --wait vm1
sudo ovs-ofctl dump-flows br-int > ovs-ofctl-dump-flows-br-int.2
sudo ovs-ofctl show br-int > ovs-ofctl-show-br-int.2
openstack server delete vm1
sudo ovs-ofctl dump-flows br-int > ovs-ofctl-dump-flows-br-int.3
sudo ovs-ofctl show br-int > ovs-ofctl-show-br-int.3
openstack server create --flavor cirros256 --image cirros-0.3.4-x86_64-uec --nic port-id=port2 --key-name key0 --wait vm2
sudo ovs-ofctl dump-flows br-int > ovs-ofctl-dump-flows-br-int.4
sudo ovs-ofctl show br-int > ovs-ofctl-show-br-int.4

for f in ovs-ofctl-dump-flows-br-int.* ; do cat $f | egrep -v 'NXST_FLOW reply' | sed -r -e 's/cookie=[^ ]+ //' -e 's/duration=[^ ]+ //' -e 's/n_packets=[^ ]+ //' -e 's/n_bytes=[^ ]+ //' -e 's/idle_age=[^ ]+ //' -e 's/^ *//' > diffable-$f ; done

diff -u diffable-ovs-ofctl-dump-flows-br-int.[34]

Revision history for this message

Bence Romsics (bence-romsics) wrote on 2016-09-22:

#8

We seem to install that rule here:

https://github.com/openstack/neutron/blob/60010a8/neutron/agent/linux/openvswitch_firewall/firewall.py#L353

Revision history for this message

Armando Migliaccio (armando-migliaccio) wrote on 2016-09-22:

#9

Bence, thanks a lot for the additional details. In relation to your points:

* if one wants to keep the bring-up logic of VLAN interfaces static (if you reuse the MAC you can have prefab images), and/or
* if one does not have access to the neutron API from inside the instance with the trunk.

Both use cases could be addressed once [1] goes in and we develop nova metadata support; however, one could simply use user-data and inject a setup script at VM boot. You don't need Neutron API access from the guest to achieve this. In fact access should be prevented to avoid exposing a potential attack surface.

[1] https://review.openstack.org/#/c/374388/

Revision history for this message

Bence Romsics (bence-romsics) wrote on 2016-09-23:

#10

Armando, I mostly agree. Once we expose subport details over the metadata API I expect that to be superior to hardcoding them in the image. However until the subport metadata becomes available hardcoding a subport vlan plan in the image could be a stopgap solution.

Embedding a setup script in user data cannot handle *runtime added subports* with arbitrary MACs.

A similar challenge may be part of the metadata support too. I don't see any technological barrier to change metadata/userdata after vm boot. However I don't know if the current nova implementation allows it or not.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-24: Related fix merged to neutron (master)

#11

Reviewed: https://review.openstack.org/374388
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=39c4f8fdc77bf81aaa8fdc9e0641c3040c0aa93b
Submitter: Jenkins
Branch: master

commit 39c4f8fdc77bf81aaa8fdc9e0641c3040c0aa93b
Author: Armando Migliaccio <email address hidden>
Date: Wed Sep 21 13:51:34 2016 -0700

Add MAC address to subports in trunk_details

    There is still time to change this without going through the hassle
    of an API bump. Having access to the subport MAC is useful when
    orchestrating the guest networking configuration.

Related-bug: 1626010
Partially-implements: blueprint vlan-aware-vms

Change-Id: I22c1ad18b9778332ca4c9c374dd3497ce03663da

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-24: Related fix proposed to neutron (stable/newton)

#12

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/375885

Ihar Hrachyshka (ihar-hrachyshka) on 2016-09-26

tags:	added: ovs-fw
Changed in neutron:
status:	New → Confirmed
milestone:	none → ocata-1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-26: Related fix merged to neutron (stable/newton)

#13

Reviewed: https://review.openstack.org/375885
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6e8d2c27ffe680f5f26649bab1edf041e9fe021e
Submitter: Jenkins
Branch: stable/newton

commit 6e8d2c27ffe680f5f26649bab1edf041e9fe021e
Author: Armando Migliaccio <email address hidden>
Date: Wed Sep 21 13:51:34 2016 -0700

Add MAC address to subports in trunk_details

    There is still time to change this without going through the hassle
    of an API bump. Having access to the subport MAC is useful when
    orchestrating the guest networking configuration.

Related-bug: 1626010
Partially-implements: blueprint vlan-aware-vms

Change-Id: I22c1ad18b9778332ca4c9c374dd3497ce03663da
(cherry picked from commit 39c4f8fdc77bf81aaa8fdc9e0641c3040c0aa93b)

tags:

added: in-stable-newton

Revision history for this message

Bence Romsics (bence-romsics) wrote on 2016-09-27: Re: Connectivity problem on trunk parent with MAC reuse and openvswitch firewall driver

#14

While looking into this bug I found a few things looking wrong.

The colliding firewall rule (beyond colliding also) seems to store the wrong vlan id into register 6.

table=0, priority=90,dl_dst=fa:16:3e:8a:48:37 actions=load:0x4b->NXM_NX_REG5[],load:0xfff->NXM_NX_REG6[],resubmit(,81)

0xfff is the DEAD_VLAN_TAG, but I think it should be the proper compute-internal vlan. The fallback to DEAD_VLAN_TAG is here:

https://github.com/openstack/neutron/blob/787c42c/neutron/agent/linux/openvswitch_firewall/firewall.py#L257

Based on that I realized that 'other_config' (ie. the ovs port attribute) was never set.

# ovs-vsctl list Port | grep other_config
other_config : {}
other_config : {}
other_config : {}
other_config : {}
other_config : {}
other_config : {}
other_config : {}
other_config : {}
other_config : {}
other_config : {}
other_config : {}

q-agt.log contains stuff like this:

2016-09-27 14:06:55.816 DEBUG neutron.agent.ovsdb.impl_idl [-] Running txn command(idx=0): DbSetCommand(table=Port, col_values=(('other_config', {'tag': 2}),), record=tapc1d40124-b4) from (pid=4948) do_commit /opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:98
2016-09-27 14:06:55.816 DEBUG neutron.agent.ovsdb.impl_idl [-] Transaction caused no change from (pid=4948) do_commit /opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:125

The culprit for this seems to be the 'native' ovsdb driver. If I return to using the 'vsctl' driver then the 'other_config' field is set again and we store the proper vlan id into register 6.

Of course using the vsctl driver did not solve the original problem of this bug report. Shall I open a new bug report for the native/idl driver swallowing some of the ovsdb transactions?

For confirmation if I use the vsctl driver other_config is set:

ml2_conf.ini:
[ovs]
ovsdb_interface = vsctl

# ovs-vsctl list Port | grep other_config
other_config : {net_uuid="47d70d80-e7a1-46cd-a248-f3217e86de31", network_type=vxlan, physical_network=None, segmentation_id="55", tag="2"}
other_config : {}
other_config : {net_uuid="f3c2363a-1a1f-4b7e-be9f-be9e18fcdd1d", network_type=vxlan, physical_network=None, segmentation_id="18", tag="1"}
other_config : {}
other_config : {net_uuid="47d70d80-e7a1-46cd-a248-f3217e86de31", network_type=vxlan, physical_network=None, segmentation_id="55", tag="2"}
other_config : {}
other_config : {}
other_config : {}
other_config : {}
other_config : {}
other_config : {net_uuid="f3c2363a-1a1f-4b7e-be9f-be9e18fcdd1d", network_type=vxlan, physical_network=None, segmentation_id="18", tag="1"}

While looking into this bug I found a few things looking wrong.

The colliding firewall rule (beyond colliding also) seems to store the wrong vlan id into register 6.

table=0, priority=90,dl_dst=fa:16:3e:8a:48:37 actions=load:0x4b->NXM_NX_REG5[],load:0xfff->NXM_NX_REG6[],resubmit(,81)

0xfff is the DEAD_VLAN_TAG, but I think it should be the proper compute-internal vlan. The fallback to DEAD_VLAN_TAG is here:

https://github.com/openstack/neutron/blob/787c42c/neutron/agent/linux/openvswitch_firewall/firewall.py#L257

Based on that I realized that 'other_config' (ie. the ovs port attribute) was never set.

# ovs-vsctl list Port | grep other_config
other_config        : {}
other_config        : {}
other_config        : {}
other_config        : {}
other_config        : {}
other_config        : {}
other_config        : {}
other_config        : {}
other_config        : {}
other_config        : {}
other_config        : {}

q-agt.log contains stuff like this:

2016-09-27 14:06:55.816 DEBUG neutron.agent.ovsdb.impl_idl [-] Running txn command(idx=0): DbSetCommand(table=Port, col_values=(('other_config', {'tag': 2}),), record=tapc1d40124-b4) from (pid=4948) do_commit /opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:98
2016-09-27 14:06:55.816 DEBUG neutron.agent.ovsdb.impl_idl [-] Transaction caused no change from (pid=4948) do_commit /opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:125

The culprit for this seems to be the 'native' ovsdb driver. If I return to using the 'vsctl' driver then the 'other_config' field is set again and we store the proper vlan id into register 6.

Of course using the vsctl driver did not solve the original problem of this bug report. Shall I open a new bug report for the native/idl driver swallowing some of the ovsdb transactions?

For confirmation if I use the vsctl driver other_config is set:

ml2_conf.ini:
[ovs]
ovsdb_interface = vsctl

# ovs-vsctl list Port | grep other_config
other_config        : {net_uuid="47d70d80-e7a1-46cd-a248-f3217e86de31", network_type=vxlan, physical_network=None, segmentation_id="55", tag="2"}
other_config        : {}
other_config        : {net_uuid="f3c2363a-1a1f-4b7e-be9f-be9e18fcdd1d", network_type=vxlan, physical_network=None, segmentation_id="18", tag="1"}
other_config        : {}
other_config        : {net_uuid="47d70d80-e7a1-46cd-a248-f3217e86de31", network_type=vxlan, physical_network=None, segmentation_id="55", tag="2"}
other_config        : {}
other_config        : {}
other_config        : {}
other_config        : {}
other_config        : {}
other_config        : {net_uuid="f3c2363a-1a1f-4b7e-be9f-be9e18fcdd1d", network_type=vxlan, physical_network=None, segmentation_id="18", tag="1"}

Revision history for this message

Jakub Libosvar (libosvar) wrote on 2016-09-30:

#15

Possibly a duplicate of https://launchpad.net/bugs/1593760

Armando Migliaccio (armando-migliaccio) on 2016-10-03

tags:

added: newton-backport-potential
removed: in-stable-newton newton-rc-potential

Revision history for this message

Bence Romsics (bence-romsics) wrote on 2016-10-06:

#16

Opened a new bug report for the problem described in comment #14, because it looks independent.
See here: https://bugs.launchpad.net/neutron/+bug/1630920

Revision history for this message

Terry Wilson (otherwiseguy) wrote on 2016-10-06:

#17

Bence: There will be ports that don't have other_config set in ovs. I'm not sure your grep is meaningful. Also, the "caused no change" message from the database just means that the value had already been set. The vsctl implementation just doesn't log this condition (because there is no way to detect it). It's just a DEBUG message to make debugging easier.

If there is an detectable issue and log messages, I'll be happy to take a look at them.

Revision history for this message

Terry Wilson (otherwiseguy) wrote on 2016-10-06:

#18

in ovs_neutron_agent:
799 port_other_config = self.int_br.db_get_val("Port", port.port_name,
800 "other_config")
801 if port_other_config is None:
802 if port.vif_id in self.deleted_ports:
803 LOG.debug("Port %s deleted concurrently", port.vif_id)
804 elif port.vif_id in self.updated_ports:
805 LOG.error(_LE("Expected port %s not found"), port.vif_id)
806 else:
807 LOG.debug("Unable to get config for port %s", port.vif_id)
808 return False

This seems like a bug to me. db_get_val won't ever return None here (vsctl or native, as long as the column is actually a map column), just an empty {}. Later we do a set_db_attribute based on port_other_config. So if it is {}, we'll be setting it to {} instead of returning False.

I have also detected a difference in behavior in set_db_attribute(). In vsctl, setting dict value merges the values, and in native it overwrites. e.g.

b.set_db_attribute('Port', port_name, 'other_config': {'a': 'b'})
b.set_db_attribute('Port', port_name, 'other_config': {'c': 'd'})

in vsctl will produce {'a': 'b', 'c': 'd'} and in native will produce {'c', 'd'}.

From the ovs-vsctl manpage:
       [--if-exists] set table record column[:key]=value...
              Sets the value of each specified column in the given record in table to value.
              For map columns, a key may optionally be specified, in which case the
              value associated with key in that column is changed (or added, if none exists),
              instead of the entire map.

So this would explain why setting other_config to {} would be a NoOp for vsctl. So it looks like a bug that exposes a difference in behavior. Simple fix is fixing ovs_neutron_agent first, then looking at set_db_attribute difference more carefully later.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-10-06: Fix proposed to neutron (master)

#19

Fix proposed to branch: master
Review: https://review.openstack.org/383054

Changed in neutron:
assignee:	Jakub Libosvar (libosvar) → Terry Wilson (otherwiseguy)
status:	Confirmed → In Progress

Revision history for this message

Terry Wilson (otherwiseguy) wrote on 2016-10-06: Re: Connectivity problem on trunk parent with MAC reuse and openvswitch firewall driver

#20

Hmm, actually, the bridge I was testing on had the other_config configured and cleared at some point and it returned {}. Apparently if the value has never been set at all, it will return None. So still have the behavior difference, but not quite what I thought the issue was.

Revision history for this message

Bence Romsics (bence-romsics) wrote on 2016-10-07:

#21

Terry, I'm happy you agree there's a behavior difference between the two ovsdb drivers.

I have not looked into the root cause of the difference before reporting it. I stumbled upon it while debugging another problem (of the openvswitch firewall driver) reported in this ticket. Can we please continue the ovsdb discussion in the other ticket? I mean this: #1630920. I think the firewall driver and ovsdb issues are independent.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-10-11: Fix proposed to neutron (master)

#22

Fix proposed to branch: master
Review: https://review.openstack.org/385085

Changed in neutron:
assignee:	Terry Wilson (otherwiseguy) → Jakub Libosvar (libosvar)

Armando Migliaccio (armando-migliaccio) on 2016-11-04

tags:	removed: trunk
summary:	- Connectivity problem on trunk parent with MAC reuse and openvswitch - firewall driver + OVS Firewall cannot handle non unique MACs

Armando Migliaccio (armando-migliaccio) on 2016-11-16

Changed in neutron:
milestone:	ocata-1 → ocata-2

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-11-19: Change abandoned on neutron (master)

#23

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/385085
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Armando Migliaccio (armando-migliaccio) on 2017-01-06

Changed in neutron:
milestone:	ocata-2 → ocata-3

Armando Migliaccio (armando-migliaccio) on 2017-01-26

Changed in neutron:
milestone:	ocata-3 → ocata-rc1

Armando Migliaccio (armando-migliaccio) on 2017-02-07

Changed in neutron:
milestone:	ocata-rc1 → ocata-rc2

Armando Migliaccio (armando-migliaccio) on 2017-02-08

Changed in neutron:
milestone:	ocata-rc2 → pike-1

Revision history for this message

Thomas Morin (tmmorin-orange) wrote on 2017-04-04:

#24

This bug happens to have another side-effect preventing [1] to work as expected.

[1] results in using the highest possible OF protocol asked by a neutron component for a given bridge, and hence may result in OVS fw code doing add_flows with OF>1.0. With OF>1.0, contrarily to OF=1.0, doing strip_vlan requires matching on dl_vlan first. Not matching on dl_vlan (this bug) is hence an issue to progress [1].

[1] https://review.openstack.org/451601

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-04-07: Fix proposed to neutron (master)

#25

Fix proposed to branch: master
Review: https://review.openstack.org/454646

Changed in neutron:
assignee:	Jakub Libosvar (libosvar) → Thomas Morin (tmmorin-orange)

Revision history for this message

Thomas Morin (tmmorin-orange) wrote on 2017-04-13:

#26

I marked bug 1593760 as a duplicate of this one (even though predating this one).

OpenStack Infra (hudson-openstack) on 2017-04-19

Changed in neutron:
assignee:	Thomas Morin (tmmorin-orange) → Jakub Libosvar (libosvar)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-05-03: Related fix proposed to neutron (master)

#27

Related fix proposed to branch: master
Review: https://review.openstack.org/462227

Armando Migliaccio (armando-migliaccio) on 2017-05-18

Changed in neutron:
milestone:	pike-1 → pike-2

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-06-13:

#28

Related fix proposed to branch: master
Review: https://review.openstack.org/473899

OpenStack Infra (hudson-openstack) on 2017-06-16

Changed in neutron:
assignee:	Jakub Libosvar (libosvar) → Thomas Morin (tmmorin-orange)

Revision history for this message

Thomas Morin (tmmorin-orange) wrote on 2017-06-16:

#29

My mistake: I had https://review.openstack.org/#/c/454646/ that was saying Partial-Bug to this bug, and I just rebased it, which made LP assign me to this bug.

I removed me as assignee, but for some reason I can't set you as assignee Jakub....

Changed in neutron:
assignee:	Thomas Morin (tmmorin-orange) → nobody

OpenStack Infra (hudson-openstack) on 2017-06-23

Changed in neutron:
assignee:	nobody → Jakub Libosvar (libosvar)

OpenStack Infra (hudson-openstack) on 2017-06-29

Changed in neutron:
assignee:	Jakub Libosvar (libosvar) → Brian Haley (brian-haley)

OpenStack Infra (hudson-openstack) on 2017-06-30

Changed in neutron:
assignee:	Brian Haley (brian-haley) → Jakub Libosvar (libosvar)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-16: Related fix merged to neutron (master)

#30

Reviewed: https://review.openstack.org/473899
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e9e8f5dccbc43ab3e07ec285a24489fdb9ac6c80
Submitter: Jenkins
Branch: master

commit e9e8f5dccbc43ab3e07ec285a24489fdb9ac6c80
Author: Jakub Libosvar <email address hidden>
Date: Tue Jun 13 12:03:31 2017 +0000

Introduce trusted ports to firewall driver API

    Currently trusted ports (DHCP & router ports) are considered trusted
    ports based on theirs owner. Trusted ports are not passed to the
    firewall driver and hence the driver doesn't have any way how to deal
    with such ports.

    This change introduces two new methods to firewall driver API:
     - process_trusted_ports
     - remove_trusted_ports

These methods will give firewall driver a chance to process traffic
coming from the trusted ports in case its needed.

For specific case, see patch depending on this one.

Change-Id: I0be64483515e45f98ffffce8346a6bff06bc0fd1
Related-bug: #1626010

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-20: Change abandoned on neutron (master)

#31

Change abandoned by Jakub Libosvar (<email address hidden>) on branch: master
Review: https://review.openstack.org/462227
Reason: https://review.openstack.org/#/c/385085/ is close to merge

OpenStack Infra (hudson-openstack) on 2017-07-20

Changed in neutron:
assignee:	Jakub Libosvar (libosvar) → Thomas Morin (tmmorin-orange)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-21: Fix merged to neutron (master)

#32

Reviewed: https://review.openstack.org/385085
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6370a0471076ccb095a90f97ffc869ae7ea2e5ed
Submitter: Jenkins
Branch: master

commit 6370a0471076ccb095a90f97ffc869ae7ea2e5ed
Author: Jakub Libosvar <email address hidden>
Date: Tue Jun 13 12:07:28 2017 +0000

ovsfw: Fix overlapping MAC addresses on integration bridge

    The patch relies on the fact that traffic not going from instance
    (and thus port not managed by firewall) is tagged. Traffic coming from
    the instance is not tagged and thus net register is used for marking
    such traffic. These two approaches make matching rules unique even if
    two ports from different networks share its' mac addressess.

Traffic coming from trusted ports is marked with network in registry
so firewall can decide later to which network traffic belongs.

Closes-bug: #1626010

Change-Id: Ia05d75a01b0469a0eaa82ada67b16a9481c50f1c

Changed in neutron:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-28: Fix included in openstack/neutron 11.0.0.0b3

#33

This issue was fixed in the openstack/neutron 11.0.0.0b3 development milestone.

neutron

OVS Firewall cannot handle non unique MACs

Bug Description

Duplicates of this bug

Other bug subscribers

Remote bug watches