neutron

DVR and floating IPs broken in latest 7.0.0.0rc1?

Bug #1792493 reported by Eric Miller
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
kolla-ansible
Invalid
Undecided
Unassigned
neutron
Fix Released
High
Unassigned

Bug Description

Kolla-Ansible 7.0.0.0rc1 with binary image build (since the source option is failing to build ceilometer images currently) on CentOS 7.5 (latest updates)

What worked previously does not appear to work anymore. I'm not sure if this is due to an update in CentOS 7.5 or OVS or other at this stage, but compute nodes are no longer ARP replying to ARP requests for who has the floating IP.

For testing, I looked for the IP assigned to the FIP namespace's fg interface (in my case, fg-ba492724-bd). This appears to be an IP on the ext-net network, but is not the floating IP assigned to a VM. Let's call this A.A.A.A and the floating IP B.B.B.B.

I can tcpdump traffic on the physical port of the compute node and see the ARP requests for both A.A.A.A and B.B.B.B with respective pings from the Internet, but no ARP replies.

I have attached a diagram showing, what I believe to be, the correct path for the packets.

There appears to be something broken between my two arrows.

Since tcpdump is not installed in the openvswitch_vswitchd container, nor is ovs-tcpdump, I can't figure out how to mirror and sniff ports on the br-ex and br-int bridges, at least in a containerized instance of OVS. If anyone knows a way to do this, I would really appreciate the help.

I haven't found any issues in the OVS configuration (ovs-vsctl show) - which matches the attached diagram.

Has anyone else had issues?

OVS returns this version info:
ovs-vsctl (Open vSwitch) 2.9.0
DB Schema 7.15.1

in case it helps.

Eric

Revision history for this message
Eric Miller (erickmiller) wrote :
Revision history for this message
Eric Miller (erickmiller) wrote :

It is interesting that OVS reports br-int's port for fg-ba492724-bd as down:

 7(fg-ba492724-bd): addr:00:00:00:00:00:00
     config: PORT_DOWN
     state: LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max

I can only assume that this is not normal?

The fg-ba492724-bd interface in the FIP namespace is UP (public IP substituted with A.A.A.A):

26: fg-ba492724-bd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:26:50:2d brd ff:ff:ff:ff:ff:ff
    inet A.A.A.A/24 brd A.A.A.255 scope global fg-ba492724-bd
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe26:502d/64 scope link
       valid_lft forever preferred_lft forever

Eric

Revision history for this message
Eric Miller (erickmiller) wrote :

I have a Queens environment that works, so I am comparing everything possible between it and the Rocky deployment.

Apparently, having the "fg" interface down is normal (why, I have no idea), but the working Queens system shows this when running "ovs-ofctl show br-int", truncated to only show the fg interface.

 6(fg-d57c7cc3-d3): addr:00:00:00:00:00:00
     config: PORT_DOWN
     state: LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max

So far, everything in OVS and the network namespaces look identical.

Also, the OVS version is identical (from "ovs-vsctl --version"):
ovs-vsctl (Open vSwitch) 2.9.0
DB Schema 7.15.1

The CentOS install on the Queens environment has a slightly older kernel:
working: Kernel: Linux 3.10.0-862.9.1.el7.x86_64
not working: Kernel: Linux 3.10.0-862.11.6.el7.x86_64

but the kernel version is only a guess - I'll continue to look for variances.

Knowing if anyone else has run into this issue would be greatly appreciated.

Eric

Revision history for this message
Eric Miller (erickmiller) wrote :

One significant difference I see is the OVS flows created on the br-ex bridge:

Non-working system:

ovs-ofctl dump-flows br-ex

 cookie=0xcf59ae700b21fb77, duration=169.257s, table=0, n_packets=16, n_bytes=944, priority=4,in_port="phy-br-ex",dl_vlan=2 actions=strip_vlan,NORMAL

Working system:

ovs-ofctl dump-flows br-ex

 cookie=0xc68f93c404034a40, duration=2.224s, table=0, n_packets=0, n_bytes=0, priority=4,in_port="phy-br-ex",dl_vlan=2 actions=strip_vlan,NORMAL
 cookie=0xc68f93c404034a40, duration=24.644s, table=0, n_packets=307, n_bytes=28483, priority=2,in_port="phy-br-ex" actions=resubmit(,1)
 cookie=0xc68f93c404034a40, duration=24.977s, table=0, n_packets=1, n_bytes=60, priority=0 actions=NORMAL
 cookie=0xc68f93c404034a40, duration=24.641s, table=0, n_packets=5719479, n_bytes=630280414, priority=1 actions=resubmit(,3)
 cookie=0xc68f93c404034a40, duration=24.638s, table=1, n_packets=307, n_bytes=28483, priority=0 actions=resubmit(,2)
 cookie=0xc68f93c404034a40, duration=24.634s, table=2, n_packets=307, n_bytes=28483, priority=2,in_port="phy-br-ex" actions=drop
 cookie=0xc68f93c404034a40, duration=24.545s, table=3, n_packets=0, n_bytes=0, priority=2,dl_src=fa:16:3f:09:b2:a0 actions=output:"phy-br-ex"
 cookie=0xc68f93c404034a40, duration=24.510s, table=3, n_packets=0, n_bytes=0, priority=2,dl_src=fa:16:3f:44:65:55 actions=output:"phy-br-ex"
 cookie=0xc68f93c404034a40, duration=24.483s, table=3, n_packets=0, n_bytes=0, priority=2,dl_src=fa:16:3f:46:ad:5b actions=output:"phy-br-ex"
 cookie=0xc68f93c404034a40, duration=24.453s, table=3, n_packets=0, n_bytes=0, priority=2,dl_src=fa:16:3f:6b:a2:0b actions=output:"phy-br-ex"
 cookie=0xc68f93c404034a40, duration=24.428s, table=3, n_packets=0, n_bytes=0, priority=2,dl_src=fa:16:3f:7f:62:05 actions=output:"phy-br-ex"
 cookie=0xc68f93c404034a40, duration=24.400s, table=3, n_packets=0, n_bytes=0, priority=2,dl_src=fa:16:3f:9c:21:19 actions=output:"phy-br-ex"
 cookie=0xc68f93c404034a40, duration=24.375s, table=3, n_packets=0, n_bytes=0, priority=2,dl_src=fa:16:3f:b0:61:1c actions=output:"phy-br-ex"
 cookie=0xc68f93c404034a40, duration=24.351s, table=3, n_packets=0, n_bytes=0, priority=2,dl_src=fa:16:3f:f5:d2:d0 actions=output:"phy-br-ex"
 cookie=0xc68f93c404034a40, duration=24.631s, table=3, n_packets=5719479, n_bytes=630280414, priority=1 actions=NORMAL

These flows appear to be created by the "neutron_openvswitch_agent" container. So, maybe something isn't working in the Rocky version of this container?

I'll keep looking...

Eric

Revision history for this message
Eric Miller (erickmiller) wrote :
Download full text (3.4 KiB)

I looked at the openvswitch log file here (from inside the openvswitch-vswitchd container):
/var/log/kolla/openvswitch/ovs-vswitchd.log

while restarting the neutron_openvswitch_agent container. It appears that the Rocky version has continuous re-connections like this (after adding a few flows to the bridges):

2018-09-14T22:43:48.311Z|63850|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connection closed by peer
2018-09-14T22:43:49.309Z|63851|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connecting...
2018-09-14T22:43:49.310Z|63852|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connected
2018-09-14T22:43:49.311Z|63853|rconn|INFO|br-prov001<->tcp:127.0.0.1:6633: connection closed by peer
2018-09-14T22:43:55.310Z|63854|rconn|INFO|br-prov002<->tcp:127.0.0.1:6633: connected
2018-09-14T22:43:55.311Z|63855|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connection closed by peer
2018-09-14T22:43:56.309Z|63856|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connecting...
2018-09-14T22:43:56.310Z|63857|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connected
2018-09-14T22:43:56.311Z|63858|rconn|INFO|br-prov002<->tcp:127.0.0.1:6633: connection closed by peer
2018-09-14T22:43:57.310Z|63859|rconn|INFO|br-prov001<->tcp:127.0.0.1:6633: connected
2018-09-14T22:43:57.311Z|63860|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connection closed by peer
2018-09-14T22:43:58.308Z|63861|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connecting...
2018-09-14T22:43:58.310Z|63862|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connected
2018-09-14T22:43:58.311Z|63863|rconn|INFO|br-prov001<->tcp:127.0.0.1:6633: connection closed by peer

Right after restart of the neutron_openvswitch_agent container, the log entry for br-ex flows added shows:

2018-09-14T22:39:43.311Z|63573|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: 2 flow_mods in the 3 s starting 5 s ago (1 adds, 1 deletes)

That's only a single flow.

Note that we have two provider networks (prov001 and prov002) in addition to the external network (br-ex).

This is what the working system log looks like - with no reconnections at all:

2018-09-14T22:38:38.455Z|00290|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: connected
2018-09-14T22:38:38.455Z|00291|rconn|INFO|br-int<->tcp:127.0.0.1:6633: connected
2018-09-14T22:38:38.455Z|00292|rconn|INFO|br-prov001<->tcp:127.0.0.1:6633: connected
2018-09-14T22:38:38.458Z|00293|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connected
2018-09-14T22:38:48.636Z|00294|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: 60 flow_mods in the 9 s starting 10 s ago (46 adds, 14 deletes)
2018-09-14T22:38:49.188Z|00295|connmgr|INFO|br-prov001<->tcp:127.0.0.1:6633: 15 flow_mods 10 s ago (15 adds)
2018-09-14T22:38:49.249Z|00296|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: 16 flow_mods in the 4 s starting 10 s ago (16 adds)
2018-09-14T22:38:49.327Z|00297|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: 47 flow_mods in the 9 s starting 10 s ago (47 adds)
2018-09-14T22:39:48.635Z|00298|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: 22 flow_mods in the 1 s starting 59 s ago (8 adds, 14 deletes)
2018-09-14T22:39:49.328Z|00299|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: 14 flow_mods in the 1 s starting 58 s ago (14 adds)

For br-ex, 16 flows were added, as opposed to 1 flow for Rocky.

I noticed that our Queens ...

Read more...

Revision history for this message
Eric Miller (erickmiller) wrote :

Rocky has been re-deployed with only br-ex and prov001 external bridges and the same problem is occurring.

The ovs-vswitchd.log continues to show reconnections over and over (in the openvswitch_vswitchd container):

tail -f /var/log/kolla/openvswitch/ovs-vswitchd.log

2018-09-15T02:22:43.305Z|13561|rconn|INFO|br-prov001<->tcp:127.0.0.1:6633: connection closed by peer
2018-09-15T02:22:44.303Z|13562|rconn|INFO|br-prov001<->tcp:127.0.0.1:6633: connecting...
2018-09-15T02:22:44.304Z|13563|rconn|INFO|br-prov001<->tcp:127.0.0.1:6633: connected
2018-09-15T02:22:44.305Z|13564|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connection closed by peer
2018-09-15T02:22:45.303Z|13565|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connecting...
2018-09-15T02:22:45.304Z|13566|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connected
2018-09-15T02:22:45.305Z|13567|rconn|INFO|br-prov001<->tcp:127.0.0.1:6633: connection closed by peer
2018-09-15T02:22:46.303Z|13568|rconn|INFO|br-prov001<->tcp:127.0.0.1:6633: connecting...
2018-09-15T02:22:46.307Z|13569|rconn|INFO|br-prov001<->tcp:127.0.0.1:6633: connected
2018-09-15T02:22:46.308Z|13570|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connection closed by peer
2018-09-15T02:22:47.303Z|13571|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connecting...
2018-09-15T02:22:47.304Z|13572|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connected
2018-09-15T02:22:47.305Z|13573|rconn|INFO|br-prov001<->tcp:127.0.0.1:6633: connection closed by peer

So it definitely appears there is something broken in Rocky.

Eric

Revision history for this message
Eric Miller (erickmiller) wrote :

Just a quick note that kernel version doesn't appear to matter. My latest deploy was with no yum updates, so the kernel was:
Kernel: Linux 3.10.0-862.el7.x86_64

and the problem still occurs.

I am removing some things, including any Octavia configurations, as well as the provider network associated with that, and redeploying, just to see if there is something in our configuration causing this. A redeploy is bare metal and up, so we're re-imaging physical servers with CentOS 7.5 (Minimal 1804 ISO).

I'll report back...

Revision history for this message
Eric Miller (erickmiller) wrote :

Ok, after simplifying the deployment (as I mentioned in the previous message), DVR is working as expected with floating IPs.

One interesting problem I had was an issue with iptables not allowing return traffic that was to be DNAT'd from the public IP back to the private IP (in the qrouter namespace). I compared everything between our Queens configuration that works to the Rocky configuration that doesn't, and there was no difference between the iptables rules.

So, I did a yum update (remember, this was running CentOS 7.5 Minimal 1804 with no updates), and all of a sudden, iptables works now! So, I was chasing this issue for a while.

Now, onto the original issue, which, I suspect is related to provider networks, or our Octavia configuration...

Revision history for this message
Eric Miller (erickmiller) wrote :

I believe I have narrowed it down to adding a provider network in the globals.yml file such as:

neutron_bridge_name: "br-ex,br-prov001"
neutron_external_interface: "eth0,eth1"

When these are set to this, DVR Floating IPs work:

neutron_bridge_name: "br-ex"
neutron_external_interface: "eth0"

I'm going to do some further testing to see if I can find why, but wanted to point out the issue in case anyone knows of the problem and possibly why it is happening.

Eric

Revision history for this message
Eric Miller (erickmiller) wrote :

I have done additional testing and the provider network configuration is the cause:

This works (floating IPs respond on the ext-net/physnet1 physical interface and SNAT traffic works as expected):
neutron_bridge_name: "br-ex"
neutron_external_interface: "eth0"

This does not work (floating IPs do NOT respond on the ext-net/physnet1 physical interface and SNAT traffic does NOT work as expected):
neutron_bridge_name: "br-ex,br-prov001"
neutron_external_interface: "eth0,eth1"

This prevents Octavia from working since it requires a provider network for communication between amphorae VMs and the Octavia worker service.

Note that there are no issues in a Queens deployment with Kolla Ansible 6.1.0, this problem only occurs in the latest Rocky release candidate of Kolla-Ansible (7.0.0.0rc1).

Eric

Revision history for this message
Eric Miller (erickmiller) wrote :

I think I may have found the issue.

Note that I wasn't sure what other project/sub-project to add this ticket to, so I will add it to the Neutron project.

We use team interfaces with VLANs, which uses the same MAC address for each VLAN (see the bottom of this note for an example of 3 VLANs on a team interface from "ip a").

It appears that the OVS DVR Neutron Agent assumes that MAC addresses will be unique across interfaces - at least what I can tell here (Note that I am not a Python programmer):
https://github.com/openstack/neutron/blob/317cdbf40850964080a0a30d6212a3b536df1caa/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py

Specifically look at the "registered_dvr_macs" variable, which is used in a few places, including the _add_dvr_mac method.

So, instead of using interface names or IDs, it looks like a MAC address is assumed to be unique, but is not unique at all, especially in VLAN configurations (obviously very common).

Anyone know if my theory is right?

Btw, the fact that it worked on Queens was likely due to the fact that we did not use team interfaces on its installation since it was installed in VMs with virtual NICs.

Eric

10: team0.1000@team0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether ec:0d:9a:d9:09:16 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ee0d:9aff:fed9:916/64 scope link
       valid_lft forever preferred_lft forever
11: team0.1001@team0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ec:0d:9a:d9:09:16 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ee0d:9aff:fed9:916/64 scope link
       valid_lft forever preferred_lft forever
12: team0.1002@team0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ec:0d:9a:d9:09:16 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ee0d:9aff:fed9:916/64 scope link
       valid_lft forever preferred_lft forever

Revision history for this message
Eric Miller (erickmiller) wrote :

It appears my theory was correct. I wrote some code to change the ifcfg files so the MAC addresses of all VLAN interfaces was unique, and after a re-deploy, floating IPs work as expected, even with a provider network defined.

Eric

Miguel Lavalle (minsel)
tags: added: l3-dvr-backlog
Changed in neutron:
importance: Undecided → High
Boden R (boden)
Changed in neutron:
status: New → Triaged
Revision history for this message
Swaminathan Vasudevan (swaminathan-vasudevan) wrote :

The unique MAC address is only used for the DVR host MAC.
We always have the same MAC for the router ports that are distributed and not sure why this should be an issue..
So I am assuming that it is with respect to your external bridge configuration that you are having this problem.
Can you confirm.

Revision history for this message
Eric Miller (erickmiller) wrote :

It is only an issue with respect to the external bridge configuration, which includes ext-net for floating IPs. If there are multiple bridges defined, such as ext-net, br-prov001, br-prov001, etc., all associated with VLANs on the same physical interface, or a team/bond interface, the same MAC address is given to all of these physical interfaces, and this appears to be where the problem lies. Once the MAC address was forcefully changed to unique MAC addresses per VLAN on a physical interface, then external bridges have worked perfectly.

Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Hello:

I think this error is caused by the error described in [1]. If several physical bridges are attached to interfaces with the same MAC, the datapath_id will be the same. When using the native interface, all datapaths must be different.

Having OVS bridges with the same datapath and using native interface will provoke the error described in [2].

This problem was solved in https://review.openstack.org/#/q/379a9faf6206039903555ce7e3fc4221e5f06a7a

[1] https://bugs.launchpad.net/neutron/+bug/1697243
[2] https://bugs.launchpad.net/neutron/+bug/1792493/comments/6

Revision history for this message
LIU Yulong (dragon889) wrote :

According to Rodolfo's explanation, so we can close this bug.

Changed in neutron:
status: Triaged → Fix Released
Michal Nasiadka (mnasiadka)
Changed in kolla-ansible:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.