| 2020-09-24 11:50:54 |
Frode Nordahl |
bug |
|
|
added bug |
| 2020-09-24 12:00:59 |
Frode Nordahl |
nominated for series |
|
Ubuntu Focal |
|
| 2020-09-24 12:00:59 |
Frode Nordahl |
bug task added |
|
ovn (Ubuntu Focal) |
|
| 2020-09-24 12:01:10 |
Frode Nordahl |
ovn (Ubuntu): status |
New |
Fix Released |
|
| 2020-09-24 12:25:38 |
Frode Nordahl |
bug |
|
|
added subscriber Thomas Parrott |
| 2020-09-24 12:33:07 |
Thomas Parrott |
description |
[Impact]
The issue I'm seeing is that if I have multiple OVN routers connected to the same external OVS uplink switch (which in turn is connected via veth to a native linux bridge), when NDP solicit packets are sent from the native linux bridge into the logical OVN switch, for some reason every OVN router that is connected to the OVN switch retransmits the NDP packet from its own MAC address. Which in turn causes them all to receive another NDP packet, which are then all retransmitted (a flood then ensues until OVS rate limiting takes effect).
This occurs even for IPs that do not exist on the network that the OVN routers dont know anything about.
It doesn't happen for ARP, nor does it occur in Groovy.
[Test Case]
[Regression Potential]
[Other Info] |
[Impact]
The issue I'm seeing is that if I have multiple OVN routers connected to the same external OVS uplink switch (which in turn is connected via veth to a native linux bridge), when NDP solicit packets are sent from the native linux bridge into the logical OVN switch, for some reason every OVN router that is connected to the OVN switch retransmits the NDP packet from its own MAC address. Which in turn causes them all to receive another NDP packet, which are then all retransmitted (a flood then ensues until OVS rate limiting takes effect).
This occurs even for IPs that do not exist on the network that the OVN routers dont know anything about.
It doesn't happen for ARP, nor does it occur in Groovy.
[Test Case]
test
[Regression Potential]
[Other Info] |
|
| 2020-09-24 12:50:00 |
Thomas Parrott |
description |
[Impact]
The issue I'm seeing is that if I have multiple OVN routers connected to the same external OVS uplink switch (which in turn is connected via veth to a native linux bridge), when NDP solicit packets are sent from the native linux bridge into the logical OVN switch, for some reason every OVN router that is connected to the OVN switch retransmits the NDP packet from its own MAC address. Which in turn causes them all to receive another NDP packet, which are then all retransmitted (a flood then ensues until OVS rate limiting takes effect).
This occurs even for IPs that do not exist on the network that the OVN routers dont know anything about.
It doesn't happen for ARP, nor does it occur in Groovy.
[Test Case]
test
[Regression Potential]
[Other Info] |
[Impact]
The issue I'm seeing is that if I have multiple OVN routers connected to the same external OVS uplink switch (which in turn is connected via veth to a native linux bridge), when NDP solicit packets are sent from the native linux bridge into the logical OVN switch, for some reason every OVN router that is connected to the OVN switch retransmits the NDP packet from its own MAC address. Which in turn causes them all to receive another NDP packet, which are then all retransmitted (a flood then ensues until OVS rate limiting takes effect).
This occurs even for IPs that do not exist on the network that the OVN routers dont know anything about.
It doesn't happen for ARP, nor does it occur in Groovy.
[Test Case]
Setup single node LXD with and OVN network and then ping a non-existant IP in the uplink network from the LXD host and watch for the OVN gateway to retransmit the IPv6 NS packet from lxdbr0 back into the uplink network.
# Install OVN and setup.
sudo apt install ovn-host ovn-central tcpdump -y
sudo ovs-vsctl set open_vswitch . \
external_ids:ovn-encap-type=geneve \
external_ids:ovn-remote="unix:/var/run/ovn/ovnsb_db.sock" \
external_ids:ovn-encap-ip=127.0.0.1
# Install LXD and get lxdbr0 address and subnet.
snap install lxd
lxd init --auto
lxc network show lxdbr0
config:
ipv4.address: 10.154.225.1/24
ipv4.nat: "true"
ipv6.address: fd42:37a6:2a0:c014::1/64
ipv6.nat: "true"
# Configure lxdbr0 as compatible OVN uplink network.
lxc network set lxdbr0 ipv4.dhcp.ranges=10.154.225.2-10.154.225.10 ipv4.ovn.ranges=10.154.225.11-10.154.225.20
# Create OVN network.
lxc network create ovn1 --type=ovn network=lxdbr0
# In separate window run tcpdump on lxsbr0.
sudo tcpdump -i lxdbr0 -n -e ip6
Now ping a non-existant IP in the subnet of lxdbr0 from the LXD host, e.g. fd42:37a6:2a0:c014::2
ping fd42:37a6:2a0:c014::2
# Check tcpdump window showing duplicate IPv6 NS (from from lxdbr0 MAC address and one from the OVN network's external router port's MAC addres):
12:46:27.123496 00:16:3e:b5:3d:5e > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:46:27.124320 00:16:3e:ab:cc:5a > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
# Create another OVN network.
lxc network create ovn2 --type=ovn network=lxdbr0
# Check tcpdump window showing a duplicate IPv6 NS storm, now from 3 MACs
12:48:30.025836 00:16:3e:b5:3d:5e > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:48:30.028238 00:16:3e:ab:cc:5a > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:48:30.028239 00:16:3e:31:9a:ff > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
[Regression Potential]
[Other Info] |
|
| 2020-09-24 12:50:25 |
Thomas Parrott |
description |
[Impact]
The issue I'm seeing is that if I have multiple OVN routers connected to the same external OVS uplink switch (which in turn is connected via veth to a native linux bridge), when NDP solicit packets are sent from the native linux bridge into the logical OVN switch, for some reason every OVN router that is connected to the OVN switch retransmits the NDP packet from its own MAC address. Which in turn causes them all to receive another NDP packet, which are then all retransmitted (a flood then ensues until OVS rate limiting takes effect).
This occurs even for IPs that do not exist on the network that the OVN routers dont know anything about.
It doesn't happen for ARP, nor does it occur in Groovy.
[Test Case]
Setup single node LXD with and OVN network and then ping a non-existant IP in the uplink network from the LXD host and watch for the OVN gateway to retransmit the IPv6 NS packet from lxdbr0 back into the uplink network.
# Install OVN and setup.
sudo apt install ovn-host ovn-central tcpdump -y
sudo ovs-vsctl set open_vswitch . \
external_ids:ovn-encap-type=geneve \
external_ids:ovn-remote="unix:/var/run/ovn/ovnsb_db.sock" \
external_ids:ovn-encap-ip=127.0.0.1
# Install LXD and get lxdbr0 address and subnet.
snap install lxd
lxd init --auto
lxc network show lxdbr0
config:
ipv4.address: 10.154.225.1/24
ipv4.nat: "true"
ipv6.address: fd42:37a6:2a0:c014::1/64
ipv6.nat: "true"
# Configure lxdbr0 as compatible OVN uplink network.
lxc network set lxdbr0 ipv4.dhcp.ranges=10.154.225.2-10.154.225.10 ipv4.ovn.ranges=10.154.225.11-10.154.225.20
# Create OVN network.
lxc network create ovn1 --type=ovn network=lxdbr0
# In separate window run tcpdump on lxsbr0.
sudo tcpdump -i lxdbr0 -n -e ip6
Now ping a non-existant IP in the subnet of lxdbr0 from the LXD host, e.g. fd42:37a6:2a0:c014::2
ping fd42:37a6:2a0:c014::2
# Check tcpdump window showing duplicate IPv6 NS (from from lxdbr0 MAC address and one from the OVN network's external router port's MAC addres):
12:46:27.123496 00:16:3e:b5:3d:5e > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:46:27.124320 00:16:3e:ab:cc:5a > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
# Create another OVN network.
lxc network create ovn2 --type=ovn network=lxdbr0
# Check tcpdump window showing a duplicate IPv6 NS storm, now from 3 MACs
12:48:30.025836 00:16:3e:b5:3d:5e > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:48:30.028238 00:16:3e:ab:cc:5a > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:48:30.028239 00:16:3e:31:9a:ff > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
[Regression Potential]
[Other Info] |
[Impact]
The issue I'm seeing is that if I have multiple OVN routers connected to the same external OVS uplink switch (which in turn is connected via veth to a native linux bridge), when NDP solicit packets are sent from the native linux bridge into the logical OVN switch, for some reason every OVN router that is connected to the OVN switch retransmits the NDP packet from its own MAC address. Which in turn causes them all to receive another NDP packet, which are then all retransmitted (a flood then ensues until OVS rate limiting takes effect).
This occurs even for IPs that do not exist on the network that the OVN routers dont know anything about.
It doesn't happen for ARP, nor does it occur in Groovy.
[Test Case]
Setup single node LXD with and OVN network and then ping a non-existant IP in the uplink network from the LXD host and watch for the OVN gateway to retransmit the IPv6 NS packet from lxdbr0 back into the uplink network.
# Install OVN and setup.
sudo apt install ovn-host ovn-central tcpdump -y
sudo ovs-vsctl set open_vswitch . \
external_ids:ovn-encap-type=geneve \
external_ids:ovn-remote="unix:/var/run/ovn/ovnsb_db.sock" \
external_ids:ovn-encap-ip=127.0.0.1
# Install LXD and get lxdbr0 address and subnet.
snap install lxd
lxd init --auto
lxc network show lxdbr0
config:
ipv4.address: 10.154.225.1/24
ipv4.nat: "true"
ipv6.address: fd42:37a6:2a0:c014::1/64
ipv6.nat: "true"
# Configure lxdbr0 as compatible OVN uplink network.
lxc network set lxdbr0 ipv4.dhcp.ranges=10.154.225.2-10.154.225.10 ipv4.ovn.ranges=10.154.225.11-10.154.225.20
# Create OVN network.
lxc network create ovn1 --type=ovn network=lxdbr0
# In separate window run tcpdump on lxdbr0 with -e flag to show MAC address.
sudo tcpdump -i lxdbr0 -n -e ip6
Now ping a non-existant IP in the subnet of lxdbr0 from the LXD host, e.g. fd42:37a6:2a0:c014::2
ping fd42:37a6:2a0:c014::2
# Check tcpdump window showing duplicate IPv6 NS (from from lxdbr0 MAC address and one from the OVN network's external router port's MAC addres):
12:46:27.123496 00:16:3e:b5:3d:5e > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:46:27.124320 00:16:3e:ab:cc:5a > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
# Create another OVN network.
lxc network create ovn2 --type=ovn network=lxdbr0
# Check tcpdump window showing a duplicate IPv6 NS storm, now from 3 MACs
12:48:30.025836 00:16:3e:b5:3d:5e > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:48:30.028238 00:16:3e:ab:cc:5a > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:48:30.028239 00:16:3e:31:9a:ff > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
[Regression Potential]
[Other Info] |
|
| 2020-09-24 12:51:13 |
Thomas Parrott |
description |
[Impact]
The issue I'm seeing is that if I have multiple OVN routers connected to the same external OVS uplink switch (which in turn is connected via veth to a native linux bridge), when NDP solicit packets are sent from the native linux bridge into the logical OVN switch, for some reason every OVN router that is connected to the OVN switch retransmits the NDP packet from its own MAC address. Which in turn causes them all to receive another NDP packet, which are then all retransmitted (a flood then ensues until OVS rate limiting takes effect).
This occurs even for IPs that do not exist on the network that the OVN routers dont know anything about.
It doesn't happen for ARP, nor does it occur in Groovy.
[Test Case]
Setup single node LXD with and OVN network and then ping a non-existant IP in the uplink network from the LXD host and watch for the OVN gateway to retransmit the IPv6 NS packet from lxdbr0 back into the uplink network.
# Install OVN and setup.
sudo apt install ovn-host ovn-central tcpdump -y
sudo ovs-vsctl set open_vswitch . \
external_ids:ovn-encap-type=geneve \
external_ids:ovn-remote="unix:/var/run/ovn/ovnsb_db.sock" \
external_ids:ovn-encap-ip=127.0.0.1
# Install LXD and get lxdbr0 address and subnet.
snap install lxd
lxd init --auto
lxc network show lxdbr0
config:
ipv4.address: 10.154.225.1/24
ipv4.nat: "true"
ipv6.address: fd42:37a6:2a0:c014::1/64
ipv6.nat: "true"
# Configure lxdbr0 as compatible OVN uplink network.
lxc network set lxdbr0 ipv4.dhcp.ranges=10.154.225.2-10.154.225.10 ipv4.ovn.ranges=10.154.225.11-10.154.225.20
# Create OVN network.
lxc network create ovn1 --type=ovn network=lxdbr0
# In separate window run tcpdump on lxdbr0 with -e flag to show MAC address.
sudo tcpdump -i lxdbr0 -n -e ip6
Now ping a non-existant IP in the subnet of lxdbr0 from the LXD host, e.g. fd42:37a6:2a0:c014::2
ping fd42:37a6:2a0:c014::2
# Check tcpdump window showing duplicate IPv6 NS (from from lxdbr0 MAC address and one from the OVN network's external router port's MAC addres):
12:46:27.123496 00:16:3e:b5:3d:5e > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:46:27.124320 00:16:3e:ab:cc:5a > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
# Create another OVN network.
lxc network create ovn2 --type=ovn network=lxdbr0
# Check tcpdump window showing a duplicate IPv6 NS storm, now from 3 MACs
12:48:30.025836 00:16:3e:b5:3d:5e > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:48:30.028238 00:16:3e:ab:cc:5a > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:48:30.028239 00:16:3e:31:9a:ff > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
[Regression Potential]
[Other Info] |
[Impact]
The issue I'm seeing is that if I have multiple OVN routers connected to the same external OVS uplink switch (which in turn is connected via veth to a native linux bridge), when NDP solicit packets are sent from the native linux bridge into the logical OVN switch, for some reason every OVN router that is connected to the OVN switch retransmits the NDP packet from its own MAC address. Which in turn causes them all to receive another NDP packet, which are then all retransmitted (a flood then ensues until OVS rate limiting takes effect).
This occurs even for IPs that do not exist on the network that the OVN routers dont know anything about.
It doesn't happen for ARP, nor does it occur in Groovy.
[Test Case]
Setup single node LXD with and OVN network and then ping a non-existant IP in the uplink network from the LXD host and watch for the OVN gateway to retransmit the IPv6 NS packet from lxdbr0 back into the uplink network.
# Install OVN and setup.
sudo apt install ovn-host ovn-central tcpdump -y
sudo ovs-vsctl set open_vswitch . \
external_ids:ovn-encap-type=geneve \
external_ids:ovn-remote="unix:/var/run/ovn/ovnsb_db.sock" \
external_ids:ovn-encap-ip=127.0.0.1
# Install LXD and get lxdbr0 address and subnet.
snap install lxd
lxd init --auto
lxc network show lxdbr0
config:
ipv4.address: 10.154.225.1/24
ipv4.nat: "true"
ipv6.address: fd42:37a6:2a0:c014::1/64
ipv6.nat: "true"
# Configure lxdbr0 as compatible OVN uplink network.
lxc network set lxdbr0 ipv4.dhcp.ranges=10.154.225.2-10.154.225.10 ipv4.ovn.ranges=10.154.225.11-10.154.225.20
# Create OVN network.
lxc network create ovn1 --type=ovn network=lxdbr0
# In separate window run tcpdump on lxdbr0 with -e flag to show MAC address.
sudo tcpdump -i lxdbr0 -n -e ip6
Now ping a non-existant IP in the subnet of lxdbr0 from the LXD host, e.g. fd42:37a6:2a0:c014::2
ping fd42:37a6:2a0:c014::2
# Check tcpdump window showing duplicate IPv6 NS (from from lxdbr0 MAC address and one from the OVN network's external router port's MAC addres):
12:46:27.123496 00:16:3e:b5:3d:5e > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:46:27.124320 00:16:3e:ab:cc:5a > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
# Create another OVN network.
lxc network create ovn2 --type=ovn network=lxdbr0
# Check tcpdump window showing a duplicate IPv6 NS storm, now from 3 MACs:
12:48:30.025836 00:16:3e:b5:3d:5e > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:48:30.028238 00:16:3e:ab:cc:5a > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:48:30.028239 00:16:3e:31:9a:ff > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
I would only expect to see the IPv6 NS packet from the LXD host's MAC address, not the OVN gateways as well.
[Regression Potential]
[Other Info] |
|
| 2020-09-24 12:56:20 |
Frode Nordahl |
nominated for series |
|
Ubuntu Groovy |
|
| 2020-09-24 12:56:20 |
Frode Nordahl |
bug task added |
|
ovn (Ubuntu Groovy) |
|
| 2020-09-25 08:36:33 |
Frode Nordahl |
ovn (Ubuntu Focal): status |
New |
Triaged |
|
| 2020-09-25 08:36:36 |
Frode Nordahl |
ovn (Ubuntu Focal): importance |
Undecided |
High |
|
| 2021-09-23 08:34:53 |
Frode Nordahl |
ovn (Ubuntu Focal): status |
Triaged |
Fix Released |
|
