| 2021-06-01 12:11:56 |
Bence Romsics |
bug |
|
|
added bug |
| 2021-06-01 14:40:18 |
Bence Romsics |
bug |
|
|
added subscriber Lajos Katona |
| 2021-06-01 17:18:13 |
Jeremy Stanley |
description |
This is a bug with potential security implications. I don't see a clear way to exploit it at the moment, but to err on the safe side, I'm opening this as private to the security team.
Short summary: Using openvswitch-agent, traffic sent on some (at least dhcp) ports before ovs-agent applies the port's vlan tag can be seen and intercepted on ports from other networks on the same integration bridge.
We observed this bug:
* using vlan and vxlan networks
* using the noop and openvswitch firewall drivers
* on openstack versions mitaka, pike and master (commit 5a6f61af4a)
The time window between the port's creation and ovs-agent applying its vlan tag is usually very short. We observed this bug in the wild on a heavily loaded host. However to make the reproduction reliable on lightly loaded systems I inserted a sleep() into ovs-agent's source (just before the port's vlan tag is set):
$ git --no-pager format-patch --stdout 5a6f61af4a
From 8389b3e8e5c60c81ff2bb262e3ae2e8aab73d3f5 Mon Sep 17 00:00:00 2001
From: Bence Romsics <bence.romsics@gmail.com>
Date: Mon, 31 May 2021 13:12:34 +0200
Subject: [PATCH] WIP
Change-Id: Ibef4278a2f6a85f52a8ffa43caef6de38cbb40cb
---
.../plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
index 2c209bd387..355584b325 100644
--- a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
+++ b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
@@ -1190,6 +1190,7 @@ class OVSNeutronAgent(l2population_rpc.L2populationRpcCallBackTunnelMixin,
self.setup_arp_spoofing_protection(self.int_br,
port, port_detail)
if cur_tag != lvm.vlan:
+ time.sleep(3000)
self.int_br.set_db_attribute(
"Port", port.port_name, "tag", lvm.vlan)
·
--·
2.31.0
We discovered the bug by the following procedure:
* a test environment created multiple neutron nets in short succession with the exact same ipv6 subnet ranges
* therefore neutron selected the exact same ipv6 address for the subnet's dhcp port
* the host running the dhcp-agent and the ovs-agent was heavily loaded
* we observed that many (when ovs-agent is made slow enough, then all but one) of these networks' services relying on the dhcp port's address were unavailable
* because duplicate address detection (DAD) for the ipv6 dhcp port address failed
* we believe DAD failed because we have some temporary crosstalk between the dhcp namespaces of different networks
* we believe that this bug is not ipv6 specific, only the default DAD of ipv6 helped us discover it
Exact reproduction steps:
$ date --iso-8601=s
2021-05-31T13:10:14+00:00
# when the ovs-agent is slow enough, even 2 networks are sufficient
$ for i in {1..5}
do
openstack network create xnet$i
openstack subnet create xsubnet$i-v6 --ip-version 6 --network xnet$i --subnet-range 2001:db8::/32
done
# for the record
$ openstack subnet list -f value -c Name -c ID | egrep xsubnet
01d614da-820b-418d-8fa4-71952713f0ad xsubnet5-v6
72158e8e-5059-4abb-98a4-5adc9e4ef39c xsubnet2-v6
8f263143-a69b-4c42-b74c-6f30aca7b19d xsubnet4-v6
9ab4159e-12f8-44ed-8947-35a56b62eaf8 xsubnet1-v6
d4ed53e2-7b70-43d7-bd9f-d45f006a8179 xsubnet3-v6
# note that all dhcp ports got the same ip
$ openstack port list --device-owner network:dhcp -f value -c id -c mac_address -c fixed_ips | egrep 2001:db8::
130855be-ead1-40bb-9ca0-5336428aa74b fa:16:3e:24:76:41 [{'subnet_id': '01d614da-820b-418d-8fa4-71952713f0ad', 'ip_address': '2001:db8::1'}]
19fcabfd-f32a-40ea-b68e-ced41f394822 fa:16:3e:43:80:fe [{'subnet_id': '9ab4159e-12f8-44ed-8947-35a56b62eaf8', 'ip_address': '2001:db8::1'}]
46963dbd-c844-4986-a07f-fb78adbd95e9 fa:16:3e:4f:23:bf [{'subnet_id': '72158e8e-5059-4abb-98a4-5adc9e4ef39c', 'ip_address': '2001:db8::1'}]
b8bf2fb1-d52a-41af-90bb-01aa23529015 fa:16:3e:90:40:8e [{'subnet_id': 'd4ed53e2-7b70-43d7-bd9f-d45f006a8179', 'ip_address': '2001:db8::1'}]
ba67f2c0-c714-45fd-aec8-7233ba379dfa fa:16:3e:35:10:8d [{'subnet_id': '8f263143-a69b-4c42-b74c-6f30aca7b19d', 'ip_address': '2001:db8::1'}]
# all but one dhcp port (and by the way metadata) addresses are in DAD failed mode
$ for net in $( openstack network list -f value -c Name -c ID | awk '/ xnet/ { print $1 }' ) ; do sudo ip netns exec qdhcp-$net ip a ; done | egrep '(link/ether|inet6 (2001:db8::|fe80::a9fe:a9fe))'
link/ether fa:16:3e:90:40:8e brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:4f:23:bf brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:24:76:41 brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:35:10:8d brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:43:80:fe brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global·
inet6 fe80::a9fe:a9fe/64 scope link·
# dmesg also shows the DAD failures
# man dmesg: Be aware that the timestamp could be inaccurate!
$ LC_TIME=en_US sudo dmesg -T
[snip]
[Mon May 31 13:10:10 2021] device tap19fcabfd-f3 entered promiscuous mode
[Mon May 31 13:10:15 2021] device tap46963dbd-c8 entered promiscuous mode
[Mon May 31 13:10:15 2021] IPv6: tap46963dbd-c8: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:15 2021] IPv6: tap46963dbd-c8: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:18 2021] device tapb8bf2fb1-d5 entered promiscuous mode
[Mon May 31 13:10:20 2021] IPv6: tapb8bf2fb1-d5: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:20 2021] IPv6: tapb8bf2fb1-d5: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:22 2021] device tapba67f2c0-c7 entered promiscuous mode
[Mon May 31 13:10:23 2021] IPv6: tapba67f2c0-c7: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:24 2021] IPv6: tapba67f2c0-c7: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:26 2021] device tap130855be-ea entered promiscuous mode
[Mon May 31 13:10:27 2021] IPv6: tap130855be-ea: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:28 2021] IPv6: tap130855be-ea: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
# while there were no errors in ovs-agent
$ sudo LC_TIME=en_US journalctl -u devstack@q-agt -S '2021-05-31 13:10:14' | egrep ERROR
[empty]
# clean up
$ openstack network list | awk '/xnet/ { print $2 }' | xargs -r openstack network delete
In a bit of further analysis I only created two networks and ran tcpdump on the first's dhcp port, while creating the second:
$ openstack network create xnet0
$ openstack subnet create xsubnet0-v6 --ip-version 6 --network xnet0 --subnet-range 2001:db8::/32
# run tcpdump on the 1st net's dhcp port
$ sudo ip netns exec qdhcp-$( openstack network show -f value -c id xnet0 ) tcpdump -n -vvv -i tapcaa92c34-53
# create the 2nd net while tcpdump is already running
$ openstack network create xnet1
$ openstack subnet create xsubnet1-v6 --ip-version 6 --network xnet1 --subnet-range 2001:db8::/32
# the 2nd net's dhcp port's mac address
$ openstack port list --device-owner network:dhcp --network xnet1 -f value -c mac_address
fa:16:3e:0d:be:aa
# tcpdump's capture contains packets with the 2nd net's dhcp port's mac address
tcpdump: listening on tapcaa92c34-53, link-type EN10MB (Ethernet), capture size 262144 bytes
^C14:24:14.541893 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 90: (hlim 1, next-header Options (0) payload length: 36) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 1 group record(s) [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.929686 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 110: (hlim 1, next-header Options (0) payload length: 56) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 2 group record(s) [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.985673 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.993930 fa:16:3e:0d:be:aa > 33:33:ff:0d:be:aa, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff0d:beaa: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::f816:3eff:fe0d:beaa
unknown option (14), length 8 (1):·
0x0000: bfdb db94 d695
14:24:15.209668 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:15.858999 fa:16:3e:0d:be:aa > 33:33:ff:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8::1
unknown option (14), length 8 (1):·
0x0000: 6e67 6a99 e72e
14:24:15.859815 fa:16:3e:6a:ed:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:db8::1 > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2001:db8::1, Flags [override]
destination link-address option (2), length 8 (1): fa:16:3e:6a:ed:cb
14:24:15.860188 fa:16:3e:0d:be:aa > 33:33:ff:fe:a9:fe, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:fffe:a9fe: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::a9fe:a9fe
unknown option (14), length 8 (1):·
0x0000: 11e1 9aab 157a
14:24:15.860731 fa:16:3e:6a:ed:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::a9fe:a9fe > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is fe80::a9fe:a9fe, Flags [override]
destination link-address option (2), length 8 (1): fa:16:3e:6a:ed:cb
14:24:16.017855 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) fe80::f816:3eff:fe0d:beaa > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:16.137879 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) fe80::f816:3eff:fe0d:beaa > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
11 packets captured
11 packets received by filter
0 packets dropped by kernel
At the moment I did not test yet:
* whether the leaked traffic can be intercepted on a vm port
* whether a vm port can similarly leak traffic
* whether ovs-agent can be attacked to intentionally slow it down |
This issue is being treated as a potential security risk under
embargo. Please do not make any public mention of embargoed
(private) security vulnerabilities before their coordinated
publication by the OpenStack Vulnerability Management Team in the
form of an official OpenStack Security Advisory. This includes
discussion of the bug or associated fixes in public forums such as
mailing lists, code review systems and bug trackers. Please also
avoid private disclosure to other individuals not already approved
for access to this information, and provide this same reminder to
those who are made aware of the issue prior to publication. All
discussion should remain confined to this private bug report, and
any proposed fixes should be added to the bug as attachments. This
embargo shall not extend past 2021-08-30 and will be made
public by or on that date even if no fix is identified.
This is a bug with potential security implications. I don't see a clear way to exploit it at the moment, but to err on the safe side, I'm opening this as private to the security team.
Short summary: Using openvswitch-agent, traffic sent on some (at least dhcp) ports before ovs-agent applies the port's vlan tag can be seen and intercepted on ports from other networks on the same integration bridge.
We observed this bug:
* using vlan and vxlan networks
* using the noop and openvswitch firewall drivers
* on openstack versions mitaka, pike and master (commit 5a6f61af4a)
The time window between the port's creation and ovs-agent applying its vlan tag is usually very short. We observed this bug in the wild on a heavily loaded host. However to make the reproduction reliable on lightly loaded systems I inserted a sleep() into ovs-agent's source (just before the port's vlan tag is set):
$ git --no-pager format-patch --stdout 5a6f61af4a
From 8389b3e8e5c60c81ff2bb262e3ae2e8aab73d3f5 Mon Sep 17 00:00:00 2001
From: Bence Romsics <bence.romsics@gmail.com>
Date: Mon, 31 May 2021 13:12:34 +0200
Subject: [PATCH] WIP
Change-Id: Ibef4278a2f6a85f52a8ffa43caef6de38cbb40cb
---
.../plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
index 2c209bd387..355584b325 100644
--- a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
+++ b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
@@ -1190,6 +1190,7 @@ class OVSNeutronAgent(l2population_rpc.L2populationRpcCallBackTunnelMixin,
self.setup_arp_spoofing_protection(self.int_br,
port, port_detail)
if cur_tag != lvm.vlan:
+ time.sleep(3000)
self.int_br.set_db_attribute(
"Port", port.port_name, "tag", lvm.vlan)
·
--·
2.31.0
We discovered the bug by the following procedure:
* a test environment created multiple neutron nets in short succession with the exact same ipv6 subnet ranges
* therefore neutron selected the exact same ipv6 address for the subnet's dhcp port
* the host running the dhcp-agent and the ovs-agent was heavily loaded
* we observed that many (when ovs-agent is made slow enough, then all but one) of these networks' services relying on the dhcp port's address were unavailable
* because duplicate address detection (DAD) for the ipv6 dhcp port address failed
* we believe DAD failed because we have some temporary crosstalk between the dhcp namespaces of different networks
* we believe that this bug is not ipv6 specific, only the default DAD of ipv6 helped us discover it
Exact reproduction steps:
$ date --iso-8601=s
2021-05-31T13:10:14+00:00
# when the ovs-agent is slow enough, even 2 networks are sufficient
$ for i in {1..5}
do
openstack network create xnet$i
openstack subnet create xsubnet$i-v6 --ip-version 6 --network xnet$i --subnet-range 2001:db8::/32
done
# for the record
$ openstack subnet list -f value -c Name -c ID | egrep xsubnet
01d614da-820b-418d-8fa4-71952713f0ad xsubnet5-v6
72158e8e-5059-4abb-98a4-5adc9e4ef39c xsubnet2-v6
8f263143-a69b-4c42-b74c-6f30aca7b19d xsubnet4-v6
9ab4159e-12f8-44ed-8947-35a56b62eaf8 xsubnet1-v6
d4ed53e2-7b70-43d7-bd9f-d45f006a8179 xsubnet3-v6
# note that all dhcp ports got the same ip
$ openstack port list --device-owner network:dhcp -f value -c id -c mac_address -c fixed_ips | egrep 2001:db8::
130855be-ead1-40bb-9ca0-5336428aa74b fa:16:3e:24:76:41 [{'subnet_id': '01d614da-820b-418d-8fa4-71952713f0ad', 'ip_address': '2001:db8::1'}]
19fcabfd-f32a-40ea-b68e-ced41f394822 fa:16:3e:43:80:fe [{'subnet_id': '9ab4159e-12f8-44ed-8947-35a56b62eaf8', 'ip_address': '2001:db8::1'}]
46963dbd-c844-4986-a07f-fb78adbd95e9 fa:16:3e:4f:23:bf [{'subnet_id': '72158e8e-5059-4abb-98a4-5adc9e4ef39c', 'ip_address': '2001:db8::1'}]
b8bf2fb1-d52a-41af-90bb-01aa23529015 fa:16:3e:90:40:8e [{'subnet_id': 'd4ed53e2-7b70-43d7-bd9f-d45f006a8179', 'ip_address': '2001:db8::1'}]
ba67f2c0-c714-45fd-aec8-7233ba379dfa fa:16:3e:35:10:8d [{'subnet_id': '8f263143-a69b-4c42-b74c-6f30aca7b19d', 'ip_address': '2001:db8::1'}]
# all but one dhcp port (and by the way metadata) addresses are in DAD failed mode
$ for net in $( openstack network list -f value -c Name -c ID | awk '/ xnet/ { print $1 }' ) ; do sudo ip netns exec qdhcp-$net ip a ; done | egrep '(link/ether|inet6 (2001:db8::|fe80::a9fe:a9fe))'
link/ether fa:16:3e:90:40:8e brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:4f:23:bf brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:24:76:41 brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:35:10:8d brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:43:80:fe brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global·
inet6 fe80::a9fe:a9fe/64 scope link·
# dmesg also shows the DAD failures
# man dmesg: Be aware that the timestamp could be inaccurate!
$ LC_TIME=en_US sudo dmesg -T
[snip]
[Mon May 31 13:10:10 2021] device tap19fcabfd-f3 entered promiscuous mode
[Mon May 31 13:10:15 2021] device tap46963dbd-c8 entered promiscuous mode
[Mon May 31 13:10:15 2021] IPv6: tap46963dbd-c8: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:15 2021] IPv6: tap46963dbd-c8: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:18 2021] device tapb8bf2fb1-d5 entered promiscuous mode
[Mon May 31 13:10:20 2021] IPv6: tapb8bf2fb1-d5: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:20 2021] IPv6: tapb8bf2fb1-d5: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:22 2021] device tapba67f2c0-c7 entered promiscuous mode
[Mon May 31 13:10:23 2021] IPv6: tapba67f2c0-c7: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:24 2021] IPv6: tapba67f2c0-c7: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:26 2021] device tap130855be-ea entered promiscuous mode
[Mon May 31 13:10:27 2021] IPv6: tap130855be-ea: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:28 2021] IPv6: tap130855be-ea: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
# while there were no errors in ovs-agent
$ sudo LC_TIME=en_US journalctl -u devstack@q-agt -S '2021-05-31 13:10:14' | egrep ERROR
[empty]
# clean up
$ openstack network list | awk '/xnet/ { print $2 }' | xargs -r openstack network delete
In a bit of further analysis I only created two networks and ran tcpdump on the first's dhcp port, while creating the second:
$ openstack network create xnet0
$ openstack subnet create xsubnet0-v6 --ip-version 6 --network xnet0 --subnet-range 2001:db8::/32
# run tcpdump on the 1st net's dhcp port
$ sudo ip netns exec qdhcp-$( openstack network show -f value -c id xnet0 ) tcpdump -n -vvv -i tapcaa92c34-53
# create the 2nd net while tcpdump is already running
$ openstack network create xnet1
$ openstack subnet create xsubnet1-v6 --ip-version 6 --network xnet1 --subnet-range 2001:db8::/32
# the 2nd net's dhcp port's mac address
$ openstack port list --device-owner network:dhcp --network xnet1 -f value -c mac_address
fa:16:3e:0d:be:aa
# tcpdump's capture contains packets with the 2nd net's dhcp port's mac address
tcpdump: listening on tapcaa92c34-53, link-type EN10MB (Ethernet), capture size 262144 bytes
^C14:24:14.541893 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 90: (hlim 1, next-header Options (0) payload length: 36) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 1 group record(s) [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.929686 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 110: (hlim 1, next-header Options (0) payload length: 56) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 2 group record(s) [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.985673 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.993930 fa:16:3e:0d:be:aa > 33:33:ff:0d:be:aa, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff0d:beaa: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::f816:3eff:fe0d:beaa
unknown option (14), length 8 (1):·
0x0000: bfdb db94 d695
14:24:15.209668 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:15.858999 fa:16:3e:0d:be:aa > 33:33:ff:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8::1
unknown option (14), length 8 (1):·
0x0000: 6e67 6a99 e72e
14:24:15.859815 fa:16:3e:6a:ed:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:db8::1 > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2001:db8::1, Flags [override]
destination link-address option (2), length 8 (1): fa:16:3e:6a:ed:cb
14:24:15.860188 fa:16:3e:0d:be:aa > 33:33:ff:fe:a9:fe, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:fffe:a9fe: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::a9fe:a9fe
unknown option (14), length 8 (1):·
0x0000: 11e1 9aab 157a
14:24:15.860731 fa:16:3e:6a:ed:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::a9fe:a9fe > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is fe80::a9fe:a9fe, Flags [override]
destination link-address option (2), length 8 (1): fa:16:3e:6a:ed:cb
14:24:16.017855 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) fe80::f816:3eff:fe0d:beaa > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:16.137879 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) fe80::f816:3eff:fe0d:beaa > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
11 packets captured
11 packets received by filter
0 packets dropped by kernel
At the moment I did not test yet:
* whether the leaked traffic can be intercepted on a vm port
* whether a vm port can similarly leak traffic
* whether ovs-agent can be attacked to intentionally slow it down |
|
| 2021-06-01 17:18:44 |
Jeremy Stanley |
bug task added |
|
ossa |
|
| 2021-06-01 17:18:54 |
Jeremy Stanley |
ossa: status |
New |
Incomplete |
|
| 2021-06-01 17:19:20 |
Jeremy Stanley |
bug |
|
|
added subscriber Neutron Core Security reviewers |
| 2021-06-08 07:20:40 |
Bence Romsics |
neutron: assignee |
|
Bence Romsics (bence-romsics) |
|
| 2021-07-06 14:52:49 |
Jeremy Stanley |
description |
This issue is being treated as a potential security risk under
embargo. Please do not make any public mention of embargoed
(private) security vulnerabilities before their coordinated
publication by the OpenStack Vulnerability Management Team in the
form of an official OpenStack Security Advisory. This includes
discussion of the bug or associated fixes in public forums such as
mailing lists, code review systems and bug trackers. Please also
avoid private disclosure to other individuals not already approved
for access to this information, and provide this same reminder to
those who are made aware of the issue prior to publication. All
discussion should remain confined to this private bug report, and
any proposed fixes should be added to the bug as attachments. This
embargo shall not extend past 2021-08-30 and will be made
public by or on that date even if no fix is identified.
This is a bug with potential security implications. I don't see a clear way to exploit it at the moment, but to err on the safe side, I'm opening this as private to the security team.
Short summary: Using openvswitch-agent, traffic sent on some (at least dhcp) ports before ovs-agent applies the port's vlan tag can be seen and intercepted on ports from other networks on the same integration bridge.
We observed this bug:
* using vlan and vxlan networks
* using the noop and openvswitch firewall drivers
* on openstack versions mitaka, pike and master (commit 5a6f61af4a)
The time window between the port's creation and ovs-agent applying its vlan tag is usually very short. We observed this bug in the wild on a heavily loaded host. However to make the reproduction reliable on lightly loaded systems I inserted a sleep() into ovs-agent's source (just before the port's vlan tag is set):
$ git --no-pager format-patch --stdout 5a6f61af4a
From 8389b3e8e5c60c81ff2bb262e3ae2e8aab73d3f5 Mon Sep 17 00:00:00 2001
From: Bence Romsics <bence.romsics@gmail.com>
Date: Mon, 31 May 2021 13:12:34 +0200
Subject: [PATCH] WIP
Change-Id: Ibef4278a2f6a85f52a8ffa43caef6de38cbb40cb
---
.../plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
index 2c209bd387..355584b325 100644
--- a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
+++ b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
@@ -1190,6 +1190,7 @@ class OVSNeutronAgent(l2population_rpc.L2populationRpcCallBackTunnelMixin,
self.setup_arp_spoofing_protection(self.int_br,
port, port_detail)
if cur_tag != lvm.vlan:
+ time.sleep(3000)
self.int_br.set_db_attribute(
"Port", port.port_name, "tag", lvm.vlan)
·
--·
2.31.0
We discovered the bug by the following procedure:
* a test environment created multiple neutron nets in short succession with the exact same ipv6 subnet ranges
* therefore neutron selected the exact same ipv6 address for the subnet's dhcp port
* the host running the dhcp-agent and the ovs-agent was heavily loaded
* we observed that many (when ovs-agent is made slow enough, then all but one) of these networks' services relying on the dhcp port's address were unavailable
* because duplicate address detection (DAD) for the ipv6 dhcp port address failed
* we believe DAD failed because we have some temporary crosstalk between the dhcp namespaces of different networks
* we believe that this bug is not ipv6 specific, only the default DAD of ipv6 helped us discover it
Exact reproduction steps:
$ date --iso-8601=s
2021-05-31T13:10:14+00:00
# when the ovs-agent is slow enough, even 2 networks are sufficient
$ for i in {1..5}
do
openstack network create xnet$i
openstack subnet create xsubnet$i-v6 --ip-version 6 --network xnet$i --subnet-range 2001:db8::/32
done
# for the record
$ openstack subnet list -f value -c Name -c ID | egrep xsubnet
01d614da-820b-418d-8fa4-71952713f0ad xsubnet5-v6
72158e8e-5059-4abb-98a4-5adc9e4ef39c xsubnet2-v6
8f263143-a69b-4c42-b74c-6f30aca7b19d xsubnet4-v6
9ab4159e-12f8-44ed-8947-35a56b62eaf8 xsubnet1-v6
d4ed53e2-7b70-43d7-bd9f-d45f006a8179 xsubnet3-v6
# note that all dhcp ports got the same ip
$ openstack port list --device-owner network:dhcp -f value -c id -c mac_address -c fixed_ips | egrep 2001:db8::
130855be-ead1-40bb-9ca0-5336428aa74b fa:16:3e:24:76:41 [{'subnet_id': '01d614da-820b-418d-8fa4-71952713f0ad', 'ip_address': '2001:db8::1'}]
19fcabfd-f32a-40ea-b68e-ced41f394822 fa:16:3e:43:80:fe [{'subnet_id': '9ab4159e-12f8-44ed-8947-35a56b62eaf8', 'ip_address': '2001:db8::1'}]
46963dbd-c844-4986-a07f-fb78adbd95e9 fa:16:3e:4f:23:bf [{'subnet_id': '72158e8e-5059-4abb-98a4-5adc9e4ef39c', 'ip_address': '2001:db8::1'}]
b8bf2fb1-d52a-41af-90bb-01aa23529015 fa:16:3e:90:40:8e [{'subnet_id': 'd4ed53e2-7b70-43d7-bd9f-d45f006a8179', 'ip_address': '2001:db8::1'}]
ba67f2c0-c714-45fd-aec8-7233ba379dfa fa:16:3e:35:10:8d [{'subnet_id': '8f263143-a69b-4c42-b74c-6f30aca7b19d', 'ip_address': '2001:db8::1'}]
# all but one dhcp port (and by the way metadata) addresses are in DAD failed mode
$ for net in $( openstack network list -f value -c Name -c ID | awk '/ xnet/ { print $1 }' ) ; do sudo ip netns exec qdhcp-$net ip a ; done | egrep '(link/ether|inet6 (2001:db8::|fe80::a9fe:a9fe))'
link/ether fa:16:3e:90:40:8e brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:4f:23:bf brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:24:76:41 brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:35:10:8d brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:43:80:fe brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global·
inet6 fe80::a9fe:a9fe/64 scope link·
# dmesg also shows the DAD failures
# man dmesg: Be aware that the timestamp could be inaccurate!
$ LC_TIME=en_US sudo dmesg -T
[snip]
[Mon May 31 13:10:10 2021] device tap19fcabfd-f3 entered promiscuous mode
[Mon May 31 13:10:15 2021] device tap46963dbd-c8 entered promiscuous mode
[Mon May 31 13:10:15 2021] IPv6: tap46963dbd-c8: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:15 2021] IPv6: tap46963dbd-c8: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:18 2021] device tapb8bf2fb1-d5 entered promiscuous mode
[Mon May 31 13:10:20 2021] IPv6: tapb8bf2fb1-d5: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:20 2021] IPv6: tapb8bf2fb1-d5: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:22 2021] device tapba67f2c0-c7 entered promiscuous mode
[Mon May 31 13:10:23 2021] IPv6: tapba67f2c0-c7: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:24 2021] IPv6: tapba67f2c0-c7: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:26 2021] device tap130855be-ea entered promiscuous mode
[Mon May 31 13:10:27 2021] IPv6: tap130855be-ea: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:28 2021] IPv6: tap130855be-ea: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
# while there were no errors in ovs-agent
$ sudo LC_TIME=en_US journalctl -u devstack@q-agt -S '2021-05-31 13:10:14' | egrep ERROR
[empty]
# clean up
$ openstack network list | awk '/xnet/ { print $2 }' | xargs -r openstack network delete
In a bit of further analysis I only created two networks and ran tcpdump on the first's dhcp port, while creating the second:
$ openstack network create xnet0
$ openstack subnet create xsubnet0-v6 --ip-version 6 --network xnet0 --subnet-range 2001:db8::/32
# run tcpdump on the 1st net's dhcp port
$ sudo ip netns exec qdhcp-$( openstack network show -f value -c id xnet0 ) tcpdump -n -vvv -i tapcaa92c34-53
# create the 2nd net while tcpdump is already running
$ openstack network create xnet1
$ openstack subnet create xsubnet1-v6 --ip-version 6 --network xnet1 --subnet-range 2001:db8::/32
# the 2nd net's dhcp port's mac address
$ openstack port list --device-owner network:dhcp --network xnet1 -f value -c mac_address
fa:16:3e:0d:be:aa
# tcpdump's capture contains packets with the 2nd net's dhcp port's mac address
tcpdump: listening on tapcaa92c34-53, link-type EN10MB (Ethernet), capture size 262144 bytes
^C14:24:14.541893 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 90: (hlim 1, next-header Options (0) payload length: 36) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 1 group record(s) [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.929686 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 110: (hlim 1, next-header Options (0) payload length: 56) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 2 group record(s) [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.985673 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.993930 fa:16:3e:0d:be:aa > 33:33:ff:0d:be:aa, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff0d:beaa: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::f816:3eff:fe0d:beaa
unknown option (14), length 8 (1):·
0x0000: bfdb db94 d695
14:24:15.209668 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:15.858999 fa:16:3e:0d:be:aa > 33:33:ff:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8::1
unknown option (14), length 8 (1):·
0x0000: 6e67 6a99 e72e
14:24:15.859815 fa:16:3e:6a:ed:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:db8::1 > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2001:db8::1, Flags [override]
destination link-address option (2), length 8 (1): fa:16:3e:6a:ed:cb
14:24:15.860188 fa:16:3e:0d:be:aa > 33:33:ff:fe:a9:fe, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:fffe:a9fe: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::a9fe:a9fe
unknown option (14), length 8 (1):·
0x0000: 11e1 9aab 157a
14:24:15.860731 fa:16:3e:6a:ed:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::a9fe:a9fe > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is fe80::a9fe:a9fe, Flags [override]
destination link-address option (2), length 8 (1): fa:16:3e:6a:ed:cb
14:24:16.017855 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) fe80::f816:3eff:fe0d:beaa > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:16.137879 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) fe80::f816:3eff:fe0d:beaa > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
11 packets captured
11 packets received by filter
0 packets dropped by kernel
At the moment I did not test yet:
* whether the leaked traffic can be intercepted on a vm port
* whether a vm port can similarly leak traffic
* whether ovs-agent can be attacked to intentionally slow it down |
This is a bug with potential security implications. I don't see a clear way to exploit it at the moment, but to err on the safe side, I'm opening this as private to the security team.
Short summary: Using openvswitch-agent, traffic sent on some (at least dhcp) ports before ovs-agent applies the port's vlan tag can be seen and intercepted on ports from other networks on the same integration bridge.
We observed this bug:
* using vlan and vxlan networks
* using the noop and openvswitch firewall drivers
* on openstack versions mitaka, pike and master (commit 5a6f61af4a)
The time window between the port's creation and ovs-agent applying its vlan tag is usually very short. We observed this bug in the wild on a heavily loaded host. However to make the reproduction reliable on lightly loaded systems I inserted a sleep() into ovs-agent's source (just before the port's vlan tag is set):
$ git --no-pager format-patch --stdout 5a6f61af4a
From 8389b3e8e5c60c81ff2bb262e3ae2e8aab73d3f5 Mon Sep 17 00:00:00 2001
From: Bence Romsics <bence.romsics@gmail.com>
Date: Mon, 31 May 2021 13:12:34 +0200
Subject: [PATCH] WIP
Change-Id: Ibef4278a2f6a85f52a8ffa43caef6de38cbb40cb
---
.../plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
index 2c209bd387..355584b325 100644
--- a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
+++ b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
@@ -1190,6 +1190,7 @@ class OVSNeutronAgent(l2population_rpc.L2populationRpcCallBackTunnelMixin,
self.setup_arp_spoofing_protection(self.int_br,
port, port_detail)
if cur_tag != lvm.vlan:
+ time.sleep(3000)
self.int_br.set_db_attribute(
"Port", port.port_name, "tag", lvm.vlan)
·
--·
2.31.0
We discovered the bug by the following procedure:
* a test environment created multiple neutron nets in short succession with the exact same ipv6 subnet ranges
* therefore neutron selected the exact same ipv6 address for the subnet's dhcp port
* the host running the dhcp-agent and the ovs-agent was heavily loaded
* we observed that many (when ovs-agent is made slow enough, then all but one) of these networks' services relying on the dhcp port's address were unavailable
* because duplicate address detection (DAD) for the ipv6 dhcp port address failed
* we believe DAD failed because we have some temporary crosstalk between the dhcp namespaces of different networks
* we believe that this bug is not ipv6 specific, only the default DAD of ipv6 helped us discover it
Exact reproduction steps:
$ date --iso-8601=s
2021-05-31T13:10:14+00:00
# when the ovs-agent is slow enough, even 2 networks are sufficient
$ for i in {1..5}
do
openstack network create xnet$i
openstack subnet create xsubnet$i-v6 --ip-version 6 --network xnet$i --subnet-range 2001:db8::/32
done
# for the record
$ openstack subnet list -f value -c Name -c ID | egrep xsubnet
01d614da-820b-418d-8fa4-71952713f0ad xsubnet5-v6
72158e8e-5059-4abb-98a4-5adc9e4ef39c xsubnet2-v6
8f263143-a69b-4c42-b74c-6f30aca7b19d xsubnet4-v6
9ab4159e-12f8-44ed-8947-35a56b62eaf8 xsubnet1-v6
d4ed53e2-7b70-43d7-bd9f-d45f006a8179 xsubnet3-v6
# note that all dhcp ports got the same ip
$ openstack port list --device-owner network:dhcp -f value -c id -c mac_address -c fixed_ips | egrep 2001:db8::
130855be-ead1-40bb-9ca0-5336428aa74b fa:16:3e:24:76:41 [{'subnet_id': '01d614da-820b-418d-8fa4-71952713f0ad', 'ip_address': '2001:db8::1'}]
19fcabfd-f32a-40ea-b68e-ced41f394822 fa:16:3e:43:80:fe [{'subnet_id': '9ab4159e-12f8-44ed-8947-35a56b62eaf8', 'ip_address': '2001:db8::1'}]
46963dbd-c844-4986-a07f-fb78adbd95e9 fa:16:3e:4f:23:bf [{'subnet_id': '72158e8e-5059-4abb-98a4-5adc9e4ef39c', 'ip_address': '2001:db8::1'}]
b8bf2fb1-d52a-41af-90bb-01aa23529015 fa:16:3e:90:40:8e [{'subnet_id': 'd4ed53e2-7b70-43d7-bd9f-d45f006a8179', 'ip_address': '2001:db8::1'}]
ba67f2c0-c714-45fd-aec8-7233ba379dfa fa:16:3e:35:10:8d [{'subnet_id': '8f263143-a69b-4c42-b74c-6f30aca7b19d', 'ip_address': '2001:db8::1'}]
# all but one dhcp port (and by the way metadata) addresses are in DAD failed mode
$ for net in $( openstack network list -f value -c Name -c ID | awk '/ xnet/ { print $1 }' ) ; do sudo ip netns exec qdhcp-$net ip a ; done | egrep '(link/ether|inet6 (2001:db8::|fe80::a9fe:a9fe))'
link/ether fa:16:3e:90:40:8e brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:4f:23:bf brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:24:76:41 brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:35:10:8d brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:43:80:fe brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global·
inet6 fe80::a9fe:a9fe/64 scope link·
# dmesg also shows the DAD failures
# man dmesg: Be aware that the timestamp could be inaccurate!
$ LC_TIME=en_US sudo dmesg -T
[snip]
[Mon May 31 13:10:10 2021] device tap19fcabfd-f3 entered promiscuous mode
[Mon May 31 13:10:15 2021] device tap46963dbd-c8 entered promiscuous mode
[Mon May 31 13:10:15 2021] IPv6: tap46963dbd-c8: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:15 2021] IPv6: tap46963dbd-c8: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:18 2021] device tapb8bf2fb1-d5 entered promiscuous mode
[Mon May 31 13:10:20 2021] IPv6: tapb8bf2fb1-d5: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:20 2021] IPv6: tapb8bf2fb1-d5: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:22 2021] device tapba67f2c0-c7 entered promiscuous mode
[Mon May 31 13:10:23 2021] IPv6: tapba67f2c0-c7: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:24 2021] IPv6: tapba67f2c0-c7: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:26 2021] device tap130855be-ea entered promiscuous mode
[Mon May 31 13:10:27 2021] IPv6: tap130855be-ea: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:28 2021] IPv6: tap130855be-ea: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
# while there were no errors in ovs-agent
$ sudo LC_TIME=en_US journalctl -u devstack@q-agt -S '2021-05-31 13:10:14' | egrep ERROR
[empty]
# clean up
$ openstack network list | awk '/xnet/ { print $2 }' | xargs -r openstack network delete
In a bit of further analysis I only created two networks and ran tcpdump on the first's dhcp port, while creating the second:
$ openstack network create xnet0
$ openstack subnet create xsubnet0-v6 --ip-version 6 --network xnet0 --subnet-range 2001:db8::/32
# run tcpdump on the 1st net's dhcp port
$ sudo ip netns exec qdhcp-$( openstack network show -f value -c id xnet0 ) tcpdump -n -vvv -i tapcaa92c34-53
# create the 2nd net while tcpdump is already running
$ openstack network create xnet1
$ openstack subnet create xsubnet1-v6 --ip-version 6 --network xnet1 --subnet-range 2001:db8::/32
# the 2nd net's dhcp port's mac address
$ openstack port list --device-owner network:dhcp --network xnet1 -f value -c mac_address
fa:16:3e:0d:be:aa
# tcpdump's capture contains packets with the 2nd net's dhcp port's mac address
tcpdump: listening on tapcaa92c34-53, link-type EN10MB (Ethernet), capture size 262144 bytes
^C14:24:14.541893 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 90: (hlim 1, next-header Options (0) payload length: 36) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 1 group record(s) [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.929686 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 110: (hlim 1, next-header Options (0) payload length: 56) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 2 group record(s) [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.985673 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.993930 fa:16:3e:0d:be:aa > 33:33:ff:0d:be:aa, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff0d:beaa: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::f816:3eff:fe0d:beaa
unknown option (14), length 8 (1):·
0x0000: bfdb db94 d695
14:24:15.209668 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:15.858999 fa:16:3e:0d:be:aa > 33:33:ff:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8::1
unknown option (14), length 8 (1):·
0x0000: 6e67 6a99 e72e
14:24:15.859815 fa:16:3e:6a:ed:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:db8::1 > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2001:db8::1, Flags [override]
destination link-address option (2), length 8 (1): fa:16:3e:6a:ed:cb
14:24:15.860188 fa:16:3e:0d:be:aa > 33:33:ff:fe:a9:fe, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:fffe:a9fe: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::a9fe:a9fe
unknown option (14), length 8 (1):·
0x0000: 11e1 9aab 157a
14:24:15.860731 fa:16:3e:6a:ed:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::a9fe:a9fe > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is fe80::a9fe:a9fe, Flags [override]
destination link-address option (2), length 8 (1): fa:16:3e:6a:ed:cb
14:24:16.017855 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) fe80::f816:3eff:fe0d:beaa > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:16.137879 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) fe80::f816:3eff:fe0d:beaa > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
11 packets captured
11 packets received by filter
0 packets dropped by kernel
At the moment I did not test yet:
* whether the leaked traffic can be intercepted on a vm port
* whether a vm port can similarly leak traffic
* whether ovs-agent can be attacked to intentionally slow it down |
|
| 2021-07-06 14:52:55 |
Jeremy Stanley |
information type |
Private Security |
Public Security |
|
| 2021-09-22 12:11:23 |
Bence Romsics |
attachment added |
|
vm0.xml https://bugs.launchpad.net/neutron/+bug/1930414/+attachment/5526974/+files/vm0.xml |
|
| 2021-09-22 13:52:24 |
Jeremy Stanley |
information type |
Public Security |
Public |
|
| 2021-09-22 13:52:31 |
Jeremy Stanley |
ossa: status |
Incomplete |
Won't Fix |
|
| 2021-09-22 13:52:47 |
Jeremy Stanley |
tags |
|
security |
|
| 2021-09-23 07:34:40 |
Bernard Cafarelli |
neutron: importance |
Undecided |
High |
|
| 2021-12-07 14:55:49 |
Bence Romsics |
attachment added |
|
vm0-dpdk.xml https://bugs.launchpad.net/neutron/+bug/1930414/+attachment/5546011/+files/vm0-dpdk.xml |
|
| 2021-12-07 15:02:30 |
OpenStack Infra |
neutron: status |
New |
In Progress |
|
| 2022-01-03 14:08:20 |
Kamil Madac |
bug |
|
|
added subscriber Kamil Madac |
| 2022-01-19 23:13:26 |
OpenStack Infra |
neutron: status |
In Progress |
Fix Released |
|
| 2022-01-20 09:08:50 |
OpenStack Infra |
tags |
security |
in-stable-xena security |
|
| 2022-01-24 22:58:19 |
OpenStack Infra |
tags |
in-stable-xena security |
in-stable-wallaby in-stable-xena security |
|
| 2022-02-02 15:41:58 |
Bence Romsics |
description |
This is a bug with potential security implications. I don't see a clear way to exploit it at the moment, but to err on the safe side, I'm opening this as private to the security team.
Short summary: Using openvswitch-agent, traffic sent on some (at least dhcp) ports before ovs-agent applies the port's vlan tag can be seen and intercepted on ports from other networks on the same integration bridge.
We observed this bug:
* using vlan and vxlan networks
* using the noop and openvswitch firewall drivers
* on openstack versions mitaka, pike and master (commit 5a6f61af4a)
The time window between the port's creation and ovs-agent applying its vlan tag is usually very short. We observed this bug in the wild on a heavily loaded host. However to make the reproduction reliable on lightly loaded systems I inserted a sleep() into ovs-agent's source (just before the port's vlan tag is set):
$ git --no-pager format-patch --stdout 5a6f61af4a
From 8389b3e8e5c60c81ff2bb262e3ae2e8aab73d3f5 Mon Sep 17 00:00:00 2001
From: Bence Romsics <bence.romsics@gmail.com>
Date: Mon, 31 May 2021 13:12:34 +0200
Subject: [PATCH] WIP
Change-Id: Ibef4278a2f6a85f52a8ffa43caef6de38cbb40cb
---
.../plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
index 2c209bd387..355584b325 100644
--- a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
+++ b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
@@ -1190,6 +1190,7 @@ class OVSNeutronAgent(l2population_rpc.L2populationRpcCallBackTunnelMixin,
self.setup_arp_spoofing_protection(self.int_br,
port, port_detail)
if cur_tag != lvm.vlan:
+ time.sleep(3000)
self.int_br.set_db_attribute(
"Port", port.port_name, "tag", lvm.vlan)
·
--·
2.31.0
We discovered the bug by the following procedure:
* a test environment created multiple neutron nets in short succession with the exact same ipv6 subnet ranges
* therefore neutron selected the exact same ipv6 address for the subnet's dhcp port
* the host running the dhcp-agent and the ovs-agent was heavily loaded
* we observed that many (when ovs-agent is made slow enough, then all but one) of these networks' services relying on the dhcp port's address were unavailable
* because duplicate address detection (DAD) for the ipv6 dhcp port address failed
* we believe DAD failed because we have some temporary crosstalk between the dhcp namespaces of different networks
* we believe that this bug is not ipv6 specific, only the default DAD of ipv6 helped us discover it
Exact reproduction steps:
$ date --iso-8601=s
2021-05-31T13:10:14+00:00
# when the ovs-agent is slow enough, even 2 networks are sufficient
$ for i in {1..5}
do
openstack network create xnet$i
openstack subnet create xsubnet$i-v6 --ip-version 6 --network xnet$i --subnet-range 2001:db8::/32
done
# for the record
$ openstack subnet list -f value -c Name -c ID | egrep xsubnet
01d614da-820b-418d-8fa4-71952713f0ad xsubnet5-v6
72158e8e-5059-4abb-98a4-5adc9e4ef39c xsubnet2-v6
8f263143-a69b-4c42-b74c-6f30aca7b19d xsubnet4-v6
9ab4159e-12f8-44ed-8947-35a56b62eaf8 xsubnet1-v6
d4ed53e2-7b70-43d7-bd9f-d45f006a8179 xsubnet3-v6
# note that all dhcp ports got the same ip
$ openstack port list --device-owner network:dhcp -f value -c id -c mac_address -c fixed_ips | egrep 2001:db8::
130855be-ead1-40bb-9ca0-5336428aa74b fa:16:3e:24:76:41 [{'subnet_id': '01d614da-820b-418d-8fa4-71952713f0ad', 'ip_address': '2001:db8::1'}]
19fcabfd-f32a-40ea-b68e-ced41f394822 fa:16:3e:43:80:fe [{'subnet_id': '9ab4159e-12f8-44ed-8947-35a56b62eaf8', 'ip_address': '2001:db8::1'}]
46963dbd-c844-4986-a07f-fb78adbd95e9 fa:16:3e:4f:23:bf [{'subnet_id': '72158e8e-5059-4abb-98a4-5adc9e4ef39c', 'ip_address': '2001:db8::1'}]
b8bf2fb1-d52a-41af-90bb-01aa23529015 fa:16:3e:90:40:8e [{'subnet_id': 'd4ed53e2-7b70-43d7-bd9f-d45f006a8179', 'ip_address': '2001:db8::1'}]
ba67f2c0-c714-45fd-aec8-7233ba379dfa fa:16:3e:35:10:8d [{'subnet_id': '8f263143-a69b-4c42-b74c-6f30aca7b19d', 'ip_address': '2001:db8::1'}]
# all but one dhcp port (and by the way metadata) addresses are in DAD failed mode
$ for net in $( openstack network list -f value -c Name -c ID | awk '/ xnet/ { print $1 }' ) ; do sudo ip netns exec qdhcp-$net ip a ; done | egrep '(link/ether|inet6 (2001:db8::|fe80::a9fe:a9fe))'
link/ether fa:16:3e:90:40:8e brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:4f:23:bf brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:24:76:41 brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:35:10:8d brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:43:80:fe brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global·
inet6 fe80::a9fe:a9fe/64 scope link·
# dmesg also shows the DAD failures
# man dmesg: Be aware that the timestamp could be inaccurate!
$ LC_TIME=en_US sudo dmesg -T
[snip]
[Mon May 31 13:10:10 2021] device tap19fcabfd-f3 entered promiscuous mode
[Mon May 31 13:10:15 2021] device tap46963dbd-c8 entered promiscuous mode
[Mon May 31 13:10:15 2021] IPv6: tap46963dbd-c8: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:15 2021] IPv6: tap46963dbd-c8: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:18 2021] device tapb8bf2fb1-d5 entered promiscuous mode
[Mon May 31 13:10:20 2021] IPv6: tapb8bf2fb1-d5: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:20 2021] IPv6: tapb8bf2fb1-d5: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:22 2021] device tapba67f2c0-c7 entered promiscuous mode
[Mon May 31 13:10:23 2021] IPv6: tapba67f2c0-c7: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:24 2021] IPv6: tapba67f2c0-c7: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:26 2021] device tap130855be-ea entered promiscuous mode
[Mon May 31 13:10:27 2021] IPv6: tap130855be-ea: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:28 2021] IPv6: tap130855be-ea: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
# while there were no errors in ovs-agent
$ sudo LC_TIME=en_US journalctl -u devstack@q-agt -S '2021-05-31 13:10:14' | egrep ERROR
[empty]
# clean up
$ openstack network list | awk '/xnet/ { print $2 }' | xargs -r openstack network delete
In a bit of further analysis I only created two networks and ran tcpdump on the first's dhcp port, while creating the second:
$ openstack network create xnet0
$ openstack subnet create xsubnet0-v6 --ip-version 6 --network xnet0 --subnet-range 2001:db8::/32
# run tcpdump on the 1st net's dhcp port
$ sudo ip netns exec qdhcp-$( openstack network show -f value -c id xnet0 ) tcpdump -n -vvv -i tapcaa92c34-53
# create the 2nd net while tcpdump is already running
$ openstack network create xnet1
$ openstack subnet create xsubnet1-v6 --ip-version 6 --network xnet1 --subnet-range 2001:db8::/32
# the 2nd net's dhcp port's mac address
$ openstack port list --device-owner network:dhcp --network xnet1 -f value -c mac_address
fa:16:3e:0d:be:aa
# tcpdump's capture contains packets with the 2nd net's dhcp port's mac address
tcpdump: listening on tapcaa92c34-53, link-type EN10MB (Ethernet), capture size 262144 bytes
^C14:24:14.541893 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 90: (hlim 1, next-header Options (0) payload length: 36) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 1 group record(s) [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.929686 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 110: (hlim 1, next-header Options (0) payload length: 56) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 2 group record(s) [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.985673 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.993930 fa:16:3e:0d:be:aa > 33:33:ff:0d:be:aa, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff0d:beaa: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::f816:3eff:fe0d:beaa
unknown option (14), length 8 (1):·
0x0000: bfdb db94 d695
14:24:15.209668 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:15.858999 fa:16:3e:0d:be:aa > 33:33:ff:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8::1
unknown option (14), length 8 (1):·
0x0000: 6e67 6a99 e72e
14:24:15.859815 fa:16:3e:6a:ed:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:db8::1 > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2001:db8::1, Flags [override]
destination link-address option (2), length 8 (1): fa:16:3e:6a:ed:cb
14:24:15.860188 fa:16:3e:0d:be:aa > 33:33:ff:fe:a9:fe, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:fffe:a9fe: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::a9fe:a9fe
unknown option (14), length 8 (1):·
0x0000: 11e1 9aab 157a
14:24:15.860731 fa:16:3e:6a:ed:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::a9fe:a9fe > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is fe80::a9fe:a9fe, Flags [override]
destination link-address option (2), length 8 (1): fa:16:3e:6a:ed:cb
14:24:16.017855 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) fe80::f816:3eff:fe0d:beaa > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:16.137879 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) fe80::f816:3eff:fe0d:beaa > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
11 packets captured
11 packets received by filter
0 packets dropped by kernel
At the moment I did not test yet:
* whether the leaked traffic can be intercepted on a vm port
* whether a vm port can similarly leak traffic
* whether ovs-agent can be attacked to intentionally slow it down |
This is a bug with potential security implications. I don't see a clear way to exploit it at the moment, but to err on the safe side, I'm opening this as private to the security team.
Short summary: Using openvswitch-agent, traffic sent on some (at least dhcp) ports before ovs-agent applies the port's vlan tag can be seen and intercepted on ports from other networks on the same integration bridge.
We observed this bug:
* using vlan and vxlan networks
* using the noop and openvswitch firewall drivers
* on openstack versions mitaka, pike and master (commit 5a6f61af4a)
The time window between the port's creation and ovs-agent applying its vlan tag is usually very short. We observed this bug in the wild on a heavily loaded host. However to make the reproduction reliable on lightly loaded systems I inserted a sleep() into ovs-agent's source (just before the port's vlan tag is set):
$ git --no-pager format-patch --stdout 5a6f61af4a
From 8389b3e8e5c60c81ff2bb262e3ae2e8aab73d3f5 Mon Sep 17 00:00:00 2001
From: Bence Romsics <bence.romsics@gmail.com>
Date: Mon, 31 May 2021 13:12:34 +0200
Subject: [PATCH] WIP
Change-Id: Ibef4278a2f6a85f52a8ffa43caef6de38cbb40cb
---
.../plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
index 2c209bd387..355584b325 100644
--- a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
+++ b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
@@ -1190,6 +1190,7 @@ class OVSNeutronAgent(l2population_rpc.L2populationRpcCallBackTunnelMixin,
self.setup_arp_spoofing_protection(self.int_br,
port, port_detail)
if cur_tag != lvm.vlan:
+ time.sleep(3)
self.int_br.set_db_attribute(
"Port", port.port_name, "tag", lvm.vlan)
·
--·
2.31.0
We discovered the bug by the following procedure:
* a test environment created multiple neutron nets in short succession with the exact same ipv6 subnet ranges
* therefore neutron selected the exact same ipv6 address for the subnet's dhcp port
* the host running the dhcp-agent and the ovs-agent was heavily loaded
* we observed that many (when ovs-agent is made slow enough, then all but one) of these networks' services relying on the dhcp port's address were unavailable
* because duplicate address detection (DAD) for the ipv6 dhcp port address failed
* we believe DAD failed because we have some temporary crosstalk between the dhcp namespaces of different networks
* we believe that this bug is not ipv6 specific, only the default DAD of ipv6 helped us discover it
Exact reproduction steps:
$ date --iso-8601=s
2021-05-31T13:10:14+00:00
# when the ovs-agent is slow enough, even 2 networks are sufficient
$ for i in {1..5}
do
openstack network create xnet$i
openstack subnet create xsubnet$i-v6 --ip-version 6 --network xnet$i --subnet-range 2001:db8::/32
done
# for the record
$ openstack subnet list -f value -c Name -c ID | egrep xsubnet
01d614da-820b-418d-8fa4-71952713f0ad xsubnet5-v6
72158e8e-5059-4abb-98a4-5adc9e4ef39c xsubnet2-v6
8f263143-a69b-4c42-b74c-6f30aca7b19d xsubnet4-v6
9ab4159e-12f8-44ed-8947-35a56b62eaf8 xsubnet1-v6
d4ed53e2-7b70-43d7-bd9f-d45f006a8179 xsubnet3-v6
# note that all dhcp ports got the same ip
$ openstack port list --device-owner network:dhcp -f value -c id -c mac_address -c fixed_ips | egrep 2001:db8::
130855be-ead1-40bb-9ca0-5336428aa74b fa:16:3e:24:76:41 [{'subnet_id': '01d614da-820b-418d-8fa4-71952713f0ad', 'ip_address': '2001:db8::1'}]
19fcabfd-f32a-40ea-b68e-ced41f394822 fa:16:3e:43:80:fe [{'subnet_id': '9ab4159e-12f8-44ed-8947-35a56b62eaf8', 'ip_address': '2001:db8::1'}]
46963dbd-c844-4986-a07f-fb78adbd95e9 fa:16:3e:4f:23:bf [{'subnet_id': '72158e8e-5059-4abb-98a4-5adc9e4ef39c', 'ip_address': '2001:db8::1'}]
b8bf2fb1-d52a-41af-90bb-01aa23529015 fa:16:3e:90:40:8e [{'subnet_id': 'd4ed53e2-7b70-43d7-bd9f-d45f006a8179', 'ip_address': '2001:db8::1'}]
ba67f2c0-c714-45fd-aec8-7233ba379dfa fa:16:3e:35:10:8d [{'subnet_id': '8f263143-a69b-4c42-b74c-6f30aca7b19d', 'ip_address': '2001:db8::1'}]
# all but one dhcp port (and by the way metadata) addresses are in DAD failed mode
$ for net in $( openstack network list -f value -c Name -c ID | awk '/ xnet/ { print $1 }' ) ; do sudo ip netns exec qdhcp-$net ip a ; done | egrep '(link/ether|inet6 (2001:db8::|fe80::a9fe:a9fe))'
link/ether fa:16:3e:90:40:8e brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:4f:23:bf brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:24:76:41 brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:35:10:8d brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:43:80:fe brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global·
inet6 fe80::a9fe:a9fe/64 scope link·
# dmesg also shows the DAD failures
# man dmesg: Be aware that the timestamp could be inaccurate!
$ LC_TIME=en_US sudo dmesg -T
[snip]
[Mon May 31 13:10:10 2021] device tap19fcabfd-f3 entered promiscuous mode
[Mon May 31 13:10:15 2021] device tap46963dbd-c8 entered promiscuous mode
[Mon May 31 13:10:15 2021] IPv6: tap46963dbd-c8: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:15 2021] IPv6: tap46963dbd-c8: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:18 2021] device tapb8bf2fb1-d5 entered promiscuous mode
[Mon May 31 13:10:20 2021] IPv6: tapb8bf2fb1-d5: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:20 2021] IPv6: tapb8bf2fb1-d5: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:22 2021] device tapba67f2c0-c7 entered promiscuous mode
[Mon May 31 13:10:23 2021] IPv6: tapba67f2c0-c7: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:24 2021] IPv6: tapba67f2c0-c7: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:26 2021] device tap130855be-ea entered promiscuous mode
[Mon May 31 13:10:27 2021] IPv6: tap130855be-ea: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:28 2021] IPv6: tap130855be-ea: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
# while there were no errors in ovs-agent
$ sudo LC_TIME=en_US journalctl -u devstack@q-agt -S '2021-05-31 13:10:14' | egrep ERROR
[empty]
# clean up
$ openstack network list | awk '/xnet/ { print $2 }' | xargs -r openstack network delete
In a bit of further analysis I only created two networks and ran tcpdump on the first's dhcp port, while creating the second:
$ openstack network create xnet0
$ openstack subnet create xsubnet0-v6 --ip-version 6 --network xnet0 --subnet-range 2001:db8::/32
# run tcpdump on the 1st net's dhcp port
$ sudo ip netns exec qdhcp-$( openstack network show -f value -c id xnet0 ) tcpdump -n -vvv -i tapcaa92c34-53
# create the 2nd net while tcpdump is already running
$ openstack network create xnet1
$ openstack subnet create xsubnet1-v6 --ip-version 6 --network xnet1 --subnet-range 2001:db8::/32
# the 2nd net's dhcp port's mac address
$ openstack port list --device-owner network:dhcp --network xnet1 -f value -c mac_address
fa:16:3e:0d:be:aa
# tcpdump's capture contains packets with the 2nd net's dhcp port's mac address
tcpdump: listening on tapcaa92c34-53, link-type EN10MB (Ethernet), capture size 262144 bytes
^C14:24:14.541893 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 90: (hlim 1, next-header Options (0) payload length: 36) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 1 group record(s) [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.929686 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 110: (hlim 1, next-header Options (0) payload length: 56) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 2 group record(s) [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.985673 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.993930 fa:16:3e:0d:be:aa > 33:33:ff:0d:be:aa, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff0d:beaa: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::f816:3eff:fe0d:beaa
unknown option (14), length 8 (1):·
0x0000: bfdb db94 d695
14:24:15.209668 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:15.858999 fa:16:3e:0d:be:aa > 33:33:ff:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8::1
unknown option (14), length 8 (1):·
0x0000: 6e67 6a99 e72e
14:24:15.859815 fa:16:3e:6a:ed:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:db8::1 > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2001:db8::1, Flags [override]
destination link-address option (2), length 8 (1): fa:16:3e:6a:ed:cb
14:24:15.860188 fa:16:3e:0d:be:aa > 33:33:ff:fe:a9:fe, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:fffe:a9fe: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::a9fe:a9fe
unknown option (14), length 8 (1):·
0x0000: 11e1 9aab 157a
14:24:15.860731 fa:16:3e:6a:ed:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::a9fe:a9fe > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is fe80::a9fe:a9fe, Flags [override]
destination link-address option (2), length 8 (1): fa:16:3e:6a:ed:cb
14:24:16.017855 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) fe80::f816:3eff:fe0d:beaa > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:16.137879 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) fe80::f816:3eff:fe0d:beaa > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
11 packets captured
11 packets received by filter
0 packets dropped by kernel
At the moment I did not test yet:
* whether the leaked traffic can be intercepted on a vm port
* whether a vm port can similarly leak traffic
* whether ovs-agent can be attacked to intentionally slow it down |
|
| 2022-02-11 09:58:06 |
OpenStack Infra |
tags |
in-stable-wallaby in-stable-xena security |
in-stable-victoria in-stable-wallaby in-stable-xena security |
|
