an instance can see other instances' unicast packets when security group firewall_driver is openvswitch

[Guangyu Suo]

We found that instances on the same host can see each others' unicast packets out to instances on the different host if these instances are on the same subnet when security group firewall_driver is openvswitch.

# How to reproduce

1. create 3 vms on the same subnet, no matter vlan or vxlan, called them vm1, vm2, vm3:

vm1: 192.168.100.3 (compute 1)
vm2: 192.168.100.12 (compute 1)
vm3: 192.168.100.17 (compute 2)

vm1 and vm2 are on the same host, while vm3 is on the other host.

2. ping vm3 from vm2

3. tcpdump eth0 on vm1, you will see icmp request packages from vm2 to vm3 are captured

# tcpdump -enni eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:01:59.361792 fa:16:3e:8e:49:5c > fa:16:3e:8f:ca:a7, ethertype IPv4 (0x0800), length 98: 192.168.100.12 > 192.168.100.17: ICMP echo request, id 1145, seq 4, length 64
09:02:00.361772 fa:16:3e:8e:49:5c > fa:16:3e:8f:ca:a7, ethertype IPv4 (0x0800), length 98: 192.168.100.12 > 192.168.100.17: ICMP echo request, id 1145, seq 5, length 64
09:02:01.361785 fa:16:3e:8e:49:5c > fa:16:3e:8f:ca:a7, ethertype IPv4 (0x0800), length 98: 192.168.100.12 > 192.168.100.17: ICMP echo request, id 1145, seq 6, length 64
09:02:02.361798 fa:16:3e:8e:49:5c > fa:16:3e:8f:ca:a7, ethertype IPv4 (0x0800), length 98: 192.168.100.12 > 192.168.100.17: ICMP echo request, id 1145, seq 7, length 64

4. ping vm2 from vm3

5. tcpdump eth0 on vm1, you will see icmp reply packages from vm2 to vm3 are captured

# tcpdump -enni eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:03:39.608748 fa:16:3e:8e:49:5c > fa:16:3e:8f:ca:a7, ethertype IPv4 (0x0800), length 98: 192.168.100.12 > 192.168.100.17: ICMP echo reply, id 1144, seq 3, length 64
09:03:40.609475 fa:16:3e:8e:49:5c > fa:16:3e:8f:ca:a7, ethertype IPv4 (0x0800), length 98: 192.168.100.12 > 192.168.100.17: ICMP echo reply, id 1144, seq 4, length 64
09:03:41.609444 fa:16:3e:8e:49:5c > fa:16:3e:8f:ca:a7, ethertype IPv4 (0x0800), length 98: 192.168.100.12 > 192.168.100.17: ICMP echo reply, id 1144, seq 5, length 64

TCP/UDP packages have the same problem, this will have performance issue and security problem on the production. This will not happen when the security group firewall driver is iptables_hybrid or disable port security.

# Versions

I am testing this on N and R release, both have the same problem, the R release neutron package versions are:

openstack-neutron-ml2-13.0.2-1.el7.noarch
openstack-neutron-openvswitch-13.0.2-1.el7.noarch
python2-neutronclient-6.9.1-1.el7.noarch
openstack-neutron-common-13.0.2-1.el7.noarch
openstack-neutron-fwaas-13.0.1-1.el7.noarch
openstack-neutron-13.0.2-1.el7.noarch
openstack-neutron-lbaas-13.0.0-1.el7.noarch
python2-neutron-lib-1.18.0-1.el7.noarch
python-neutron-lbaas-13.0.0-1.el7.noarch
python-neutron-13.0.2-1.el7.noarch
python-neutron-fwaas-13.0.1-1.el7.noarch

and the operating system and kernel are:

[root@node-30 ~]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)

[root@node-30 ~]# uname -a
Linux node-30 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 16:29:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

and the openvswitch version is :

openvswitch-2.9.0-3.el7.x86_64

[Tristan Cacqueray (tristan-cacqueray)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

Is this a mis-configuration from neutron or is it an ovs issue?

[Guangyu Suo (yugsuo)]

I am not sure, this is my neutron ml2_config.ini:

[ml2]
type_drivers = flat,vlan,vxlan
tenant_network_types = vxlan
mechanism_drivers = openvswitch,l2population
extension_drivers = port_security

[ml2_type_vlan]
network_vlan_ranges = physnet1:3026:3027

[ml2_type_flat]
flat_networks = physnet1

[ml2_type_vxlan]
vni_ranges = 1:1000
vxlan_group = 239.1.1.1

[securitygroup]
firewall_driver = openvswitch

[agent]
tunnel_types = vxlan
l2_population = true
arp_responder = true

[ovs]
bridge_mappings = physnet1:br-ex
datapath_type = system
ovsdb_connection = tcp:127.0.0.1:6640
local_ip = 10.0.125.30

[Lajos Katona (lajos-katona)]

Hi, Finally I managed to have a multihost devstack (everything from master), but still with some weird config problems...
For me it seems that even with openvswitch firewall driver there is no visible icmp traffic in the vm (VM2 or VM1 in your setup).
Could you please send the security-groups you used?

[Guangyu Suo (yugsuo)]

Hi, this is my security group rule:

| id | name | tenant_id | security_group_rules |
+--------------------------------------+---------+----------------------------------+----------------------------------------------------------------------+
| 0f84365f-f94a-45d1-9a60-ad87f0c2b3ab | default | 94bd455f22b44c1dbb1ffff8c88e8d78 | egress, IPv4 |
| | | | egress, IPv6 |
| | | | ingress, IPv4, 22/tcp, remote_ip_prefix: 0.0.0.0/0 |
| | | | ingress, IPv4, 443/tcp, remote_ip_prefix: 0.0.0.0/0 |
| | | | ingress, IPv4, icmp, remote_ip_prefix: 0.0.0.0/0 |
| | | | ingress, IPv4, remote_group_id: 0f84365f-f94a-45d1-9a60-ad87f0c2b3ab |
| | | | ingress, IPv6, remote_group_id: 0f84365f-f94a-45d1-9a60-ad87f0c2b3ab |

[Lajos Katona (lajos-katona)]

Thanks, I checked and my group rules are similar, and finally I got a working environment, but still can't see icmp in the vm not directed by ping.
Do you have a chance to check on master?

[Guangyu Suo (yugsuo)]

OK, I will check it on master using devstack, My Rocky OpenStack environment is deployed by kolla.

[Tristan Cacqueray (tristan-cacqueray)]

@kolla-coresec, I've subscribed you to this bug as it may be affecting kolla ovs firewall user.

[yao ning (mslovy11022)]

hi, Lajos katona, do you check it on stable/rokcy? we find that every clusters with openvswitch firewall driver have this issues. but it works well with iptables firewall driver.

can you send us your configuration of your devstack environment and we can do some deeper troubleshooting about this issue? we think this is a critical security issues and would impact widely.

[James Denton (james-denton)]

This could be a duplicate of: https://bugs.launchpad.net/neutron/+bug/1732067

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

I agree with James, this bug is a duplicate of https://bugs.launchpad.net/neutron/+bug/1732067.