an instance can see other instances' unicast packets when security group firewall_driver is openvswitch

Bug #1813439 reported by Guangyu Suo on 2019-01-27
274
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Security Advisory
Undecided
Unassigned
neutron
Medium
Unassigned

Bug Description

We found that instances on the same host can see each others' unicast packets out to instances on the different host if these instances are on the same subnet when security group firewall_driver is openvswitch.

# How to reproduce

1. create 3 vms on the same subnet, no matter vlan or vxlan, called them vm1, vm2, vm3:

vm1: 192.168.100.3 (compute 1)
vm2: 192.168.100.12 (compute 1)
vm3: 192.168.100.17 (compute 2)

vm1 and vm2 are on the same host, while vm3 is on the other host.

2. ping vm3 from vm2

3. tcpdump eth0 on vm1, you will see icmp request packages from vm2 to vm3 are captured

# tcpdump -enni eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:01:59.361792 fa:16:3e:8e:49:5c > fa:16:3e:8f:ca:a7, ethertype IPv4 (0x0800), length 98: 192.168.100.12 > 192.168.100.17: ICMP echo request, id 1145, seq 4, length 64
09:02:00.361772 fa:16:3e:8e:49:5c > fa:16:3e:8f:ca:a7, ethertype IPv4 (0x0800), length 98: 192.168.100.12 > 192.168.100.17: ICMP echo request, id 1145, seq 5, length 64
09:02:01.361785 fa:16:3e:8e:49:5c > fa:16:3e:8f:ca:a7, ethertype IPv4 (0x0800), length 98: 192.168.100.12 > 192.168.100.17: ICMP echo request, id 1145, seq 6, length 64
09:02:02.361798 fa:16:3e:8e:49:5c > fa:16:3e:8f:ca:a7, ethertype IPv4 (0x0800), length 98: 192.168.100.12 > 192.168.100.17: ICMP echo request, id 1145, seq 7, length 64

4. ping vm2 from vm3

5. tcpdump eth0 on vm1, you will see icmp reply packages from vm2 to vm3 are captured

# tcpdump -enni eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:03:39.608748 fa:16:3e:8e:49:5c > fa:16:3e:8f:ca:a7, ethertype IPv4 (0x0800), length 98: 192.168.100.12 > 192.168.100.17: ICMP echo reply, id 1144, seq 3, length 64
09:03:40.609475 fa:16:3e:8e:49:5c > fa:16:3e:8f:ca:a7, ethertype IPv4 (0x0800), length 98: 192.168.100.12 > 192.168.100.17: ICMP echo reply, id 1144, seq 4, length 64
09:03:41.609444 fa:16:3e:8e:49:5c > fa:16:3e:8f:ca:a7, ethertype IPv4 (0x0800), length 98: 192.168.100.12 > 192.168.100.17: ICMP echo reply, id 1144, seq 5, length 64

TCP/UDP packages have the same problem, this will have performance issue and security problem on the production. This will not happen when the security group firewall driver is iptables_hybrid or disable port security.

# Versions

I am testing this on N and R release, both have the same problem, the R release neutron package versions are:

openstack-neutron-ml2-13.0.2-1.el7.noarch
openstack-neutron-openvswitch-13.0.2-1.el7.noarch
python2-neutronclient-6.9.1-1.el7.noarch
openstack-neutron-common-13.0.2-1.el7.noarch
openstack-neutron-fwaas-13.0.1-1.el7.noarch
openstack-neutron-13.0.2-1.el7.noarch
openstack-neutron-lbaas-13.0.0-1.el7.noarch
python2-neutron-lib-1.18.0-1.el7.noarch
python-neutron-lbaas-13.0.0-1.el7.noarch
python-neutron-13.0.2-1.el7.noarch
python-neutron-fwaas-13.0.1-1.el7.noarch

and the operating system and kernel are:

[root@node-30 ~]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)

[root@node-30 ~]# uname -a
Linux node-30 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 16:29:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

and the openvswitch version is :

openvswitch-2.9.0-3.el7.x86_64

Guangyu Suo (yugsuo) on 2019-01-27
information type: Public → Public Security

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

Is this a mis-configuration from neutron or is it an ovs issue?

Changed in ossa:
status: New → Incomplete
Guangyu Suo (yugsuo) wrote :

I am not sure, this is my neutron ml2_config.ini:

[ml2]
type_drivers = flat,vlan,vxlan
tenant_network_types = vxlan
mechanism_drivers = openvswitch,l2population
extension_drivers = port_security

[ml2_type_vlan]
network_vlan_ranges = physnet1:3026:3027

[ml2_type_flat]
flat_networks = physnet1

[ml2_type_vxlan]
vni_ranges = 1:1000
vxlan_group = 239.1.1.1

[securitygroup]
firewall_driver = openvswitch

[agent]
tunnel_types = vxlan
l2_population = true
arp_responder = true

[ovs]
bridge_mappings = physnet1:br-ex
datapath_type = system
ovsdb_connection = tcp:127.0.0.1:6640
local_ip = 10.0.125.30

Changed in neutron:
status: New → Confirmed
importance: Undecided → Medium
Lajos Katona (lajos-katona) wrote :

Hi, Finally I managed to have a multihost devstack (everything from master), but still with some weird config problems...
For me it seems that even with openvswitch firewall driver there is no visible icmp traffic in the vm (VM2 or VM1 in your setup).
Could you please send the security-groups you used?

Guangyu Suo (yugsuo) wrote :

Hi, this is my security group rule:

| id | name | tenant_id | security_group_rules |
+--------------------------------------+---------+----------------------------------+----------------------------------------------------------------------+
| 0f84365f-f94a-45d1-9a60-ad87f0c2b3ab | default | 94bd455f22b44c1dbb1ffff8c88e8d78 | egress, IPv4 |
| | | | egress, IPv6 |
| | | | ingress, IPv4, 22/tcp, remote_ip_prefix: 0.0.0.0/0 |
| | | | ingress, IPv4, 443/tcp, remote_ip_prefix: 0.0.0.0/0 |
| | | | ingress, IPv4, icmp, remote_ip_prefix: 0.0.0.0/0 |
| | | | ingress, IPv4, remote_group_id: 0f84365f-f94a-45d1-9a60-ad87f0c2b3ab |
| | | | ingress, IPv6, remote_group_id: 0f84365f-f94a-45d1-9a60-ad87f0c2b3ab |

Lajos Katona (lajos-katona) wrote :

Thanks, I checked and my group rules are similar, and finally I got a working environment, but still can't see icmp in the vm not directed by ping.
Do you have a chance to check on master?

Guangyu Suo (yugsuo) wrote :

OK, I will check it on master using devstack, My Rocky OpenStack environment is deployed by kolla.

@kolla-coresec, I've subscribed you to this bug as it may be affecting kolla ovs firewall user.

yao ning (mslovy11022) wrote :

hi, Lajos katona, do you check it on stable/rokcy? we find that every clusters with openvswitch firewall driver have this issues. but it works well with iptables firewall driver.

can you send us your configuration of your devstack environment and we can do some deeper troubleshooting about this issue? we think this is a critical security issues and would impact widely.

no longer affects: ossn
James Denton (james-denton) wrote :
Jeremy Stanley (fungi) on 2019-07-22
summary: - an instance can see other instances' unicast packages when security
- group firewall_driver is openvswitch
+ an instance can see other instances' unicast packets when security group
+ firewall_driver is openvswitch
description: updated

I agree with James, this bug is a duplicate of https://bugs.launchpad.net/neutron/+bug/1732067.

To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Other bug subscribers