| 2017-11-14 23:02:21 |
Sarah Newman |
bug |
|
|
added bug |
| 2017-11-14 23:04:29 |
Sarah Newman |
description |
We experienced a DOS yesterday on a system (not openstack based) which would have been mitigated if a mac address whitelist in ebtables had occurred in the nat PREROUTING chain rather than the filter FORWARD chain.
At least with kernel version 4.9.39, with rapidly cycling mac addresses the linux bridge appears to get bogged down in learning new MAC addresses if this is not explicitly turned off with brctl setageing <bridge> 0.
We deployed a workaround to our own infrastructure but I believe https://git.openstack.org/cgit/openstack/neutron/tree/neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py#n158 means that openstack has the same vulnerability.
It should be possible to move all logic related to checking the inpout to the ebtables nat PREROUTING chain using the ebtables_nat module.
To duplicate, in a VM on a host with bridged networking and mac spoofing protection in place, install dsniff and run:
macof -i <ethernet device> -s <valid local IP> -d <valid remote IP> -n 50000000 &> /dev/null
Observe on the host that ksoftirqd usage goes to near 100% on one core, that 'perf top' will show br_fdb_update as taking significant resources, and that 'brctl showmacs <bridge>' will probably hang. |
We experienced a DOS yesterday on a system (not openstack based) which would have been mitigated if a mac address whitelist in ebtables had occurred in the nat PREROUTING chain rather than the filter FORWARD chain.
At least with kernel version 4.9.39, with rapidly cycling mac addresses the linux bridge appears to get bogged down in learning new MAC addresses if this is not explicitly turned off with brctl setageing <bridge> 0.
We deployed a workaround to our own infrastructure but I believe https://git.openstack.org/cgit/openstack/neutron/tree/neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py#n158 means that openstack has the same vulnerability.
It should be possible to move all logic related to checking the input to the ebtables nat PREROUTING chain using the ebtables_nat module.
To duplicate, in a VM on a host with bridged networking and mac spoofing protection in place, install dsniff and run:
macof -i <ethernet device> -s <valid local IP> -d <valid remote IP> -n 50000000 &> /dev/null
Observe on the host that ksoftirqd usage goes to near 100% on one core, that 'perf top' will show br_fdb_update as taking significant resources, and that 'brctl showmacs <bridge>' will probably hang. |
|
| 2017-11-14 23:04:51 |
Sarah Newman |
description |
We experienced a DOS yesterday on a system (not openstack based) which would have been mitigated if a mac address whitelist in ebtables had occurred in the nat PREROUTING chain rather than the filter FORWARD chain.
At least with kernel version 4.9.39, with rapidly cycling mac addresses the linux bridge appears to get bogged down in learning new MAC addresses if this is not explicitly turned off with brctl setageing <bridge> 0.
We deployed a workaround to our own infrastructure but I believe https://git.openstack.org/cgit/openstack/neutron/tree/neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py#n158 means that openstack has the same vulnerability.
It should be possible to move all logic related to checking the input to the ebtables nat PREROUTING chain using the ebtables_nat module.
To duplicate, in a VM on a host with bridged networking and mac spoofing protection in place, install dsniff and run:
macof -i <ethernet device> -s <valid local IP> -d <valid remote IP> -n 50000000 &> /dev/null
Observe on the host that ksoftirqd usage goes to near 100% on one core, that 'perf top' will show br_fdb_update as taking significant resources, and that 'brctl showmacs <bridge>' will probably hang. |
We experienced a DOS yesterday on a system (not openstack based) which would have been mitigated if a mac address whitelist in ebtables had occurred in the nat PREROUTING chain rather than the filter FORWARD chain.
At least with kernel version 4.9, with rapidly cycling mac addresses the linux bridge appears to get bogged down in learning new MAC addresses if this is not explicitly turned off with brctl setageing <bridge> 0.
We deployed a workaround to our own infrastructure but I believe https://git.openstack.org/cgit/openstack/neutron/tree/neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py#n158 means that openstack has the same vulnerability.
It should be possible to move all logic related to checking the input to the ebtables nat PREROUTING chain using the ebtables_nat module.
To duplicate, in a VM on a host with bridged networking and mac spoofing protection in place, install dsniff and run:
macof -i <ethernet device> -s <valid local IP> -d <valid remote IP> -n 50000000 &> /dev/null
Observe on the host that ksoftirqd usage goes to near 100% on one core, that 'perf top' will show br_fdb_update as taking significant resources, and that 'brctl showmacs <bridge>' will probably hang. |
|
| 2017-11-15 00:56:15 |
Tristan Cacqueray |
description |
We experienced a DOS yesterday on a system (not openstack based) which would have been mitigated if a mac address whitelist in ebtables had occurred in the nat PREROUTING chain rather than the filter FORWARD chain.
At least with kernel version 4.9, with rapidly cycling mac addresses the linux bridge appears to get bogged down in learning new MAC addresses if this is not explicitly turned off with brctl setageing <bridge> 0.
We deployed a workaround to our own infrastructure but I believe https://git.openstack.org/cgit/openstack/neutron/tree/neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py#n158 means that openstack has the same vulnerability.
It should be possible to move all logic related to checking the input to the ebtables nat PREROUTING chain using the ebtables_nat module.
To duplicate, in a VM on a host with bridged networking and mac spoofing protection in place, install dsniff and run:
macof -i <ethernet device> -s <valid local IP> -d <valid remote IP> -n 50000000 &> /dev/null
Observe on the host that ksoftirqd usage goes to near 100% on one core, that 'perf top' will show br_fdb_update as taking significant resources, and that 'brctl showmacs <bridge>' will probably hang. |
This issue is being treated as a potential security risk under embargo. Please do not make any public mention of embargoed (private) security vulnerabilities before their coordinated publication by the OpenStack Vulnerability Management Team in the form of an official OpenStack Security Advisory. This includes discussion of the bug or associated fixes in public forums such as mailing lists, code review systems and bug trackers. Please also avoid private disclosure to other individuals not already approved for access to this information, and provide this same reminder to those who are made aware of the issue prior to publication. All discussion should remain confined to this private bug report, and any proposed fixes should be added to the bug as attachments.
--
We experienced a DOS yesterday on a system (not openstack based) which would have been mitigated if a mac address whitelist in ebtables had occurred in the nat PREROUTING chain rather than the filter FORWARD chain.
At least with kernel version 4.9, with rapidly cycling mac addresses the linux bridge appears to get bogged down in learning new MAC addresses if this is not explicitly turned off with brctl setageing <bridge> 0.
We deployed a workaround to our own infrastructure but I believe https://git.openstack.org/cgit/openstack/neutron/tree/neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py#n158 means that openstack has the same vulnerability.
It should be possible to move all logic related to checking the input to the ebtables nat PREROUTING chain using the ebtables_nat module.
To duplicate, in a VM on a host with bridged networking and mac spoofing protection in place, install dsniff and run:
macof -i <ethernet device> -s <valid local IP> -d <valid remote IP> -n 50000000 &> /dev/null
Observe on the host that ksoftirqd usage goes to near 100% on one core, that 'perf top' will show br_fdb_update as taking significant resources, and that 'brctl showmacs <bridge>' will probably hang. |
|
| 2017-11-15 00:56:57 |
Tristan Cacqueray |
bug task added |
|
ossa |
|
| 2017-11-15 00:57:09 |
Tristan Cacqueray |
bug |
|
|
added subscriber Neutron Core Security reviewers |
| 2017-11-15 20:36:17 |
Jeremy Stanley |
description |
This issue is being treated as a potential security risk under embargo. Please do not make any public mention of embargoed (private) security vulnerabilities before their coordinated publication by the OpenStack Vulnerability Management Team in the form of an official OpenStack Security Advisory. This includes discussion of the bug or associated fixes in public forums such as mailing lists, code review systems and bug trackers. Please also avoid private disclosure to other individuals not already approved for access to this information, and provide this same reminder to those who are made aware of the issue prior to publication. All discussion should remain confined to this private bug report, and any proposed fixes should be added to the bug as attachments.
--
We experienced a DOS yesterday on a system (not openstack based) which would have been mitigated if a mac address whitelist in ebtables had occurred in the nat PREROUTING chain rather than the filter FORWARD chain.
At least with kernel version 4.9, with rapidly cycling mac addresses the linux bridge appears to get bogged down in learning new MAC addresses if this is not explicitly turned off with brctl setageing <bridge> 0.
We deployed a workaround to our own infrastructure but I believe https://git.openstack.org/cgit/openstack/neutron/tree/neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py#n158 means that openstack has the same vulnerability.
It should be possible to move all logic related to checking the input to the ebtables nat PREROUTING chain using the ebtables_nat module.
To duplicate, in a VM on a host with bridged networking and mac spoofing protection in place, install dsniff and run:
macof -i <ethernet device> -s <valid local IP> -d <valid remote IP> -n 50000000 &> /dev/null
Observe on the host that ksoftirqd usage goes to near 100% on one core, that 'perf top' will show br_fdb_update as taking significant resources, and that 'brctl showmacs <bridge>' will probably hang. |
We experienced a DOS yesterday on a system (not openstack based) which would have been mitigated if a mac address whitelist in ebtables had occurred in the nat PREROUTING chain rather than the filter FORWARD chain.
At least with kernel version 4.9, with rapidly cycling mac addresses the linux bridge appears to get bogged down in learning new MAC addresses if this is not explicitly turned off with brctl setageing <bridge> 0.
We deployed a workaround to our own infrastructure but I believe https://git.openstack.org/cgit/openstack/neutron/tree/neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py#n158 means that openstack has the same vulnerability.
It should be possible to move all logic related to checking the input to the ebtables nat PREROUTING chain using the ebtables_nat module.
To duplicate, in a VM on a host with bridged networking and mac spoofing protection in place, install dsniff and run:
macof -i <ethernet device> -s <valid local IP> -d <valid remote IP> -n 50000000 &> /dev/null
Observe on the host that ksoftirqd usage goes to near 100% on one core, that 'perf top' will show br_fdb_update as taking significant resources, and that 'brctl showmacs <bridge>' will probably hang. |
|
| 2017-11-15 20:36:29 |
Jeremy Stanley |
information type |
Private Security |
Public |
|
| 2017-11-15 20:36:41 |
Jeremy Stanley |
tags |
|
security |
|
| 2017-11-15 20:36:59 |
Jeremy Stanley |
bug task added |
|
ossn |
|
| 2017-11-15 20:37:24 |
Jeremy Stanley |
ossa: status |
New |
Won't Fix |
|
| 2017-11-16 00:28:07 |
Brian Haley |
bug |
|
|
added subscriber Brian Haley |
| 2017-11-16 00:28:12 |
Brian Haley |
neutron: importance |
Undecided |
Critical |
|
| 2017-11-16 00:28:30 |
OpenStack Infra |
neutron: status |
New |
In Progress |
|
| 2017-11-16 00:28:30 |
OpenStack Infra |
neutron: assignee |
|
Brian Haley (brian-haley) |
|
| 2017-11-16 18:08:19 |
James Denton |
bug |
|
|
added subscriber James Denton |
| 2018-01-26 21:04:10 |
Miguel Lavalle |
neutron: milestone |
|
queens-rc1 |
|
| 2018-02-02 16:13:24 |
OpenStack Infra |
neutron: status |
In Progress |
Fix Released |
|
