Activity log for bug #1501206

Date Who What changed Old value New value Message
2015-09-30 07:47:05 Tore Anderson bug added bug
2015-09-30 09:04:02 Tore Anderson bug added subscriber Pip Oomen
2015-09-30 11:35:56 Jeremy Stanley bug added subscriber Neutron Core Security reviewers
2015-09-30 11:36:51 Jeremy Stanley bug task added ossa
2015-09-30 11:37:27 Jeremy Stanley ossa: status New Incomplete
2015-09-30 11:38:29 Jeremy Stanley description When configuring an public IPv4 subnet with DHCP enabled inside Neutron (and attaching it to an Internet-connected router), the DNS recursive resolver service provided by dnsmasq inside the qdhcp network namespace will respond to DNS queries from the entire Internet. This is a huge problem from a security standpoint, as open resolvers are very likely to be abused for DDoS purposes. This does not only cause significant damage to third parties (i.e., the true destination of the DDoS attack and every network in between), but also on the local network or servers (due to saturation of all the available network bandwidth and/or the processing capacity of the node running the dnsmasq instance). Quoting from http://openresolverproject.org/: «Open Resolvers pose a significant threat to the global network infrastructure by answering recursive queries for hosts outside of its domain. They are utilized in DNS Amplification attacks and pose a similar threat as those from Smurf attacks commonly seen in the late 1990s. [...] What can I do? If you operate a DNS server, please check the settings. Recursive servers should be restricted to your enterprise or customer IP ranges to prevent abuse. Directions on securing BIND and Microsoft nameservers can be found on the Team CYMRU Website - If you operate BIND, you can deploy the TCP-ANY patch» It seems reasonable to expect that the dnsmasq instance within Neutron would only respond to DNS queries from the subnet prefixes it is associated with and ignore all others. Note that this only occurs for IPv4. That is however likely just a symptom of bug #1499170, which breaks all IPv6 DNS queries (external as well as internal). I would assume that when bug #1499170 is fixed, the router:dhcp ports will immediately start being open resolvers over IPv6 too. For what it's worth, the reason I noticed this issue in the first place was that NorCERT (the national Norwegian Computer Emergency Response Team - http://www.cert.no/) got in touch with us, notifying us about the open resolvers they had observed in our network and insisted that we lock them down ASAP. It only took NorCERT couple of days after the subnet was first created to do so. Tore This issue is being treated as a potential security risk under embargo. Please do not make any public mention of embargoed (private) security vulnerabilities before their coordinated publication by the OpenStack Vulnerability Management Team in the form of an official OpenStack Security Advisory. This includes discussion of the bug or associated fixes in public forums such as mailing lists, code review systems and bug trackers. Please also avoid private disclosure to other individuals not already approved for access to this information, and provide this same reminder to those who are made aware of the issue prior to publication. All discussion should remain confined to this private bug report, and any proposed fixes should be added to the bug as attachments. When configuring an public IPv4 subnet with DHCP enabled inside Neutron (and attaching it to an Internet-connected router), the DNS recursive resolver service provided by dnsmasq inside the qdhcp network namespace will respond to DNS queries from the entire Internet. This is a huge problem from a security standpoint, as open resolvers are very likely to be abused for DDoS purposes. This does not only cause significant damage to third parties (i.e., the true destination of the DDoS attack and every network in between), but also on the local network or servers (due to saturation of all the available network bandwidth and/or the processing capacity of the node running the dnsmasq instance). Quoting from http://openresolverproject.org/: «Open Resolvers pose a significant threat to the global network infrastructure by answering recursive queries for hosts outside of its domain. They are utilized in DNS Amplification attacks and pose a similar threat as those from Smurf attacks commonly seen in the late 1990s. [...] What can I do? If you operate a DNS server, please check the settings. Recursive servers should be restricted to your enterprise or customer IP ranges to prevent abuse. Directions on securing BIND and Microsoft nameservers can be found on the Team CYMRU Website - If you operate BIND, you can deploy the TCP-ANY patch» It seems reasonable to expect that the dnsmasq instance within Neutron would only respond to DNS queries from the subnet prefixes it is associated with and ignore all others. Note that this only occurs for IPv4. That is however likely just a symptom of bug #1499170, which breaks all IPv6 DNS queries (external as well as internal). I would assume that when bug #1499170 is fixed, the router:dhcp ports will immediately start being open resolvers over IPv6 too. For what it's worth, the reason I noticed this issue in the first place was that NorCERT (the national Norwegian Computer Emergency Response Team - http://www.cert.no/) got in touch with us, notifying us about the open resolvers they had observed in our network and insisted that we lock them down ASAP. It only took NorCERT couple of days after the subnet was first created to do so. Tore
2015-10-14 11:01:43 Tore Anderson bug added subscriber Anders Einar Hilden
2015-10-19 23:12:08 Mark McClain neutron: importance Undecided High
2015-10-19 23:12:13 Mark McClain neutron: assignee Mark McClain (markmcclain)
2015-11-09 15:45:32 Tristan Cacqueray description This issue is being treated as a potential security risk under embargo. Please do not make any public mention of embargoed (private) security vulnerabilities before their coordinated publication by the OpenStack Vulnerability Management Team in the form of an official OpenStack Security Advisory. This includes discussion of the bug or associated fixes in public forums such as mailing lists, code review systems and bug trackers. Please also avoid private disclosure to other individuals not already approved for access to this information, and provide this same reminder to those who are made aware of the issue prior to publication. All discussion should remain confined to this private bug report, and any proposed fixes should be added to the bug as attachments. When configuring an public IPv4 subnet with DHCP enabled inside Neutron (and attaching it to an Internet-connected router), the DNS recursive resolver service provided by dnsmasq inside the qdhcp network namespace will respond to DNS queries from the entire Internet. This is a huge problem from a security standpoint, as open resolvers are very likely to be abused for DDoS purposes. This does not only cause significant damage to third parties (i.e., the true destination of the DDoS attack and every network in between), but also on the local network or servers (due to saturation of all the available network bandwidth and/or the processing capacity of the node running the dnsmasq instance). Quoting from http://openresolverproject.org/: «Open Resolvers pose a significant threat to the global network infrastructure by answering recursive queries for hosts outside of its domain. They are utilized in DNS Amplification attacks and pose a similar threat as those from Smurf attacks commonly seen in the late 1990s. [...] What can I do? If you operate a DNS server, please check the settings. Recursive servers should be restricted to your enterprise or customer IP ranges to prevent abuse. Directions on securing BIND and Microsoft nameservers can be found on the Team CYMRU Website - If you operate BIND, you can deploy the TCP-ANY patch» It seems reasonable to expect that the dnsmasq instance within Neutron would only respond to DNS queries from the subnet prefixes it is associated with and ignore all others. Note that this only occurs for IPv4. That is however likely just a symptom of bug #1499170, which breaks all IPv6 DNS queries (external as well as internal). I would assume that when bug #1499170 is fixed, the router:dhcp ports will immediately start being open resolvers over IPv6 too. For what it's worth, the reason I noticed this issue in the first place was that NorCERT (the national Norwegian Computer Emergency Response Team - http://www.cert.no/) got in touch with us, notifying us about the open resolvers they had observed in our network and insisted that we lock them down ASAP. It only took NorCERT couple of days after the subnet was first created to do so. Tore When configuring an public IPv4 subnet with DHCP enabled inside Neutron (and attaching it to an Internet-connected router), the DNS recursive resolver service provided by dnsmasq inside the qdhcp network namespace will respond to DNS queries from the entire Internet. This is a huge problem from a security standpoint, as open resolvers are very likely to be abused for DDoS purposes. This does not only cause significant damage to third parties (i.e., the true destination of the DDoS attack and every network in between), but also on the local network or servers (due to saturation of all the available network bandwidth and/or the processing capacity of the node running the dnsmasq instance). Quoting from http://openresolverproject.org/: «Open Resolvers pose a significant threat to the global network infrastructure by answering recursive queries for hosts outside of its domain. They are utilized in DNS Amplification attacks and pose a similar threat as those from Smurf attacks commonly seen in the late 1990s. [...] What can I do? If you operate a DNS server, please check the settings. Recursive servers should be restricted to your enterprise or customer IP ranges to prevent abuse. Directions on securing BIND and Microsoft nameservers can be found on the Team CYMRU Website - If you operate BIND, you can deploy the TCP-ANY patch» It seems reasonable to expect that the dnsmasq instance within Neutron would only respond to DNS queries from the subnet prefixes it is associated with and ignore all others. Note that this only occurs for IPv4. That is however likely just a symptom of bug #1499170, which breaks all IPv6 DNS queries (external as well as internal). I would assume that when bug #1499170 is fixed, the router:dhcp ports will immediately start being open resolvers over IPv6 too. For what it's worth, the reason I noticed this issue in the first place was that NorCERT (the national Norwegian Computer Emergency Response Team - http://www.cert.no/) got in touch with us, notifying us about the open resolvers they had observed in our network and insisted that we lock them down ASAP. It only took NorCERT couple of days after the subnet was first created to do so. Tore
2015-11-09 15:46:05 Tristan Cacqueray information type Private Security Public Security
2015-11-17 16:59:36 Tristan Cacqueray ossa: status Incomplete Won't Fix
2015-11-17 16:59:40 Tristan Cacqueray information type Public Security Public
2016-01-21 21:27:43 OpenStack Infra neutron: status New In Progress
2016-01-21 21:27:43 OpenStack Infra neutron: assignee Mark McClain (markmcclain) Cedric Brandily (cbrandily)
2016-02-18 07:11:07 OpenStack Infra neutron: assignee Cedric Brandily (cbrandily) Benoît Knecht (benoit-knecht)
2016-03-12 00:58:15 Armando Migliaccio neutron: milestone mitaka-rc1
2016-03-15 08:55:17 Armando Migliaccio neutron: milestone mitaka-rc1 newton-1
2016-04-14 17:58:47 Armando Migliaccio neutron: status In Progress Confirmed
2016-04-14 17:58:50 Armando Migliaccio neutron: assignee Benoît Knecht (benoit-knecht)
2016-04-14 17:58:57 Armando Migliaccio neutron: milestone newton-1
2016-04-19 04:03:13 Scott Wilson bug added subscriber Scott Wilson
2016-05-17 16:48:52 Lance Albertson bug added subscriber Lance Albertson
2016-05-30 09:26:14 OpenStack Infra neutron: status Confirmed In Progress
2016-05-30 09:26:14 OpenStack Infra neutron: assignee Benoît Knecht (benoit-knecht)
2016-06-23 12:45:27 Dr. Jens Harbott bug added subscriber Dr. Jens Rosenboom
2016-09-29 22:03:54 OpenStack Infra neutron: assignee Benoît Knecht (benoit-knecht) Dustin Lundquist (dlundquist)
2016-10-05 14:21:22 OpenStack Infra neutron: assignee Dustin Lundquist (dlundquist) Dr. Jens Rosenboom (j-rosenboom-j)
2016-10-24 18:37:10 Armando Migliaccio neutron: status In Progress Incomplete
2016-10-24 18:37:14 Armando Migliaccio neutron: assignee Dr. Jens Rosenboom (j-rosenboom-j)
2016-10-24 18:37:24 Armando Migliaccio tags dns dns l3-bgp
2016-10-24 18:37:32 Armando Migliaccio tags dns l3-bgp l3-bgp
2016-11-07 07:17:40 Sreekumar S neutron: status Incomplete In Progress
2016-11-07 07:19:24 Sreekumar S neutron: status In Progress Incomplete
2017-02-09 14:37:24 Dr. Jens Harbott neutron: status Incomplete Confirmed
2017-06-28 07:08:49 Crazik bug added subscriber Crazik
2017-11-27 06:04:26 Dr. Jens Harbott bug task added neutron (Ubuntu)
2017-11-30 13:54:52 OpenStack Infra neutron: status Confirmed In Progress
2017-11-30 13:54:52 OpenStack Infra neutron: assignee David Homolka (davidhomolka)
2017-12-05 14:25:39 James Page neutron (Ubuntu): status New Triaged
2017-12-05 14:25:43 James Page neutron (Ubuntu): importance Undecided High
2017-12-05 14:27:24 James Page neutron (Ubuntu): status Triaged Invalid
2018-03-02 15:56:35 new neutron (Ubuntu): assignee new (cloudie)
2018-03-02 15:58:30 new neutron (Ubuntu): status Invalid In Progress
2018-03-06 18:09:12 Atif neutron (Ubuntu): status In Progress Confirmed
2018-10-27 03:50:59 Vladimir Astafiev bug added subscriber Vladimir Astafiev
2018-10-29 12:37:17 OpenStack Infra neutron: assignee David Homolka (davidhomolka) Dr. Jens Harbott (j-harbott)
2018-11-28 20:49:52 OpenStack Infra neutron: assignee Dr. Jens Harbott (j-harbott) Brian Haley (brian-haley)
2018-11-30 21:57:40 OpenStack Infra neutron: status In Progress Fix Released
2019-01-04 13:54:57 Bernard Cafarelli tags l3-bgp l3-bgp neutron-proactive-backport-potential
2019-01-16 19:24:30 Corey Bryant neutron (Ubuntu): status Confirmed Triaged
2019-01-30 04:50:05 OpenStack Infra tags l3-bgp neutron-proactive-backport-potential in-stable-queens l3-bgp neutron-proactive-backport-potential
2019-01-31 10:38:57 OpenStack Infra tags in-stable-queens l3-bgp neutron-proactive-backport-potential in-stable-pike in-stable-queens l3-bgp neutron-proactive-backport-potential
2019-02-01 15:04:03 OpenStack Infra tags in-stable-pike in-stable-queens l3-bgp neutron-proactive-backport-potential in-stable-pike in-stable-queens in-stable-rocky l3-bgp neutron-proactive-backport-potential
2019-10-19 04:00:19 Mathew Hodson neutron (Ubuntu): status Triaged Fix Released
2019-10-19 04:00:29 Mathew Hodson nominated for series Ubuntu Bionic
2019-10-19 04:00:29 Mathew Hodson bug task added neutron (Ubuntu Bionic)
2019-10-19 04:00:40 Mathew Hodson neutron (Ubuntu Bionic): status New Fix Released
2019-10-19 04:00:47 Mathew Hodson neutron (Ubuntu Bionic): importance Undecided High
2019-10-19 04:01:30 Mathew Hodson neutron (Ubuntu): assignee new (cloudie)