| 2020-08-25 10:11:21 |
Sergio Morant |
bug |
|
|
added bug |
| 2020-08-25 16:27:52 |
Jeremy Stanley |
description |
We have recently installer an OpenStack cluster in Train release and we have noticed an unexpected behavior when Neutron contacts memcached upon some specific API requests. In fact, in our environment lbaas is not configured in Neutron (I don't know if this still possible in the current Neutron version), but we have deployed a monitoring service based on Prometheus openstack-exporter which, by default, checks the accessibility of the lbaas neutron part of the API.
Each time the check have the neutron server generates a 404, as this part of the API is not available. An example can be found at http://paste.openstack.org/show/797113/
The actual issue happens at the socket management level. Each time the check is performed, the socket between the Neutron server and memcached is not released as expected. This leads to a continuous increment of the established sockets between Neutron an Memcached until all the available sockets are exhausted and the cluster authentication locks.
Removing the checking on the lbaas service from Prometheus openstack-exporter allows to avoid the issue but we think that an issue like this (if confirmed) have quite important security concerns as it can lead quite easily to a DoS (wanted or not).
For the record, We have tried to explicitly configure the memcached socket monitoring options in Neutron but it looks like that they are not applied under the current conditions :
memcache_pool_socket_timeout = 3
memcache_pool_unused_timeout = 60
Here under is the environment setup:
Deployer : OpenStack Ansible 20.1.2
Neutron version: neutron 15.1.1.dev2
Neutron config file http://paste.openstack.org/show/797117/
Memcached config file http://paste.openstack.org/show/797118/
Prometheus OpenStack Exporter : https://github.com/openstack-exporter/openstack-exporter
Prometheus OpenStack Exporter config : http://paste.openstack.org/show/797119/
Prometheus command WITH lbaas checking : =/usr/local/bin/openstack-exporter \
--os-client-config=/etc/openstack-exporter/openstack-exporter.yml \
--web.telemetry-path="/metrics" \
--web.listen-address="0.0.0.0:9180" \
--prefix="openstack" \
--endpoint-type="public"
Prometheus command WITHOUT lbaas checking:
/usr/local/bin/openstack-exporter \
--os-client-config=/etc/openstack-exporter/openstack-exporter.yml \
--web.telemetry-path="/metrics" \
--web.listen-address="0.0.0.0:9180" \
--prefix="openstack" \
--endpoint-type="public" \
--disable-metric neutron-loadbalancers \
--disable-metric neutron-loadbalancers_not_active \
--disable-service.object-store \
--disable-service.load-balancer \
I hope the provided information is enough to reproduce the issue.
Cheers
Sergio |
This issue is being treated as a potential security risk under
embargo. Please do not make any public mention of embargoed
(private) security vulnerabilities before their coordinated
publication by the OpenStack Vulnerability Management Team in the
form of an official OpenStack Security Advisory. This includes
discussion of the bug or associated fixes in public forums such as
mailing lists, code review systems and bug trackers. Please also
avoid private disclosure to other individuals not already approved
for access to this information, and provide this same reminder to
those who are made aware of the issue prior to publication. All
discussion should remain confined to this private bug report, and
any proposed fixes should be added to the bug as attachments. This
embargo shall not extend past 2020-11-23 and will be made
public by or on that date even if no fix is identified.
We have recently installer an OpenStack cluster in Train release and we have noticed an unexpected behavior when Neutron contacts memcached upon some specific API requests. In fact, in our environment lbaas is not configured in Neutron (I don't know if this still possible in the current Neutron version), but we have deployed a monitoring service based on Prometheus openstack-exporter which, by default, checks the accessibility of the lbaas neutron part of the API.
Each time the check have the neutron server generates a 404, as this part of the API is not available. An example can be found at http://paste.openstack.org/show/797113/
The actual issue happens at the socket management level. Each time the check is performed, the socket between the Neutron server and memcached is not released as expected. This leads to a continuous increment of the established sockets between Neutron an Memcached until all the available sockets are exhausted and the cluster authentication locks.
Removing the checking on the lbaas service from Prometheus openstack-exporter allows to avoid the issue but we think that an issue like this (if confirmed) have quite important security concerns as it can lead quite easily to a DoS (wanted or not).
For the record, We have tried to explicitly configure the memcached socket monitoring options in Neutron but it looks like that they are not applied under the current conditions :
memcache_pool_socket_timeout = 3
memcache_pool_unused_timeout = 60
Here under is the environment setup:
Deployer : OpenStack Ansible 20.1.2
Neutron version: neutron 15.1.1.dev2
Neutron config file http://paste.openstack.org/show/797117/
Memcached config file http://paste.openstack.org/show/797118/
Prometheus OpenStack Exporter : https://github.com/openstack-exporter/openstack-exporter
Prometheus OpenStack Exporter config : http://paste.openstack.org/show/797119/
Prometheus command WITH lbaas checking : =/usr/local/bin/openstack-exporter \
--os-client-config=/etc/openstack-exporter/openstack-exporter.yml \
--web.telemetry-path="/metrics" \
--web.listen-address="0.0.0.0:9180" \
--prefix="openstack" \
--endpoint-type="public"
Prometheus command WITHOUT lbaas checking:
/usr/local/bin/openstack-exporter \
--os-client-config=/etc/openstack-exporter/openstack-exporter.yml \
--web.telemetry-path="/metrics" \
--web.listen-address="0.0.0.0:9180" \
--prefix="openstack" \
--endpoint-type="public" \
--disable-metric neutron-loadbalancers \
--disable-metric neutron-loadbalancers_not_active \
--disable-service.object-store \
--disable-service.load-balancer \
I hope the provided information is enough to reproduce the issue.
Cheers
Sergio |
|
| 2020-08-25 16:29:25 |
Jeremy Stanley |
affects |
neutron |
keystonemiddleware |
|
| 2020-08-25 16:29:55 |
Jeremy Stanley |
bug task added |
|
ossa |
|
| 2020-08-25 16:30:05 |
Jeremy Stanley |
ossa: status |
New |
Incomplete |
|
| 2020-08-25 16:30:30 |
Jeremy Stanley |
bug |
|
|
added subscriber Keystone Core security contacts |
| 2020-09-14 15:50:53 |
Jeremy Stanley |
bug |
|
|
added subscriber Oslo Core security contacts |
| 2020-09-14 15:51:11 |
Jeremy Stanley |
bug task added |
|
oslo.cache |
|
| 2020-09-29 16:40:45 |
Jeremy Stanley |
description |
This issue is being treated as a potential security risk under
embargo. Please do not make any public mention of embargoed
(private) security vulnerabilities before their coordinated
publication by the OpenStack Vulnerability Management Team in the
form of an official OpenStack Security Advisory. This includes
discussion of the bug or associated fixes in public forums such as
mailing lists, code review systems and bug trackers. Please also
avoid private disclosure to other individuals not already approved
for access to this information, and provide this same reminder to
those who are made aware of the issue prior to publication. All
discussion should remain confined to this private bug report, and
any proposed fixes should be added to the bug as attachments. This
embargo shall not extend past 2020-11-23 and will be made
public by or on that date even if no fix is identified.
We have recently installer an OpenStack cluster in Train release and we have noticed an unexpected behavior when Neutron contacts memcached upon some specific API requests. In fact, in our environment lbaas is not configured in Neutron (I don't know if this still possible in the current Neutron version), but we have deployed a monitoring service based on Prometheus openstack-exporter which, by default, checks the accessibility of the lbaas neutron part of the API.
Each time the check have the neutron server generates a 404, as this part of the API is not available. An example can be found at http://paste.openstack.org/show/797113/
The actual issue happens at the socket management level. Each time the check is performed, the socket between the Neutron server and memcached is not released as expected. This leads to a continuous increment of the established sockets between Neutron an Memcached until all the available sockets are exhausted and the cluster authentication locks.
Removing the checking on the lbaas service from Prometheus openstack-exporter allows to avoid the issue but we think that an issue like this (if confirmed) have quite important security concerns as it can lead quite easily to a DoS (wanted or not).
For the record, We have tried to explicitly configure the memcached socket monitoring options in Neutron but it looks like that they are not applied under the current conditions :
memcache_pool_socket_timeout = 3
memcache_pool_unused_timeout = 60
Here under is the environment setup:
Deployer : OpenStack Ansible 20.1.2
Neutron version: neutron 15.1.1.dev2
Neutron config file http://paste.openstack.org/show/797117/
Memcached config file http://paste.openstack.org/show/797118/
Prometheus OpenStack Exporter : https://github.com/openstack-exporter/openstack-exporter
Prometheus OpenStack Exporter config : http://paste.openstack.org/show/797119/
Prometheus command WITH lbaas checking : =/usr/local/bin/openstack-exporter \
--os-client-config=/etc/openstack-exporter/openstack-exporter.yml \
--web.telemetry-path="/metrics" \
--web.listen-address="0.0.0.0:9180" \
--prefix="openstack" \
--endpoint-type="public"
Prometheus command WITHOUT lbaas checking:
/usr/local/bin/openstack-exporter \
--os-client-config=/etc/openstack-exporter/openstack-exporter.yml \
--web.telemetry-path="/metrics" \
--web.listen-address="0.0.0.0:9180" \
--prefix="openstack" \
--endpoint-type="public" \
--disable-metric neutron-loadbalancers \
--disable-metric neutron-loadbalancers_not_active \
--disable-service.object-store \
--disable-service.load-balancer \
I hope the provided information is enough to reproduce the issue.
Cheers
Sergio |
We have recently installer an OpenStack cluster in Train release and we have noticed an unexpected behavior when Neutron contacts memcached upon some specific API requests. In fact, in our environment lbaas is not configured in Neutron (I don't know if this still possible in the current Neutron version), but we have deployed a monitoring service based on Prometheus openstack-exporter which, by default, checks the accessibility of the lbaas neutron part of the API.
Each time the check have the neutron server generates a 404, as this part of the API is not available. An example can be found at http://paste.openstack.org/show/797113/
The actual issue happens at the socket management level. Each time the check is performed, the socket between the Neutron server and memcached is not released as expected. This leads to a continuous increment of the established sockets between Neutron an Memcached until all the available sockets are exhausted and the cluster authentication locks.
Removing the checking on the lbaas service from Prometheus openstack-exporter allows to avoid the issue but we think that an issue like this (if confirmed) have quite important security concerns as it can lead quite easily to a DoS (wanted or not).
For the record, We have tried to explicitly configure the memcached socket monitoring options in Neutron but it looks like that they are not applied under the current conditions :
memcache_pool_socket_timeout = 3
memcache_pool_unused_timeout = 60
Here under is the environment setup:
Deployer : OpenStack Ansible 20.1.2
Neutron version: neutron 15.1.1.dev2
Neutron config file http://paste.openstack.org/show/797117/
Memcached config file http://paste.openstack.org/show/797118/
Prometheus OpenStack Exporter : https://github.com/openstack-exporter/openstack-exporter
Prometheus OpenStack Exporter config : http://paste.openstack.org/show/797119/
Prometheus command WITH lbaas checking : =/usr/local/bin/openstack-exporter \
--os-client-config=/etc/openstack-exporter/openstack-exporter.yml \
--web.telemetry-path="/metrics" \
--web.listen-address="0.0.0.0:9180" \
--prefix="openstack" \
--endpoint-type="public"
Prometheus command WITHOUT lbaas checking:
/usr/local/bin/openstack-exporter \
--os-client-config=/etc/openstack-exporter/openstack-exporter.yml \
--web.telemetry-path="/metrics" \
--web.listen-address="0.0.0.0:9180" \
--prefix="openstack" \
--endpoint-type="public" \
--disable-metric neutron-loadbalancers \
--disable-metric neutron-loadbalancers_not_active \
--disable-service.object-store \
--disable-service.load-balancer \
I hope the provided information is enough to reproduce the issue.
Cheers
Sergio |
|
| 2020-09-29 16:40:53 |
Jeremy Stanley |
information type |
Private Security |
Public Security |
|
| 2020-10-08 17:13:29 |
Radosław Piliszek |
bug |
|
|
added subscriber Radosław Piliszek |
| 2020-10-20 13:04:43 |
s10 |
bug |
|
|
added subscriber s10 |
| 2021-02-09 06:48:54 |
Summer Long |
bug |
|
|
added subscriber Summer Long |
| 2021-02-17 20:39:06 |
Jeremy Stanley |
ossa: status |
Incomplete |
Won't Fix |
|
| 2021-02-17 23:52:50 |
Summer Long |
removed subscriber Summer Long |
|
|
|
| 2021-02-18 15:20:43 |
Ben Nemec |
oslo.cache: status |
New |
Fix Released |
|
