[FG-VD-16-015] Openstack Glance Authenticated User DoS Vulnerability Notification

[chro eric]

Vulnerability Notification
March 7, 2016
Tracking Case #: FG-VD-16-015

Dear Openstack,

The following information pertains to information discovered by Fortinet's FortiGuard Labs. It has been determined that a vulnerability exists in Openstack Glance module. To streamline the disclosure process, we have created a preliminary advisory which you can find below. This upcoming advisory is purely intended as a reference, and does not contain sensitive information such as proof of concept code.

As a mature corporation involved in security research, we strive to responsibly disclose vulnerability information. We will not post an advisory until we determine it is appropriate to do so in co-ordination with the vendor unless a resolution cannot be reached. We will not disclose full proof of concept, only details relevant to the advisory.

We look forward to working closely with you to resolve this issue, and kindly ask for your co-operation during this time. Please let us know if you have any further questions, and we will promptly respond to address any issues.

If this message is not encrypted, it is because we could not find your key to do so. If you have one available for use, please notify us and we will ensure that this is used in future correspondence. We ask you use our public PGP key to encrypt and communicate any sensitive information with us. You may find the key on our FortiGuard center at: http://www.fortiguard.com/pgp_key.html.

Type of Vulnerability & Repercussions:
  DoS

Affected Software:
  Ubuntu 14.04.3 with latest repository installed
  # apt-get install software-properties-common
  # add-apt-repository cloud-archive:liberty

Upcoming Advisory Reference:
  http://www.fortiguard.com/advisory/UpcomingAdvisories.html

Credits:
  This vulnerability was discovered by Fortinet's FortiGuard Labs.

Proof of Concept/How to Reproduce:
  1. Run script "sh curl_get_token_demo_work.txt" to get a valid non-admin or admin user token. Need to replace "tenantName", "username", "password" with your Openstack credential.
  2. Open script glance_DoS.py, and replace the line 30 "x-auth-token" value with the above token value, also replace the IP in url "http://10.0.0.11:9191/images" with your Openstack control node IP address.
  3. Run script glance_DoS.py which will keep running forever. You can check the images added by the script using console command "glance image-list" or clicking Dashboard images column. You will notice you cannnot delete the images added by the script. It prompts failure. Refer to the screenshots glance_cli_delete_fail.png and dashboard_delete_garbage_image_fail.png.
  4. Because either non-admin or admin user cannot delete the garbage images, with the above PoC running forever, more and more garbage images are added. So finally DoS can be caused because resource is exhausted or glance database query is very very slow.

 Notes:
     1) Run the PoC glance_DoS.py in Windows 7.

 Additional Information:

[Tristan Cacqueray (tristan-cacqueray)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

This seems like a duplicate of bug 1545092, which is still private. Can you confirm this glance-coresec ?

[Kairat Kushaev (kkushaev)]

Hello, I think this is not a duplicate.
Here we see that images are modified directly in glance-registry (so glance api is not involved at all).
Here is 2 questions I am wondering:
how to prevent creation of these records
why these images shown in image-list output.
Still analyzing the bug.

[Stuart McLaren (stuart-mclaren)]

The glance registry (like the database) should be firewalled off from regular users. A regular user should only be able to access the glance api endpoint (typically on port 9292), not the glance registry.

Any deployment which allows direct access to the glance registry for standard users has been misconfigured.

Assuming a well configured deployment, this bug covers the ability of an admin, on their private network, to DOS their own deployment by sending bogus input to the registry.

I don't think a properly deployed system (where only the Glance API, not the registry, is accessible) will allow a regular user to use this registry behaviour as an attack.

That said, I think the registry should do a better job here, eg in returning 400 for the input in question (which seems to create image records which can't be deleted).

[Flavio Percoco (flaper87)]

I tend to agree with Stuart that this might be a deployment issue. It's true the registry is not super secure but again, it's not meant to be put as a public endpoint and users should *NEVER* talk to it directly.

[chro eric (chrorxu)]

I think have many people have chance attack glance-registery , because Firewall lies in network gateway, in internal network, still have many different privilege user interactive with Openstack. as we known, insider attack is common, see http://www.defenddemocracy.org/media-hit/pasi-eronen-every-corporation-has-an-insider-cyberthreat/, so I think the DoS should also be a vulnerability which need fixed.

[Stuart McLaren (stuart-mclaren)]

If you have a rogue admin on your network I think there is much greater scope for malicious activity than just filling up the database with bogus records.

If possible, I agree it would be better if the registry rejected requests that created new image records that can't be deleted via the API.

(It's not strictly true that those records can't be deleted by an admin. They can be deleted by accessing the database directly.)

[Jeremy Stanley (fungi)]

This seems like a security hardening opportunity. We already classify bugs which require access to service management networks or which need admin privileges as security hardening opportunities because we have so many possible avenues of attack against management interfaces (including but not limited to lack of SSL certificate validation, automatically trusting SSH host keys, a completely unsecured message bus) that we wouldn't ever have enough time to track and publish formal advisories for them all in OpenStack's current state. Instead we publish secure deployment recommendations in http://docs.openstack.org/sec/ to cover those sorts of challenges.

I have hopes this situation will change and eventually deployers might expect to be able to distribute OpenStack service backends across untrusted network links, expose management interfaces to the untrusted parties/the Internet without filtering, or have confident levels of privilege and trust separation between various service administrator roles... but getting there is a massive undertaking which should not be encumbered by bottlenecks like vulnerability management teams, embargoes or coordinated disclosures.

So with the above, I'm recommending we switch this bug to public and tag it as a security hardening opportunity consistent with a class D report in our taxonomy ( https://security.openstack.org/vmt-process.html#incident-report-taxonomy ).

[Flavio Percoco (flaper87)]

I agree with Jeremy's assessment. Before we do the switch, I'd like to get confirmation from at least one more member of Glance's vulnerability team.

Thanks Jeremy.

[chro eric (chrorxu)]

I want to mention in the internal network, it not only admin can attack but
also non-admin also can attack glance-registery via the vulnerability.

On Fri, Mar 11, 2016 at 3:01 AM, Flavio Percoco <email address hidden>
wrote:

> I agree with Jeremy's assessment. Before we do the switch, I'd like to
> get confirmation from at least one more member of Glance's vulnerability
> team.
>
> Thanks Jeremy.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1554288
>
> Title:
> [FG-VD-16-015] Openstack Glance Authenticated User DoS Vulnerability
> Notification
>
> Status in Glance:
> New
> Status in OpenStack Security Advisory:
> Incomplete
>
> Bug description:
> This issue is being treated as a potential security risk under
> embargo. Please do not make any public mention of embargoed (private)
> security vulnerabilities before their coordinated publication by the
> OpenStack Vulnerability Management Team in the form of an official
> OpenStack Security Advisory. This includes discussion of the bug or
> associated fixes in public forums such as mailing lists, code review
> systems and bug trackers. Please also avoid private disclosure to
> other individuals not already approved for access to this information,
> and provide this same reminder to those who are made aware of the
> issue prior to publication. All discussion should remain confined to
> this private bug report, and any proposed fixes should be added to the
> bug as attachments.
>
> --
>
> Vulnerability Notification
> March 7, 2016
> Tracking Case #: FG-VD-16-015
>
> Dear Openstack,
>
> The following information pertains to information discovered by
> Fortinet's FortiGuard Labs. It has been determined that a
> vulnerability exists in Openstack Glance module. To streamline the
> disclosure process, we have created a preliminary advisory which you
> can find below. This upcoming advisory is purely intended as a
> reference, and does not contain sensitive information such as proof of
> concept code.
>
> As a mature corporation involved in security research, we strive to
> responsibly disclose vulnerability information. We will not post an
> advisory until we determine it is appropriate to do so in co-
> ordination with the vendor unless a resolution cannot be reached. We
> will not disclose full proof of concept, only details relevant to the
> advisory.
>
> We look forward to working closely with you to resolve this issue, and
> kindly ask for your co-operation during this time. Please let us know
> if you have any further questions, and we will promptly respond to
> address any issues.
>
> If this message is not encrypted, it is because we could not find your
> key to do so. If you have one available for use, please notify us and
> we will ensure that this is used in future correspondence. We ask you
> use our public PGP key to encrypt and communicate any sensitive
> information with us. You may find the key on our FortiGuard center at:
> http://www.fortiguard.com/pgp_key.html.
>
> Type of Vulnerability & Repercussions:
> ...

[Jeremy Stanley (fungi)]

Are you implying that there's no way to block access from tenant/project networks to the network on which Glance's registry service is exposed? Or are you saying in your particular test environment you deployed Glance in such a way that you exposed the registry endpoint to untrusted (non-admin/non-management) systems?

[chro eric (chrorxu)]

I agree deploy glance-registery should be protected environment, I think
block glance-registery service via Firewall in gateway only can protected
from internet attack, In internal network, I mean for theose insider, still
have admin user also have non-admin user, if a malcious inside attacker
which is a non-admin user(he has not admin priviledge), he can still cause
DoS attack.

BTW, because I see from above discussion, most foucs in admin user can
attack, but for the vulnerability, non-admin insider user also can attack.

On Fri, Mar 11, 2016 at 6:55 AM, Jeremy Stanley <email address hidden> wrote:

> Are you implying that there's no way to block access from tenant/project
> networks to the network on which Glance's registry service is exposed?
> Or are you saying in your particular test environment you deployed
> Glance in such a way that you exposed the registry endpoint to untrusted
> (non-admin/non-management) systems?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1554288
>
> Title:
> [FG-VD-16-015] Openstack Glance Authenticated User DoS Vulnerability
> Notification
>
> Status in Glance:
> New
> Status in OpenStack Security Advisory:
> Incomplete
>
> Bug description:
> This issue is being treated as a potential security risk under
> embargo. Please do not make any public mention of embargoed (private)
> security vulnerabilities before their coordinated publication by the
> OpenStack Vulnerability Management Team in the form of an official
> OpenStack Security Advisory. This includes discussion of the bug or
> associated fixes in public forums such as mailing lists, code review
> systems and bug trackers. Please also avoid private disclosure to
> other individuals not already approved for access to this information,
> and provide this same reminder to those who are made aware of the
> issue prior to publication. All discussion should remain confined to
> this private bug report, and any proposed fixes should be added to the
> bug as attachments.
>
> --
>
> Vulnerability Notification
> March 7, 2016
> Tracking Case #: FG-VD-16-015
>
> Dear Openstack,
>
> The following information pertains to information discovered by
> Fortinet's FortiGuard Labs. It has been determined that a
> vulnerability exists in Openstack Glance module. To streamline the
> disclosure process, we have created a preliminary advisory which you
> can find below. This upcoming advisory is purely intended as a
> reference, and does not contain sensitive information such as proof of
> concept code.
>
> As a mature corporation involved in security research, we strive to
> responsibly disclose vulnerability information. We will not post an
> advisory until we determine it is appropriate to do so in co-
> ordination with the vendor unless a resolution cannot be reached. We
> will not disclose full proof of concept, only details relevant to the
> advisory.
>
> We look forward to working closely with you to resolve this issue, and
> kindly ask for your co-operation during this time. Please let us know
> if you have any furth...

[chro eric (chrorxu)]

I think because glance-registery service listen in 0.0.0.0 address, except
those people blocked by Firewall access 9191 port, glance-registery service
still can be accessed by any people which is not firewall blocked, if these
people include a insider attacker, he may not be admin user, he has not
admin priviledge, he is still can launch DoS attack. that is to say, for
the vulnerability, non-admin insider user also can attack, so impact is
greater than only admin-user attack.

On Fri, Mar 11, 2016 at 7:13 AM, Ch Chror <email address hidden> wrote:

> I agree deploy glance-registery should be protected environment, I think
> block glance-registery service via Firewall in gateway only can protected
> from internet attack, In internal network, I mean for theose insider, still
> have admin user also have non-admin user, if a malcious inside attacker
> which is a non-admin user(he has not admin priviledge), he can still cause
> DoS attack.
>
> BTW, because I see from above discussion, most foucs in admin user can
> attack, but for the vulnerability, non-admin insider user also can attack.
>
>
>
> On Fri, Mar 11, 2016 at 6:55 AM, Jeremy Stanley <email address hidden> wrote:
>
>> Are you implying that there's no way to block access from tenant/project
>> networks to the network on which Glance's registry service is exposed?
>> Or are you saying in your particular test environment you deployed
>> Glance in such a way that you exposed the registry endpoint to untrusted
>> (non-admin/non-management) systems?
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1554288
>>
>> Title:
>> [FG-VD-16-015] Openstack Glance Authenticated User DoS Vulnerability
>> Notification
>>
>> Status in Glance:
>> New
>> Status in OpenStack Security Advisory:
>> Incomplete
>>
>> Bug description:
>> This issue is being treated as a potential security risk under
>> embargo. Please do not make any public mention of embargoed (private)
>> security vulnerabilities before their coordinated publication by the
>> OpenStack Vulnerability Management Team in the form of an official
>> OpenStack Security Advisory. This includes discussion of the bug or
>> associated fixes in public forums such as mailing lists, code review
>> systems and bug trackers. Please also avoid private disclosure to
>> other individuals not already approved for access to this information,
>> and provide this same reminder to those who are made aware of the
>> issue prior to publication. All discussion should remain confined to
>> this private bug report, and any proposed fixes should be added to the
>> bug as attachments.
>>
>> --
>>
>> Vulnerability Notification
>> March 7, 2016
>> Tracking Case #: FG-VD-16-015
>>
>> Dear Openstack,
>>
>> The following information pertains to information discovered by
>> Fortinet's FortiGuard Labs. It has been determined that a
>> vulnerability exists in Openstack Glance module. To streamline the
>> disclosure process, we have created a preliminary advisory which you
>> can find below. This upcoming advisory is purely intended as a
>> reference, and ...

[Stuart McLaren (stuart-mclaren)]

> 4. Because either non-admin or admin user cannot delete the garbage images, with the above PoC running forever, more and more garbage images are added. So finally DoS can be caused because resource is exhausted or glance database query is very very slow.

Note that it is possible for an admin to delete the garbage images using the Registry API:

 $ curl -v -H "x-auth-token: $TOKEN" -X DELETE http://localhost:9191/images/dffde883-e723-45ab-8be7-be7fc2afb8e4

 $ glance --os-image-api-version 1 image-show dffde883-e723-45ab-8be7-be7fc2afb8e4
 No image with a name or ID of 'dffde883-e723-45ab-8be7-be7fc2afb8e4' exists.

It looks to me that the proposed attack was based on an admin not being able to delete the garbage images using API requests -- but this doesn't seem to be the case.

This is starting to feel quite similar to https://bugs.launchpad.net/glance/+bug/1545092, but related to the registry rather than the api.

[chro eric (chrorxu)]

The following is another DoS, it can be infer from PoC glance_dos.py.
This is a create random specified uuid in glance image, then deleted it as if nothing happended, but if glance system re-create image with uuid same as above uuid, the image create action will failed, If openstack glance verbose log have not enabled, the create-delete image action in PoC glance_dos2.py is unnoticed, but if the create specified uuid image and delete the image action is loop always, for example last 1 year or two, the cannot used image uuid more and more, the image create DoS should occure.

note: 1. replace PoC glance_dos2.py chro_token use a valid non-admin or admin user token, for example demo.
      2. the create-delete image call glance-api.

[chro eric (chrorxu)]

For the glance_dos2, if a malcious user launch a distributed DoS attack, I mean using over ten thousands node run glance_dos2.py for long long time, the DoS should come to real.

[chro eric (chrorxu)]

For the glance_dos2.py, seems is not possibiliy, http://stackoverflow.com/questions/1155008/how-unique-is-uuid

[Stuart McLaren (stuart-mclaren)]

@chro eric

Yes, I think, as long as the UUIDs are being generated in a sufficiently random way there should be enough that it is safe to assume duplicates are extremely unlikely.

(BTW I think it's great that someone is looking at finding vulnerabilities in Glance -- keep it up!)

[Jeremy Stanley (fungi)]

Any vulnerability hinging on attempts to trigger a UUID collision would fall under class C1 in our report taxonomy: https://security.openstack.org/vmt-process.html#incident-report-taxonomy

[chro eric (chrorxu)]

I have got that glance_dos2.py is an class C1 vulnerability(Not considered a practical vulnerability (but some people might assign a CVE for it).

Could I request a CVE assign to it ? and when you post security advisory could you please just mention us for some short words in it ?

[Jeremy Stanley (fungi)]

If it ends up being agreed upon as C1 (or anything other than class A) there won't be an OpenStack Security Advisory and the OpenStack Vulnerability Management Team won't request a CVE. It's possible the OpenStack Security Notes editors will publish some information on it, in which case it will be up to them to include credit for the discovery. Anyone can request a CVE for tracking this once it's public, even if the VMT does not deem it a practical vulnerability.

Anyway, given the low risk/impact of the reported issue, I recommend making this public by the end of the week unless there are any objections.

[chro eric (chrorxu)]

Thanks comment, We will try to request a CVE number for the C1 vulnerability.

[Jeremy Stanley (fungi)]

Since there are no objections, I'm switching this to public and marking as a hardening opportunity.

[Travis McPeak (travis-mcpeak)]

Sorry I'm playing catch up a bit here but UUID collision definitely should be infeasible. Also image creation for non-admin users should be subject to quotas, correct?