nova boot succeeds for deleted glance image

Bug #1054163 reported by kirk sticken
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Michael Still

Bug Description

calling this a security exposure since delete data is resurrected

an image existed with uuid ending in af79:

ID Name Disk Format Container Format Size
------------------------------------ ------------------------------ -------------------- -------------------- --------------
dcaf164d-7295-4a5a-9878-c2e4b0d3af79 ub.vdi vdi ovf 965775360
0698b15f-160a-48aa-bd97-8ac696b8f8d6 avutil-51.dll ami ami 144424

then it was deleted:

ID Name Disk Format Container Format Size
------------------------------------ ------------------------------ -------------------- -------------------- --------------
0698b15f-160a-48aa-bd97-8ac696b8f8d6 avutil-51.dll ami ami 144424

then a new image with the same text name was added, but with uuid ending in 28f9:
ID Name Disk Format Container Format Size
------------------------------------ ------------------------------ -------------------- -------------------- --------------
b19e1ca7-256e-4aff-8ece-e3ea770e28f9 ub.vdi vdi ovf 0
0698b15f-160a-48aa-bd97-8ac696b8f8d6 avutil-51.dll ami ami 144424

then I booted the deleted image using nova

 n boot --flavor 1 --image dcaf164d-7295-4a5a-9878-c2e4b0d3af79 kirk5
+-------------------------------------+--------------------------------------+
| Property | Value |
+-------------------------------------+--------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | instance-00000006 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | 97SjHC9vZQaD |
| config_drive | |
| created | 2012-09-21T15:43:03Z |
| flavor | standard.xsmall |
| hostId | |
| id | 90114972-2386-480c-8e51-2cb2a41828d0 |
| image | ub.vdi |
| key_name | |
| metadata | {} |
| name | kirk5 |
| progress | 0 |
| status | BUILD |
| tenant_id | aeb143d32b734e6dae0145cbfe7561ba |
| updated | 2012-09-21T15:43:03Z |
| user_id | 804e32e815c146b7b955f19355b3b7a9 |
+-------------------------------------+--------------------------------------+
[root@sticken-isc1 ~]# n list
+--------------------------------------+-------+--------+------------------+
| ID | Name | Status | Networks |
+--------------------------------------+-------+--------+------------------+
| 90114972-2386-480c-8e51-2cb2a41828d0 | kirk5 | ACTIVE | novanet=10.1.0.6 |
+--------------------------------------+-------+--------+------------------+
[root@sticken-isc1 ~]# n list
+--------------------------------------+-------+--------+-------------------------------+
| ID | Name | Status | Networks |
+--------------------------------------+-------+--------+-------------------------------+
| 90114972-2386-480c-8e51-2cb2a41828d0 | kirk5 | ACTIVE | novanet=10.1.0.6, 192.168.1.7 |
+--------------------------------------+-------+--------+-------------------------------

Revision history for this message
kirk sticken (william-kir-sticken) wrote :

opened against glance at first, changed to nova compute

affects: glance → nova
Revision history for this message
Brian Waldon (bcwaldon) wrote :

So here is what seems to be happening:

1) A non-admin user boots the image in question, caching it on the nova-compute node (by uuid)
2) That non-admin user deletes said image - the image is *not* removed from the cache, however
3) An admin user can still see the metadata in Glance, and since the data is cached locally, it is still bootable by that admin user

It is key to realize that Glance allows admin users to make HEAD (metadata-only) requests of images through the v1 API. When Nova checks for image existence before trying to boot the vm and everything seems to check out. It doesnt attempt to download the data since it has a local cached copy.

I tried creating a new vm from an image that had never been cached and the vm did go to ERROR.

Revision history for this message
Russell Bryant (russellb) wrote :

If only an admin can boot the deleted image, I'd say this is just a regular bug and not a vulnerability. Thoughts?

Revision history for this message
Brian Waldon (bcwaldon) wrote :

That sounds fair. We should probably delete images from the cache if the image was deleted explicitly through the OpenStack Compute API.

Revision history for this message
kirk sticken (william-kir-sticken) wrote :

my thought: it is important that proper data clensing be done. having a hidden image (on the file system?) that does not show in "glance index" seems dicey to me. at least make a "flush the cache" parameter part of the delete image call

Changed in nova:
status: New → Confirmed
importance: Undecided → High
importance: High → Medium
Revision history for this message
Russell Bryant (russellb) wrote :

The cache is distributed across all compute nodes, though. Forcing it to be deleted from the cache would require broadcasting a message to all compute nodes every time an image is deleted, though. Is there a way for nova to know that an image has been deleted based on the metadata the admin user can see about that image?

On a related note, right now if the image is no longer in use (no instances running on that compute node that used it as their base), it will get deleted from the cache by a periodic task that runs every 40 to 60 minutes or so (40 ticks, so 40 * 60 seconds plus however long it takes for 40 runs of periodic tasks).

Revision history for this message
Michael Still (mikal) wrote :

We can't just delete the image from the cache -- it might still be in use for instances which were started before the delete. It seems like we need a check to make sure the image still exists before we create a new instance.

Revision history for this message
Michael Still (mikal) wrote :

Ok, so attached is a possible fix for this problem. I have two questions:

 - what do people think of this approach?
 - what's the right way to push this out for review? Just plonk it into gerrit with no bug comment? Something else?

Revision history for this message
Thierry Carrez (ttx) wrote :

I agree that this is not a vulnerability per se (due do it only being accessible to admins) but should definitely be strengthened. Will open the bug to public soon unless someone objects.

Revision history for this message
Vish Ishaya (vishvananda) wrote :

Yes open to public. Only admins can do it so I don't think there is a security issue here. Mikal: patch seems reasonable to me.

Michael Still (mikal)
Changed in nova:
assignee: nobody → Michael Still (mikalstill)
Thierry Carrez (ttx)
security vulnerability: yes → no
visibility: private → public
Revision history for this message
Michael Still (mikal) wrote :

I messed up the commit message tag for this bug, so here's me doing it manually. The patch just merged as https://review.openstack.org/#/c/13692/

Changed in nova:
status: Confirmed → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-1 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.