[Nova] Strange logic for shared images for different tenants/users

[Timur Nurlygayanov]

Reproduced on MOS 7.0, on scale lab (but it doesn't require scale lab for repro/verification)
upstream bug: https://bugs.launchpad.net/nova/+bug/1490528

Steps To Reproduce:
1. Login to OpenStack horizon dashboard as admin user
2. Upload Ubuntu cloud image into Glance
3. Boot VM 'test1' from Ubuntu image
4. Install stress tool on the VM: 'sudo apt-get install stress'
5. Add 'stress' tool in autorun: 'sudo echo "stress --cpu 10 &" > /etc/rc.local'
6. Reboot VM 'test1'
7. Make a snapshot of 'test1' VM
8. Mark this snapshot as 'public' in Glance
9. Create 10 VMs with this image (snapshot) from admin user. All VMs became Active in several seconds.
10. Login as non-admin user in another tenant (for example, user 'test' in tenant 'my-project')
11. Boot 10 VMs with public image 'TestImage'

Expected Result:
VMs will start as quickly as for the admin tenant.

Observed Result:
VMs hang in "Downloading Image" operation, it takes more than 20 minutes to run VM from the snapshot in other tenants (but in the same time it takes a few seconds to run VMs from the same image from tenant where we have created the image).

Looks like Nova tries to copy this image for each new tenant, and it doesn't looks correct.

[Timur Nurlygayanov (tnurlygayanov)]

I'm not sure that the issue in Nova, probably the root of the issue in Glance, but anyway it looks strange, we shouldn't have difference for shared images / snapshots.

[Roman Podoliaka (rpodolyaka)]

Timur, are you sure that these new instances did not happen to be spawned on *other* compute nodes?

I just double checked this on my devstack: images are cached per compute node, not per tenant. Once you have spawned an instance from this particular image on this compute, for all subsequent VMs nova-compute will *not* download the image from Glance again, but will use the cached one instead (technically, nova-compute uses it as a backing file for QCOW2 images and copies a RAW image locally).

[Timur Nurlygayanov (tnurlygayanov)]

Roman, yes, it will boot VM on other compute nodes. But under the admin tenant it will take several seconds to run VMs on any compute nodes, and will take 20+ minutes to run VMs under the other tenant.

Probably this issue not reproduced on the devstack, I can provide the access to my environment if it is required.

[Timur Nurlygayanov (tnurlygayanov)]

Not reproduced on the latest MOS 7.0 image #265.

[Roman Podoliaka (rpodolyaka)]

I can't reproduce it either:

http://paste.openstack.org/show/445371/