Cannot resize instance if base image is not available

[Belmiro Moreira]

Nova fails to resize instances if the base image is not available anymore in glance.

nova resize instance01 "m1.medium"
ERROR: The resource could not be found. (HTTP 404) (Request-ID: req-37976706-cb02-46df-b7d8-11ceb96da63f)

In this cases I would expect that a complete image was copied to the target compute node and not only the "delta".

[Chuck Short (zulcss)]

Which version and can you attach the traceback that you get if any?

chuck

[Belmiro Moreira (moreira-belmiro-email-lists)]

I'm using Folsom.
But Grizzly seems to have the same problem to me.
It fails when gets image info from glance:
2013-03-22 16:16:58 1662 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/compute/api.py", line 1716, in resize
2013-03-22 16:16:58 1662 TRACE nova.api.openstack image = self.image_service.show(context, instance['image_ref'])

Complete Traceback:

2013-03-22 16:16:58 ERROR nova.api.openstack [req-37976706-cb02-46df-b7d8-11ceb96da63f USER ad7ad3ebda374bdfb40ee6f797ad6d01] Caught error: Image 8c60f4be-1189-43f8-ba74-d432
e619d6f1 could not be found.
2013-03-22 16:16:58 1662 TRACE nova.api.openstack Traceback (most recent call last):
2013-03-22 16:16:58 1662 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 78, in __call__
2013-03-22 16:16:58 1662 TRACE nova.api.openstack return req.get_response(self.application)
2013-03-22 16:16:58 1662 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/request.py", line 1053, in get_response
2013-03-22 16:16:58 1662 TRACE nova.api.openstack application, catch_exc_info=False)
2013-03-22 16:16:58 1662 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/request.py", line 1022, in call_application
2013-03-22 16:16:58 1662 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2013-03-22 16:16:58 1662 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2013-03-22 16:16:58 1662 TRACE nova.api.openstack return resp(environ, start_response)
2013-03-22 16:16:58 1662 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/keystone/middleware/auth_token.py", line 278, in __call__
2013-03-22 16:16:58 1662 TRACE nova.api.openstack return self.app(env, start_response)
2013-03-22 16:16:58 1662 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2013-03-22 16:16:58 1662 TRACE nova.api.openstack return resp(environ, start_response)
2013-03-22 16:16:58 1662 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2013-03-22 16:16:58 1662 TRACE nova.api.openstack return resp(environ, start_response)
2013-03-22 16:16:58 1662 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2013-03-22 16:16:58 1662 TRACE nova.api.openstack return resp(environ, start_response)
2013-03-22 16:16:58 1662 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/middleware.py", line 131, in __call__
2013-03-22 16:16:58 1662 TRACE nova.api.openstack response = self.app(environ, start_response)
2013-03-22 16:16:58 1662 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2013-03-22 16:16:58 1662 TRACE nova.api.openstack return resp(environ, start_response)
2013-03-22 16:16:58 1662 TRACE nova.api.openstack File "/usr...

[Michael Still (mikal)]

What compute driver are you using?

[Belmiro Moreira (moreira-belmiro-email-lists)]

Hi,
I'm using the LibvirtDriver.

[wangpan (hzwangpan)]

I think the reason of this issue may be that,
1. create an instance with an image on glance
2. delete the image on glance
3. resize the instance
and then this issue occurs,
we may need to forbid deleting an in-use image on glance.

[Belmiro Moreira (moreira-belmiro-email-lists)]

Hi wangpan,
yes, that is the reason.

In my option there are two possible solutions:
1) if the image is removed from glance then nova copies everything from the compute node where the instance is running.
2) or, as you said not allow delete images from glance if they are being used.

The option 2) has several consequences:
In a big infrastructure with hundreds of users, not allowing delete an image in use means that "virtually" no image can be deleted. Also, it wouldn't be possible to update images (keep the same name and change the image file) which is a problem for public images.

In my opinion we should go through option 1)

[Aimon Bustardo (aimonb)]

+1 to option 1 from Belmiro.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/29534

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/29793

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/29793
Committed: http://github.com/openstack/nova/commit/65a6264c236a779712694d2379cfe4f9e46e2732
Submitter: Jenkins
Branch: master

commit 65a6264c236a779712694d2379cfe4f9e46e2732
Author: Nikola Dipanov <email address hidden>
Date: Mon May 20 17:36:49 2013 +0200

    Disallow resize if image not available

    If a resize is attempted on an instance that was started from an image
    that has since been deleted, a resize will fail. This change makes error
    reporting a bit cleaner.

    This change is needed since in order to actually properly support
    resize/migrate when the image is deleted - it is necessary for nova to keep
    a copy of the image metadata and re-use it in case of migration/resize.

    Fixes bug: 1160773
    Fixes bug: 1177001

    Change-Id: Ifaea71f79c97046a4cde094e3a5e676772fcceb4

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/30442

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/30442
Committed: http://github.com/openstack/nova/commit/ab51ac688e3019c068e99814ae766d994ff41d00
Submitter: Jenkins
Branch: stable/grizzly

commit ab51ac688e3019c068e99814ae766d994ff41d00
Author: Nikola Dipanov <email address hidden>
Date: Mon May 20 17:36:49 2013 +0200

    Disallow resize if image not available

    If a resize is attempted on an instance that was started from an image
    that has since been deleted, a resize will fail. This change makes error
    reporting a bit cleaner.

    This change is needed since in order to actually properly support
    resize/migrate when the image is deleted - it is necessary for nova to keep
    a copy of the image metadata and re-use it in case of migration/resize.

    Fixes bug: 1160773
    Fixes bug: 1177001

    Change-Id: Ifaea71f79c97046a4cde094e3a5e676772fcceb4
    (cherry picked from commit 65a6264c236a779712694d2379cfe4f9e46e2732)