shelve/unshelve does not work when instance created by bootable volume

[kwonho lee]

When instance created by bootable volume, "shelve/unshelve" does not work properly.

In case of "shelve" instance, attached root volume does not detached.

In case of "unshelve" instance, the instance that created by bootable volume does not have
image ref information. Also sys_meta does not have any shelved information such as
"shelved_image_id" . Because when execute the "unshelve" command, "shelve_offload_instance" method executed. In that case, Nova Conductor raised errors belows.

 2014-03-03 12:17:37 ERROR [nova.openstack.common.rpc.amqp] Exception during message handling
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
    **args)
  File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
    result = getattr(proxyobj, method)(ctxt, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/nova/conductor/manager.py", line 793, in unshelve_instance
    if sys_meta['shelved_image_id'] is not None :
KeyError: 'shelved_image_id'

Here is my test step.

1. Instance create using bootable colume

==> nova boot --flavor 1 klee_test_autoha1 --block-device id=e9494670-fb19-48c7-9154-20b136ce6356,source=image,dest=volume,device=vda,bootindex=0,shutdown=remove,size=1

2. Execute shelve/unshelve command

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/77803

[Nikola Đipanov (ndipanov)]

It wold be really good to know what "does not work properly" means in this case.

Anything like a stack trace from the log files or a brief explanation saying what happened would be really good. The bug in this form is not as useful as it could be.

I will also try to reproduce this and add some details.

[Mehdi Abaakouk (sileht)]

Step to reproduce:

I boot from volume like this:

$ nova boot --flavor 2 --key-name sileht --nic net-id=d4021bdd-ac52-44cb-8943-1587c6fe9081 --block-device source=image,id=6f797908-b006-4036-9670-cc3c12c65d56,dest=volume,size=2,shutdown=PRESERVE,bootindex=0 NAME

And then

$ nova shelve 76f4c00d-707a-4426-baba-13217250cec0

Shelving looks ok, now I unshelve:

$ nova unshelve 76f4c00d-707a-4426-baba-13217250cec0

The task state stuck at 'unshelving'

$ nova list
+--------------------------------------+------+-------------------+------------+-------------+---------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+-------------------+------------+-------------+---------------------------------+
| 76f4c00d-707a-4426-baba-13217250cec0 | NAME | SHELVED_OFFLOADED | unshelving | Shutdown | admpriv=192.168.0.5, 172.24.4.4 |
+--------------------------------------+------+-------------------+------------+-------------+---------------------------------+

nova-conductor logs:

2014-04-02 09:51:15.669 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: 'shelved_image_id'
2014-04-02 09:51:15.669 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-04-02 09:51:15.669 TRACE oslo.messaging.rpc.dispatcher File "/vagrant/stack-master/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-04-02 09:51:15.669 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-04-02 09:51:15.669 TRACE oslo.messaging.rpc.dispatcher File "/vagrant/stack-master/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-04-02 09:51:15.669 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-04-02 09:51:15.669 TRACE oslo.messaging.rpc.dispatcher File "/vagrant/stack-master/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-04-02 09:51:15.669 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-04-02 09:51:15.669 TRACE oslo.messaging.rpc.dispatcher File "/vagrant/stack-master/nova/nova/conductor/manager.py", line 833, in unshelve_instance
2014-04-02 09:51:15.669 TRACE oslo.messaging.rpc.dispatcher sys_meta['shelved_image_id'])
2014-04-02 09:51:15.669 TRACE oslo.messaging.rpc.dispatcher KeyError: 'shelved_image_id'
2014-04-02 09:51:15.669 TRACE oslo.messaging.rpc.dispatcher
2014-04-02 09:51:15.669 ERROR oslo.messaging._drivers.common [-] Returning exception 'shelved_image_id' to caller
2014-04-02 09:51:15.669 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', ' File "/vagrant/stack-master/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n incoming.message))\n', ' File "/vagrant/stack-master/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/vagrant/stack-master/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 122, in _do_disp...

[kwonho lee (lkhysy)]

Modified source code does not merged yet, so master code does not work properly.

I try to merge the code ASAP.

[haruka tanizawa (h-tanizawa)]

This problem is still happen?

[Tushar Patil (tpatil)]

I have verified this problem using the latest master code and it seems that it is fixed in this patch https://review.openstack.org/#/c/71907/13

LP bug https://bugs.launchpad.net/nova/+bug/1307416 is marked as duplicate but the above patch is merged under this bug somehow.

Lee, Michael: Request you to please set the status of this bug to duplicate after removing duplicate status from LP Bug 1307416.
Thanks.

[Sean Dague (sdague)]

Tushar I don't understand. The code you are referencing was merged a long time ago. Is this still an issue?

[Tushar Patil (tpatil)]

Sean: Sorry for the confusion.
I couldn't reproduce this problem anymore with the latest master code.
This bug should be marked as duplicate of 1307416 or invalid whichever is appropriate.

[Sean Dague (sdague)]

Thanks!