Nova shelve creates duplicated images in cells environment

[Ryo Miki]

Description
===========

Nova shelve creates duplicated images in cells environment.
I think, this behavior is caused by below patch.
(not reproduced in icehouse version)

commit 394ac5eea215831e82f96d43177442af67b40bb4

I modified source code like below, and restart nova-api. It seems working fine.

diff --git a/nova/compute/cells_api.py b/nova/compute/cells_api.py
index 175ece2..4fe44a3 100644
--- a/nova/compute/cells_api.py
+++ b/nova/compute/cells_api.py
@@ -322,8 +322,6 @@ class ComputeCellsAPI(compute_api.API):
     @check_instance_cell
     def shelve(self, context, instance, clean_shutdown=True):
         """Shelve the given instance."""
- super(ComputeCellsAPI, self).shelve(context, instance,
- clean_shutdown=clean_shutdown)
         self._cast_to_cells(context, instance, 'shelve',
                 clean_shutdown=clean_shutdown)

Steps to reproduce
==================

1. Create cells environment (ex. using devstack)

2. Create vm by Nova

openstack@localhost:~$ nova list
+--------------------------------------+---------+--------+------------+-------------+--------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+--------+------------+-------------+--------------------------------+
| a520a4a7-6991-40e1-885a-bd3138c2b5dc | test-vm | ACTIVE | - | Running | public=2001:db8::4, 172.24.4.4 |
+--------------------------------------+---------+--------+------------+-------------+--------------------------------+

3. Execute nova shelve to vm

openstack@localhost:~$ nova shelve test-vm
openstack@localhost:~$ nova list
+--------------------------------------+---------+-------------------+------------+-------------+--------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+-------------------+------------+-------------+--------------------------------+
| a520a4a7-6991-40e1-885a-bd3138c2b5dc | test-vm | SHELVED_OFFLOADED | - | Shutdown | public=2001:db8::4, 172.24.4.4 |
+--------------------------------------+---------+-------------------+------------+-------------+--------------------------------+

4. Two images which is named <vmname>-shelved are created in glance

openstack@localhost:~$ glance image-list
+--------------------------------------+---------------------------------+
| ID | Name |
+--------------------------------------+---------------------------------+
| 3a106d4e-0b55-4e84-990a-a8aa4ae68c2a | cirros-0.3.4-x86_64-uec |
| ffc958ac-7435-42a4-9dc6-b42a7c518c2e | cirros-0.3.4-x86_64-uec-kernel |
| 51378b3f-86f6-4d70-a22e-768599eafc09 | cirros-0.3.4-x86_64-uec-ramdisk |
| 42a6f2b0-c1b4-43d2-95c8-45e7c2d6ca3e | test-vm-shelved |
| cc06acba-0911-49ec-86c2-d08c1084fbbe | test-vm-shelved |
+--------------------------------------+---------------------------------+

5. Only an glance image is used for nova-shelve.

openstack@localhost:~$ glance image-show 42a6f2b0-c1b4-43d2-95c8-45e7c2d6ca3e
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| base_image_ref | 3a106d4e-0b55-4e84-990a-a8aa4ae68c2a |
| checksum | None |
| container_format | ami |
| created_at | 2016-06-03T06:16:32Z |
| disk_format | ami |
| id | 42a6f2b0-c1b4-43d2-95c8-45e7c2d6ca3e |
| image_type | snapshot |
| instance_uuid | a520a4a7-6991-40e1-885a-bd3138c2b5dc |
| kernel_id | ffc958ac-7435-42a4-9dc6-b42a7c518c2e |
| min_disk | 1 |
| min_ram | 0 |
| name | test-vm-shelved |
| owner | 73af4afafeae4d5189d8190a8e876418 |
| protected | False |
| ramdisk_id | 51378b3f-86f6-4d70-a22e-768599eafc09 |
| size | 0 |
| status | queued |
| tags | [] |
| updated_at | 2016-06-03T06:16:32Z |
| user_id | c4954683e8244a0f84f509b8f3d63a4d |
| virtual_size | None |
| visibility | private |
+------------------+--------------------------------------+
openstack@localhost:~$ glance image-show cc06acba-0911-49ec-86c2-d08c1084fbbe
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| base_image_ref | 3a106d4e-0b55-4e84-990a-a8aa4ae68c2a |
| checksum | ac94fabcc66952b36066eebc6af577d6 |
| container_format | ami |
| created_at | 2016-06-03T06:16:33Z |
| disk_format | ami |
| id | cc06acba-0911-49ec-86c2-d08c1084fbbe |
| image_location | snapshot |
| image_state | available |
| image_type | snapshot |
| instance_uuid | a520a4a7-6991-40e1-885a-bd3138c2b5dc |
| kernel_id | ffc958ac-7435-42a4-9dc6-b42a7c518c2e |
| min_disk | 1 |
| min_ram | 0 |
| name | test-vm-shelved |
| owner | 73af4afafeae4d5189d8190a8e876418 |
| owner_id | 73af4afafeae4d5189d8190a8e876418 |
| protected | False |
| ramdisk_id | 51378b3f-86f6-4d70-a22e-768599eafc09 |
| size | 10682368 |
| status | active |
| tags | [] |
| updated_at | 2016-06-03T06:17:02Z |
| user_id | c4954683e8244a0f84f509b8f3d63a4d |
| virtual_size | None |
| visibility | private |
+------------------+--------------------------------------+

Expected result
===============

* Only an image is created

openstack@localhost:~$ glance image-list
+--------------------------------------+---------------------------------+
| ID | Name |
+--------------------------------------+---------------------------------+
| 3a106d4e-0b55-4e84-990a-a8aa4ae68c2a | cirros-0.3.4-x86_64-uec |
| ffc958ac-7435-42a4-9dc6-b42a7c518c2e | cirros-0.3.4-x86_64-uec-kernel |
| 51378b3f-86f6-4d70-a22e-768599eafc09 | cirros-0.3.4-x86_64-uec-ramdisk |
| cc06acba-0911-49ec-86c2-d08c1084fbbe | test-vm-shelved |
+--------------------------------------+---------------------------------+

Actual result
=============

* Duplicated image is created on glance

openstack@localhost:~$ glance image-list
+--------------------------------------+---------------------------------+
| ID | Name |
+--------------------------------------+---------------------------------+
| 3a106d4e-0b55-4e84-990a-a8aa4ae68c2a | cirros-0.3.4-x86_64-uec |
| ffc958ac-7435-42a4-9dc6-b42a7c518c2e | cirros-0.3.4-x86_64-uec-kernel |
| 51378b3f-86f6-4d70-a22e-768599eafc09 | cirros-0.3.4-x86_64-uec-ramdisk |
| 42a6f2b0-c1b4-43d2-95c8-45e7c2d6ca3e | test-vm-shelved |
| cc06acba-0911-49ec-86c2-d08c1084fbbe | test-vm-shelved |
+--------------------------------------+---------------------------------+

Environment
===========

* Devstack with cell
** This is snippet of localrc

ENABLED_SERVICES=g-api,g-reg,key,n-api,n-cpu,n-cell,n-net,n-cond,n-sch,n-novnc,n-cauth,mysql,neutron,q-svc,q-agt,q-dhcp,q-l3,q-meta,horizon,s-proxy,s-object,s-container,s-account,rabbit

* nova version
** This is on stable/liberty

git log -1
c5d0d2d0f4940d2e9f5888d7229fa4ac202ff161

Logs & Configs
==============

* Duplicated image is created on nova-api

2016-06-03 02:16:32.537 DEBUG glanceclient.common.http [req-c7963aa0-f38a-4580-98b2-fc1464adabed admin admin] curl -g -i -X POST -H 'x-image-meta-property-user_id: c4954683e8244a0f84f509b8f3d63a4d' -H 'X-Auth-Token: {SHA1}0d0ff6398ca7d37
52cf85f625b8009e3373c20af' -H 'Accept: */*' -H 'User-Agent: python-glanceclient' -H 'x-image-meta-name: test-vm-shelved' -H 'x-image-meta-container_format: ami' -H 'X-Tenant-Id: 73af4afafeae4d5189d8190a8e876418' -H 'x-image-meta-property
-image_type: snapshot' -H 'X-User-Id: c4954683e8244a0f84f509b8f3d63a4d' -H 'x-image-meta-min_ram: 0' -H 'Accept-Encoding: gzip, deflate' -H 'x-image-meta-property-instance_uuid: a520a4a7-6991-40e1-885a-bd3138c2b5dc' -H 'x-image-meta-prop
erty-ramdisk_id: 51378b3f-86f6-4d70-a22e-768599eafc09' -H 'Connection: keep-alive' -H 'x-image-meta-property-kernel_id: ffc958ac-7435-42a4-9dc6-b42a7c518c2e' -H 'x-image-meta-min_disk: 1' -H 'X-Roles: admin' -H 'x-image-meta-property-bas
e_image_ref: 3a106d4e-0b55-4e84-990a-a8aa4ae68c2a' -H 'x-image-meta-is_public: False' -H 'X-Identity-Status: Confirmed' -H 'Content-Type: application/octet-stream' -H 'x-image-meta-disk_format: ami' http://192.168.56.101:9292/v1/images from (pid=16454) log_curl_request /usr/lib/python2.7/site-packages/glanceclient/common/http.py:189
2016-06-03 02:16:32.770 DEBUG glanceclient.common.http [req-c7963aa0-f38a-4580-98b2-fc1464adabed admin admin]
HTTP/1.1 201 Created
Content-Length: 755
Connection: keep-alive
Location: http://192.168.56.101:9292/v1/images/42a6f2b0-c1b4-43d2-95c8-45e7c2d6ca3e
Date: Fri, 03 Jun 2016 06:16:32 GMT
Content-Type: application/json
X-Openstack-Request-Id: req-d1bd0257-5771-4f4b-8ca2-25e1284664d9

{"image": {"status": "queued", "deleted": false, "container_format": "ami", "min_ram": 0, "updated_at": "2016-06-03T06:16:32.000000", "owner": "73af4afafeae4d5189d8190a8e876418", "min_disk": 1, "is_public": false, "deleted_at": null, "id
": "42a6f2b0-c1b4-43d2-95c8-45e7c2d6ca3e", "size": 0, "virtual_size": null, "name": "test-vm-shelved", "checksum": null, "created_at": "2016-06-03T06:16:32.000000", "disk_format": "ami", "properties": {"instance_uuid": "a520a4a7-6991-40e
1-885a-bd3138c2b5dc", "user_id": "c4954683e8244a0f84f509b8f3d63a4d", "image_type": "snapshot", "ramdisk_id": "51378b3f-86f6-4d70-a22e-768599eafc09", "kernel_id": "ffc958ac-7435-42a4-9dc6-b42a7c518c2e", "base_image_ref": "3a106d4e-0b55-4e
84-990a-a8aa4ae68c2a"}, "protected": false}}

* nova-cell(child) also creates the image

2016-06-03 02:16:32.973 DEBUG glanceclient.common.http [req-c7963aa0-f38a-4580-98b2-fc1464adabed admin admin] curl -g -i -X POST -H 'x-image-meta-property-user_id: c4954683e8244a0f84f509b8f3d63a4d' -H 'X-Auth-Token: {SHA1}0d0ff6398ca7d37
52cf85f625b8009e3373c20af' -H 'Accept: */*' -H 'User-Agent: python-glanceclient' -H 'x-image-meta-name: test-vm-shelved' -H 'x-image-meta-container_format: ami' -H 'X-Tenant-Id: 73af4afafeae4d5189d8190a8e876418' -H 'x-image-meta-property
-image_type: snapshot' -H 'X-User-Id: c4954683e8244a0f84f509b8f3d63a4d' -H 'x-image-meta-min_ram: 0' -H 'Accept-Encoding: gzip, deflate' -H 'x-image-meta-property-instance_uuid: a520a4a7-6991-40e1-885a-bd3138c2b5dc' -H 'x-image-meta-prop
erty-ramdisk_id: 51378b3f-86f6-4d70-a22e-768599eafc09' -H 'Connection: keep-alive' -H 'x-image-meta-property-kernel_id: ffc958ac-7435-42a4-9dc6-b42a7c518c2e' -H 'x-image-meta-min_disk: 1' -H 'X-Roles: admin' -H 'x-image-meta-property-bas
e_image_ref: 3a106d4e-0b55-4e84-990a-a8aa4ae68c2a' -H 'x-image-meta-is_public: False' -H 'X-Identity-Status: Confirmed' -H 'Content-Type: application/octet-stream' -H 'x-image-meta-disk_format: ami' http://192.168.56.101:9292/v1/images f
rom (pid=31995) log_curl_request /usr/lib/python2.7/site-packages/glanceclient/common/http.py:189
2016-06-03 02:16:33.045 DEBUG glanceclient.common.http [req-c7963aa0-f38a-4580-98b2-fc1464adabed admin admin]
HTTP/1.1 201 Created
Content-Length: 755
Connection: keep-alive
Location: http://192.168.56.101:9292/v1/images/cc06acba-0911-49ec-86c2-d08c1084fbbe
Date: Fri, 03 Jun 2016 06:16:33 GMT
Content-Type: application/json
X-Openstack-Request-Id: req-fa0bde4b-c741-4922-8c40-741d3a4576c2

{"image": {"status": "queued", "deleted": false, "container_format": "ami", "min_ram": 0, "updated_at": "2016-06-03T06:16:33.000000", "owner": "73af4afafeae4d5189d8190a8e876418", "min_disk": 1, "is_public": false, "deleted_at": null, "id
": "cc06acba-0911-49ec-86c2-d08c1084fbbe", "size": 0, "virtual_size": null, "name": "test-vm-shelved", "checksum": null, "created_at": "2016-06-03T06:16:33.000000", "disk_format": "ami", "properties": {"instance_uuid": "a520a4a7-6991-40e
1-885a-bd3138c2b5dc", "user_id": "c4954683e8244a0f84f509b8f3d63a4d", "image_type": "snapshot", "ramdisk_id": "51378b3f-86f6-4d70-a22e-768599eafc09", "kernel_id": "ffc958ac-7435-42a4-9dc6-b42a7c518c2e", "base_image_ref": "3a106d4e-0b55-4e
84-990a-a8aa4ae68c2a"}, "protected": false}}

* Config
** set 'glanceclient=DEBUG' to show glanceclient log on nova-cell

nova-cells.conf

 [DEFAULT]
 default_log_levels=amqp=DEBUG,amqplib=DEBUG,boto=WARN,qpid=WARN,sqlalchemy=WARN,suds=INFO,oslo.messaging=DEBUG,iso8601=WARN,glanceclient=DEBUG

[Matt Riedemann (mriedem)]

For the most part we've said we won't fix latent bugs in cells v1:

http://docs.openstack.org/developer/nova/cells.html#status

However, this looks like a regression and potentially trivial fix, so we should let it in.

[Vipul Nayyar (vipulnayyar)]

Hey Ryo,

If you think that your changes to source code fix all the problems described in this bug, then please commit your changes so that I can unassign myself from this bug.

If not, I'd like to spend some more time as a newbie, learning about Nova and Cells codebase as a part of resolving this bug.

[Ryo Miki (miki-ryo-e)]

Hello Vipul,

I think my code fixes all of the problem, so I can push the patch.
(but, I need some time to add a unit test too.)

If it's alright with you, I'll assign myself.

[Vipul Nayyar (vipulnayyar)]

Sure, please go ahead and assign yourself.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/327001

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/327001
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=32be8108366bf47f0066deb53a5330379f6fd10c
Submitter: Jenkins
Branch: master

commit 32be8108366bf47f0066deb53a5330379f6fd10c
Author: Ryo Miki <email address hidden>
Date: Wed Jun 8 19:29:22 2016 +0900

    Nova shelve creates duplicated images in cells

    Calling super method in cells_api.py causes duplicated method call in
    nova-api and nova-cell. I found the problem in shelve and unshelve,
    so fix about them. (Nova cell v1 may have similar problems)

    Change-Id: I8030b6ffc1710bca733e38a9a6fb0d7de5f24268
    Closes-bug: #1588657

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/375565

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/375565
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=50cff0ddf11dda7171ed9db776d24393928b590d
Submitter: Jenkins
Branch: stable/mitaka

commit 50cff0ddf11dda7171ed9db776d24393928b590d
Author: Ryo Miki <email address hidden>
Date: Wed Jun 8 19:29:22 2016 +0900

    Nova shelve creates duplicated images in cells

    Calling super method in cells_api.py causes duplicated method call in
    nova-api and nova-cell. I found the problem in shelve and unshelve,
    so fix about them. (Nova cell v1 may have similar problems)

    Change-Id: I8030b6ffc1710bca733e38a9a6fb0d7de5f24268
    Closes-bug: #1588657
    (cherry picked from commit 32be8108366bf47f0066deb53a5330379f6fd10c)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 13.1.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.0.0.0b1 development milestone.