OpenStack Heat

Restore from snapshot is not working with resource references

Bug #1687009 reported by Syed Armani
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
Medium
Rabi Mishra
OpenStack Heat pike-2
Newton
Fix Committed
Undecided
Rabi Mishra
Ocata
Fix Committed
Undecided
Rabi Mishra

Bug Description

OpenStack Release: Newton
OS: Ubuntu 16.04

While trying to restore a heat stack from its snapshot i.e. `openstack stack snapshot restore`, the stack status says " RESTORE_COMPLETE" but it actually does nothing. REBUILD never happens and the image of the instance never changes to snapshot image.

snapshot creation works just fine but restore functionality appears to be broken in Newton.

"openstack stack show" shows the "stack_status" as "RESTORE_COMPLETE" but the "resource_status" for "OS::Nova::Server" in "openstack stack resource list" is "SNAPSHOT_COMPLETE" while it should have been "UPDATE COMPLETE".

http://paste.openstack.org/show/608318/

$ openstack stack resource list syed_rollbacktest
+-----------------------+----------------------------------------------------------+------------------------------+-------------------+----------------------+
| resource_name | physical_resource_id | resource_type | resource_status | updated_time |
+-----------------------+----------------------------------------------------------+------------------------------+-------------------+----------------------+
| mysub_net | bacfc4a4-fa37-4727-ae79-e831afcc5bc0 | OS::Neutron::Subnet | SNAPSHOT_COMPLETE | 2017-04-28T10:53:28Z |
| mybox | 44f533e0-80d2-4729-a347-d3b194e2cdad | OS::Nova::Server | SNAPSHOT_COMPLETE | 2017-04-28T10:53:28Z |
| router_interface | 40361f97-277f-4906-9a6d-2b17ca3ae956:subnet_id=bacfc4a4- | OS::Neutron::RouterInterface | SNAPSHOT_COMPLETE | 2017-04-28T10:53:28Z |
| | fa37-4727-ae79-e831afcc5bc0 | | | |
| mybox_management_port | b4fb4ef4-e4aa-4de6-8f81-9fe8ef9505d8 | OS::Neutron::Port | SNAPSHOT_COMPLETE | 2017-04-28T10:53:28Z |
| myfloating_ip | fe066cc6-aa36-43b3-8863-6abafa177ee8 | OS::Neutron::FloatingIP | SNAPSHOT_COMPLETE | 2017-04-28T10:53:28Z |
| myconfig | 998b0339-464b-49d6-a0a5-02ca5cbf1715 | OS::Heat::CloudConfig | SNAPSHOT_COMPLETE | 2017-04-28T10:53:28Z |
| router_gateway | 40361f97-277f-4906-9a6d-2b17ca3ae956:6fc73bfd-c0b5-418c- | OS::Neutron::RouterGateway | SNAPSHOT_COMPLETE | 2017-04-28T10:53:28Z |
| | 84a1-ffa719257ad1 | | | |
| mysecurity_group | d21aade4-e43d-4a11-b7b5-27c7fa39aa64 | OS::Neutron::SecurityGroup | SNAPSHOT_COMPLETE | 2017-04-28T10:53:28Z |
| router | 40361f97-277f-4906-9a6d-2b17ca3ae956 | OS::Neutron::Router | SNAPSHOT_COMPLETE | 2017-04-28T10:53:28Z |
| mynet | f854e5f1-1432-44d1-a35e-75746b975612 | OS::Neutron::Net | SNAPSHOT_COMPLETE | 2017-04-28T10:53:28Z |
+-----------------------+----------------------------------------------------------+------------------------------+-------------------+----------------------+

$ openstack server show 44f533e0-80d2-4729-a347-d3b194e2cdad
+-----------------------------+------------------------------------------------------------------+
| Field | Value |
+-----------------------------+------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2017-04-28T10:53:42.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | management-net=192.168.101.9, 89.46.85.20 |
| config_drive | |
| created | 2017-04-28T10:53:32Z |
| flavor | 1C-1GB-20GB (b6560dce-f87e-4175-8224-68b6654f9ad5) |
| hostId | dffe6d4398a25f8cd75b74bed1e508b590ab3127a0c790384d6f9333 |
| id | 44f533e0-80d2-4729-a347-d3b194e2cdad |
| image | Ubuntu 16.04 Xenial Xerus (2324095e-649b-40b0-98e3-9da5a9618a69) |
| key_name | syed |
| name | mybox |
| progress | 0 |
| project_id | 42d9af4c2d724838a8ef6628566aefec |
| properties | |
| security_groups | name='mysecurity_group' |
| status | ACTIVE |
| updated | 2017-04-28T10:55:20Z |
| user_id | 8ae69ffb933249ebb3290a484caf0373 |
| volumes_attached | |
+-----------------------------+------------------------------------------------------------------+

$ openstack image list | grep rollback

| 0dde39e3-4374-4029-93dc-faec62a624b4 | syed_rollbacktest-mybox-5xcggcdwkj7z | active |

$ openstack stack show syed_rollbacktest
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------+
| id | 2d26d002-709d-4b7c-9c8a-1dde03426a11 |
| stack_name | syed_rollbacktest |
| description | No description |
| creation_time | 2017-04-28T10:53:28Z |
| updated_time | 2017-04-28T11:00:13Z |
| stack_status | RESTORE_COMPLETE |
| stack_status_reason | Stack RESTORE completed successfully |
| parameters | OS::project_id: 42d9af4c2d724838a8ef6628566aefec |
| | OS::stack_id: 2d26d002-709d-4b7c-9c8a-1dde03426a11 |
| | OS::stack_name: syed_rollbacktest |
| | flavor: 1C-1GB-20GB |
| | image: Ubuntu 16.04 Xenial Xerus |
| | key_name: syed |
| | packages: '[]' |
| | public_net: ext-net |
| | |
| outputs | - description: Floating IP address in public network |
| | output_key: public_ip |
| | output_value: 89.46.85.20 |
| | |
| links | - href: https://dx1.citycloud.com:8004/v1/42d9af4c2d724838a8ef6628566aefec/stacks/syed_rollbacktest/2d26d002-709d-4b7c-9c8a- |
| | 1dde03426a11 |
| | rel: self |
| | |
| parent | None |
| disable_rollback | True |
| deletion_time | None |
| stack_user_project_id | e67650773cee4df5a306d0078bbea719 |
| capabilities | [] |
| notification_topics | [] |
| stack_owner | None |
| timeout_mins | None |
| tags | None |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------+

$ openstack stack event list 2d26d002-709d-4b7c-9c8a-1dde03426a11

2017-04-28 10:53:28Z [syed_rollbacktest]: CREATE_IN_PROGRESS Stack CREATE started
2017-04-28 10:53:29Z [syed_rollbacktest.myconfig]: CREATE_IN_PROGRESS state changed
2017-04-28 10:53:29Z [syed_rollbacktest.mynet]: CREATE_IN_PROGRESS state changed
2017-04-28 10:53:29Z [syed_rollbacktest.myconfig]: CREATE_COMPLETE state changed
2017-04-28 10:53:29Z [syed_rollbacktest.router]: CREATE_IN_PROGRESS state changed
2017-04-28 10:53:29Z [syed_rollbacktest.mysecurity_group]: CREATE_IN_PROGRESS state changed
2017-04-28 10:53:29Z [syed_rollbacktest.mynet]: CREATE_COMPLETE state changed
2017-04-28 10:53:29Z [syed_rollbacktest.mysecurity_group]: CREATE_COMPLETE state changed
2017-04-28 10:53:29Z [syed_rollbacktest.mysub_net]: CREATE_IN_PROGRESS state changed
2017-04-28 10:53:30Z [syed_rollbacktest.mysub_net]: CREATE_COMPLETE state changed
2017-04-28 10:53:30Z [syed_rollbacktest.router]: CREATE_COMPLETE state changed
2017-04-28 10:53:30Z [syed_rollbacktest.mybox_management_port]: CREATE_IN_PROGRESS state changed
2017-04-28 10:53:30Z [syed_rollbacktest.router_interface]: CREATE_IN_PROGRESS state changed
2017-04-28 10:53:30Z [syed_rollbacktest.router_gateway]: CREATE_IN_PROGRESS state changed
2017-04-28 10:53:30Z [syed_rollbacktest.mybox_management_port]: CREATE_COMPLETE state changed
2017-04-28 10:53:31Z [syed_rollbacktest.mybox]: CREATE_IN_PROGRESS state changed
2017-04-28 10:53:31Z [syed_rollbacktest.router_gateway]: CREATE_COMPLETE state changed
2017-04-28 10:53:32Z [syed_rollbacktest.router_interface]: CREATE_COMPLETE state changed
2017-04-28 10:53:32Z [syed_rollbacktest.myfloating_ip]: CREATE_IN_PROGRESS state changed
2017-04-28 10:53:33Z [syed_rollbacktest.myfloating_ip]: CREATE_COMPLETE state changed
2017-04-28 10:53:44Z [syed_rollbacktest.mybox]: CREATE_COMPLETE state changed
2017-04-28 10:53:44Z [syed_rollbacktest]: CREATE_COMPLETE Stack CREATE completed successfully
2017-04-28 10:54:56Z [syed_rollbacktest]: SNAPSHOT_IN_PROGRESS Stack SNAPSHOT started
2017-04-28 10:54:56Z [syed_rollbacktest.mynet]: SNAPSHOT_IN_PROGRESS state changed
2017-04-28 10:54:56Z [syed_rollbacktest.mynet]: SNAPSHOT_COMPLETE state changed
2017-04-28 10:54:56Z [syed_rollbacktest.router]: SNAPSHOT_IN_PROGRESS state changed
2017-04-28 10:54:56Z [syed_rollbacktest.router]: SNAPSHOT_COMPLETE state changed
2017-04-28 10:54:56Z [syed_rollbacktest.mysub_net]: SNAPSHOT_IN_PROGRESS state changed
2017-04-28 10:54:56Z [syed_rollbacktest.mysub_net]: SNAPSHOT_COMPLETE state changed
2017-04-28 10:54:56Z [syed_rollbacktest.myconfig]: SNAPSHOT_IN_PROGRESS state changed
2017-04-28 10:54:56Z [syed_rollbacktest.myconfig]: SNAPSHOT_COMPLETE state changed
2017-04-28 10:54:56Z [syed_rollbacktest.mysecurity_group]: SNAPSHOT_IN_PROGRESS state changed
2017-04-28 10:54:56Z [syed_rollbacktest.mysecurity_group]: SNAPSHOT_COMPLETE state changed
2017-04-28 10:54:56Z [syed_rollbacktest.mybox_management_port]: SNAPSHOT_IN_PROGRESS state changed
2017-04-28 10:54:56Z [syed_rollbacktest.mybox_management_port]: SNAPSHOT_COMPLETE state changed
2017-04-28 10:54:56Z [syed_rollbacktest.mybox]: SNAPSHOT_IN_PROGRESS state changed
2017-04-28 10:54:57Z [syed_rollbacktest.router_interface]: SNAPSHOT_IN_PROGRESS state changed
2017-04-28 10:54:57Z [syed_rollbacktest.router_interface]: SNAPSHOT_COMPLETE state changed
2017-04-28 10:54:57Z [syed_rollbacktest.router_gateway]: SNAPSHOT_IN_PROGRESS state changed
2017-04-28 10:54:57Z [syed_rollbacktest.router_gateway]: SNAPSHOT_COMPLETE state changed
2017-04-28 10:54:57Z [syed_rollbacktest.myfloating_ip]: SNAPSHOT_IN_PROGRESS state changed
2017-04-28 10:54:57Z [syed_rollbacktest.myfloating_ip]: SNAPSHOT_COMPLETE state changed
2017-04-28 10:55:20Z [syed_rollbacktest.mybox]: SNAPSHOT_COMPLETE state changed
2017-04-28 10:55:20Z [syed_rollbacktest]: SNAPSHOT_COMPLETE Stack SNAPSHOT completed successfully
2017-04-28 11:00:13Z [syed_rollbacktest]: RESTORE_IN_PROGRESS Stack RESTORE started
2017-04-28 11:00:19Z [syed_rollbacktest]: RESTORE_COMPLETE Stack RESTORE completed successfully
2017-04-28 11:00:19Z [syed_rollbacktest]: RESTORE_COMPLETE Stack RESTORE completed successfully

See original description
Syed Armani (dce3062)
description: updated
Revision history for this message
Rabi Mishra (rabi) wrote :

I could reproduce this in my setup. So both snapshot and restore do not work for nested stacks( i'e OS::Nova::Server in a nested stack). We don't seem to do anything, in case of snapshot we set the status to SNAPSHOT_COMPLETE for the nested stack resource and ignore it completely for restore.

Revision history for this message
Rabi Mishra (rabi) wrote :

The above comment applies to master too.

Changed in heat:
assignee: nobody → Rabi Mishra (rabi)
status: New → Triaged
Revision history for this message
Florian Haas (fghaas) wrote :

@rabi For the benefit of other subscribers following this, I'd like to emphasize that at least the restore problem is *not* limited to nested stacks.

If anyone wants to reproduce this, here is the template that Syed used to create the stack mentioned in the description:
https://github.com/fghaas/ccmc-20170420/blob/master/hot/intermediate.yaml

But I think what you're saying is that for nested stacks both snapshot and restore are broken, whereas for non-nested ones snapshot works but restore breaks. Did I understand that correctly?

Revision history for this message
Rabi Mishra (rabi) wrote :

> But I think what you're saying is that for nested stacks both snapshot and restore are broken, whereas for non-nested ones snapshot works but restore breaks. Did I understand that correctly?

It works if there are no resource references(get_resource/get_attr) is used in the OS::Nova::Server resource for non-nested stacks.

The issue is we're using the frozen resource definitions[1] when adding the resource to the new stack and these references are resolved to None.

[1] https://git.openstack.org/cgit/openstack/heat/tree/heat/engine/resources/openstack/nova/server.py#n1590

Revision history for this message
Rabi Mishra (rabi) wrote :

It seems to be happening before calling defn.freeze(properties=props), when doing get_item to create props as we override defn properties and it doesn't resolve any intrinsic functions.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/462426

Changed in heat:
status: Triaged → In Progress
Rabi Mishra (rabi)
Changed in heat:
importance: Undecided → Medium
milestone: none → pike-2
summary: - Restore from snapshot is not working in Newton
+ Restore from snapshot is not working with resource references
Revision history for this message
Florian Haas (fghaas) wrote :

That looks fantastic, thanks @rabi! Will this one be backported to Newton?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/462426
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=a8129573d221284a482e84e62a7e8b0732318bfe
Submitter: Jenkins
Branch: master

commit a8129573d221284a482e84e62a7e8b0732318bfe
Author: rabi <email address hidden>
Date: Thu May 4 13:02:28 2017 +0530

    Fix handle_restore for server and volume resources

    When resource references are used for properties restore
    does not work for nova server and volume resources.

    This patch fixes it and adds a functional test for it.

    Change-Id: I34efb3f4e0d3578d95134793f5c5d413667790e0
    Closes-Bug: #1687009

Changed in heat:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/468010

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/heat 9.0.0.0b2

This issue was fixed in the openstack/heat 9.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/472524

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/ocata)

Reviewed: https://review.openstack.org/468010
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=437c5e7c4fe2f0fc3eaa034348384e7cd7cc81a6
Submitter: Jenkins
Branch: stable/ocata

commit 437c5e7c4fe2f0fc3eaa034348384e7cd7cc81a6
Author: rabi <email address hidden>
Date: Thu May 4 13:02:28 2017 +0530

    Fix handle_restore for server and volume resources

    When resource references are used for properties restore
    does not work for nova server and volume resources.

    This patch fixes it and adds a functional test for it.

     Conflicts:
     heat/engine/resources/openstack/nova/server.py

    There is very minor conflict in server.py as the code
    has been changed in pike for translation refactoring.

    Change-Id: I34efb3f4e0d3578d95134793f5c5d413667790e0
    Closes-Bug: #1687009
    (cherry picked from commit a8129573d221284a482e84e62a7e8b0732318bfe)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/newton)

Reviewed: https://review.openstack.org/472524
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=f461a0b36215d5d830899c65ac4070a39d7c3b69
Submitter: Jenkins
Branch: stable/newton

commit f461a0b36215d5d830899c65ac4070a39d7c3b69
Author: rabi <email address hidden>
Date: Thu May 4 13:02:28 2017 +0530

    Fix handle_restore for server and volume resources

    When resource references are used for properties restore
    does not work for nova server and volume resources.

    This patch fixes it and adds a functional test for it.

     Conflicts:
     heat/engine/resources/openstack/nova/server.py

    There is very minor conflict in server.py as the code
    has been changed in pike for translation refactoring.

    Change-Id: I34efb3f4e0d3578d95134793f5c5d413667790e0
    Closes-Bug: #1687009
    (cherry picked from commit a8129573d221284a482e84e62a7e8b0732318bfe)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/heat 7.0.5

This issue was fixed in the openstack/heat 7.0.5 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/heat 8.0.3

This issue was fixed in the openstack/heat 8.0.3 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.