HTTP 500 for assisted volume snapshot on shelved instance

[Eric Harney]

Nova throws an HTTP 500 when trying to create an assisted volume snapshot for Cinder NFS if the instance is shelved. (Has no "host" field, presumably.)

To reproduce:

1. Pull https://review.openstack.org/#/c/147186/48 for Cinder NFS snapshot support.
2. Create instance.
3. Attach NFS volume to instance.
4. Shelve instance.
5. Cinder snapshot-create on the volume.

2017-01-18 16:43:38.002 DEBUG nova.api.openstack.wsgi [req-e441340d-8147-4a03-b401-198ecb0e760d nova service] Action: 'create', calling method: <bound method AssistedVolumeSnapshotsController.create of <nova.api.openstack.compute.assisted_volume_snapshots.AssistedVolumeSnapshotsController object at 0x7f0474b1f9d0>>, body: {"snapshot": {"create_info": {"snapshot_id": "6e9292a6-ddaf-42f5-9cc7-374f9470e406", "type": "qcow2", "new_file": "volume-924ae600-6bfc-47f9-ae48-87eb34fe3c21.6e9292a6-ddaf-42f5-9cc7-374f9470e406"}, "volume_id": "924ae600-6bfc-47f9-ae48-87eb34fe3c21"}} from (pid=13329) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:623
2017-01-18 16:43:38.080 ERROR nova.api.openstack.extensions [req-e441340d-8147-4a03-b401-198ecb0e760d nova service] Unexpected exception in API method
2017-01-18 16:43:38.080 TRACE nova.api.openstack.extensions Traceback (most recent call last):
2017-01-18 16:43:38.080 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/extensions.py", line 338, in wrapped
2017-01-18 16:43:38.080 TRACE nova.api.openstack.extensions return f(*args, **kwargs)
2017-01-18 16:43:38.080 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
2017-01-18 16:43:38.080 TRACE nova.api.openstack.extensions return func(*args, **kwargs)
2017-01-18 16:43:38.080 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/assisted_volume_snapshots.py", line 55, in create
2017-01-18 16:43:38.080 TRACE nova.api.openstack.extensions create_info)
2017-01-18 16:43:38.080 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 3935, in volume_snapshot_create
2017-01-18 16:43:38.080 TRACE nova.api.openstack.extensions volume_id, create_info)
2017-01-18 16:43:38.080 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/rpcapi.py", line 1044, in volume_snapshot_create
2017-01-18 16:43:38.080 TRACE nova.api.openstack.extensions server=_compute_host(None, instance), version=version)
2017-01-18 16:43:38.080 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/rpcapi.py", line 53, in _compute_host
2017-01-18 16:43:38.080 TRACE nova.api.openstack.extensions 'Instance %s') % instance.uuid)
2017-01-18 16:43:38.080 TRACE nova.api.openstack.extensions NovaException: Unable to find host for Instance 875480c0-8f5e-44e9-9778-b39d6256cfb9
2017-01-18 16:43:38.080 TRACE nova.api.openstack.extensions

[Matt Riedemann (mriedem)]

Yeah the instance is shelved so it's literally deleted from the hypervisor and doesn't have a host/node set in the instances table. For guest-assisted snapshots we have to cast from n-api to n-cpu that the instance is running on, but it's not running in this case so things fail.

We should just check the status and handle it in the API.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/429476

[Ken'ichi Ohmichi (oomichi)]

https://review.openstack.org/#/c/429918/ is the related patch on Tempest side.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/429476
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=70afc0d5408aaae8beb587682fe26c124c0cacee
Submitter: Jenkins
Branch: master

commit 70afc0d5408aaae8beb587682fe26c124c0cacee
Author: Matt Riedemann <email address hidden>
Date: Sun Feb 5 16:32:44 2017 -0500

    Handle conflicts for os-assisted-volume-snapshots

    Since a guest-assisted disk snapshot is performed on the compute
    that the instance is running on, there are only certain states
    that the instance can be in to perform this operation. For example,
    if the instance is shelved_offloaded then the instance does not
    have a host and we can't cast to a compute to perform the snapshot.

    Given how unrestrictive this API was before, the only restriction
    we place on the state is that the instance does not have a task_state
    set. We allow any vm_state for performing the operation as long as
    there is a host and no task_state.

    As noted in the code, we'd normally return a 409 in this case but
    according to our microversion docs [1] that would be a new error
    code and require a version bump, so this change just uses 400 and
    leaves a TODO to make this 409 in a later mass return code update
    microversion.

    [1] https://docs.openstack.org/developer/nova/api_microversion_dev.html#f1

    Change-Id: I1dc54a38f02bb48921bcbc4c2fdcc2c946e783c1
    Closes-Bug: #1657585

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.0.0.0b1 development milestone.