Snapshot not created with any container_format column

[Jay Pipes]

======UPDATE=============

Got some more information on that problem:
What happened was something like this, not entirely sure about all sequences:
- create VM1 from image, do something with VM1, create snapshot of VM1
- create VM2 from snapshot, stop VM1, do something with VM2, create snapshot of VM2
- create VM3 from snapshot, stop VM2, delete snapshot of VM2

So the situation was this: There's a VM created from a snapshot and the snapshot image was deleted. Creating a snapshot in that situation was impossible and resulted in the stacktrace

=======ORIGINAL=============

I've got an Openstack setup with one controller node (glance, keystone, horizon, nova-* (except nova-compute)) and three compute nodes. I wanted to create a snapshot of a vm running on one of the compute nodes, by using the option in the horizon dashboard.

What happens is that the snapshot gets listed as a snapshot, but with the status queued. And from there on nothing happens anymore.

In the nova-compute.log on the compute host I can see these entries:
2012-07-25 12:45:56 TRACE nova.rpc.amqp Traceback (most recent call last):
2012-07-25 12:45:56 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 253, in _process_data
2012-07-25 12:45:56 TRACE nova.rpc.amqp rval = node_func(context=ctxt, **node_args)
2012-07-25 12:45:56 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
2012-07-25 12:45:56 TRACE nova.rpc.amqp return f(*args, **kw)
2012-07-25 12:45:56 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 183, in decorated_function
2012-07-25 12:45:56 TRACE nova.rpc.amqp sys.exc_info())
2012-07-25 12:45:56 TRACE nova.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-07-25 12:45:56 TRACE nova.rpc.amqp self.gen.next()
2012-07-25 12:45:56 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 177, in decorated_function
2012-07-25 12:45:56 TRACE nova.rpc.amqp return function(self, context, instance_uuid, *args, **kwargs)
2012-07-25 12:45:56 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 952, in snapshot_instance
2012-07-25 12:45:56 TRACE nova.rpc.amqp self.driver.snapshot(context, instance_ref, image_id)
2012-07-25 12:45:56 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
2012-07-25 12:45:56 TRACE nova.rpc.amqp return f(*args, **kw)
2012-07-25 12:45:56 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 714, in snapshot
2012-07-25 12:45:56 TRACE nova.rpc.amqp image_file)
2012-07-25 12:45:56 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 306, in update
2012-07-25 12:45:56 TRACE nova.rpc.amqp _reraise_translated_image_exception(image_id)
2012-07-25 12:45:56 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 304, in update
2012-07-25 12:45:56 TRACE nova.rpc.amqp image_meta = client.update_image(image_id, image_meta, data)
2012-07-25 12:45:56 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/glance/client.py", line 195, in update_image
2012-07-25 12:45:56 TRACE nova.rpc.amqp res = self.do_request("PUT", "/images/%s" % image_id, body, headers)
2012-07-25 12:45:56 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 58, in wrapped
2012-07-25 12:45:56 TRACE nova.rpc.amqp return func(self, *args, **kwargs)
2012-07-25 12:45:56 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 420, in do_request
2012-07-25 12:45:56 TRACE nova.rpc.amqp headers=headers)
2012-07-25 12:45:56 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 75, in wrapped
2012-07-25 12:45:56 TRACE nova.rpc.amqp return func(self, method, url, body, headers)
2012-07-25 12:45:56 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 547, in _do_request
2012-07-25 12:45:56 TRACE nova.rpc.amqp raise exception.Invalid(res.read())
2012-07-25 12:45:56 TRACE nova.rpc.amqp Invalid: Data supplied was not valid.
2012-07-25 12:45:56 TRACE nova.rpc.amqp Details: 400 Bad Request
2012-07-25 12:45:56 TRACE nova.rpc.amqp
2012-07-25 12:45:56 TRACE nova.rpc.amqp The server could not comply with the request since it is either malformed or otherwise incorrect.
2012-07-25 12:45:56 TRACE nova.rpc.amqp
2012-07-25 12:45:56 TRACE nova.rpc.amqp Failed to update image metadata. Got error: Data supplied was not valid. Details: 400 Bad Request The server could not comply with the request since it is either malformed or otherwise incorrect. Failed to update image metadata. Got error: Data supplied was not valid. Details: Invalid container format 'None' for image.

I checked with the api.log of glance and found these entries:

2012-07-25 13:13:22 2371 ERROR [glance.api.v1.images] Failed to update image metadata. Got error: Data supplied was not valid.
2012-07-25 13:13:22 2371 ERROR [glance.api.v1.images] Details: 400 Bad Request
2012-07-25 13:13:22 2371 ERROR [glance.api.v1.images]
2012-07-25 13:13:22 2371 ERROR [glance.api.v1.images] The server could not comply with the request since it is either malformed or otherwise incorrect.
2012-07-25 13:13:22 2371 ERROR [glance.api.v1.images]
2012-07-25 13:13:22 2371 ERROR [glance.api.v1.images] Failed to update image metadata. Got error: Data supplied was not valid. Details: Invalid container format 'None' for image.

Any idea what's going wrong here?

[Jay Pipes (jaypipes)]

Hi guys, this came from here: https://answers.launchpad.net/glance/+question/204133

[Gabriel Hurley (gabriel-hurley)]

I'm not sure where that's stemming from but it deserves further investigation. In general snapshotting works, so this may be an edge case somewhere.

[Max Schilling (mx-chilly)]

Got some more information on that problem:
What happened was something like this, not entirely sure about all sequences:
- create VM1 from image, do something with VM1, create snapshot of VM1
- create VM2 from snapshot, stop VM1, do something with VM2, create snapshot of VM2
- create VM3 from snapshot, stop VM2, delete snapshot of VM2

So the situation was this: There's a VM created from a snapshot and the snapshot image was deleted. Creating a snapshot in that situation was impossible and resulted in the stacktrace

[Gabriel Hurley (gabriel-hurley)]

Hmmm... I'm not certain what the appropriate fix for that is.. I'm gonna defer this to RC1 since it's quite the edge-case.

[Gabriel Hurley (gabriel-hurley)]

I was able to reproduce this by doing the following:

1. create instance A
2. create snapshot A from instance A
3. launch instance B from snapshot A
4. delete snapshot A
5. create snapshot B from instance B

This results in a queued snapshot in Glance which will never resolve and can only be deleted.

However, I don't think this is a horizon bug, really. I presume this is just as possible through the clients, CLIs, etc. since that's all Horizon is doing. Can anyone confirm that?

Moreover, does anyone have suggestions on where this should be fixed?

[Brian Waldon (bcwaldon)]

The right way to fix this is to break the tie between Nova and Glance. When an instance is built Nova should store a copy of the necessary metadata - Nova can then refer back to that copy when a snapshot is created from that instance.

[Gabriel Hurley (gabriel-hurley)]

There doesn't appear to be anything for Horizon to do here that isn't a kludge. Brian's suggestion sounds reasonable to me.

[Brian Waldon (bcwaldon)]

The core of the problem is that data is going away and Nova doesn't know what the hell to do without it. This isn't a Glance bug either. I'll fix this for Nova in Grizzly.

[Justin Shepherd (jshepher)]

Marking this issue as invalid, as it is filed against Folsom, and has not been updated in over a year and a half. If you believe this to be in error, please update this issue.

[Aron Wong (aronivicki)]

This issue still in icehouse, please update this issue as soon as possible.