Unhandled exception during 'nova image-create' caused by wrong destination format in qemu-img create

[Valeriy Belavin]

Environment:
- Ubuntu Server 11.04
- OpenStack Diablo release (2011.3)

Steps to reproduce:
- create an image with qcow2 disk
- upload an image:
  glance-upload --disk-format=aki --container-format=aki --type=kernel ...
  glance-upload --disk-format=ari --container-format=ari --type=ramdisk ...
  glance-upload --disk-format=ami --container-format=ami --type=machine ...
- start instance of the uploaded image
- try to create a snaphot with 'nova image-create' command

Problem:
Error occured during nova-image create command, here is a fragment of nova-api.log:

2011-10-13 10:41:07,611 AUDIT nova.compute.manager [a92a2892-b9ad-464b-b261-38f9d8e709a6 cloudroot sandbox] instance 7: snapshotting
2011-10-13 10:41:25,504 DEBUG nova.utils [-] Running cmd (subprocess): qemu-img convert -f qcow2 -O ami -s c8b77874aacb40018955f96935e850d1 /var/lib/nova/instances/instance-00000007/disk /tmp/tmp1KtwHx/c8b77874aacb40018955f96935e850d1 from (pid=29030) execute /usr/lib/python2.7/dist-packages/nova/utils.py:165
2011-10-13 10:41:25,559 DEBUG nova.utils [-] Result was 1 from (pid=29030) execute /usr/lib/python2.7/dist-packages/nova/utils.py:180
2011-10-13 10:41:25,560 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 98, in wrapped
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 459, in snapshot
(nova.exception): TRACE: utils.execute(*qemu_img_cmd)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 188, in execute
(nova.exception): TRACE: cmd=' '.join(cmd))
(nova.exception): TRACE: ProcessExecutionError: Unexpected error while running command.
(nova.exception): TRACE: Command: qemu-img convert -f qcow2 -O ami -s c8b77874aacb40018955f96935e850d1 /var/lib/nova/instances/instance-00000007/disk /tmp/tmp1KtwHx/c8b77874aacb40018955f96935e850d1
(nova.exception): TRACE: Exit code: 1
(nova.exception): TRACE: Stdout: ''
(nova.exception): TRACE: Stderr: "qemu-img: Unknown file format 'ami'\n"
(nova.exception): TRACE:
2011-10-13 10:41:25,566 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 98, in wrapped
(nova.rpc): TRACE: return f(*args, **kw)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 655, in snapshot_instance
(nova.rpc): TRACE: self.driver.snapshot(context, instance_ref, image_id)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 129, in wrapped
(nova.rpc): TRACE: raise Error(str(e))
(nova.rpc): TRACE: Error: Unexpected error while running command.
(nova.rpc): TRACE: Command: qemu-img convert -f qcow2 -O ami -s c8b77874aacb40018955f96935e850d1 /var/lib/nova/instances/instance-00000007/disk /tmp/tmp1KtwHx/c8b77874aacb40018955f96935e850d1
(nova.rpc): TRACE: Exit code: 1
(nova.rpc): TRACE: Stdout: ''
(nova.rpc): TRACE: Stderr: "qemu-img: Unknown file format 'ami'\n"
(nova.rpc): TRACE:
2011-10-13 10:41:53,455 INFO nova.compute.manager [-] Updating host status

Original image description (glance show):

Public: Yes
Name: ttylinux-uec-i386-11.2_2.6.35-16_1
Status: active
Size: 15663104
Disk format: ami
Container format: ami
Minimum Ram Required (MB): 0
Minimum Disk Required (GB): 0
Property 'kernel_id': 22
Property 'ramdisk_id': 23
Property 'type': machine

Workaroung:
Edit /usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py
Change destination format in ''qemu_img_cmd' in function 'def snapshot'. I've made the following:

        # Export the snapshot to a raw image
        temp_dir = tempfile.mkdtemp()
        out_path = os.path.join(temp_dir, snapshot_name)
        qemu_img_cmd = ('qemu-img',
                        'convert',
                        '-f',
                        source_format,
                        '-O',
                        #>>>> image_format,
                        source_format,
                        '-s',
                        snapshot_name,
                        disk_path,
                        out_path)
        utils.execute(*qemu_img_cmd)

[Vish Ishaya (vishvananda)]

Proposed a fix for this. Really, glance should not force disk format of 'ami' images to be 'ami'. I think container format is enough, then you could have a qcow2 ami for example.

[JuanPM (juanpm)]

I had the same issue. I've tried the workaround proposed by Valery, but it doesn't work for me using Diablo release,

nova version 2011.3 (2011.3-nova-milestone-tarball:tarmac-20110922115702-k9nkvxqzhj130av2)
glance version 2011.3

In my case, the problemas appears after doing the next steps:

 1) First, I've uploaded the images using euca-upload-bundle. Below is my image metadata, displayed by glance:

Id: 3
Public: No
Name: None
Status: active
Size: 5370494976
Disk format: ami
Container format: ami
Minimum Ram Required (MB): 0
Minimum Disk Required (GB): 0
Property 'image_location': mybucket/ubuntu-10.10-jeos-0.1rev14.img.manifest.xml
Property 'image_state': available
Property 'project_id': CloudZEN
Property 'architecture': x86_64

2) Then, I create a new instance based on this image, and do some changes.

3) After that I tried to capture a new image with "nova image-create" command". Below the nova-comput.log optput:

2011-10-17 16:49:43,765 DEBUG nova.compute.manager [-] Checking state of instance-00000003 from (pid=622) _get_power_state /usr/lib/python2.7/dist-packages/nova/compute/manager.py:188
2011-10-17 16:49:44,640 AUDIT nova.compute.manager [fcb49d9e-ef04-41f0-a2aa-c2e509a29853 sensei CloudZEN] instance 3: snapshotting
2011-10-17 16:51:28,499 DEBUG nova.utils [-] Running cmd (subprocess): qemu-img convert -f qcow2 -O qcow2 -s 0f5fa0a9eaad4665a910c5e6f5b839d6 /var/lib/nova/instances/instance-00000003/disk /tmp/tmp8tsZ4f/0f5fa0a9eaad4665a910c5e6f5b839d6 from (pid=622) execute /usr/lib/python2.7/dist-packages/nova/utils.py:165
2011-10-17 16:51:28,548 INFO nova.compute.manager [-] Updating host status
2011-10-17 16:52:31,843 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 98, in wrapped
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 466, in snapshot
(nova.exception): TRACE: image_file)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 287, in update
(nova.exception): TRACE: image_meta = client.update_image(image_id, image_meta, data)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/glance/client.py", line 157, in update_image
(nova.exception): TRACE: res = self.do_request("PUT", "/images/%s" % image_id, body, headers)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 140, in do_request
(nova.exception): TRACE: method, action, body=body, headers=headers, params=params)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 230, in _do_request
(nova.exception): TRACE: raise exception.Invalid(res.read())
(nova.exception): TRACE: Invalid: 400 Bad Request
(nova.exception): TRACE:
(nova.exception): TRACE: The server could not comply with the request since it is either malformed or otherwise incorrect.
(nova.exception): TRACE:
(nova.exception): TRACE: Failed to update image metadata. Got error: 400 Bad Request The ser...

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/889
Committed: http://github.com/openstack/nova/commit/b931d51ce47203ee6a4433dc7577e0779ab94710
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit b931d51ce47203ee6a4433dc7577e0779ab94710
Author: Vishvananda Ishaya <email address hidden>
Date: Fri Oct 14 10:06:00 2011 -0700

    Makes snapshots work for amis. Fixes bug 873156

    Change-Id: I6ceb714f31afaf59c28c5ab3b2ab85409dbe89c6

[JuanPM (juanpm)]

I've applied the fix and it works for me! Thank you very much.

Th nova-compute.log ouput now, shows:

2011-10-18 20:25:29,968 DEBUG nova.compute.manager [-] Checking state of instance-00000003 from (pid=32205) _get_power_state /usr/lib/python2.7/dist-packages/nova/compute/manager.py:188
2011-10-18 20:25:31,080 AUDIT nova.compute.manager [f3aa092a-2d16-46ba-813b-30a1f16098a9 user PROJECT] instance 3: snapshotting
2011-10-18 20:28:10,897 DEBUG nova.utils [-] Running cmd (subprocess): qemu-img convert -f qcow2 -O raw -s cd6e87433a3441b09e4e2f5bf244721f /var/lib/nova/instances/instance-00000003/disk /tmp/tmpXSA77w/cd6e87433a3441b09e4e2f5bf244721f from (pid=32205) execute /usr/lib/python2.7/dist-packages/nova/utils.py:165
2011-10-18 20:28:10,926 INFO nova.compute.manager [-] Updating host status

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/996
Committed: http://github.com/openstack/nova/commit/d46f6e095b43fc00099d87f73b098a137830b167
Submitter: Jenkins
Branch: stable/diablo

 status fixcommitted
 done

commit d46f6e095b43fc00099d87f73b098a137830b167
Author: Vishvananda Ishaya <email address hidden>
Date: Fri Oct 14 10:06:00 2011 -0700

    Makes snapshots work for amis. Fixes bug 873156

    (cherry picked from commit b931d51ce47203ee6a4433dc7577e0779ab94710)

    Change-Id: I3adee5aed8a500602fb938f27fa5096ec376cbe2

[Martin Pitt (pitti)]

Hello Valeriy, or anyone else affected,

Accepted nova into oneiric-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!