OpenStack Compute (nova)

  1. Bug #1244694
  2. Activity log

Activity log for bug #1244694

Date Who What changed Old value New value Message
2013-10-25 14:57:36 Daniel Speichert bug added bug
2013-10-25 14:58:25 Daniel Speichert bug task added nova (Ubuntu)
2013-11-05 16:17:13 Dirk Grunwald bug added subscriber Dirk Grunwald
2013-11-05 18:35:35 Launchpad Janitor nova (Ubuntu): status New Confirmed
2013-11-21 13:25:23 Semyon Deviatkin bug added subscriber Semeyon Deviatkin
2013-11-23 21:03:34 Matt Riedemann tags libvirt
2013-12-12 04:09:19 wingwj bug added subscriber wingwj
2013-12-31 01:53:47 Chuck Short affects nova (Ubuntu) libvirt (Ubuntu)
2014-01-08 04:59:52 Guangyu Suo bug added subscriber Guangyu Suo
2014-01-10 02:21:48 Launchpad Janitor libvirt (Ubuntu): status Confirmed Fix Released
2014-01-16 18:48:24 Adam Gandelman description In some cases (not for all instances, just for some) the following error prevents creating the snapshot: 2013-10-25 14:49:30.724 22980 AUDIT nova.compute.manager [req-6e9326d7-64df-40f7-bc81-190ec5234de2 657f1aca48d24eaf9655e0b77b2bc6d9 35b2b08cc3f44a538cf3535043793a2a] [instance: db9c8a72-6ce2-41b7-8f7a-be0be8468667] instance snapshotting 2013-10-25 14:49:30.944 22980 INFO nova.virt.libvirt.driver [req-6e9326d7-64df-40f7-bc81-190ec5234de2 657f1aca48d24eaf9655e0b77b2bc6d9 35b2b08cc3f44a538cf3535043793a2a] [instance: db9c8a72-6ce2-41b7-8f7a-be0be8468667] Beginning live snapshot process 2013-10-25 14:49:32.006 22980 INFO nova.virt.libvirt.driver [req-6e9326d7-64df-40f7-bc81-190ec5234de2 657f1aca48d24eaf9655e0b77b2bc6d9 35b2b08cc3f44a538cf3535043793a2a] [instance: db9c8a72-6ce2-41b7-8f7a-be0be8468667] Snapshot extracted, beginning image upload 2013-10-25 14:49:32.329 22980 ERROR nova.openstack.common.rpc.amqp [req-6e9326d7-64df-40f7-bc81-190ec5234de2 657f1aca48d24eaf9655e0b77b2bc6d9 35b2b08cc3f44a538cf3535043793a2a] Exception during message handling 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last): 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp **args) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 353, in decorated_function 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp payload) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 243, in decorated_function 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp pass 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 229, in decorated_function 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 271, in decorated_function 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info()) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 258, in decorated_function 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 319, in decorated_function 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp % image_id, instance=instance) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 309, in decorated_function 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp *args, **kwargs) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2293, in snapshot_instance 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp task_states.IMAGE_SNAPSHOT) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2324, in _snapshot_instance 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp update_task_state) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1397, in snapshot 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp image_format) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1483, in _live_snapshot 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp libvirt.VIR_DOMAIN_BLOCK_REBASE_SHALLOW) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp result = proxy_call(self._autowrap, f, *args, **kwargs) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp rv = execute(f,*args,**kwargs) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp rv = meth(*args,**kwargs) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/libvirt.py", line 646, in blockRebase 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp if ret == -1: raise libvirtError ('virDomainBlockRebase() failed', dom=self) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp libvirtError: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/var/lib/nova/instances/snapshots/tmp5DdrIO/236df3e170e64fabaeb3c7601e2d6c47.delta' 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp The directory /var/lib/nova/instances/snapshots/tmp5DdrIO does not exist immidiately after this error shows up. Permissions to the parent directory are nova:nova 755, I believe they are correct. Package: nova-compute Version: 1:2013.2-0ubuntu1~cloud0 ---- SRU Justification --- [Impact] In a libvirt-based OpenStack deployment, Nova fails to snapshot instances, failing with error: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/var/lib/nova/instances/snapshots/tmp5DdrIO/236df3e170e64fabaeb3c7601e2d6c47.delta' I had originally discovered this bug using the Tempset test suite while verifying an unrelated OpenStack SRU, but other users are experiencing this in the wild. [Test Case] Deploy an OpenStack cloud based on Ubuntu Saucy and OpenStack Havana, then attempt to snapshot a running instance. The Tempest integration test suite contains a snapshot test case: tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_delete_image [1] [Regression Potential] The proposed fix is isolated to the libvirt packaging and simply appends an additional directory exception to the packages apparmor configuration, so that libvirt has appropriate access to the directory used during the process of snapshotting an instance. [1]https://github.com/openstack/tempest/blob/master/tempest/api/compute/images/test_images_oneserver.py#L92 --- Original Bug --- In some cases (not for all instances, just for some) the following error prevents creating the snapshot: 2013-10-25 14:49:30.724 22980 AUDIT nova.compute.manager [req-6e9326d7-64df-40f7-bc81-190ec5234de2 657f1aca48d24eaf9655e0b77b2bc6d9 35b2b08cc3f44a538cf3535043793a2a] [instance: db9c8a72-6ce2-41b7-8f7a-be0be8468667] instance snapshotting 2013-10-25 14:49:30.944 22980 INFO nova.virt.libvirt.driver [req-6e9326d7-64df-40f7-bc81-190ec5234de2 657f1aca48d24eaf9655e0b77b2bc6d9 35b2b08cc3f44a538cf3535043793a2a] [instance: db9c8a72-6ce2-41b7-8f7a-be0be8468667] Beginning live snapshot process 2013-10-25 14:49:32.006 22980 INFO nova.virt.libvirt.driver [req-6e9326d7-64df-40f7-bc81-190ec5234de2 657f1aca48d24eaf9655e0b77b2bc6d9 35b2b08cc3f44a538cf3535043793a2a] [instance: db9c8a72-6ce2-41b7-8f7a-be0be8468667] Snapshot extracted, beginning image upload 2013-10-25 14:49:32.329 22980 ERROR nova.openstack.common.rpc.amqp [req-6e9326d7-64df-40f7-bc81-190ec5234de2 657f1aca48d24eaf9655e0b77b2bc6d9 35b2b08cc3f44a538cf3535043793a2a] Exception during message handling 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last): 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp **args) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 353, in decorated_function 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp payload) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 243, in decorated_function 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp pass 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 229, in decorated_function 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 271, in decorated_function 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info()) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 258, in decorated_function 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 319, in decorated_function 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp % image_id, instance=instance) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 309, in decorated_function 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp *args, **kwargs) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2293, in snapshot_instance 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp task_states.IMAGE_SNAPSHOT) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2324, in _snapshot_instance 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp update_task_state) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1397, in snapshot 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp image_format) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1483, in _live_snapshot 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp libvirt.VIR_DOMAIN_BLOCK_REBASE_SHALLOW) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp result = proxy_call(self._autowrap, f, *args, **kwargs) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp rv = execute(f,*args,**kwargs) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp rv = meth(*args,**kwargs) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/libvirt.py", line 646, in blockRebase 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp if ret == -1: raise libvirtError ('virDomainBlockRebase() failed', dom=self) 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp libvirtError: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/var/lib/nova/instances/snapshots/tmp5DdrIO/236df3e170e64fabaeb3c7601e2d6c47.delta' 2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp The directory /var/lib/nova/instances/snapshots/tmp5DdrIO does not exist immidiately after this error shows up. Permissions to the parent directory are nova:nova 755, I believe they are correct. Package: nova-compute Version: 1:2013.2-0ubuntu1~cloud0
2014-01-16 18:48:38 Adam Gandelman summary Creating snapshot fails due to nonexistent temporary directory [SRU] Creating snapshot fails due to nonexistent temporary directory
2014-01-17 01:29:07 Brian Murray nominated for series Ubuntu Saucy
2014-01-17 01:29:07 Brian Murray bug task added libvirt (Ubuntu Saucy)
2014-01-17 01:29:34 Brian Murray libvirt (Ubuntu Saucy): importance Undecided High
2014-01-17 01:29:37 Brian Murray libvirt (Ubuntu Saucy): status New Triaged
2014-01-17 01:31:34 Brian Murray libvirt (Ubuntu Saucy): status Triaged Fix Committed
2014-01-17 01:31:37 Brian Murray bug added subscriber Ubuntu Stable Release Updates Team
2014-01-17 01:31:40 Brian Murray bug added subscriber SRU Verification
2014-01-17 01:31:44 Brian Murray tags libvirt libvirt verification-needed
2014-01-22 21:30:05 Marc Deslauriers bug added subscriber Marc Deslauriers
2014-02-07 18:14:30 Russell Bryant nova: status New Invalid
2014-04-05 08:38:06 xiaofa bug added subscriber xiaofa
2014-04-07 15:45:38 Serge Hallyn libvirt (Ubuntu Saucy): status Fix Committed Confirmed
2014-04-26 06:19:51 snowch bug added subscriber snowch
2014-05-15 15:59:48 cristi1979 bug added subscriber cristi1979
2014-12-05 06:35:22 Rolf Leggewie libvirt (Ubuntu Saucy): status Confirmed Won't Fix