OpenStack Compute (nova)

Failed to shelve paused instance with libvirt: "libvirtError: Requested operation is not valid: domain is not running"

Bug #1745529 reported by Brin Zhang on 2018-01-26

This bug affects 1 person

	Status	Importance	Assigned to
OpenStack Compute (nova)	Fix Released	Medium	Leopard Ma
Ocata	Fix Committed	Medium	Ghanshyam Mann
Pike	Fix Committed	Medium	Ghanshyam Mann
Queens	Fix Committed	Medium	Ghanshyam Mann

Bug Description

You can use the following steps to reproduce(This problem also exists in the Ocata version)：
1.nova pause 5e824732-c557-434f-9902-a6d9abce7bda
2.nova list
+--------------------------------------+-------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+---------------------+
| 5e824732-c557-434f-9902-a6d9abce7bda | test | PAUSED | - | Paused | network=192.168.0.9 |
+--------------------------------------+-------+--------+------------+-------------+---------------------+
3.nova shelve 5e824732-c557-434f-9902-a6d9abce7bda

Error log is as follows：
oslo_concurrency.lockutils [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin admin] Lock "compute_resources" acquired by "nova.compute.resource_tracker.update_usage" :: waited 0.000s {{(pid=8304) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273}}
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: DEBUG oslo_concurrency.lockutils [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin admin] Lock "compute_resources" released by "nova.compute.resource_tracker.update_usage" :: held 0.038s {{(pid=8304) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285}}
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: INFO nova.compute.manager [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin admin] [instance: 5e824732-c557-434f-9902-a6d9abce7bda] Successfully reverted task state from shelving on failure for instance.
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin admin] Exception during message handling: libvirtError: Requested operation is not valid: domain is not running
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 76, in wrapped
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server self.force_reraise()
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 67, in wrapped
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 186, in decorated_function
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server "Error: %s", e, instance=instance)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server self.force_reraise()
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 156, in decorated_function
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/utils.py", line 944, in decorated_function
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 214, in decorated_function
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server self.force_reraise()
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 202, in decorated_function
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 4678, in shelve_instance
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server do_shelve_instance()
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 4677, in do_shelve_instance
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server self._shelve_instance(context, instance, image_id, clean_shutdown)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 4711, in _shelve_instance
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server self._power_off_instance(context, instance, clean_shutdown)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 2318, in _power_off_instance
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server self.driver.power_off(instance, timeout, retry_interval)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2684, in power_off
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server self._clean_shutdown(instance, timeout, retry_interval)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2639, in _clean_shutdown
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server guest.shutdown()
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 589, in shutdown
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server self._domain.shutdown()
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server result = proxy_call(self._autowrap, f, *args, **kwargs)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server six.reraise(c, e, tb)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2399, in shutdown
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server if ret == -1: raise libvirtError ('virDomainShutdown() failed', dom=self)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server libvirtError: Requested operation is not valid: domain is not running
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server
Jan 26 10:13:16 localhost.localdomain nova-compute[8304]: DEBUG oslo_service.periodic_task [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin admin] Running periodic task ComputeManager._poll_rebooting_instances {{(pid=8304) run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215}}

See original description

Tags:

Brin Zhang (zhangbailin) on 2018-01-26

description:	updated
Changed in nova:
assignee:	nobody → Brin Zhang (zhangbailin)

zhiyuan su (lawrencesu) on 2018-01-26

Changed in nova:
status:	New → Confirmed
tags:	added: shelve
tags:	added: ocata removed: shelve

Leopard Ma (leopardma) on 2018-01-26

Changed in nova:
assignee:	Brin Zhang (zhangbailin) → Leopard Ma (leopardma)

OpenStack Infra (hudson-openstack) on 2018-02-12

Changed in nova:
status:	Confirmed → In Progress

Matt Riedemann (mriedem) on 2018-03-27

tags:

added: libvirt shelve

Matt Riedemann (mriedem) on 2018-03-27

Changed in nova:
importance:	Undecided → Medium
summary:	- The pause status of the virtual machine to perform the shelve operation - reported error + Failed to shelve paused instance with libvirt
summary:	- Failed to shelve paused instance with libvirt + Failed to shelve paused instance with libvirt: "libvirtError: Requested + operation is not valid: domain is not running"

OpenStack Infra (hudson-openstack) on 2018-05-01

Changed in nova:
assignee:	Leopard Ma (leopardma) → Matt Riedemann (mriedem)

Matt Riedemann (mriedem) on 2018-05-01

Changed in nova:
assignee:	Matt Riedemann (mriedem) → Leopard Ma (leopardma)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-05-11: Fix merged to nova (master)

Reviewed: https://review.openstack.org/541660
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d5901f613cf98f61b5253a1568b22af1d9dd1a08
Submitter: Zuul
Branch: master

commit d5901f613cf98f61b5253a1568b22af1d9dd1a08
Author: LeopardMa <email address hidden>
Date: Wed Feb 7 10:21:08 2018 +0800

Fix shelving a paused instance

    It is possible to shelve a paused instance, but in that case the
    guest is already shutdown, and some hypervisors will fail when
    trying to perform a clean shutdown of a non-running guest. For
    example, attempting to shelve a paused libvirt instance will
    result in this error:

libvirtError: Requested operation is not valid: domain is not running

Therefore, if the instance is paused, we don't attempt a clean
shutdown while shelving.

Related Tempest test: https://review.openstack.org/564127/

Closes-Bug: #1745529
Change-Id: I8ca25d9847d50001fbe8513a6c1dba8b697c24e4

Changed in nova:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-05-20: Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/569650

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-05-20: Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/569651

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-05-20: Fix proposed to nova (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/569655

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-05-29: Fix merged to nova (stable/queens)

Reviewed: https://review.openstack.org/569650
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f521e03f8e1524f5abfef6497b5b1a57daba7ced
Submitter: Zuul
Branch: stable/queens

commit f521e03f8e1524f5abfef6497b5b1a57daba7ced
Author: LeopardMa <email address hidden>
Date: Wed Feb 7 10:21:08 2018 +0800

Fix shelving a paused instance

libvirtError: Requested operation is not valid: domain is not running

Therefore, if the instance is paused, we don't attempt a clean
shutdown while shelving.

Related Tempest test: https://review.openstack.org/564127/

    Closes-Bug: #1745529
    Change-Id: I8ca25d9847d50001fbe8513a6c1dba8b697c24e4
    (cherry picked from commit d5901f613cf98f61b5253a1568b22af1d9dd1a08)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-05-31: Fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/569651
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ce740643686bcf4ed8a1e089082547cf8fcef8fc
Submitter: Zuul
Branch: stable/pike

commit ce740643686bcf4ed8a1e089082547cf8fcef8fc
Author: LeopardMa <email address hidden>
Date: Wed Feb 7 10:21:08 2018 +0800

Fix shelving a paused instance

libvirtError: Requested operation is not valid: domain is not running

Therefore, if the instance is paused, we don't attempt a clean
shutdown while shelving.

Related Tempest test: https://review.openstack.org/564127/

    Closes-Bug: #1745529
    Change-Id: I8ca25d9847d50001fbe8513a6c1dba8b697c24e4
    (cherry picked from commit d5901f613cf98f61b5253a1568b22af1d9dd1a08)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-05-31: Fix merged to nova (stable/ocata)

Reviewed: https://review.openstack.org/569655
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e0534442b531c3f89c9c19d95491dcddd5b0fe41
Submitter: Zuul
Branch: stable/ocata

commit e0534442b531c3f89c9c19d95491dcddd5b0fe41
Author: LeopardMa <email address hidden>
Date: Wed Feb 7 10:21:08 2018 +0800

Fix shelving a paused instance

libvirtError: Requested operation is not valid: domain is not running

Therefore, if the instance is paused, we don't attempt a clean
shutdown while shelving.

Related Tempest test: https://review.openstack.org/564127/

    Closes-Bug: #1745529
    Change-Id: I8ca25d9847d50001fbe8513a6c1dba8b697c24e4
    (cherry picked from commit d5901f613cf98f61b5253a1568b22af1d9dd1a08)