Illegal DB access in nova-compute _sync_db_power_states method

Bug #1123219 reported by Daniel Berrange
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Dan Smith

Bug Description

Boot an instance with the libvirt driver, and then use 'virsh destroy [instancename]' to kill it off behind Nova's back.

Now wait a while (upto 10 minutes) for the '_sync_db_power_states' periodic task to run in nova-compute. When it does run it will generate the following exception

2013-02-12 16:25:52.509 ERROR nova.compute [-] No db access allowed in nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 192, in main
    result = function(*args, **kwargs)
  File "/home/berrange/src/cloud/nova/nova/utils.py", line 630, in _inner
    idle = self.f(*self.args, **self.kw)
  File "/home/berrange/src/cloud/nova/nova/service.py", line 571, in periodic_tasks
    return self.manager.periodic_tasks(ctxt, raise_on_error=raise_on_error)
  File "/home/berrange/src/cloud/nova/nova/manager.py", line 229, in periodic_tasks
    task(self, context)
  File "/home/berrange/src/cloud/nova/nova/compute/manager.py", line 3425, in _sync_power_states
    vm_power_state)
  File "/home/berrange/src/cloud/nova/nova/compute/manager.py", line 3492, in _sync_instance_power_state
    self.compute_api.stop(context, db_instance)
  File "/home/berrange/src/cloud/nova/nova/compute/api.py", line 158, in wrapped
    return func(self, context, target, *args, **kwargs)
  File "/home/berrange/src/cloud/nova/nova/compute/api.py", line 148, in inner
    return function(self, context, instance, *args, **kwargs)
  File "/home/berrange/src/cloud/nova/nova/compute/api.py", line 139, in wrapped
    return function(self, context, instance, *args, **kwargs)
  File "/home/berrange/src/cloud/nova/nova/compute/api.py", line 129, in inner
    return f(self, context, instance, *args, **kw)
  File "/home/berrange/src/cloud/nova/nova/compute/api.py", line 1236, in stop
    progress=0)
  File "/home/berrange/src/cloud/nova/nova/compute/api.py", line 158, in wrapped
    return func(self, context, target, *args, **kwargs)
  File "/home/berrange/src/cloud/nova/nova/compute/api.py", line 950, in update
    _, updated = self._update(context, instance, **kwargs)
  File "/home/berrange/src/cloud/nova/nova/compute/api.py", line 957, in _update
    context, instance['uuid'], kwargs)
  File "/home/berrange/src/cloud/nova/nova/db/api.py", line 697, in instance_update_and_get_original
    rv = IMPL.instance_update_and_get_original(context, instance_uuid, values)
  File "/home/berrange/src/cloud/nova/bin/nova-compute", line 65, in __call__
    stacktrace = "".join(traceback.format_stack())

Dan Smith (danms)
Changed in nova:
assignee: nobody → Dan Smith (danms)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/21797

Changed in nova:
status: New → In Progress
Dan Smith (danms)
Changed in nova:
importance: Undecided → High
milestone: none → grizzly-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/21797
Committed: http://github.com/openstack/nova/commit/959e65f320093a6b17945f52ad160da5aa042ff6
Submitter: Jenkins
Branch: master

commit 959e65f320093a6b17945f52ad160da5aa042ff6
Author: Dan Smith <email address hidden>
Date: Tue Feb 12 12:34:39 2013 -0500

    Make compute manager use conductor for stopping instances

    During _sync_power_states, the manager will call to compute_api to
    stop instances as necessary. This takes a database hit, which isn't
    allowed. This directs that call through conductor, following the
    pattern of late.

    Fixes bug 1123219

    Change-Id: I5d789dd73390cb1c41a245f6594b1c3aecf7efde

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-3 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.