resize auto-confirmation failed with nova-conductor because of db access in nova-compute

[wangpan]

I can reproduce this bug in master and stable havana.

this bug is similar to https://bugs.launchpad.net/nova/+bug/1158897
but this bug is cause by _reserve_quota_delta() method, which will need to access DB if quota delta is not empty,
the trace log is:
2013-12-19 01:39:40.879 ERROR nova.compute [-] No db access allowed in nova-compute:
File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/opt/stack/nova/nova/openstack/common/loopingcall.py", line 125, in _inner
    idle = self.f(*self.args, **self.kw)
  File "/opt/stack/nova/nova/service.py", line 314, in periodic_tasks
    return self.manager.periodic_tasks(ctxt, raise_on_error=raise_on_error)
  File "/opt/stack/nova/nova/manager.py", line 101, in periodic_tasks
    return self.run_periodic_tasks(context, raise_on_error=raise_on_error)
  File "/opt/stack/nova/nova/openstack/common/periodic_task.py", line 180, in run_periodic_tasks
    task(self, context)
  File "/opt/stack/nova/nova/compute/manager.py", line 4535, in _poll_unconfirmed_resizes
    migration=migration)
  File "/opt/stack/nova/nova/compute/api.py", line 199, in wrapped
    return func(self, context, target, *args, **kwargs)
  File "/opt/stack/nova/nova/compute/api.py", line 189, in inner
    return function(self, context, instance, *args, **kwargs)
  File "/opt/stack/nova/nova/compute/api.py", line 216, in _wrapped
    return fn(self, context, instance, *args, **kwargs)
  File "/opt/stack/nova/nova/compute/api.py", line 170, in inner
    return f(self, context, instance, *args, **kw)
  File "/opt/stack/nova/nova/compute/api.py", line 2158, in confirm_resize
    reservations = self._reserve_quota_delta(context, deltas)
  File "/opt/stack/nova/nova/compute/api.py", line 2238, in _reserve_quota_delta
    return QUOTAS.reserve(context, project_id=project_id, **deltas)
  File "/opt/stack/nova/nova/quota.py", line 1272, in reserve
    user_id=user_id)
  File "/opt/stack/nova/nova/quota.py", line 487, in reserve
    has_sync=True, project_id=project_id)
  File "/opt/stack/nova/nova/quota.py", line 354, in _get_quotas
    usages=False)
  File "/opt/stack/nova/nova/quota.py", line 264, in get_project_quotas
    project_quotas = db.quota_get_all_by_project(context, project_id)
  File "/opt/stack/nova/nova/db/api.py", line 1023, in quota_get_all_by_project
    return IMPL.quota_get_all_by_project(context, project_id)
  File "/opt/stack/nova/nova/cmd/compute.py", line 48, in __call__
    stacktrace = "".join(traceback.format_stack())

2013-12-19 01:39:40.879 ERROR nova.compute.manager [-] [instance: 4ffaabaf-a480-421f-a7a7-efd148d7c956] Error auto-confirming resize: nova-compute. Will retry later.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/63305

[wangpan (hzwangpan)]

there also has a bug of quota reservation and usage in auto confirmation process, even the db access is allowed.
the reservation in db is like below(project_id and user_id is NULL):
| created_at | updated_at | deleted_at | id | uuid | usage_id | project_id | resource | delta | expire | deleted | user_id
| 2013-12-20 03:39:01 | NULL | 2013-12-20 03:39:12 | 26 | 1fda83bf-0cdf-47f2-b88e-78527444d070 | 8 | NULL | ram | -448 | 2013-12-21 03:39:01 | 26 | NULL

it will be commited to NULL project after confirmation, not the instance's project:
| created_at | updated_at | deleted_at | id | project_id | resource | in_use | reserved | until_refresh | deleted | user_id
| 2013-12-20 03:39:01 | 2013-12-20 03:39:12 | NULL | 8 | NULL | ram | -448 | 0 | NULL | 0 | NULL

this is because the context is generated in nova-compute(in the periodic task process),
so the project_id and user_id are all None, but the qutoa reserve method need them(project_id at least).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/81099
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5ebc60c48d4a8b6c7bac96d923626df9ae67a2b2
Submitter: Jenkins
Branch: master

commit 5ebc60c48d4a8b6c7bac96d923626df9ae67a2b2
Author: Matt Riedemann <email address hidden>
Date: Mon Mar 17 05:14:10 2014 -0700

    Make compute API resize methods use Quotas objects

    The compute API's various resize methods are using the old-style quota
    class to reserve and commit quota changes directly to the database. This
    change uses quotas objects to do the reserve and commit instead so those
    operations go through conductor for the database access.

    Note that the long-term goal is to pass the quota object over RPC
    rather than the reservations list, but that's a larger patch which
    is not necessary for the bug fix.

    Closes-Bug: #1262461

    Change-Id: Id18357f449193d25287f32bb8ffbf26ee2194a22