OpenStack Compute (nova)

Booting instance cause infinite recursion in the case of enough user quota and unlimited project quota

  1. Series liberty
  2. Bug #1491222
Bug #1491222 reported by Rui Chen
18
This bug affects 5 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Rui Chen
Liberty
Fix Released
Medium
Jordan Callicoat

Bug Description

1. code base

$ git log -1
commit e041f3ee625b2737a1788304916589e3cc74db3d
Merge: c310255 bef481b
Author: Jenkins <email address hidden>
Date: Tue Sep 1 21:53:31 2015 +0000

    Merge "Fix three typos on nova/pci directory"

2. nova-api.log file:

2015-09-01 18:03:32.490 DEBUG nova.db.sqlalchemy.api [req-b506dbaa-7157-4001-a79d-af1c9d642aa4 admin admin] Request is over project quota for resource "instances". Project limit: -1, delta: 1, current total project usage: 0 _calculate_overquota /opt/stack/nova/nova/db/sqlalchemy/api.py:3497
........
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 427, in _check_num_instances_quota
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack min_count, allowed)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 427, in _check_num_instances_quota
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack min_count, allowed)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 427, in _check_num_instances_quota
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack min_count, allowed)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 427, in _check_num_instances_quota
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack min_count, allowed)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 427, in _check_num_instances_quota
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack min_count, allowed)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 401, in _check_num_instances_quota
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack cores=req_cores, ram=req_ram)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 213, in wrapper
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack return fn(self, *args, **kwargs)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/opt/stack/nova/nova/objects/quotas.py", line 92, in reserve
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack **deltas)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/opt/stack/nova/nova/quota.py", line 1343, in reserve
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack user_id=user_id)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/opt/stack/nova/nova/quota.py", line 569, in reserve
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack project_quotas=project_quotas)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/opt/stack/nova/nova/quota.py", line 406, in _get_quotas
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack project_quotas=project_quotas)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/opt/stack/nova/nova/quota.py", line 246, in get_user_quotas
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack user_id)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/opt/stack/nova/nova/db/api.py", line 1032, in quota_get_all_by_project_and_user
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack return IMPL.quota_get_all_by_project_and_user(context, project_id, user_id)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 216, in wrapper
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack return f(*args, **kwargs)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 3128, in quota_get_all_by_project_and_user
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack filter_by(user_id=user_id).\
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2320, in all
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack return list(self)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2438, in __iter__
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack return self._execute_and_instances(context)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2453, in _execute_and_instances
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack result = conn.execute(querycontext.statement, self._params)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 729, in execute
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack return meth(self, multiparams, params)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack return connection._execute_clauseelement(self, multiparams, params)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 819, in _execute_clauseelement
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack inline=len(distilled_params) > 1)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "<string>", line 1, in <lambda>
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 493, in compile
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack return self._compiler(dialect, bind=bind, **kw)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 499, in _compiler
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack return dialect.statement_compiler(dialect, self, **kw)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 395, in __init__
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack Compiled.__init__(self, dialect, statement, **kwargs)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 199, in __init__
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack self.string = self.process(self.statement, **compile_kwargs)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 222, in process
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack return obj._compiler_dispatch(self, **kwargs)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/visitors.py", line 80, in _compiler_dispatch
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack return meth(self, **kw)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 1544, in visit_select
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack t = select._whereclause._compiler_dispatch(self, **kwargs)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/visitors.py", line 92, in _compiler_dispatch
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack return meth(self, **kw)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 662, in visit_clauselist
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack for c in clauselist.clauses)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 659, in <genexpr>
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack s for s in
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 662, in <genexpr>
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack for c in clauselist.clauses)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/visitors.py", line 92, in _compiler_dispatch
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack return meth(self, **kw)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 662, in visit_clauselist
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack for c in clauselist.clauses)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 659, in <genexpr>
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack s for s in
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 662, in <genexpr>
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack for c in clauselist.clauses)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/visitors.py", line 92, in _compiler_dispatch
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack return meth(self, **kw)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 847, in visit_binary
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack return self._generate_generic_binary(binary, opstring, **kw)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 864, in _generate_generic_binary
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack binary.right._compiler_dispatch(self, **kw)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/visitors.py", line 92, in _compiler_dispatch
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack return meth(self, **kw)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 993, in visit_bindparam
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack name = self._truncate_bindparam(bindparam)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 1049, in _truncate_bindparam
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack bind_name = self._truncated_identifier("bindparam", bind_name)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 1060, in _truncated_identifier
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack anonname = name.apply_map(self.anon_map)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 3331, in apply_map
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack return self % map_
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/_collections.py", line 680, in __missing__
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack self[key] = val = self.creator(key)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 1079, in _process_anon
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack return derived + "_" + str(anonymous_counter)
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack RuntimeError: maximum recursion depth exceeded while getting the str of an object
2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack
2015-09-01 18:05:34.310 INFO nova.api.openstack [req-82c1beaa-d618-4ceb-8941-105df8720ad1 admin admin] http://localhost:8774/v2/4ffccaf0c84a44a29799b7bacab52334/servers returned with HTTP 500

3. Reproduce steps:

#Step 1: set project quota to unlimited
$ nova quota-update --instances -1 $tenant

#Step 2: set user quota to 1 (current usage 0)
$ nova quota-update --user $tenantUser --instances 1 $tenant

#Step 3: boot instance
$ nova boot --image cirros-0.3.2-x86_64-disk --nic net-id=84140e3f-f20d-4ec5-9eac-d455dba7c569 --flavor m1.tiny chenrui_vm
ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-82c1beaa-d618-4ceb-8941-105df8720ad1)

Expected result:
* booting instance success

Actual result:
* booting instance failed

See original description
Rui Chen (kiwik-chenrui)
Changed in nova:
assignee: nobody → Rui Chen (kiwik-chenrui)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/219596

Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/219596
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b630d1471b9f7efc0949fcd8042e86641a997762
Submitter: Jenkins
Branch: master

commit b630d1471b9f7efc0949fcd8042e86641a997762
Author: Rui Chen <email address hidden>
Date: Wed Sep 2 16:07:11 2015 +0800

    Fix booting fail when unlimited project quota

    If we launch a instance in the case of unlimited project quota
    and limited user quota, API would return internal error 500.
    The code logic wouldn't check the case of unlimited project quota
    in _calculate_overquota() and OverQuota exception was raised,
    it cause infinite recursion in _check_num_instances_quota()

    Change-Id: I7831266b6269ea5d9a3aaa32041158293a1d601e
    Closes-Bug: #1491222

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/nova 13.0.0.0b1

This issue was fixed in the openstack/nova 13.0.0.0b1 development milestone.

Doug Hellmann (doug-hellmann)
Changed in nova:
status: Fix Committed → Fix Released
Jordan Callicoat (jcallicoat)
tags: added: liberty-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/329231

Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/liberty)

Reviewed: https://review.openstack.org/329231
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9ab10b8c99f18d4546a7bf8dcefed37121414f2a
Submitter: Jenkins
Branch: stable/liberty

commit 9ab10b8c99f18d4546a7bf8dcefed37121414f2a
Author: Rui Chen <email address hidden>
Date: Wed Sep 2 16:07:11 2015 +0800

    Fix booting fail when unlimited project quota

    If we launch a instance in the case of unlimited project quota
    and limited user quota, API would return internal error 500.
    The code logic wouldn't check the case of unlimited project quota
    in _calculate_overquota() and OverQuota exception was raised,
    it cause infinite recursion in _check_num_instances_quota()

    Change-Id: I7831266b6269ea5d9a3aaa32041158293a1d601e
    Closes-Bug: #1491222
    (cherry picked from commit b630d1471b9f7efc0949fcd8042e86641a997762)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 12.0.5

This issue was fixed in the openstack/nova 12.0.5 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.