Failed to commit reservations in gate

[Jamie Lennox]

From: http://logs.openstack.org/31/105031/14/gate/gate-tempest-dsvm-full/c05b927/console.html

2014-08-05 02:54:01.131 | Log File Has Errors: n-cond
2014-08-05 02:54:01.132 | *** Not Whitelisted *** 2014-08-05 02:25:47.799 ERROR nova.quota [req-19feeaa2-e1d4-419b-a7bb-a19bb7000b1d AggregatesAdminTestJSON-2075387658 AggregatesAdminTestJSON-270189725] Failed to commit reservations [u'ceaa6ce7-db8d-4ba6-871a-b29c59f4a338', u'10d7550d-d791-44dd-8396-2fa6eaea7c20', u'e7a322e2-948d-45f7-892f-7ea4d9aa0e7c']

There are a number of errors happening in that file that arent whitelisted.

This one *seems* to be a possible cause of others.as there is then a number of InstanceNotFound errors.

[Joe Gordon (jogo)]

the gate will fail anytime n-cond has a stacktrace, which is why this is causing jobs to fail.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/112466

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/112466
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=f3ef417122cec2907980d3e755de815c8536e3f4
Submitter: Jenkins
Branch: master

commit f3ef417122cec2907980d3e755de815c8536e3f4
Author: Joe Gordon <email address hidden>
Date: Wed Aug 6 21:11:48 2014 -0700

    Add fingerprint for nova bug 1353131

    This bug causes tempest jobs to fail when its seen in nova-conductor
    since the gate is set up to fail on any nova-conductor stacktrace.

    This stacktrace also occurs in nova-api.

    142 hits in last 10 days with 14 successes due to seeing this stacktrace
    in grenade jobs.

    Change-Id: I0894b76bf483031103faea3740c83cbcb749ca58
    Related-Bug: #1353131

[Kashyap Chamarthy (kashyapc)]

Just noticed this across this from Nova Conductor logs of "gate-tempest-dsvm-full" job[1]

------------------------
.
.
.
2014-08-27 12:44:02.131 ERROR nova.quota [req-c5907f41-fd16-426c-875f-ced42bd18945 ImagesOneServerTestJSON-1604589649 ImagesOneServerTestJSON-1332055474] Failed to commit reservations [u'2d72229d-7f20-4af0-b89b-d0f352690ffa', u'324abcba-288f-437a-9895-f0ca3ced683b', u'9738d565-9122-4fb7-b96f-971fcdd079f7']
2014-08-27 12:44:02.131 27194 TRACE nova.quota Traceback (most recent call last):
2014-08-27 12:44:02.131 27194 TRACE nova.quota File "/opt/stack/new/nova/nova/quota.py", line 1326, in commit
2014-08-27 12:44:02.131 27194 TRACE nova.quota user_id=user_id)
2014-08-27 12:44:02.131 27194 TRACE nova.quota File "/opt/stack/new/nova/nova/quota.py", line 569, in commit
2014-08-27 12:44:02.131 27194 TRACE nova.quota user_id=user_id)
2014-08-27 12:44:02.131 27194 TRACE nova.quota File "/opt/stack/new/nova/nova/db/api.py", line 1150, in reservation_commit
2014-08-27 12:44:02.131 27194 TRACE nova.quota user_id=user_id)
2014-08-27 12:44:02.131 27194 TRACE nova.quota File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 167, in wrapper
2014-08-27 12:44:02.131 27194 TRACE nova.quota return f(*args, **kwargs)
2014-08-27 12:44:02.131 27194 TRACE nova.quota File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 205, in wrapped
2014-08-27 12:44:02.131 27194 TRACE nova.quota return f(*args, **kwargs)
2014-08-27 12:44:02.131 27194 TRACE nova.quota File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 3296, in reservation_commit
2014-08-27 12:44:02.131 27194 TRACE nova.quota for reservation in reservation_query.all():
2014-08-27 12:44:02.131 27194 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2241, in all
2014-08-27 12:44:02.131 27194 TRACE nova.quota return list(self)
2014-08-27 12:44:02.131 27194 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2353, in __iter__
2014-08-27 12:44:02.131 27194 TRACE nova.quota return self._execute_and_instances(context)
2014-08-27 12:44:02.131 27194 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2368, in _execute_and_instances
2014-08-27 12:44:02.131 27194 TRACE nova.quota result = conn.execute(querycontext.statement, self._params)
2014-08-27 12:44:02.131 27194 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 662, in execute
2014-08-27 12:44:02.131 27194 TRACE nova.quota params)
2014-08-27 12:44:02.131 27194 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
2014-08-27 12:44:02.131 27194 TRACE nova.quota compiled_sql, distilled_params
2014-08-27 12:44:02.131 27194 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
2014-08-27 12:44:02.131 27194 TRACE nova.quota context)
2014-08-27 12:44:02.131 27194 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1024, in _handle_dbapi_exception
2014-08-27 12:44:0...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/118470

[Matt Riedemann (mriedem)]

Sounds like oslo.db should fix this:

https://review.openstack.org/#/c/101901/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Joe Gordon (<email address hidden>) on branch: master
Review: https://review.openstack.org/118470
Reason: fix https://review.openstack.org/#/c/101901/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/101901
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1b83b2f11054ddd3f72fe75c5cef496060afcb3a
Submitter: Jenkins
Branch: master

commit 1b83b2f11054ddd3f72fe75c5cef496060afcb3a
Author: Andrey Kurilin <email address hidden>
Date: Sat Jun 21 13:56:42 2014 +0300

    Move to oslo.db

    Replace common oslo code nova.openstack.common.db by usage
    of oslo.db library and remove common code.

    Replaced catching of raw sqlalchemy exceptions by catches
    of oslo.db exceptions(such as DBError, DBDuplicateEntry, etc).

    Co-Authored-By: Eugeniya Kudryashova <email address hidden>

    Closes-Bug: #1364986
    Closes-Bug: #1353131
    Closes-Bug: #1283987
    Closes-Bug: #1274523
    Change-Id: I0649539e071b2318ec85ed5d70259c949408e64b