bw_usage_update() mysql innodb Deadlock

[Chris Behrens]

UPDATE + INSERT inside a transaction can deadlock when running in parallel in some cases. bw_usage_update() code does this. We should check for deadlock and retry.

[Chris Behrens (cbehrens)]

ERROR:nova.openstack.common.db.sqlalchemy.session:DB exception wrapped.
Traceback (most recent call last):
  File "/home/cbehrens/git/nova/comstud.git/nova/openstack/common/db/sqlalchemy/session.py", line 416, in _wrap
    return f(*args, **kwargs)
  File "/home/cbehrens/git/nova/comstud.git/nova/openstack/common/db/sqlalchemy/session.py", line 610, in flush
    return super(Session, self).flush(*args, **kwargs)
  File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/orm/session.py", line 1720, in flush
    self._flush(objects)
  File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/orm/session.py", line 1791, in _flush
    flush_context.execute()
  File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/orm/unitofwork.py", line 331, in execute
    rec.execute(self)
  File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/orm/unitofwork.py", line 475, in execute
    uow
  File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/orm/persistence.py", line 64, in save_obj
    table, insert)
  File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/orm/persistence.py", line 558, in _emit_insert_statements
    execute(statement, params)
  File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/engine/base.py", line 1449, in execute
    params)
  File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
    context)
  File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
    context)
  File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/engine/default.py", line 331, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python2.6/dist-packages/MySQLdb/cursors.py", line 201, in execute
    self.errorhandler(self, exc, value)
  File "/usr/local/lib/python2.6/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'INSERT INTO bw_usage_cache (created_at, updated_at, deleted_at, deleted, uuid, mac, start_period, last_refreshed, bw_in, bw_out, last_ctr_in, last_ctr_out) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)' (datetime.datetime(2013, 2, 19, 1, 5, 55, 199067), None, None, 0, 'comstud0-11a6-429b-b728-fd489cb06cea', 'e6:82:a0:8d:b0:ed', datetime.datetime(2013, 2, 19, 1, 5, 54, 534992), datetime.datetime(2013, 2, 19, 1, 5, 55, 190637), 1, 2, 3, 4)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/22276

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/22276
Committed: http://github.com/openstack/nova/commit/e53e22271ed7e7a9b919d817a8eb50a1ecce16f8
Submitter: Jenkins
Branch: master

commit e53e22271ed7e7a9b919d817a8eb50a1ecce16f8
Author: Chris Behrens <email address hidden>
Date: Tue Feb 19 01:04:37 2013 +0000

    Retry bw_usage_update() on innodb Deadlock

    Adds a new decorator _retry_on_deadlock() to sqlalchemy api. This patch
    makes bw_usage_update() use it.

    Fixes bug 1129622

    Change-Id: I0293c62d2dd5ac036445bc639cabbd05ba016e83