Comment 9 for bug 1649616

Revision history for this message
Peter Sabaini (peter-sabaini) wrote :

We seem to be hitting this in production.

Mysql log has:

170309 9:06:23 [Warning] WSREP: transaction size limit (1073774592) exceeded: 1073807360
170309 9:06:23 [ERROR] WSREP: rbr write fail, data_len: 0, 2

The leftover GRA log files contain only DELETE statements for keystone.token afaict.

Keystone log contains:

(oslo_db.sqlalchemy.exc_filters): 2017-03-09 09:07:06,576 ERROR DBAPIError exception wrapped from (pymysql.err.InternalError) (1180, u'Got error 5 during COMMIT')
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 688, in _commit_impl
    self.engine.dialect.do_commit(self.connection)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/dialects/mysql/base.py", line 2514, in do_commit
    dbapi_connection.commit()
  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 758, in commit
    self._read_ok_packet()
  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 737, in _read_ok_packet
    pkt = self._read_packet()
  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 966, in _read_packet
    packet.check_error()
  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 394, in check_error
    err.raise_mysql_exception(self._data)
  File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 120, in raise_mysql_exception
    _check_mysql_exception(errinfo)
  File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 115, in _check_mysql_exception
    raise InternalError(errno, errorvalue)
InternalError: (1180, u'Got error 5 during COMMIT')

The token table has ~500k entries.