The leftover GRA log files contain only DELETE statements for keystone.token afaict.
Keystone log contains:
(oslo_db.sqlalchemy.exc_filters): 2017-03-09 09:07:06,576 ERROR DBAPIError exception wrapped from (pymysql.err.InternalError) (1180, u'Got error 5 during COMMIT')
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 688, in _commit_impl
self.engine.dialect.do_commit(self.connection)
File "/usr/lib/python2.7/dist-packages/sqlalchemy/dialects/mysql/base.py", line 2514, in do_commit
dbapi_connection.commit()
File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 758, in commit
self._read_ok_packet()
File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 737, in _read_ok_packet
pkt = self._read_packet()
File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 966, in _read_packet
packet.check_error()
File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 394, in check_error
err.raise_mysql_exception(self._data)
File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 120, in raise_mysql_exception
_check_mysql_exception(errinfo)
File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 115, in _check_mysql_exception
raise InternalError(errno, errorvalue)
InternalError: (1180, u'Got error 5 during COMMIT')
We seem to be hitting this in production.
Mysql log has:
170309 9:06:23 [Warning] WSREP: transaction size limit (1073774592) exceeded: 1073807360
170309 9:06:23 [ERROR] WSREP: rbr write fail, data_len: 0, 2
The leftover GRA log files contain only DELETE statements for keystone.token afaict.
Keystone log contains:
(oslo_db. sqlalchemy. exc_filters) : 2017-03-09 09:07:06,576 ERROR DBAPIError exception wrapped from (pymysql. err.InternalErr or) (1180, u'Got error 5 during COMMIT') python2. 7/dist- packages/ sqlalchemy/ engine/ base.py" , line 688, in _commit_impl engine. dialect. do_commit( self.connection ) python2. 7/dist- packages/ sqlalchemy/ dialects/ mysql/base. py", line 2514, in do_commit connection. commit( ) python2. 7/dist- packages/ pymysql/ connections. py", line 758, in commit _read_ok_ packet( ) python2. 7/dist- packages/ pymysql/ connections. py", line 737, in _read_ok_packet python2. 7/dist- packages/ pymysql/ connections. py", line 966, in _read_packet check_error( ) python2. 7/dist- packages/ pymysql/ connections. py", line 394, in check_error raise_mysql_ exception( self._data) python2. 7/dist- packages/ pymysql/ err.py" , line 120, in raise_mysql_ exception mysql_exception (errinfo) python2. 7/dist- packages/ pymysql/ err.py" , line 115, in _check_ mysql_exception errno, errorvalue)
Traceback (most recent call last):
File "/usr/lib/
self.
File "/usr/lib/
dbapi_
File "/usr/lib/
self.
File "/usr/lib/
pkt = self._read_packet()
File "/usr/lib/
packet.
File "/usr/lib/
err.
File "/usr/lib/
_check_
File "/usr/lib/
raise InternalError(
InternalError: (1180, u'Got error 5 during COMMIT')
The token table has ~500k entries.