This is similar to older bug https://bugs.launchpad.net/bugs/1543937 but in 'nova-manage db purge'. Purging a large dataset causes this failure in Galera cluster. Mariadb log has following error when this happens:
2022-07-30 20:22:10 1567 [Warning] WSREP: transaction size limit (2147483647) exceeded: 2147483648
2022-07-30 20:22:10 1567 [ERROR] WSREP: rbr write fail, data_len: 0, 2
This happens because the transaction is too large for mariadb to handle. 'nova-manage db archive_deleted_rows' works around this by limiting the number of max_rows using --max_rows argument and checking it against db.MAX_INT variable, we might have to do similar thing for purge
Traceback:
nova-manage --config-file /etc/nova/nova.conf db purge --verbose --all-cells --all
An error has occurred:
Traceback (most recent call last):
File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/cmd/manage.py", line 2793, in main
ret = fn(*fn_args, **fn_kwargs)
File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/cmd/manage.py", line 454, in purge
status_fn=status)
File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 4426, in purge_shadow_tables
deleted = conn.execute(delete)
File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
distilled_params,
File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
e, statement, parameters, cursor, context
File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1508, in _handle_dbapi_exception
util.raise_(newraise, with_traceback=exc_info[2], from_=e)
File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1301, in _execute_context
self._root._commit_impl(autocommit=True)
File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 773, in _commit_impl
self._handle_dbapi_exception(e, None, None, None, None)
File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1508, in _handle_dbapi_exception
util.raise_(newraise, with_traceback=exc_info[2], from_=e)
File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 771, in _commit_impl
self.engine.dialect.do_commit(self.connection)
File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/dialects/mysql/base.py", line 2463, in do_commit
dbapi_connection.commit()
File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 422, in commit
self._read_ok_packet()
File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 396, in _read_ok_packet
pkt = self._read_packet()
File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 676, in _read_packet
packet.raise_for_error()
File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/protocol.py", line 223, in raise_for_error
err.raise_mysql_exception(self._data)
File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
raise errorclass(errno, errval)
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1180, 'Got error 90 "Message too long" during COMMIT')
(Background on this error at: http://sqlalche.me/e/13/e3q8)
MariaDB log has following error:
2022-07-30 20:22:10 1567 [Warning] WSREP: transaction size limit (2147483647) exceeded: 2147483648
2022-07-30 20:22:10 1567 [ERROR] WSREP: rbr write fail, data_len: 0, 2
More information on how to replicate this and workaround this using mariadb is available here
https://www.percona.com/blog/2015/10/26/how-big-can-your-galera-transactions-be/
Triaging as Wishlist since we mean to add a specific parameter for the nova-manage command.