nova-manage db purge fails on large datasets

Bug #1983188 reported by Arun S A G
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
In Progress
Wishlist
Rajesh Tailor

Bug Description

This is similar to older bug https://bugs.launchpad.net/bugs/1543937 but in 'nova-manage db purge'. Purging a large dataset causes this failure in Galera cluster. Mariadb log has following error when this happens:

2022-07-30 20:22:10 1567 [Warning] WSREP: transaction size limit (2147483647) exceeded: 2147483648
2022-07-30 20:22:10 1567 [ERROR] WSREP: rbr write fail, data_len: 0, 2

This happens because the transaction is too large for mariadb to handle. 'nova-manage db archive_deleted_rows' works around this by limiting the number of max_rows using --max_rows argument and checking it against db.MAX_INT variable, we might have to do similar thing for purge

Traceback:
nova-manage --config-file /etc/nova/nova.conf db purge --verbose --all-cells --all

An error has occurred:
Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/cmd/manage.py", line 2793, in main
    ret = fn(*fn_args, **fn_kwargs)
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/cmd/manage.py", line 454, in purge
    status_fn=status)
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 4426, in purge_shadow_tables
    deleted = conn.execute(delete)
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
    distilled_params,
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
    e, statement, parameters, cursor, context
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1508, in _handle_dbapi_exception
    util.raise_(newraise, with_traceback=exc_info[2], from_=e)
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1301, in _execute_context
    self._root._commit_impl(autocommit=True)
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 773, in _commit_impl
    self._handle_dbapi_exception(e, None, None, None, None)
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1508, in _handle_dbapi_exception
    util.raise_(newraise, with_traceback=exc_info[2], from_=e)
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 771, in _commit_impl
    self.engine.dialect.do_commit(self.connection)
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/dialects/mysql/base.py", line 2463, in do_commit
    dbapi_connection.commit()
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 422, in commit
    self._read_ok_packet()
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 396, in _read_ok_packet
    pkt = self._read_packet()
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 676, in _read_packet
    packet.raise_for_error()
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/protocol.py", line 223, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
    raise errorclass(errno, errval)
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1180, 'Got error 90 "Message too long" during COMMIT')
(Background on this error at: http://sqlalche.me/e/13/e3q8)

MariaDB log has following error:
2022-07-30 20:22:10 1567 [Warning] WSREP: transaction size limit (2147483647) exceeded: 2147483648
2022-07-30 20:22:10 1567 [ERROR] WSREP: rbr write fail, data_len: 0, 2

More information on how to replicate this and workaround this using mariadb is available here
https://www.percona.com/blog/2015/10/26/how-big-can-your-galera-transactions-be/

Tags: nova-manage
Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

Triaging as Wishlist since we mean to add a specific parameter for the nova-manage command.

Changed in nova:
importance: Undecided → Wishlist
status: New → Confirmed
Rajesh Tailor (ratailor)
Changed in nova:
assignee: nobody → Rajesh Tailor (ratailor)
Revision history for this message
Arun S A G (sagarun) wrote :

Thanks for working on this bug. Please let me know if you need reviews. I am happy to provider them.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/927339

Changed in nova:
status: Confirmed → In Progress
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.