Upgrading from train(el7) to ussuri(el8): Packet sequence number wrong - got 2 expected 1

[Igor Gnatenko]

We are trying to upgrade our OpenStack from Train (based on CentOS 7) to the Ussuri (based on CentOS 8) and during `neutron-db-manage upgrade --expand` I see it failing with a following traceback. I've tried googling this, but only thing I found was "wrong usage of PyMySQL" which sounds like a bug somewhere in OpenStack.

python3-PyMySQL-0.9.3-2.el8.rdo.1.noarch
python3-alembic-1.3.1-1.el8.noarch
python3-sqlalchemy-1.3.2-1.module_el8.1.0+245+c39af44f.x86_64
python3-neutron-16.0.0-1.el8+gdc.noarch

Traceback: http://paste.openstack.org/show/797412/

We run MariaDB 10.3.17 with ProxySQL + Galera cluster.

[Bernard Cafarelli (bcafarel)]

Pasting the initial portion for reference:
[NAX][NO TICKET][root@novatron01a ~]# neutron-db-manage upgrade --expand
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
  Running upgrade (expand) for neutron ...
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade c613d0b82681 -> c3e9d13c4367
INFO [alembic.runtime.migration] Running upgrade c3e9d13c4367 -> 86274d77933e
INFO [alembic.runtime.migration] Running upgrade 86274d77933e -> f4b9654dd40c
INFO [alembic.runtime.migration] Running upgrade f4b9654dd40c -> a010322604bc
INFO [alembic.runtime.migration] Running upgrade a010322604bc -> 263d454a9655
INFO [alembic.runtime.migration] Running upgrade 263d454a9655 -> Ibac91d24da2
ERROR [sqlalchemy.pool.impl.QueuePool] Exception during reset or similar
Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 170, in execute
    result = self._query(query)
  File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 328, in _query
    conn.query(q)
  File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 517, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 732, in _read_query_result
    result.read()
  File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1077, in read
    first_packet = self.connection._read_packet()
  File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 684, in _read_packet
    packet.check_error()
  File "/usr/lib/python3.6/site-packages/pymysql/protocol.py", line 220, in check_error
    err.raise_mysql_exception(self._data)
  File "/usr/lib/python3.6/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.InternalError: (-1, '\x01\x02\x02\x00\x00\x00\rEnabling keys')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 739, in _rollback_impl
    self.engine.dialect.do_rollback(self.connection)
  File "/usr/lib64/python3.6/site-packages/sqlalchemy/dialects/mysql/base.py", line 2227, in do_rollback
    dbapi_connection.rollback()
  File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 430, in rollback
    self._read_ok_packet()
  File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 394, in _read_ok_packet
    pkt = self._read_packet()
  File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 671, in _read_packet
    % (packet_number, self._next_seq_id))
pymysql.err.InternalError: Packet sequence number wrong - got 2 expected 1

[Bernard Cafarelli (bcafarel)]

This is the relevant migration:
https://opendev.org/openstack/neutron/src/branch/master/neutron/db/migration/alembic_migrations/versions/ussuri/expand/Ibac91d24da2_port_forwarding_description.py

I have not found similar issue, especially on install with clean tables this runs properly. So I think the issue is not in the alter_column/create_foreign_key/create_unique_constraint calls, but in the existing records update: some incorrect entry in either "portforwardings" or "standardattributes" tables. Can you check these tables?

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]