Upgrade to Ussuri cinder charm in error require manual online_data_migrations

[Diko Parvanov]

Happened independantly on 2 different clouds during upgrade from X to ussuri (one started from queens, the other from train) - both had the same error on train -> ussuri

cloud: bionic (both clouds)
cinder revision: 306 (latest on both clouds)

Charm in error state with:

unit-cinder-0: 09:33:26 INFO unit.cinder/0.juju-log Retrying 'migrate_database' 1 more times (delay=15)
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.680 768484 INFO migrate.versioning.api [-] 135 -> 136...
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 WARNING oslo_db.sqlalchemy.exc_filters [-] DBAPIError exception wrapped.: pymysql.err.InternalError: (1138, 'Invalid use of NULL value')
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters cursor, statement, parameters, context
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 581, in do_execute
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters)
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 165, in execute
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters result = self._query(query)
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 321, in _query
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters conn.query(q)
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 860, in query
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters self._affected_rows = self._read_query_result(unbuffered=unbuffered)
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1061, in _read_query_result
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters result.read()
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1349, in read
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters first_packet = self.connection._read_packet()
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1018, in _read_packet
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters packet.check_error()
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 384, in check_error
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters err.raise_mysql_exception(self._data)
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters raise errorclass(errno, errval)
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters pymysql.err.InternalError: (1138, 'Invalid use of NULL value')
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed 2020-12-11 09:33:42.707 768484 ERROR oslo_db.sqlalchemy.exc_filters
unit-cinder-0: 09:33:42 DEBUG unit.cinder/0.config-changed Error during database migration: Migration cannot continue until all volumes have been migrated to the `__DEFAULT__` volume type. Please run `cinder-manage db online_data_migrations`. There are still untyped volumes unmigrated.
unit-cinder-0: 09:33:43 WARNING unit.cinder/0.config-changed Traceback (most recent call last):
unit-cinder-0: 09:33:43 WARNING unit.cinder/0.config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/config-changed", line 700, in <module>
unit-cinder-0: 09:33:43 WARNING unit.cinder/0.config-changed hooks.execute(sys.argv)
unit-cinder-0: 09:33:43 WARNING unit.cinder/0.config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/charmhelpers/core/hookenv.py", line 945, in execute
unit-cinder-0: 09:33:43 WARNING unit.cinder/0.config-changed self._hooks[hook_name]()
unit-cinder-0: 09:33:43 WARNING unit.cinder/0.config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/charmhelpers/contrib/openstack/utils.py", line 1720, in wrapped_f
unit-cinder-0: 09:33:43 WARNING unit.cinder/0.config-changed stopstart, restart_functions)
unit-cinder-0: 09:33:43 WARNING unit.cinder/0.config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/charmhelpers/core/host.py", line 747, in restart_on_change_helper
unit-cinder-0: 09:33:43 WARNING unit.cinder/0.config-changed r = lambda_f()
unit-cinder-0: 09:33:43 WARNING unit.cinder/0.config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/charmhelpers/contrib/openstack/utils.py", line 1719, in <lambda>
unit-cinder-0: 09:33:43 WARNING unit.cinder/0.config-changed (lambda: f(*args, **kwargs)), __restart_map_cache['cache'],
unit-cinder-0: 09:33:43 WARNING unit.cinder/0.config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/charmhelpers/contrib/hardening/harden.py", line 93, in _harden_inner2
unit-cinder-0: 09:33:43 WARNING unit.cinder/0.config-changed return f(*args, **kwargs)
unit-cinder-0: 09:33:43 WARNING unit.cinder/0.config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/config-changed", line 214, in config_changed
unit-cinder-0: 09:33:43 WARNING unit.cinder/0.config-changed do_openstack_upgrade(configs=CONFIGS)

openstack volume type list returned empty results

Manually running cinder-manage db online_data_migrations on the cinder until and juju resolved on all cinder units and openstack-upgrade completed successfully on all 3 units.

[Trent Lloyd (lathiat)]

This seems likely related to https://bugs.launchpad.net/cinder/+bug/1893107 - we will need special handling for this.

Basically if you upgraded from Stein->Train 15.3 or earlier, it setup a failure that only triggers on the Train->Ussuri upgrade. Detailed in the release notes for 15.4.0 and 16.2.0 under "Upgrade Notes" here:
https://docs.openstack.org/releasenotes/cinder/train.html

This release modifies the online database migrations to address an an upgrade issue (Bug #1893107). The issue does not manifest itself in the Train release of cinder, but under specific circumstances it can prevent a cinder database upgrade from Train to Ussuri.

This upgrade notice applies to you only if all of the following conditions are met:

You upgraded to Train from Stein

Before upgrading from Stein, you did not purge the cinder database

Your original upgrade from Stein was to cinder version 15.3.0 or earlier.

 Note

If you are upgrading a Stein installation directly to this release (cinder 15.4.0) or later, this notice does not apply to you.

If all the above three items apply to you, as part of your upgrade to cinder 15.4.0 you should re-run the online database migrations contained in this release. This will prepare your cinder database for an eventual upgrade to the Ussuri release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-cinder/+/815387

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/815387
Committed: https://opendev.org/openstack/charm-cinder/commit/d33a2d056729d007b6bbd91896ac3798a8b4d658
Submitter: "Zuul (22348)"
Branch: master

commit d33a2d056729d007b6bbd91896ac3798a8b4d658
Author: Billy Olsen <email address hidden>
Date: Thu Oct 21 18:55:18 2021 -0700

    Check for online_data_migrations when syncing db

    Due to bug #1908037, the database sync can fail if the volume types
    have not been migrated to the __DEFAULT__ volume type. When this fails,
    a message will be sent telling the user to run online_data_migrations.
    Check for this error, and when it occurs, run the online_data_migrations
    and then retry the db sync.

    Closes-Bug: #1908037

    Change-Id: I0cd9a9db256b0adffbb244222a66c85d055840c4