Seeing this in a grenade run upgrading to Rocky:
http://logs.openstack.org/38/544238/4/check/neutron-grenade/d101bba/logs/grenade.sh.txt.gz#_2018-04-05_09_41_21_078
2018-04-05 09:41:21.078 | + /opt/stack/new/grenade/projects/60_nova/upgrade.sh:main:74 : /usr/local/bin/nova-manage --config-file /etc/nova/nova.conf db sync
2018-04-05 09:41:21.528 | /usr/local/lib/python2.7/dist-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
2018-04-05 09:41:21.528 | """)
2018-04-05 09:41:25.030 | [01;31mERROR oslo_db.sqlalchemy.exc_filters [[01;36mNone req-955c4479-2900-4425-bb3e-2161ed85d6ed [00;36mNone None[01;31m] [01;35m[01;31mDBAPIError exception wrapped from (pymysql.err.InternalError) (1054, u"Unknown column 'cell_mappings.disabled' in 'field list'") [SQL: u'SELECT cell_mappings.created_at AS cell_mappings_created_at, cell_mappings.updated_at AS cell_mappings_updated_at, cell_mappings.id AS cell_mappings_id, cell_mappings.uuid AS cell_mappings_uuid, cell_mappings.name AS cell_mappings_name, cell_mappings.transport_url AS cell_mappings_transport_url, cell_mappings.database_connection AS cell_mappings_database_connection, cell_mappings.disabled AS cell_mappings_disabled \nFROM cell_mappings \nWHERE cell_mappings.uuid = %(uuid_1)s \n LIMIT %(param_1)s'] [parameters: {u'uuid_1': '00000000-0000-0000-0000-000000000000', u'param_1': 1}] (Background on this error at: http://sqlalche.me/e/2j85)[00m: InternalError: (1054, u"Unknown column 'cell_mappings.disabled' in 'field list'")
2018-04-05 09:41:25.030 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00mTraceback (most recent call last):
2018-04-05 09:41:25.030 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
2018-04-05 09:41:25.030 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m context)
2018-04-05 09:41:25.030 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 507, in do_execute
2018-04-05 09:41:25.030 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m cursor.execute(statement, parameters)
2018-04-05 09:41:25.030 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 165, in execute
2018-04-05 09:41:25.030 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m result = self._query(query)
2018-04-05 09:41:25.030 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 321, in _query
2018-04-05 09:41:25.030 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m conn.query(q)
2018-04-05 09:41:25.030 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 860, in query
2018-04-05 09:41:25.030 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2018-04-05 09:41:25.030 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1061, in _read_query_result
2018-04-05 09:41:25.031 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m result.read()
2018-04-05 09:41:25.031 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1349, in read
2018-04-05 09:41:25.031 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m first_packet = self.connection._read_packet()
2018-04-05 09:41:25.031 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1018, in _read_packet
2018-04-05 09:41:25.031 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m packet.check_error()
2018-04-05 09:41:25.031 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 384, in check_error
2018-04-05 09:41:25.031 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m err.raise_mysql_exception(self._data)
2018-04-05 09:41:25.031 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
2018-04-05 09:41:25.031 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m raise errorclass(errno, errval)
2018-04-05 09:41:25.031 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00mInternalError: (1054, u"Unknown column 'cell_mappings.disabled' in 'field list'")
2018-04-05 09:41:25.031 | ERROR oslo_db.sqlalchemy.exc_filters [01;35m[00m
2018-04-05 09:41:25.032 | ERROR: Could not access cell0.
2018-04-05 09:41:25.032 | Has the nova_api database been created?
2018-04-05 09:41:25.032 | Has the nova_cell0 database been created?
2018-04-05 09:41:25.032 | Has "nova-manage api_db sync" been run?
2018-04-05 09:41:25.032 | Has "nova-manage cell_v2 map_cell0" been run?
2018-04-05 09:41:25.032 | Is [api_database]/connection set in nova.conf?
2018-04-05 09:41:25.032 | Is the cell0 database connection URL correct?
That's from this call:
https://github.com/openstack-dev/grenade/blob/b7dbc632b6a0fe2b625096b3a53576e746150b0a/projects/60_nova/upgrade.sh#L74
That doesn't fail because the nova-manage db sync command catches the exception, logs the error but doesn't return a non-zero exit code:
https://github.com/openstack/nova/blob/887dc23a1775a8e64eee62ac3f94a04cba0052c8/nova/cmd/manage.py#L468
This is failing in Rocky because the cell_mappings.disabled column is new, but the nova_api db sync hasn't happened yet since grenade syncs the nova db before the nova_api db, which follows the nova upgrade docs:
https://docs.openstack.org/nova/latest/user/upgrade.html#rolling-upgrade-process
So we could fix this in one of two ways:
1. Swap the order in the grenade upgrade.sh script to sync the nova_api db first (and update the nova docs to flip that order also).
2. Add a from-queens script to grenade for upgrading nova to sync the API DB first.
Option #1 seems a bit shady, but option #2 seems excessive, especially if we're going to be changing things in the API DB in future releases.
Alternatively, we could pass the --local_cell flag to the nova-manage db sync command in grenade to only sync the cell (nova) database, not cell0. But that's another behavior change.