Comment 2 for bug 1832860

Digging into this with John G, it looks like it's the nova_cell0 DB that does not have the trusted_certs field in the instance_extra table.

I was looking through the nova-manage code and found an error message that made me wonder if we had the ordering of DB syncs wrong in kolla. I switched the order of the 'nova-manage api_db sync' and 'nova-manage db sync' commands, and this did indeed fix the problem.

I then restored back my DB to queens and investigated. Running the api_db sync then db sync commands gives no error. However, it we run the db sync first (as we do in kolla), then we see this error:

()[nova@72f9e8e99a94 /]$ nova-manage db sync
ERROR: Could not access cell0.
Has the nova_api database been created?
Has the nova_cell0 database been created?
Has "nova-manage api_db sync" been run?
Has "nova-manage cell_v2 map_cell0" been run?
Is [api_database]/connection set in nova.conf?
Is the cell0 database connection URL correct?
Error: (pymysql.err.InternalError) (1054, u"Unknown column 'cell_mappings.disabled' in 'field list'") [SQL: u'SELECT cell_mappings.created_at AS cell_mappings_created_at, cell_mappings.updated_at AS cell_mappings_updated_at, cell_mappings.id AS cell_mappings_id, cell_mappings.uuid AS cell_mappings_uuid, cell_mappings.name AS cell_mappings_name, cell_mappings.transport_url AS cell_mappings_transport_url, cell_mappings.database_connection AS cell_mappings_database_connection, cell_mappings.disabled AS cell_mappings_disabled \nFROM cell_mappings \nWHERE cell_mappings.uuid = %(uuid_1)s \n LIMIT %(param_1)s'] [parameters: {u'uuid_1': '00000000-0000-0000-0000-000000000000', u'param_1': 1}] (Background on this error at: http://sqlalche.me/e/2j85)

However, the command exits zero, so we do not notice this error message.

()[nova@72f9e8e99a94 /]$ echo $?
0

As a general comment, these tools (e.g. nova-manage and nova-status) are typically not run by humans, so text output is not too useful unless accompanied by a non-zero exit code.

So I think we need two fixes.

Kolla: run api_db sync, db sync.
Nova: exit non-zero on the above error, make ordering requirement more clear in the upgrade docs.