Nova and Neutron DB syncs fail in the upgrade CI, not being able to find schema version definitions. Given our current CI is doing master->master, it's likely that the RPM on the overcloud before upgrading is newer than the RPM in the container being upgraded to, which could confuse the DB migration code.
http://logs.openstack.org/84/450784/18/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/b3d61f9/logs/subnode-2/var/log/messages.txt.gz#_Apr_21_15_04_04
Nova trace:
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: ERROR: could not access cell mapping database - has api db been created?
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: An error has occurred:
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: Traceback (most recent call last):
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 1606, in main
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: ret = fn(*fn_args, **fn_kwargs)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 645, in sync
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: return migration.db_sync(version)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/nova/db/migration.py", line 26, in db_sync
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: return IMPL.db_sync(version=version, database=database, context=context)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/migration.py", line 57, in db_sync
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: repository, version)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: return _migrate(url, repository, version, upgrade=True, err=err, **opts)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "<string>", line 2, in _migrate
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: return f(*a, **kw)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 345, in _migrate
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: changeset = schema.changeset(version)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/migrate/versioning/schema.py", line 82, in changeset
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: changeset = self.repository.changeset(database, start_ver, version)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/migrate/versioning/repository.py", line 225, in changeset
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: changes = [self.version(v).script(database, op) for v in versions]
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/migrate/versioning/repository.py", line 189, in version
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: return self.versions.version(*p, **k)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/migrate/versioning/version.py", line 163, in version
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: return self.versions[VerNum(vernum)]
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: KeyError: <VerNum(359)>
Neutron trace:
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: Traceback (most recent call last):
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/bin/neutron-db-manage", line 10, in <module>
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: sys.exit(main())
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 687, in main
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: return_val |= bool(CONF.command.func(config, CONF.command.name))
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 206, in do_upgrade
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: run_sanity_checks(config, revision)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 671, in run_sanity_checks
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: script_dir.run_env()
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/alembic/script/base.py", line 407, in run_env
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: util.load_python_file(self.dir, 'env.py')
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: module = load_module_py(module_id, path)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/alembic/util/compat.py", line 79, in load_module_py
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: mod = imp.load_source(module_id, path, fp)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py", line 120, in <module>
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: run_migrations_online()
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py", line 114, in run_migrations_online
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: context.run_migrations()
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "<string>", line 8, in run_migrations
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/alembic/runtime/environment.py", line 797, in run_migrations
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: self.get_context().run_migrations(**kw)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/alembic/runtime/migration.py", line 303, in run_migrations
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: for step in self._migrations_fn(heads, self):
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 662, in check_sanity
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: revision, rev, implicit_base=True):
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/alembic/script/revision.py", line 641, in _iterate_revisions
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: requested_lowers = self.get_revisions(lower)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/alembic/script/revision.py", line 298, in get_revisions
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: return sum([self.get_revisions(id_elem) for id_elem in id_], ())
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/alembic/script/revision.py", line 303, in get_revisions
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: for rev_id in resolved_id)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/alembic/script/revision.py", line 303, in <genexpr>
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: for rev_id in resolved_id)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/alembic/script/revision.py", line 358, in _revision_for_ident
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: resolved_id)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: alembic.script.revision.ResolutionError: No such revision or branch '804a3c76314c'
Change abandoned by Jiri Stransky (<email address hidden>) on branch: master /review. openstack. org/459770
Review: https:/