containers upgrade CI -- DB syncs fail, could not find schema versions

[Jiří Stránský]

Nova and Neutron DB syncs fail in the upgrade CI, not being able to find schema version definitions. Given our current CI is doing master->master, it's likely that the RPM on the overcloud before upgrading is newer than the RPM in the container being upgraded to, which could confuse the DB migration code.

http://logs.openstack.org/84/450784/18/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/b3d61f9/logs/subnode-2/var/log/messages.txt.gz#_Apr_21_15_04_04

Nova trace:

Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: ERROR: could not access cell mapping database - has api db been created?
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: An error has occurred:
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: Traceback (most recent call last):
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 1606, in main
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: ret = fn(*fn_args, **fn_kwargs)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 645, in sync
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: return migration.db_sync(version)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/nova/db/migration.py", line 26, in db_sync
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: return IMPL.db_sync(version=version, database=database, context=context)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/migration.py", line 57, in db_sync
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: repository, version)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: return _migrate(url, repository, version, upgrade=True, err=err, **opts)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "<string>", line 2, in _migrate
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: return f(*a, **kw)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 345, in _migrate
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: changeset = schema.changeset(version)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/migrate/versioning/schema.py", line 82, in changeset
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: changeset = self.repository.changeset(database, start_ver, version)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/migrate/versioning/repository.py", line 225, in changeset
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: changes = [self.version(v).script(database, op) for v in versions]
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/migrate/versioning/repository.py", line 189, in version
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: return self.versions.version(*p, **k)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: File "/usr/lib/python2.7/site-packages/migrate/versioning/version.py", line 163, in version
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: return self.versions[VerNum(vernum)]
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 journal: KeyError: <VerNum(359)>

Neutron trace:

Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: Traceback (most recent call last):
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/bin/neutron-db-manage", line 10, in <module>
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: sys.exit(main())
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 687, in main
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: return_val |= bool(CONF.command.func(config, CONF.command.name))
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 206, in do_upgrade
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: run_sanity_checks(config, revision)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 671, in run_sanity_checks
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: script_dir.run_env()
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/alembic/script/base.py", line 407, in run_env
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: util.load_python_file(self.dir, 'env.py')
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: module = load_module_py(module_id, path)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/alembic/util/compat.py", line 79, in load_module_py
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: mod = imp.load_source(module_id, path, fp)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py", line 120, in <module>
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: run_migrations_online()
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py", line 114, in run_migrations_online
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: context.run_migrations()
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "<string>", line 8, in run_migrations
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/alembic/runtime/environment.py", line 797, in run_migrations
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: self.get_context().run_migrations(**kw)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/alembic/runtime/migration.py", line 303, in run_migrations
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: for step in self._migrations_fn(heads, self):
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 662, in check_sanity
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: revision, rev, implicit_base=True):
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/alembic/script/revision.py", line 641, in _iterate_revisions
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: requested_lowers = self.get_revisions(lower)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/alembic/script/revision.py", line 298, in get_revisions
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: return sum([self.get_revisions(id_elem) for id_elem in id_], ())
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/alembic/script/revision.py", line 303, in get_revisions
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: for rev_id in resolved_id)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/alembic/script/revision.py", line 303, in <genexpr>
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: for rev_id in resolved_id)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: File "/usr/lib/python2.7/site-packages/alembic/script/revision.py", line 358, in _revision_for_ident
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: resolved_id)
Apr 21 15:04:04 centos-7-2-node-osic-cloud1-s3500-8527278-539389 os-collect-config: alembic.script.revision.ResolutionError: No such revision or branch '804a3c76314c'

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jiri Stransky (<email address hidden>) on branch: master
Review: https://review.openstack.org/459770

[Jiří Stránský (jistr)]

Will be solved by implementing O->P upgrade instead of master->master. The top level patch is https://review.openstack.org/#/c/460061/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/462664
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=951d66abe4e0811c6dc23d0d7281be7279d283f8
Submitter: Jenkins
Branch: master

commit 951d66abe4e0811c6dc23d0d7281be7279d283f8
Author: Jiri Stransky <email address hidden>
Date: Thu May 4 18:26:18 2017 +0200

    multinode-container-upgrade.yaml usable for mixed upgrade

    We need Docker service mapping defined and set to OS::Heat::None so that
    we can reuse multinode-container-upgrade.yaml service list both for
    initial deployment and for the upgrade. The upgrade will not be broken
    by this as its env files are being passed later on the command line, and
    they'll take priority and effectively enable the Docker service on
    upgrade.

    Another change we need for mixed upgrade is to add the TripleoPackages
    service, which will take care of updating RPMs on the bare metal and
    prevent docker installation from failing with outdated
    puppet-tripleo ("Could not find class ::tripleo::profile::base::docker").

    Related-Bug: #1685795
    Closes-Bug: #1689772
    Change-Id: Idb6917f22d0e9f74f8853972c6a08bffb01be410

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/459789
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=5c4aaf3d6f1b92077a84ee335c87e7ff612a4b69
Submitter: Jenkins
Branch: master

commit 5c4aaf3d6f1b92077a84ee335c87e7ff612a4b69
Author: Jiri Stransky <email address hidden>
Date: Tue Apr 25 19:04:16 2017 +0200

    TripleO CI mixed release master UC / ocata OC

    For testing upstream upgrades O -> master(P).

    Related-Bug: #1685795
    Co-Authored-By: Mathieu Bultel <email address hidden>
    Change-Id: I2c30c560f7e0b5ffd9fff3b567b3c80f65698e89