DB migration broken

[Slawek Kaplonski]

Probably since we merged https://review.opendev.org/#/c/741676/ in Neutron db migration is broken for networking-bgpvpn project. Errors are like in https://zuul.opendev.org/t/openstack/build/141ddec140174c32a575ac70e38ee97a

Running upgrade for neutron ...
OK
Traceback (most recent call last):
  File "/usr/local/bin/neutron-db-manage", line 10, in <module>
    sys.exit(main())
  File "/opt/stack/neutron/neutron/db/migration/cli.py", line 635, in main
    return_val |= bool(CONF.command.func(config, CONF.command.name))
  File "/opt/stack/neutron/neutron/db/migration/cli.py", line 179, in do_upgrade
    run_sanity_checks(config, revision)
  File "/opt/stack/neutron/neutron/db/migration/cli.py", line 619, in run_sanity_checks
    script_dir.run_env()
  File "/usr/local/lib/python3.6/dist-packages/alembic/script/base.py", line 489, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.6/dist-packages/alembic/util/pyfiles.py", line 98, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.6/dist-packages/alembic/util/compat.py", line 184, in load_module_py
    spec.loader.exec_module(module)

  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/stack/networking-bagpipe/networking_bagpipe/db/migration/alembic_migrations/env.py", line 86, in <module>
    run_migrations_online()
  File "/opt/stack/networking-bagpipe/networking_bagpipe/db/migration/alembic_migrations/env.py", line 77, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python3.6/dist-packages/alembic/runtime/environment.py", line 846, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.6/dist-packages/alembic/runtime/migration.py", line 509, in run_migrations
    for step in self._migrations_fn(heads, self):
  File "/opt/stack/neutron/neutron/db/migration/cli.py", line 610, in check_sanity
    revision, rev, implicit_base=True):
  File "/usr/local/lib/python3.6/dist-packages/alembic/script/revision.py", line 777, in _iterate_revisions
    uppers = util.dedupe_tuple(self.get_revisions(upper))
  File "/usr/local/lib/python3.6/dist-packages/alembic/script/revision.py", line 321, in get_revisions
    resolved_id, branch_label = self._resolve_revision_number(id_)
  File "/usr/local/lib/python3.6/dist-packages/alembic/script/revision.py", line 501, in _resolve_revision_number
    self._revision_map
  File "/usr/local/lib/python3.6/dist-packages/alembic/util/langhelpers.py", line 230, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/usr/local/lib/python3.6/dist-packages/alembic/script/revision.py", line 156, in _revision_map
    down_revision = map_[downrev]
KeyError: 'd4d4d7f03b21'

[Radosław Piliszek (yoctozepto)]

It broke kolla-ansible gates as well. :/

[Radosław Piliszek (yoctozepto)]

Thought we get this on VPNaaS:

"+++ neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --subproject neutron-vpnaas upgrade head",
    "INFO [alembic.runtime.migration] Context impl MySQLImpl.",
    "INFO [alembic.runtime.migration] Will assume non-transactional DDL.",
    "/var/lib/kolla/venv/lib/python3.8/site-packages/alembic/script/revision.py:152: UserWarning: Revision 24f28869838b referenced from 24f28869838b -> 41b509d10b5e (head), VPNaaS endpoint groups is not present",
    " util.warn(",
    "Traceback (most recent call last):",
    " File \"/var/lib/kolla/venv/bin/neutron-db-manage\", line 8, in <module>",
    " sys.exit(main())",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/db/migration/cli.py\", line 635, in main",
    " return_val |= bool(CONF.command.func(config, CONF.command.name))",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/db/migration/cli.py\", line 179, in do_upgrade",
    " run_sanity_checks(config, revision)",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/db/migration/cli.py\", line 619, in run_sanity_checks",
    " script_dir.run_env()",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/alembic/script/base.py\", line 489, in run_env",
    " util.load_python_file(self.dir, \"env.py\")",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/alembic/util/pyfiles.py\", line 98, in load_python_file",
    " module = load_module_py(module_id, path)",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/alembic/util/compat.py\", line 184, in load_module_py",
    " spec.loader.exec_module(module)",
    " File \"<frozen importlib._bootstrap_external>\", line 783, in exec_module",
    " File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_vpnaas/db/migration/alembic_migrations/env.py\", line 88, in <module>",
    " run_migrations_online()",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_vpnaas/db/migration/alembic_migrations/env.py\", line 79, in run_migrations_online",
    " context.run_migrations()",
    " File \"<string>\", line 8, in run_migrations",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/alembic/runtime/environment.py\", line 846, in run_migrations",
    " self.get_context().run_migrations(**kw)",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/alembic/runtime/migration.py\", line 509, in run_migrations",
    " for step in self._migrations_fn(heads, self):",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/db/migration/cli.py\", line 609, in check_sanity",
    " for script in script_dir.revision_map.iterate_revisions(",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/alembic/script/revision.py\", line 777, in _iterate_revisions",
    " uppers = util.dedupe_tuple(self.get_revisions(upper))",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/alembic/script/revision.py\", line 321, in get_revisions",
    " resolved_id,...

[Lajos Katona (lajos-katona)]

By the exception in https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_141/749060/3/check/neutron-tempest-plugin-bgpvpn-bagpipe/141ddec/job-output.txt , it seems that bgpvpn and bagpipe liberty alembic branch must be squashed to Pike, but not sure about if that is really the only solution.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/749696

[Lajos Katona (lajos-katona)]

For bagpipe the patch from Rodolfo: https://review.opendev.org/749687

[Radosław Piliszek (yoctozepto)]

How's it going?

[Slawek Kaplonski (slaweq)]

@Radosław: It should be fixed with revert https://review.opendev.org/#/c/749866/

[Slawek Kaplonski (slaweq)]

Actually https://review.opendev.org/#/c/749866/ just avoids failures but I will keep this bug open as to fix it properly we will need to squash db migrations in that stadium project also and we can use this bug to track that.

[Radosław Piliszek (yoctozepto)]

Ok, thanks @Sławek for confirming. It means we (kolla) can restore these optional migrations.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rodolfo Alonso Hernandez (<email address hidden>) on branch: master
Review: https://review.opendev.org/749696
Reason: We have too many external dependencies. [1] is not going to happen.

[1]https://review.opendev.org/#/c/741676/