Newton->Ocata upgrade fails during neutron-db-manage upgrade head (caused by python-networking-bigswitch)

[Marius Cornea]

[root@overcloud-controller-0 ~]# rpm -qa | grep bigswitch
openstack-neutron-bigswitch-agent-10.0.0-0.20170126212240.6962731.el7.centos.noarch
openstack-neutron-bigswitch-lldp-10.0.0-0.20170126212240.6962731.el7.centos.noarch
python-networking-bigswitch-10.0.0-0.20170126212240.6962731.el7.centos.noarch

[root@overcloud-controller-0 ~]# neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
  Running upgrade for neutron ...
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
  OK
Traceback (most recent call last):
  File "/bin/neutron-db-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 687, in main
    return_val |= bool(CONF.command.func(config, CONF.command.name))
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 206, in do_upgrade
    run_sanity_checks(config, revision)
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 671, in run_sanity_checks
    script_dir.run_env()
  File "/usr/lib/python2.7/site-packages/alembic/script/base.py", line 407, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/usr/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/lib/python2.7/site-packages/alembic/util/compat.py", line 79, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "/usr/lib/python2.7/site-packages/bsnstacklib/plugins/bigswitch/db/migration/alembic_migrations/env.py", line 59, in <module>
    @db_api.sqla_listen(sa.Table, 'after_parent_attach')
  File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 247, in sqla_listen
    event.listen(*args)
TypeError: listen() takes at least 3 arguments (2 given)

[Brent Eagles (beagles)]

There was a similar issue here: https://bugs.launchpad.net/tripleo/+bug/1649284
We worked around it by removing the package in the u/s images - of course this doesn't help those that have images that do include it or are upgrading deployments with it already installed. While this is evident as part of the tripleo testing/work - this seems to be more of an issue with that particular package and should be targeted accordingly.

I suspect that this has been already resolved in bsnstacklib as there have patches "around this area" recently.

[Brent Eagles (beagles)]

My mistake, this was quite possibly *broken* recently - not *fixed* recently. Discussing with some of the devs on #rdo.

[Brent Eagles (beagles)]

Possible fix posted and merged: https://review.openstack.org/#/c/431699/. Marius, any chance you could pull this in somehow and test it?

[Marius Cornea (mcornea)]

Yes, sure. I'll do it today and get back with my results. Thanks!

[Marius Cornea (mcornea)]

After applying https://review.openstack.org/#/c/431699/ it fails with:

INFO [alembic.runtime.migration] Running upgrade 1ef57200f387 -> 34f888285695, support multiple logical routers
Traceback (most recent call last):
  File "/bin/neutron-db-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 687, in main
    return_val |= bool(CONF.command.func(config, CONF.command.name))
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 208, in do_upgrade
    desc=branch, sql=CONF.command.sql)
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 109, in do_alembic_command
    getattr(alembic_command, cmd)(config, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/alembic/command.py", line 174, in upgrade
    script.run_env()
  File "/usr/lib/python2.7/site-packages/alembic/script/base.py", line 407, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/usr/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/lib/python2.7/site-packages/alembic/util/compat.py", line 79, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "/usr/lib/python2.7/site-packages/bsnstacklib/plugins/bigswitch/db/migration/alembic_migrations/env.py", line 86, in <module>
    run_migrations_online()
  File "/usr/lib/python2.7/site-packages/bsnstacklib/plugins/bigswitch/db/migration/alembic_migrations/env.py", line 77, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/lib/python2.7/site-packages/alembic/runtime/environment.py", line 797, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/lib/python2.7/site-packages/alembic/runtime/migration.py", line 312, in run_migrations
    step.migration_fn(**kw)
  File "/usr/lib/python2.7/site-packages/bsnstacklib/plugins/bigswitch/db/migration/alembic_migrations/versions/34f888285695_support_multiple_logical_routers.py", line 40, in upgrade
    connection.execute("UPDATE bsn_routerrules bsn "
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 906, in execute
    return self._execute_text(object, multiparams, params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1054, in _execute_text
    statement, parameters
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
    util.raise_from_cause(newraise, exc_info)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
    ...

[Emilien Macchi (emilienm)]

Marius, I think the bug can be closes on TripleO side, it has been fixed by https://review.rdoproject.org/r/#/c/5174/.

Can you please confirm and close this bug please.

[Assaf Muller (amuller)]

Emilien, the patch you linked to removes the ACI plugin, this bug is about BigSwitch. I'm not following why you're saying this bug can be closed? It seems as if the next step here would be to send a similar patch that removes BigSwitch from the base image.

[Brent Eagles (beagles)]

Actually this particular issue should be fixed in the bigswitch package. There are other issues involving database upgrades that may present themselves but as this specific bug was fixed I'm closing to avoid further confusion.

[Brent Eagles (beagles)]

Bigswitch patch for fix https://review.openstack.org/#/c/431699/