[SRU] Missing sqlalchemy-utils dep on ussuri

[Liam Young]

[Impact]
Package seems to be missing a dependency on sqlalchemy-utils *1. The issue shows itself when running masakari-manage with the new 'taskflow' section enabled *2

*1 https://opendev.org/openstack/masakari/src/branch/stable/ussuri/requirements.txt#L29
*2 https://review.opendev.org/734450

I saw this with bionic ussuri but I assume affects focal too.

[Test Case]
apt install --install-suggests python3-masakari
ensure this installs python3-sqlalchemy-utils
ensure "su -s /bin/sh -c 'masakari-manage db sync' masakari" no longer fails

[Regression Potential]
This is just adding a new Suggested package so really can't think of any way this would regress any current functionality.

[Liam Young (gnuoy)]

It seems sqlalchemy-utils may have been removed recently in error https://git.launchpad.net/ubuntu/+source/masakari/tree/debian/changelog?id=4d933765965f3d02cd68c696cc69cf53b7c6390d#n3

[Corey Bryant (corey.bryant)]

Thanks for reporting this Liam. I think I was able to recreate the same issue (focal ussuri):

root@f1:~# grep -A 1 "\[taskflow\]" /etc/masakari/masakari.conf
[taskflow]
connection = sqlite:////var/lib/masakari/masakari.sqlite
root@f1:~# su -s /bin/sh -c 'masakari-manage db sync' masakari
2020-06-10 12:38:48.164 9027 INFO migrate.versioning.api [-] 5 -> 6...
2020-06-10 12:38:48.166 9027 INFO masakari.engine.driver [-] Loading masakari notification driver 'taskflow_driver'
2020-06-10 12:38:48.333 9027 INFO keyring.backend [-] Loading KWallet
2020-06-10 12:38:48.342 9027 INFO keyring.backend [-] Loading SecretService
2020-06-10 12:38:48.345 9027 INFO keyring.backend [-] Loading Windows
2020-06-10 12:38:48.346 9027 INFO keyring.backend [-] Loading chainer
2020-06-10 12:38:48.346 9027 INFO keyring.backend [-] Loading macOS
Invalid input received: No module named 'sqlalchemy_utils'
root@f1:~# apt policy masakari-common
masakari-common:
  Installed: 9.0.0-0ubuntu0.20.04.2
  Candidate: 9.0.0-0ubuntu0.20.04.2
  Version table:
 *** 9.0.0-0ubuntu0.20.04.2 500

[Corey Bryant (corey.bryant)]

And error goes away after install python3-sqlalchemy-utils:

root@f1:~# sudo apt install python3-sqlalchemy-utils
root@f1:~# su -s /bin/sh -c 'masakari-manage db sync' masakari
2020-06-10 12:41:48.709 9204 INFO migrate.versioning.api [-] 5 -> 6...
2020-06-10 12:41:48.712 9204 INFO masakari.engine.driver [-] Loading masakari notification driver 'taskflow_driver'
2020-06-10 12:41:49.022 9204 INFO keyring.backend [-] Loading KWallet
2020-06-10 12:41:49.032 9204 INFO keyring.backend [-] Loading SecretService
2020-06-10 12:41:49.036 9204 INFO keyring.backend [-] Loading Windows
2020-06-10 12:41:49.037 9204 INFO keyring.backend [-] Loading chainer
2020-06-10 12:41:49.038 9204 INFO keyring.backend [-] Loading macOS
2020-06-10 12:41:49.845 9204 INFO migrate.versioning.api [-] done

[Corey Bryant (corey.bryant)]

It looks like sqlalchemy-utils should be a run-time dependency of taskflow:

https://opendev.org/openstack/taskflow/src/branch/master/taskflow/persistence/backends/sqlalchemy/tables.py#L22

I'm not sure why but all of the sqlalchemy dependencies for taskflow are listed in test-requirements.txt. So it looks like it could use a fix upstream as well.

[Corey Bryant (corey.bryant)]

Ok nevermind, they are listed in setup.cfg [extras] in the upstream source. Dropping upstream from this bug.

[Corey Bryant (corey.bryant)]

Removing masakari, never know how to handle that. Bug affects masakari but is not in masakari. Feel free to add back if you want.

[Liam Young (gnuoy)]

Yep thats the traceback I'm seeing.

Charm shows:

2020-06-10 12:45:57 ERROR juju-log amqp:40: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-masakari-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-masakari-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-masakari-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-masakari-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-masakari-0/charm/reactive/masakari_handlers.py", line 50, in init_db
    charm_class.db_sync()
  File "/var/lib/juju/agents/unit-masakari-0/.venv/lib/python3.6/site-packages/charms_openstack/charm/core.py", line 849, in db_sync
    subprocess.check_call(self.sync_cmd)
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['masakari-manage', '--config-file', '/etc/masakari/masakari.conf', 'db', 'sync']' returned non-zero exit status 1.

2020-06-10 12:45:57 DEBUG amqp-relation-changed Traceback (most recent call last):
2020-06-10 12:45:57 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-masakari-0/charm/hooks/amqp-relation-changed", line 22, in <module>
2020-06-10 12:45:57 DEBUG amqp-relation-changed main()
2020-06-10 12:45:57 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-masakari-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 74, in main
2020-06-10 12:45:57 DEBUG amqp-relation-changed bus.dispatch(restricted=restricted_mode)
2020-06-10 12:45:57 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-masakari-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
2020-06-10 12:45:57 DEBUG amqp-relation-changed _invoke(other_handlers)
2020-06-10 12:45:57 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-masakari-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
2020-06-10 12:45:57 DEBUG amqp-relation-changed handler.invoke()
2020-06-10 12:45:57 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-masakari-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
2020-06-10 12:45:57 DEBUG amqp-relation-changed self._action(*args)
2020-06-10 12:45:57 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-masakari-0/charm/reactive/masakari_handlers.py", line 50, in init_db
2020-06-10 12:45:57 DEBUG amqp-relation-changed charm_class.db_sync()
2020-06-10 12:45:57 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-masakari-0/.venv/lib/python3.6/site-packages/charms_openstack/charm/core.py", line 849, in db_sync
2020-06-10 12:45:57 DEBUG amqp-relation-changed subprocess.check_call(self.sync_cmd)
2020-06-10 12:45:57 DEBUG amqp-relation-changed File "/usr/lib/python3.6/subp...

[Corey Bryant (corey.bryant)]

We may need to revive the MIR bug for sqlalchemy-utils: pad.lv/1543641

The problem we have now is sqlalchemy-utils is in universe still and tasklfow is in main.

[Corey Bryant (corey.bryant)]

https://pad.lv/1543641

[Corey Bryant (corey.bryant)]

We'll try to pursue the MIR for groovy and get this added as a dependency. For prior releases we can add it as a 'Suggests' dependency as they don't need to be in main.

[Corey Bryant (corey.bryant)]

This is fixed in python-taskflow 4.2.0-0ubuntu2 for groovy.

[Corey Bryant (corey.bryant)]

Uploaded to focal unapproved queue.

[Robie Basak (racb)]

I don't understand how adding a Suggests to the package in Focal will make any difference to the user impact. Please could you explain? What problem that the user is facing will adding a Suggests solve, and how?

[Chris MacNaughton (chris.macnaughton)]

I think that the core of the issue is that has a runtime depends on sqlalchemy-utils but sqlalchemy-utils isn't yet in main. A MIR has been re-opened for Groovy but sqlalchemy-utils (likely) won't get pushed into main for Focal.

[Brian Murray (brian-murray)]

Robie's question was how does adding python3-sqlachemy-utils to "Suggests" actually help users? They can still install python-taskflow (without --install-suggests) and not have python3-sqlalchemy-utils installed and still encounter the original crash.

[Brian Murray (brian-murray)]

An upload of python-taskflow to focal-proposed has been rejected from the upload queue for the following reason: "This has been rejected due to unanswered questions about the utility of providing this change as an update.".