jammy-yoga neutron-db-manage results in AttributeError: module 'collections' has no attribute 'MutableMapping'

[Corey Bryant]

neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head

results in traceback: https://paste.ubuntu.com/p/SZgSHmmnym/plain/

Pasting traceback here too:

Traceback (most recent call last):
  File "/usr/bin/neutron-db-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python3/dist-packages/neutron/db/migration/cli.py", line 661, in main
    return_val |= bool(CONF.command.func(config, CONF.command.name))
  File "/usr/lib/python3/dist-packages/neutron/db/migration/cli.py", line 183, in do_upgrade
    run_sanity_checks(config, revision)
  File "/usr/lib/python3/dist-packages/neutron/db/migration/cli.py", line 645, in run_sanity_checks
    script_dir.run_env()
  File "/usr/lib/python3/dist-packages/alembic/script/base.py", line 563, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/lib/python3/dist-packages/alembic/util/pyfiles.py", line 92, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/lib/python3/dist-packages/alembic/util/pyfiles.py", line 108, in load_module_py
    spec.loader.exec_module(module) # type: ignore
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3/dist-packages/neutron/db/migration/alembic_migrations/env.py", line 16, in <module>
    from neutron_lib.db import model_base
  File "/usr/lib/python3/dist-packages/neutron_lib/__init__.py", line 17, in <module>
    from neutron_lib.db import api # noqa
  File "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 18, in <module>
    from oslo_concurrency import lockutils
  File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 28, in <module>
    import fasteners
  File "/usr/lib/python3/dist-packages/fasteners/__init__.py", line 23, in <module>
    from fasteners.lock import locked # noqa
  File "/usr/lib/python3/dist-packages/fasteners/lock.py", line 31, in <module>
    import eventlet
  File "/usr/lib/python3/dist-packages/eventlet/__init__.py", line 17, in <module>
    from eventlet import convenience
  File "/usr/lib/python3/dist-packages/eventlet/convenience.py", line 7, in <module>
    from eventlet.green import socket
  File "/usr/lib/python3/dist-packages/eventlet/green/socket.py", line 21, in <module>
    from eventlet.support import greendns
  File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 66, in <module>
    setattr(dns, pkg, import_patched('dns.' + pkg))
  File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 61, in import_patched
    return patcher.import_patched(module_name, **modules)
  File "/usr/lib/python3/dist-packages/eventlet/patcher.py", line 132, in import_patched
    return inject(
  File "/usr/lib/python3/dist-packages/eventlet/patcher.py", line 109, in inject
    module = __import__(module_name, {}, {}, module_name.split('.')[:-1])
  File "/usr/local/lib/python3.10/dist-packages/dns/namedict.py", line 35, in <module>
    class NameDict(collections.MutableMapping):
AttributeError: module 'collections' has no attribute 'MutableMapping'

The issue looks to be in the dnspython package. It needs to be using collections.abc.MutableMapping

[Corey Bryant (corey.bryant)]

I don't think this is a package issue as the code in the jammy+ dnspython packages includes the collections.abc fix. Also note the subtle difference in the traceback:

usr/local/lib/python3.10/dist-packages/dns/namedict.py

There is a pip installed package on the unit as well as an apt installed package and the pip-installed version is getting picked up which causes the issue:

ubuntu@juju-c42f04-zaza-e3abc6e946c3-6:/var/lib/juju/agents/unit-neutron-api-0/charm$ pip freeze|grep -i dns
dnspython==1.16.0

ubuntu@juju-c42f04-zaza-e3abc6e946c3-6:/var/lib/juju/agents/unit-neutron-api-0/charm$ apt policy python3-dnspython
python3-dnspython:
  Installed: 2.1.0-1ubuntu1
  Candidate: 2.1.0-1ubuntu1
  Version table:
 *** 2.1.0-1ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu jammy/main amd64 Packages

Running the neutron-db-manage command is successful after 'pip uninstall dnspython'.

[Corey Bryant (corey.bryant)]

The conficting version of dnspython is located in:

ubuntu@juju-c42f04-zaza-e3abc6e946c3-6:/var/lib/juju/agents/unit-neutron-api-plugin-arista-0/charm/wheelhouse$ ls dnspython-1.16.0.zip
dnspython-1.16.0.zip

which comes from:

https://github.com/openstack/charm-layer-openstack/blob/master/wheelhouse.txt

While fixing the pin in the wheelhouse.txt would fix this, we'd still end up with use of a mix of apt/pip installed packages.