ERROR neutron.manager [-] Plugin 'calico' not found.

Bug #1856485 reported by Vasili
36
This bug affects 8 people
Affects Status Importance Assigned to Milestone
OpenStack Community Project
New
Undecided
Unassigned
Python Package Index
New
Undecided
Unassigned
Ubuntu Server
New
Undecided
Unassigned
networking-calico
Fix Released
Undecided
Unassigned
neutron
Invalid
Undecided
Unassigned
Ubuntu
Confirmed
Undecided
Unassigned

Bug Description

Hi, I tried to install Openstack Train with neutron calico driver and getting the following exception:
ERROR neutron.manager [-] Plugin ‘calico’ not found.
Could someone help me understand if calico driver is ready for Openstack Train or Stein which is running with Python3.
If it's still not ready and not working after few years, so what is the developing and QA cycles in calico for Openstack? And what is the solution? Or ETA for solution? Maybe someone can advise for better solution than going with calico to Production and latest Openstack release?
Thanks..

see neuton-server log

2019-12-15 22:47:00.418 8031 INFO neutron.common.config [-] Logging enabled!
2019-12-15 22:47:00.418 8031 INFO neutron.common.config [-] /usr/bin/neutron-server version 15.0.0
2019-12-15 22:47:00.419 8031 INFO neutron.common.config [-] Logging enabled!
2019-12-15 22:47:00.419 8031 INFO neutron.common.config [-] /usr/bin/neutron-server version 15.0.0
2019-12-15 22:47:00.793 8031 INFO keyring.backend [-] Loading Gnome
2019-12-15 22:47:00.811 8031 INFO keyring.backend [-] Loading Google
2019-12-15 22:47:00.814 8031 INFO keyring.backend [-] Loading Windows (alt)
2019-12-15 22:47:00.817 8031 INFO keyring.backend [-] Loading file
2019-12-15 22:47:00.819 8031 INFO keyring.backend [-] Loading keyczar
2019-12-15 22:47:00.819 8031 INFO keyring.backend [-] Loading multi
2019-12-15 22:47:00.821 8031 INFO keyring.backend [-] Loading pyfs
2019-12-15 22:47:00.855 8031 INFO neutron.manager [-] Loading core plugin: calico
2019-12-15 22:47:00.856 8031 ERROR neutron_lib.utils.runtime [-] Error loading class by alias: stevedore.exception.NoMatches: No 'neutron.core_plugins' driver found, looking for 'calico'
2019-12-15 22:47:00.856 8031 ERROR neutron_lib.utils.runtime Traceback (most recent call last):
2019-12-15 22:47:00.856 8031 ERROR neutron_lib.utils.runtime File "/usr/lib/python3/dist-packages/neutron_lib/utils/runtime.py", line 114, in load_class_by_alias_or_classname
2019-12-15 22:47:00.856 8031 ERROR neutron_lib.utils.runtime namespace, name, warn_on_missing_entrypoint=False)
2019-12-15 22:47:00.856 8031 ERROR neutron_lib.utils.runtime File "/usr/lib/python3/dist-packages/stevedore/driver.py", line 61, in __init__
2019-12-15 22:47:00.856 8031 ERROR neutron_lib.utils.runtime warn_on_missing_entrypoint=warn_on_missing_entrypoint
2019-12-15 22:47:00.856 8031 ERROR neutron_lib.utils.runtime File "/usr/lib/python3/dist-packages/stevedore/named.py", line 89, in __init__
2019-12-15 22:47:00.856 8031 ERROR neutron_lib.utils.runtime self._init_plugins(extensions)
2019-12-15 22:47:00.856 8031 ERROR neutron_lib.utils.runtime File "/usr/lib/python3/dist-packages/stevedore/driver.py", line 113, in _init_plugins
2019-12-15 22:47:00.856 8031 ERROR neutron_lib.utils.runtime (self.namespace, name))
2019-12-15 22:47:00.856 8031 ERROR neutron_lib.utils.runtime stevedore.exception.NoMatches: No 'neutron.core_plugins' driver found, looking for 'calico'
2019-12-15 22:47:00.856 8031 ERROR neutron_lib.utils.runtime
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime [-] Error loading class by class name: ValueError: Empty module name
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime Traceback (most recent call last):
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime File "/usr/lib/python3/dist-packages/neutron_lib/utils/runtime.py", line 114, in load_class_by_alias_or_classname
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime namespace, name, warn_on_missing_entrypoint=False)
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime File "/usr/lib/python3/dist-packages/stevedore/driver.py", line 61, in __init__
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime warn_on_missing_entrypoint=warn_on_missing_entrypoint
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime File "/usr/lib/python3/dist-packages/stevedore/named.py", line 89, in __init__
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime self._init_plugins(extensions)
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime File "/usr/lib/python3/dist-packages/stevedore/driver.py", line 113, in _init_plugins
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime (self.namespace, name))
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime stevedore.exception.NoMatches: No 'neutron.core_plugins' driver found, looking for 'calico'
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime During handling of the above exception, another exception occurred:
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime Traceback (most recent call last):
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime File "/usr/lib/python3/dist-packages/neutron_lib/utils/runtime.py", line 120, in load_class_by_alias_or_classname
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime class_to_load = importutils.import_class(name)
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime File "/usr/lib/python3/dist-packages/oslo_utils/importutils.py", line 30, in import_class
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime __import__(mod_str)
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime ValueError: Empty module name
2019-12-15 22:47:00.857 8031 ERROR neutron_lib.utils.runtime
2019-12-15 22:47:00.858 8031 ERROR neutron.manager [-] Plugin 'calico' not found.
2019-12-15 22:47:00.858 8031 ERROR neutron.service [-] Unrecoverable error: please check log for details.: ImportError: Class not found.
2019-12-15 22:47:00.858 8031 ERROR neutron.service Traceback (most recent call last):
2019-12-15 22:47:00.858 8031 ERROR neutron.service File "/usr/lib/python3/dist-packages/neutron_lib/utils/runtime.py", line 114, in load_class_by_alias_or_classname
2019-12-15 22:47:00.858 8031 ERROR neutron.service namespace, name, warn_on_missing_entrypoint=False)
2019-12-15 22:47:00.858 8031 ERROR neutron.service File "/usr/lib/python3/dist-packages/stevedore/driver.py", line 61, in __init__
2019-12-15 22:47:00.858 8031 ERROR neutron.service warn_on_missing_entrypoint=warn_on_missing_entrypoint
2019-12-15 22:47:00.858 8031 ERROR neutron.service File "/usr/lib/python3/dist-packages/stevedore/named.py", line 89, in __init__
2019-12-15 22:47:00.858 8031 ERROR neutron.service self._init_plugins(extensions)
2019-12-15 22:47:00.858 8031 ERROR neutron.service File "/usr/lib/python3/dist-packages/stevedore/driver.py", line 113, in _init_plugins
2019-12-15 22:47:00.858 8031 ERROR neutron.service (self.namespace, name))
2019-12-15 22:47:00.858 8031 ERROR neutron.service stevedore.exception.NoMatches: No 'neutron.core_plugins' driver found, looking for 'calico'
2019-12-15 22:47:00.858 8031 ERROR neutron.service
2019-12-15 22:47:00.858 8031 ERROR neutron.service During handling of the above exception, another exception occurred:

Vasili (vasili.namatov)
tags: added: openstack
Revision history for this message
Vasili (vasili.namatov) wrote :

ok, I installed now the networking-calico with pip instead with apt and it looks better but neutron-server service failing every 2 seconds...

CRITICAL neutron.plugins.ml2.managers [-] The 'calico = networking_calico.plugins.ml2.drivers.calico.mech_calico:CalicoMechanismDriver' entrypoint could not be loaded for the following reason: 'cannot import name 'StringTypes''.

2019-12-15 23:29:18.100 41228 INFO neutron.common.config [-] Logging enabled!
2019-12-15 23:29:18.100 41228 INFO neutron.common.config [-] /usr/bin/neutron-server version 15.0.0
2019-12-15 23:29:18.101 41228 INFO neutron.common.config [-] Logging enabled!
2019-12-15 23:29:18.101 41228 INFO neutron.common.config [-] /usr/bin/neutron-server version 15.0.0
2019-12-15 23:29:18.474 41228 INFO keyring.backend [-] Loading Gnome
2019-12-15 23:29:18.488 41228 INFO keyring.backend [-] Loading Google
2019-12-15 23:29:18.490 41228 INFO keyring.backend [-] Loading Windows (alt)
2019-12-15 23:29:18.492 41228 INFO keyring.backend [-] Loading file
2019-12-15 23:29:18.493 41228 INFO keyring.backend [-] Loading keyczar
2019-12-15 23:29:18.493 41228 INFO keyring.backend [-] Loading multi
2019-12-15 23:29:18.494 41228 INFO keyring.backend [-] Loading pyfs
2019-12-15 23:29:18.522 41228 INFO neutron.manager [-] Loading core plugin: calico
2019-12-15 23:29:19.189 41228 INFO networking_calico.plugins.calico.plugin [-] Forcing ML2 mechanism_drivers to 'calico'
2019-12-15 23:29:19.190 41228 INFO networking_calico.plugins.calico.plugin [-] Forcing ML2 type_drivers to 'local, flat'
2019-12-15 23:29:19.190 41228 INFO networking_calico.plugins.calico.plugin [-] Forcing ML2 tenant_network_types to 'local'
2019-12-15 23:29:19.190 41228 INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['local', 'flat']
2019-12-15 23:29:19.196 41228 INFO neutron.plugins.ml2.drivers.type_flat [-] Arbitrary flat physical_network names allowed
2019-12-15 23:29:19.196 41228 INFO neutron.plugins.ml2.drivers.type_local [-] ML2 LocalTypeDriver initialization complete
2019-12-15 23:29:19.196 41228 INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'local']
2019-12-15 23:29:19.197 41228 INFO neutron.plugins.ml2.managers [-] Registered types: dict_keys(['flat', 'local'])
2019-12-15 23:29:19.198 41228 INFO neutron.plugins.ml2.managers [-] Tenant network_types: ['local']
2019-12-15 23:29:19.198 41228 INFO neutron.plugins.ml2.managers [-] Configured extension driver names: []
2019-12-15 23:29:19.198 41228 INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: []
2019-12-15 23:29:19.199 41228 INFO neutron.plugins.ml2.managers [-] Registered extension drivers: []
2019-12-15 23:29:19.199 41228 INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['calico']
2019-12-15 23:29:19.251 41228 CRITICAL neutron.plugins.ml2.managers [-] The 'calico = networking_calico.plugins.ml2.drivers.calico.mech_calico:CalicoMechanismDriver' entrypoint could not be loaded for the following reason: 'cannot import name 'StringTypes''.

Revision history for this message
Vasili (vasili.namatov) wrote :

could someone assist to understand the issue and maybe provide some workaround?

'cannot import name 'StringTypes'

Revision history for this message
Vasili (vasili.namatov) wrote :

in /usr/local/lib/python3.6/dist-packages/networking_calico/etcdutils.py

Revision history for this message
Vasili (vasili.namatov) wrote :

I fixed this error in code an d few errors afterwards..

could you assist how to fix this?

Dec 16 14:38:50 ltpr-ctr01 neutron-server[49614]: File "/usr/lib/python3/dist-packages/eventlet/hubs/poll.py", line 111, in wait
Dec 16 14:38:50 ltpr-ctr01 neutron-server[49614]: listener.cb(fileno)
Dec 16 14:38:50 ltpr-ctr01 neutron-server[49614]: File "/usr/lib/python3/dist-packages/eventlet/greenthread.py", line 221, in main
Dec 16 14:38:50 ltpr-ctr01 neutron-server[49614]: result = function(*args, **kwargs)
Dec 16 14:38:50 ltpr-ctr01 neutron-server[49614]: File "/usr/local/lib/python3.6/dist-packages/networking_calico/etcdutils.py", line 189, in start
Dec 16 14:38:50 ltpr-ctr01 neutron-server[49614]: self.dispatcher.handle_event(response)
Dec 16 14:38:50 ltpr-ctr01 neutron-server[49614]: File "/usr/local/lib/python3.6/dist-packages/networking_calico/etcdutils.py", line 79, in handle_event
Dec 16 14:38:50 ltpr-ctr01 neutron-server[49614]: key_parts = response.key.strip("/").split("/")
Dec 16 14:38:50 ltpr-ctr01 neutron-server[49614]: TypeError: a bytes-like object is required, not 'str'
Dec 16 14:38:50 ltpr-ctr01 neutron-server[49614]: Removing descriptor: 10

Revision history for this message
Nell Jerram (neil-jerram) wrote :

Vasili, the problem here is Python 3, which we don't yet support. Can you try running (the whole of OpenStack, plus Calico) with Python 2 instead?

Revision history for this message
Vasili (vasili.namatov) wrote :

I installed Openstack Queens with Calico successfully and it works fine.
BUT We have to move to Python3 due to since beginning of 2020 the python2.x will be EOL and will not be supported anymore.

I need calico driver working on Openstack Train which is fully working with python3.
I tried to fix some issues but there are many.
Could you advise if calico Driver is going to continue to work with Openstack. And if not so we have to look for alternatives for calico driver. Its not so serious for the component which is working with openstack not to be ready for latest openstack version and python3.

Python3 available since stein, since April there is no working calico plugin with python3?

Best,
Vasili

Revision history for this message
Vasili (vasili.namatov) wrote :

btw... the pip project says that networking-calico have python3 support....
https://pypi.org/project/networking-calico/

when you saying you don't yet support python3.. where it mentioned that you don't support it?

Vasili (vasili.namatov)
description: updated
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ubuntu:
status: New → Confirmed
Revision history for this message
Nell Jerram (neil-jerram) wrote :

Vasili, I completely agree with you, and I am trying to schedule time for the work needed for it. BTW, you can see the work that I have started, at https://github.com/neiljerram/networking-calico/commits/ci-fixups. (The next class of problems is where the test code expects normal strings and gets b"..." strings instead.)

Regarding the PyPI declaration - yes, I'm sorry, this is wrong. I did previously think that our code was already Python 3 compatible, but I'm afraid I misunderstood the situation.

I will try to fix the PyPI declaration for the next release...

Revision history for this message
Vasili (vasili.namatov) wrote :

Thank a lot Neil, hopefully the next release will be already on this year.

Revision history for this message
Bence Romsics (bence-romsics) wrote :

FYI: Running openstack (master) with python2 will not work for long any more since in ussuri we dropped support for it:

https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html

Most projects did not willingly broke python2 compatibility but dropped python2 gate jobs, so it's only a question of time when it will become actually broken.

On the other hand the train release is still supposed to work with python2, so that maybe a workaround until calico makes the changes mentioned by Neil above.

Revision history for this message
Nell Jerram (neil-jerram) wrote :

Thanks Bence. FYI, in Neutron I believe https://review.opendev.org/#/c/691874/ has already broken Python 2, so the ship has now sailed for almost all new OpenStack deployments (excepting any that are Swift-only, or still using nova-net if that is possible).

Revision history for this message
Vasili (vasili.namatov) wrote :

Thanks Neil, so not yet in 3.11 release...
Waiting for the next one..

As discussed I'm trying to fix some peaces of code by myself and now neutron-server service doesn't fails, but I assume that it didn't load some data as expected and maybe that why it not failed. Meanwhile it's just a theory. :)

Could you advise if I need to open the new bugs for each and each errors I found or I can put all my findings here in this bug?

Just opened another one:
https://bugs.launchpad.net/networking-calico/+bug/1857030

Thanks,
Vasili

Revision history for this message
Akihiro Motoki (amotoki) wrote :

python3 support is completely up to the networking-calico project.
From the perspective of the neutron project, there is nothing to do, so I am marking it as Invalid in "neutron".
All other associated projects (OpenStack Community Project, Python Packaging Index, Ubuntu and Ubuntu Server) are unrelated as they are projects which develop/maintain networking-calilo.

Changed in neutron:
status: New → Invalid
Revision history for this message
Vasili (vasili.namatov) wrote :

Hey, sorry what do you mean the python3 support is completely up to the networking-calico project? Could you clarify?

Revision history for this message
Vasili (vasili.namatov) wrote :

Hi,
Could someone advise when the bug will be in progress state?

Revision history for this message
Vasili (vasili.namatov) wrote :
Download full text (22.9 KiB)

Hi Bence,

In addition to your comment:
On the other hand the train release is still supposed to work with python2, so that maybe a workaround until calico makes the changes mentioned by Neil above.

Looks like there is no workarounds for calico beginning from Stein! The other hand is to wait ages for calico python3 support or try to find the alternatives. In May 2020 we already will have Ussuri and Calico is still not in Stein.

I tried to install Stein or Train and I see that neutron is already based python3.
You have some proper way/guyide how you can install neutron in stein release by using python2.7?

Stein and Train components are essentially written on python3 in those releases.
Looks like its nothing to do with the components which are already written on python3 on those releases.
I asked today the community about that, let's wait a few days for the answer:
https://ask.openstack.org/en/question/125627/is-it-possible-to-install-openstack-stein-or-train-with-python-27-and-not-default-python3/

Meanwhile you can see some notes about it:
TRAIN https://wrestlingpenguins.wordpress.com/2019/10/23/openstack-train-for-ubuntu-18-04-lts/
STEIN https://javacruft.wordpress.com/2019/04/17/openstack-stein-for-ubuntu-18-04-lts/

Rocky is the last guy who still based on python2.7 code.
ROCKY https://wrestlingpenguins.wordpress.com/2018/09/07/openstack-rocky-for-ubuntu-18-04-lts/

From the research found:

1. While running neutron installation on stein it uses python3 packages:

add-apt-repository cloud-archive:stein

root@qtor-ctr101:~# apt install neutron-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  alembic docutils-common ieee-data ipset javascript-common libipset3 libjbig0 libjpeg-turbo8 libjpeg8 libjs-jquery libjs-sphinxdoc libjs-underscore liblcms2-2 libpaper-utils libpaper1 libtiff5 libwebp6 libwebpdemux2 libwebpmux3
  neutron-common neutron-plugin-ml2 pycadf-common python-babel-localedata python-pastedeploy-tpl python3-alembic python3-amqp python3-anyjson python3-appdirs python3-asn1crypto python3-babel python3-blinker python3-bs4
  python3-cachetools python3-certifi python3-cffi python3-cffi-backend python3-chardet python3-cliff python3-cmd2 python3-contextlib2 python3-crypto python3-cryptography python3-dateutil python3-debtcollector python3-decorator
  python3-deprecation python3-designateclient python3-distutils python3-dnspython python3-docutils python3-dogpile.cache python3-eventlet python3-extras python3-fasteners python3-fixtures python3-funcsigs python3-futurist
  python3-greenlet python3-html5lib python3-httplib2 python3-idna python3-iso8601 python3-jinja2 python3-jmespath python3-json-pointer python3-jsonpatch python3-jsonschema python3-jwt python3-keyring python3-keyrings.alt
  python3-keystoneauth1 python3-keystoneclient python3-keystonemiddleware python3-kombu python3-lib2to3 python3-linecache2 python3-logutils python3-lxml python3-mako python3-markupsafe python3-memcache python3-migrate python3-mimeparse
  python3-monotonic python3-msgpack python3-munch python3-netaddr python3-netifaces python3-neutron python3-neut...

Revision history for this message
Vasili (vasili.namatov) wrote :

Python support

The Stein and Train releases of Ubuntu OpenStack is Python 3 only; all Python 2 packages have been dropped in Stein and Train.

As mentioned here:
TRAIN https://wrestlingpenguins.wordpress.com/2019/10/23/openstack-train-for-ubuntu-18-04-lts/
STEIN https://javacruft.wordpress.com/2019/04/17/openstack-stein-for-ubuntu-18-04-lts/

Revision history for this message
Nell Jerram (neil-jerram) wrote :

Python 3 support was added in Calico v3.15 https://docs.projectcalico.org/release-notes/#v3150

The master networking-calico code now supports Python 3, and no longer Python 2. There is a 'release-v3.15-python2' branch that still supports Python 2, but we are hoping that maintenance of that branch can now be short-lived.

Changed in networking-calico:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.