AttributeError: 'NoneType' object has no attribute 'get_segments' when neutron-server is restarted

[Flavio Fernandes]

Using master neutron on devstack deployment, I see this after restarting neutron service

sudo systemctl restart devstack@q-svc

...

AttributeError: 'NoneType' object has no attribute 'get_segments'

Jun 12 01:03:56 stack neutron-server[15077]: DEBUG neutron.db.agents_db [None req-14ec19bf-1bf8-4cd6-a8cb-6e9b49be9e8e None None] Agent healthcheck: found 2 active agents {{(pid=15105) agent_health_check /opt/stack/neutron/neutron/db/agents_db.py:327}}
Jun 12 01:03:57 stack neutron-server[15077]: DEBUG futurist.periodics [None req-d99096de-f66c-4d38-91f7-384397e65e67 None None] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.HashRingHealthCheckPeriodics.touch_hash_ring_nodes' {{(pid=15104) _process_scheduled /usr/local/lib/python3.6/dist-packages/futurist/periodics.py:642}}
n relationship Tag.standard_attr to refer to viewonly relationship StandardAttribute.tags should include sync_backref=False set on the Tag.standard_attr relationship. (this warning may be suppressed after 10 occurrences)
Jun 12 01:01:37 stack neutron-server[15077]: (rel_b, rel_a, rel_b),
Jun 12 01:01:37 stack neutron-server[15077]: Traceback (most recent call last):
Jun 12 01:01:37 stack neutron-server[15077]: File "/usr/local/lib/python3.6/dist-packages/eventlet/hubs/hub.py", line 461, in fire_timers
Jun 12 01:01:37 stack neutron-server[15077]: timer()
Jun 12 01:01:37 stack neutron-server[15077]: File "/usr/local/lib/python3.6/dist-packages/eventlet/hubs/timer.py", line 59, in __call__
Jun 12 01:01:37 stack neutron-server[15077]: cb(*args, **kw)
Jun 12 01:01:37 stack neutron-server[15077]: File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 221, in main
Jun 12 01:01:37 stack neutron-server[15077]: result = function(*args, **kwargs)
Jun 12 01:01:37 stack neutron-server[15077]: File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 92, in do_sync
Jun 12 01:01:37 stack neutron-server[15077]: self.sync_networks_ports_and_dhcp_opts(ctx)
Jun 12 01:01:37 stack neutron-server[15077]: File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 857, in sync_networks_ports_and_dhcp_opts
Jun 12 01:01:37 stack neutron-server[15077]: db_segments = self.segments_plugin.get_segments(
Jun 12 01:01:37 stack neutron-server[15077]: AttributeError: 'NoneType' object has no attribute 'get_segments'

vagrant@stack:~$ sudo journalctl --unit devstack@q-svc --since today > neutron.log

[Flavio Fernandes (ffernand)]

vagrant@stack:/opt/stack/neutron$ git log | head -10
commit 4de18104ae88a835544cefbf30c878aa49efc31f
Merge: 76b2ace079 483f468fdd
Author: Zuul <email address hidden>
Date: Thu Jun 11 21:02:59 2020 +0000

    Merge "[OVN] Create localnet port for each created segment"

commit 76b2ace0790e3f5df19f7d4425aa26c966b373a6
Merge: 646d2db6d1 eb46081150
Author: Zuul <email address hidden>
vagrant@stack:/opt/stack/neutron$ git log | head -20
commit 4de18104ae88a835544cefbf30c878aa49efc31f
Merge: 76b2ace079 483f468fdd
Author: Zuul <email address hidden>
Date: Thu Jun 11 21:02:59 2020 +0000

    Merge "[OVN] Create localnet port for each created segment"

[Flavio Fernandes (ffernand)]

vagrant@stack:/opt/stack/neutron$ python3
Python 3.6.9 (default, Apr 18 2020, 01:56:04)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from neutron_lib.plugins import directory
>>> directory.get_plugin('segments') is None
True
>>>

[Flavio Fernandes (ffernand)]

I am seeing failure in the following 6 functional tests:

==============================
Failed 6 tests - output below:
==============================

neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_sync.TestOvnNbSyncOverSsl.test_ovn_nb_sync_repair
--------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/opt/stack/neutron/neutron/tests/base.py", line 182, in func
    return f(self, *args, **kwargs)

      File "/opt/stack/neutron/neutron/tests/base.py", line 182, in func
    return f(self, *args, **kwargs)

      File "/opt/stack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovn_db_sync.py", line 1463, in test_ovn_nb_sync_repair
    self._test_ovn_nb_sync_helper('repair')

      File "/opt/stack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovn_db_sync.py", line 1460, in _test_ovn_nb_sync_helper
    self._validate_resources(should_match=should_match_after_sync)

      File "/opt/stack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovn_db_sync.py", line 1424, in _validate_resources
    self._validate_networks(should_match=should_match)

      File "/opt/stack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovn_db_sync.py", line 853, in _validate_networks
    self.assertItemsEqual(db_provnet_ports, plugin_provnet_ports)

      File "/opt/stack/neutron/.tox/functional/lib/python3.6/site-packages/unittest2/case.py", line 1182, in assertItemsEqual
    return self.assertSequenceEqual(expected, actual, msg=msg)

      File "/opt/stack/neutron/.tox/functional/lib/python3.6/site-packages/unittest2/case.py", line 1014, in assertSequenceEqual
    self.fail(msg)

      File "/opt/stack/neutron/.tox/functional/lib/python3.6/site-packages/unittest2/case.py", line 690, in fail
    raise self.failureException(msg)

    AssertionError: Sequences differ: ['provnet-bf349500-9342-49b0-8625-d11499805bb0'] != []

First sequence contains 1 additional elements.
First extra element 0:
provnet-bf349500-9342-49b0-8625-d11499805bb0

- ['provnet-bf349500-9342-49b0-8625-d11499805bb0']
+ []

neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_sync.TestOvnNbSync.test_ovn_nb_sync_repair
-------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/opt/stack/neutron/neutron/tests/base.py", line 182, in func
    return f(self, *args, **kwargs)

      File "/opt/stack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovn_db_sync.py", line 1463, in test_ovn_nb_sync_repair
    self._test_ovn_nb_sync_helper('repair')

      File "/opt/stack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovn_db_sync.py", line 1460, in _test_ovn_nb_sync_helper
    self._validate_resources(should_match=should_match_after_sync)

      F...

[Maciej Jozefczyk (maciejjozefczyk)]

Flavio,

I remember I saw the same problem, but while I added line:
https://review.opendev.org/#/c/715157/18/neutron/cmd/ovn/neutron_ovn_db_sync_util.py

It started working - the segments plugin gets registered properly. Could you please check it by running `pip install -e` on your neutron environment?

[Maciej Jozefczyk (maciejjozefczyk)]

Ok, we reproduced it on Flavio environment. In the maint task the segments plugin seems not to be loaded...

[Maciej Jozefczyk (maciejjozefczyk)]

Ok, so while there is no 'segments' service plugin set in configuration, like:

neutron.conf:

[DEFAULT]
service_plugins = segments

The maint task starts to fail.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/737819

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/737819
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=4bd5b57da46be36c956faecacbd3eb4a171593ac
Submitter: Zuul
Branch: stable/train

commit 4bd5b57da46be36c956faecacbd3eb4a171593ac
Author: Maciej Jozefczyk <email address hidden>
Date: Mon Jun 15 12:19:43 2020 +0000

    [OVN] Load segments plugin in case not loaded in maintanance task

    While the segments plugin is not loaded in neutron config, it should
    be loaded anyways in OVN maintanance task, to operate on the first
    default segment of each network.

    Change-Id: Ideffacc2f478c95eeec881c82d1d5bae46ecdc74
    Closes-Bug: 1883193
    (cherry picked from commit 56f519f472adf235856853bf12bfeac0d16a8b3f)