[L3] TooManyExternalNetworks: More than one external network exists.

[LIU Yulong]

Code: master with nothing changed.

Exception:
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server [None req-9b3e8e62-b6b3-4506-8950-f73c3e5e2be3 None None] Exception during message handling: TooManyExternalNetworks: More than one external network exists.
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/api/rpc/handlers/l3_rpc.py", line 253, in get_external_network_id
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server net_id = self.plugin.get_external_network_id(context)
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/db/external_net_db.py", line 149, in get_external_network_id
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server raise n_exc.TooManyExternalNetworks()
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server TooManyExternalNetworks: More than one external network exists.
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server

Upstream log search:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22TooManyExternalNetworks%3A%20More%20than%20one%20external%20network%20exists.%5C%22

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/682418

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/682418
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f51e5ce9241c9bbd0e2ed786f41dc11ed0146b88
Submitter: Zuul
Branch: master

commit f51e5ce9241c9bbd0e2ed786f41dc11ed0146b88
Author: LIU Yulong <email address hidden>
Date: Tue Sep 17 00:58:08 2019 +0800

    Remove get_external_network_id for router

    L3 agent supports multiple external networks from a long
    time ago, so remove this RPC call since it is not used.
    According to codesearch of [1] and [2], we just remove
    neutron built-in L3 agent RPC. For neutron server side
    or RPC callback classes, the function is still remained.

    [1] http://codesearch.openstack.org/?q=get_external_network_id
    [2] http://codesearch.openstack.org/?q=L3RpcCallback

    Change-Id: I764423e175d6e82729a647e415a9f267f495916f
    Closes-Bug: #1844168

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.0.0.0rc1 release candidate.

[Laurent Dumont (baconpackets)]

What is the status of this bug? I seem to be hitting the same with Stein deployed with Kolla-Ansible. As soon as I create two external networks, I'm seeing the following logs :

Exception during message handling: TooManyExternalNetworks: More than one external network exists.

2019-10-12 21:08:48.407 29 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-10-12 21:08:48.407 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2019-10-12 21:08:48.407 29 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2019-10-12 21:08:48.407 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch

Router '5d5cda12-bdc7-48a8-9477-6ea37a71918e' is not compatible with this agent.: RouterNotCompatibleWithAgent: Router '5d5cda12-bdc7-48a8-9477-6ea37a71918e' is not compatible with this agent.
2019-10-12 21:08:48.412 48 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2019-10-12 21:08:48.412 48 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 710, in _process_routers_if_compatible
2019-10-12 21:08:48.412 48 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2019-10-12 21:08:48.412 48 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 551, in _process_router_if_compatible
2019-10-12 21:08:48.412 48 ERROR neutron.agent.l3.agent router_id=router['id'])
2019-10-12 21:08:48.412 48 ERROR neutron.agent.l3.agent RouterNotCompatibleWithAgent: Router '5d5cda12-bdc7-48a8-9477-6ea37a71918e' is not compatible with this agent.
2019-10-12 21:08:48.412 48 ERROR neutron.agent.l3.agent

[Slawek Kaplonski (slaweq)]

@Laurent: I don't think You hit the same bug. In original bug report TooManyExternalNetworks exception was raised. In stack trace from You it is RouterNotCompatibleWithAgent exception.
Can You report this as new bug and share Your neutron-l3 agent's config, information how Your routers are created exactly and how to reproduce this issue?

[Laurent Dumont (baconpackets)]

@slaweq

That is a bit strange. I'm using Terraform to spawn the Networks and if I create both my PROC and EXT network with the "external" flag, the router will not spawn and will show both the "RouterNotCompatibleWithAgent" and "TooManyExternalNetworks" errors.

If I create one network with the External flag and one without, the router will spawn properly.

[Slawek Kaplonski (slaweq)]

Hi Laurent,

I'm trying to reproduce Your issue on latest Stein release (14.0.4.dev1) and I can't.
Can You try it on latest Stein code? If it still happens for You can You share Your exact configs of neutron-server and L3 agents? And details about network and routers created in neutron API.

[chalansonnet (schalans)]

Hello,

We also use Kolla for deployment.
When migrating under Stein, the same problem appeared.
We also confirm the problem with the latest Stein updates

Greeting,

[James Denton (james-denton)]

We are able to reproduce the errors mentioned by Laurent under version 14.0.4.dev11.

Steps to replicate:

1. Create an external provider network (extnet1)
2. Create a router and attach to extnet1
3. Create a second external provider network (extnet2)
4. Create a second router and attach to extnet2

You will see the following in the neutron-server log:

root@1041074-controller2:~# tail -f /var/log/neutron/neutron-server.log | grep TooMany
2020-01-14 16:24:00.216 3936722 ERROR oslo_messaging.rpc.server [req-202b2398-0de8-4db3-9458-5cfcf5efceed - - - - -] Exception during message handling: TooManyExternalNetworks: More than one external network exists.
2020-01-14 16:24:00.216 3936722 ERROR oslo_messaging.rpc.server TooManyExternalNetworks: More than one external network exists.

You will see the following in the l3 agent log at the same time (ignore timestamp):

2020-01-14 16:42:12.655 3823794 ERROR neutron.agent.l3.agent [-] Router '6144e0d1-762f-4839-9511-ccac4058bad8' is not compatible with this agent.: RouterNotCompatibleWithAgent: Router '6144e0d1-762f-4839-9511-ccac4058bad8' is not compatible with this agent.

These errors appear to originate from this block:

https://github.com/openstack/neutron/blob/stable/stein/neutron/agent/l3/agent.py#L557-L564

        # If target_ex_net_id and ex_net_id are set they must be equal
        target_ex_net_id = self._fetch_external_net_id()
        if (target_ex_net_id and ex_net_id and ex_net_id != target_ex_net_id):
            # Double check that our single external_net_id has not changed
            # by forcing a check by RPC.
            if ex_net_id != self._fetch_external_net_id(force=True):
                raise l3_exc.RouterNotCompatibleWithAgent(
                    router_id=router['id'])

This code has been removed in master via patch https://review.opendev.org/#/c/682418/, but I could not cleanly cherry pick that to stable/stein. Any chance this could be addressed for Stein?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/703067

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/703067
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=197c57cf23728614cf37c16ed3f6080e19fdacb3
Submitter: Zuul
Branch: stable/stein

commit 197c57cf23728614cf37c16ed3f6080e19fdacb3
Author: LIU Yulong <email address hidden>
Date: Tue Sep 17 00:58:08 2019 +0800

    Stop verifying unique external_net_id

    This cherry-pick is modified to omit the RPC modification of the
    original patch. Instead it only drops the code that would generate an
    exception if there are two different external networks. Tests are
    amended accordingly.

    Original patch description:

    Remove get_external_network_id for router

    L3 agent supports multiple external networks from a long
    time ago, so remove this RPC call since it is not used.
    According to codesearch of [1] and [2], we just remove
    neutron built-in L3 agent RPC. For neutron server side
    or RPC callback classes, the function is still remained.

    [1] http://codesearch.openstack.org/?q=get_external_network_id
    [2] http://codesearch.openstack.org/?q=L3RpcCallback

    Conflicts:
            neutron/agent/l3/agent.py

    Change-Id: I764423e175d6e82729a647e415a9f267f495916f
    Closes-Bug: #1844168
    (cherry picked from commit f51e5ce9241c9bbd0e2ed786f41dc11ed0146b88)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.1.0 release.