[L3] TooManyExternalNetworks: More than one external network exists.

Bug #1844168 reported by LIU Yulong on 2019-09-16
20
This bug affects 4 people
Affects Status Importance Assigned to Milestone
neutron
High
LIU Yulong

Bug Description

Code: master with nothing changed.

Exception:
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server [None req-9b3e8e62-b6b3-4506-8950-f73c3e5e2be3 None None] Exception during message handling: TooManyExternalNetworks: More than one external network exists.
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/api/rpc/handlers/l3_rpc.py", line 253, in get_external_network_id
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server net_id = self.plugin.get_external_network_id(context)
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/db/external_net_db.py", line 149, in get_external_network_id
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server raise n_exc.TooManyExternalNetworks()
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server TooManyExternalNetworks: More than one external network exists.
Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server

Upstream log search:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22TooManyExternalNetworks%3A%20More%20than%20one%20external%20network%20exists.%5C%22

Fix proposed to branch: master
Review: https://review.opendev.org/682418

Changed in neutron:
assignee: nobody → LIU Yulong (dragon889)
status: New → In Progress
tags: added: l3-ipam-dhcp
Changed in neutron:
importance: Undecided → Low
status: In Progress → Confirmed
Changed in neutron:
status: Confirmed → In Progress
Changed in neutron:
importance: Low → High
milestone: none → train-rc1

Reviewed: https://review.opendev.org/682418
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f51e5ce9241c9bbd0e2ed786f41dc11ed0146b88
Submitter: Zuul
Branch: master

commit f51e5ce9241c9bbd0e2ed786f41dc11ed0146b88
Author: LIU Yulong <email address hidden>
Date: Tue Sep 17 00:58:08 2019 +0800

    Remove get_external_network_id for router

    L3 agent supports multiple external networks from a long
    time ago, so remove this RPC call since it is not used.
    According to codesearch of [1] and [2], we just remove
    neutron built-in L3 agent RPC. For neutron server side
    or RPC callback classes, the function is still remained.

    [1] http://codesearch.openstack.org/?q=get_external_network_id
    [2] http://codesearch.openstack.org/?q=L3RpcCallback

    Change-Id: I764423e175d6e82729a647e415a9f267f495916f
    Closes-Bug: #1844168

Changed in neutron:
status: In Progress → Fix Released

This issue was fixed in the openstack/neutron 15.0.0.0rc1 release candidate.

tags: added: neutron-proactive-backport-potential
Laurent Dumont (baconpackets) wrote :

What is the status of this bug? I seem to be hitting the same with Stein deployed with Kolla-Ansible. As soon as I create two external networks, I'm seeing the following logs :

Exception during message handling: TooManyExternalNetworks: More than one external network exists.

2019-10-12 21:08:48.407 29 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-10-12 21:08:48.407 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2019-10-12 21:08:48.407 29 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2019-10-12 21:08:48.407 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch

Router '5d5cda12-bdc7-48a8-9477-6ea37a71918e' is not compatible with this agent.: RouterNotCompatibleWithAgent: Router '5d5cda12-bdc7-48a8-9477-6ea37a71918e' is not compatible with this agent.
2019-10-12 21:08:48.412 48 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2019-10-12 21:08:48.412 48 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 710, in _process_routers_if_compatible
2019-10-12 21:08:48.412 48 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2019-10-12 21:08:48.412 48 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 551, in _process_router_if_compatible
2019-10-12 21:08:48.412 48 ERROR neutron.agent.l3.agent router_id=router['id'])
2019-10-12 21:08:48.412 48 ERROR neutron.agent.l3.agent RouterNotCompatibleWithAgent: Router '5d5cda12-bdc7-48a8-9477-6ea37a71918e' is not compatible with this agent.
2019-10-12 21:08:48.412 48 ERROR neutron.agent.l3.agent

Slawek Kaplonski (slaweq) wrote :

@Laurent: I don't think You hit the same bug. In original bug report TooManyExternalNetworks exception was raised. In stack trace from You it is RouterNotCompatibleWithAgent exception.
Can You report this as new bug and share Your neutron-l3 agent's config, information how Your routers are created exactly and how to reproduce this issue?

Laurent Dumont (baconpackets) wrote :

@slaweq

That is a bit strange. I'm using Terraform to spawn the Networks and if I create both my PROC and EXT network with the "external" flag, the router will not spawn and will show both the "RouterNotCompatibleWithAgent" and "TooManyExternalNetworks" errors.

If I create one network with the External flag and one without, the router will spawn properly.

Slawek Kaplonski (slaweq) wrote :

Hi Laurent,

I'm trying to reproduce Your issue on latest Stein release (14.0.4.dev1) and I can't.
Can You try it on latest Stein code? If it still happens for You can You share Your exact configs of neutron-server and L3 agents? And details about network and routers created in neutron API.

chalansonnet (schalans) wrote :

Hello,

We also use Kolla for deployment.
When migrating under Stein, the same problem appeared.
We also confirm the problem with the latest Stein updates

Greeting,

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers