Neutron - Stein - Having two external networks will prevent the router from being created.

[Laurent Dumont]

I am seeing an issue where creating two networks marked as external will prevent a Neutron router from spawning on the Controllers. Ports attached to those networks will be shown as down.

Environment:
- Openstack Stein.
- Deployed through Kolla.

1) One network+subnet (EXT) marked as external with X vlan configured on physnet1. This is expected to be used as the SNAT for a router and floating IPs. Not marked as shared.
2) One network+subnet (PROV) marked as external with Y vlan configured on physnet1. Not currently in use on any qrouters.
3) Create one router with SNAT and the external network set as the EXT network.
4) The router is created within Openstack but is never created on the Controllers namespaces. All port are marked as down.
5) I've attached the logs.
6) If I clear the gateway from the router, the router is spawned and the interfaces come up.
7) If I re-add the gateway with the EXT network. The router goes down and the interfaces are deleted.

-----------------

2019-10-22 18:52:27.199 49 ERROR neutron.agent.l3.agent [-] Router 'b6dba893-1454-4c0f-9088-e61edef31902' is not compatible with this agent.: RouterNotCompatibleWithAgent: Router 'b6dba893-1454-4c0f-9088-e61edef31902' is not compatible with this agent.
2019-10-22 18:52:27.199 49 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2019-10-22 18:52:27.199 49 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 710, in _process_routers_if_compatible
2019-10-22 18:52:27.199 49 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2019-10-22 18:52:27.199 49 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 551, in _process_router_if_compatible
2019-10-22 18:52:27.199 49 ERROR neutron.agent.l3.agent router_id=router['id'])
2019-10-22 18:52:27.199 49 ERROR neutron.agent.l3.agent RouterNotCompatibleWithAgent: Router 'b6dba893-1454-4c0f-9088-e61edef31902' is not compatible with this agent.
2019-10-22 18:52:27.199 49 ERROR neutron.agent.l3.agent
2019-10-22 18:52:27.200 49 DEBUG neutron.agent.l3.agent [-] Payload:

2019-10-22 18:52:20.296 31 ERROR oslo_messaging.rpc.server [req-f62c802f-d047-45d7-a3eb-b40e8c44f6ae - - - - -] Exception during message handling: TooManyExternalNetworks: More than one external network exists.
2019-10-22 18:52:20.296 31 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 254, in get_external_network_id
2019-10-22 18:52:20.296 31 ERROR oslo_messaging.rpc.server net_id = self.plugin.get_external_network_id(context)
2019-10-22 18:52:20.296 31 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/external_net_db.py", line 149, in get_external_network_id
2019-10-22 18:52:20.296 31 ERROR oslo_messaging.rpc.server TooManyExternalNetworks: More than one external network exists.

[Laurent Dumont (baconpackets)]

Sorry about that, I cliked submit a bit too quickly. We can can close this one, I will create a more clear bug asap. :)

[Laurent Dumont (baconpackets)]

And I just noticed the edit body...

I'll use this one for the bug.

[Laurent Dumont (baconpackets)]

I can also provide access to the environment where I see the issue.

[Marek Grudzinski (ivve)]

Are you using DVR and HA?

[Donny Davis (donny-g)]

Mine does the same thing, but if I restart the l3-agent it will end up creating the routers and then complaining about the existence of more than one external network

[Laurent Dumont (baconpackets)]

@ivve The setup is with HA and no DVR. I've also tried using a non-HA router but encountered the same issue.

I've tried Donny's trick and it seems to work. Restarting the L3 agent on all controllers does allow the namespace to be created and the qrouter to exist. I see that the ports are up as well. I'll see if connectivity works.

[Slawek Kaplonski (slaweq)]

Hi Laurent,

I'm trying to reproduce Your issue on latest Stein release (14.0.4.dev1) and I can't.
Can You try it on latest Stein code? If it still happens for You can You share Your exact configs of neutron-server and L3 agents? And details about network and routers created in neutron API.

[Laurent Dumont (baconpackets)]

Hey Slawek,

I tried an upgrade for the kolla container to the last images for Stein but that seems to have killed the Mariadb cluster. I'm doing a fresh deploy to see how it behaves.

Just be be on the safe side, are there any inherent limitations for External networks with the following setup?

- 1 interface for API/MGMT.
- 1 interface for External networks - defined in neutron_external_interface.

Thanks

[Laurent Dumont (baconpackets)]

I've re-deployed Stein and I seem to have the same issue. I've attached the different configuration files from Neutron and the l3-agent.

[Laurent Dumont (baconpackets)]

This also seem to be the exact same issue :

https://ask.openstack.org/en/question/121209/external-network-with-more-than-one-vlan-not-working-in-stein-release/

[Laurent Dumont (baconpackets)]

It's a bit annoying that I can't find a rhyme or a specific pattern to the issue.

- Right now, if I restart all 3 l3_agent after marking a network as External, I can attach that network to a router and the ports will spawn properly.

I'm not sure if there is something wrong with the l3 agent when they first spawn but a restart after the initial setup seems to fix the issue.