BGP: multiple AZ neutron env has no routes to gateway IPs

Bug #1609296 reported by LIU Yulong
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Undecided
Unassigned

Bug Description

The neutron DR agent only advertise floating IP routes as destination(floating IP/32) - next_hop(gateway IP).
Such routes may cause link unreachable in a layer 3 isolated multi-AZ network environment.

For instances:
ISP1 - DC1
ISP2 - DC2
ISP3 - DC3

For the floating IPs, we let all the ISP to advise the IP range, such as 172.16.1.0/24.
And each DC is layer 3 isolated in a separate AZ managed by one neutron deployment.

So for a floating IP (172.16.1.10) bgp route will be:
+-----------------+--------------+
| destination | next_hop |
+-----------------+--------------+
| 172.16.1.10/32 | 172.16.1.2 |
+-----------------+--------------+

We use more than one DR agent to advise that route to all DC physic device (router).
Then assuming that 172.16.1.10 is served at DC1. Some incoming traffic from DC2 or DC3 now does
not know how to route the 172.16.1.10, because the 172.16.1.2 is stranger to them, aka link unreachable.

So advising the gateway routes may be needed in such scenario.

Example topology:
https://docs.google.com/document/d/1rlQEbD-Wr99surhKN3O29f29KSfmmplilEvb9ysFBzE/pub

LIU Yulong (dragon889)
tags: added: l3-bgp
Revision history for this message
Na Zhu (nazhu) wrote :

I think the physical router should advertise gateway subnet to other datacenter.
For example, the upstream physical router ip address 172.16.1.1, so the gateway of neutron router is 172.16.1.1, the upstream physical router should advertise 172.16.1.0/24 to other datacenter, to make the underlay knows how to route 172.16.1.0/24.

Revision history for this message
LIU Yulong (dragon889) wrote :

@Na Zhu (nazhu), thank you for your reply.
We need the following routes to physical device (router):
+-----------------+--------------+
| destination | next_hop |
+-----------------+--------------+
| 172.16.1.10/32 | 172.16.1.2 |
| 172.16.1.2/32 | new_ip[1] | (new, the routes for gateway IPs)
+-----------------+--------------+

[1] the new_ip could be:
(1) l3_agent host br-ex related "NIC IP", this may also need to add some routes which destination is the "NIC IP" to physic router.
(2) the physical router corresponding port IP to another DC.

Revision history for this message
Na Zhu (nazhu) wrote :

Hi,

The "NIC IP" is out of neutron's scope, how bgp knows the new_ip.
Normally, the new_ip is in the same subnet of gateway interface ip.
If the upstream physical router can advertise 172.16.1.0/24, the underlay is reachable.

Revision history for this message
LIU Yulong (dragon889) wrote :

(1) Add br-ex IP to L3 agent_config, maybe a potential solution. Then let the bgpspeaker process gateway routes from the DB query. The br-ex IP need network manager to add to physical router.
(2) For all the AZ, the physical router may assume that 172.16.1.0/24 IPs are all handled by themself, aka something called route black hole. And the BGP speaker in neutron now does not use the subnet gateway IP as the next_hop IP.

LIU Yulong (dragon889)
description: updated
description: updated
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers