BGP: multiple AZ neutron env has no routes to gateway IPs

[LIU Yulong]

The neutron DR agent only advertise floating IP routes as destination(floating IP/32) - next_hop(gateway IP).
Such routes may cause link unreachable in a layer 3 isolated multi-AZ network environment.

For instances:
ISP1 - DC1
ISP2 - DC2
ISP3 - DC3

For the floating IPs, we let all the ISP to advise the IP range, such as 172.16.1.0/24.
And each DC is layer 3 isolated in a separate AZ managed by one neutron deployment.

So for a floating IP (172.16.1.10) bgp route will be:
+-----------------+--------------+
| destination | next_hop |
+-----------------+--------------+
| 172.16.1.10/32 | 172.16.1.2 |
+-----------------+--------------+

We use more than one DR agent to advise that route to all DC physic device (router).
Then assuming that 172.16.1.10 is served at DC1. Some incoming traffic from DC2 or DC3 now does
not know how to route the 172.16.1.10, because the 172.16.1.2 is stranger to them, aka link unreachable.

So advising the gateway routes may be needed in such scenario.

Example topology:
https://docs.google.com/document/d/1rlQEbD-Wr99surhKN3O29f29KSfmmplilEvb9ysFBzE/pub

[Na Zhu (nazhu)]

I think the physical router should advertise gateway subnet to other datacenter.
For example, the upstream physical router ip address 172.16.1.1, so the gateway of neutron router is 172.16.1.1, the upstream physical router should advertise 172.16.1.0/24 to other datacenter, to make the underlay knows how to route 172.16.1.0/24.

[LIU Yulong (dragon889)]

@Na Zhu (nazhu), thank you for your reply.
We need the following routes to physical device (router):
+-----------------+--------------+
| destination | next_hop |
+-----------------+--------------+
| 172.16.1.10/32 | 172.16.1.2 |
| 172.16.1.2/32 | new_ip[1] | (new, the routes for gateway IPs)
+-----------------+--------------+

[1] the new_ip could be:
(1) l3_agent host br-ex related "NIC IP", this may also need to add some routes which destination is the "NIC IP" to physic router.
(2) the physical router corresponding port IP to another DC.

[Na Zhu (nazhu)]

Hi,

The "NIC IP" is out of neutron's scope, how bgp knows the new_ip.
Normally, the new_ip is in the same subnet of gateway interface ip.
If the upstream physical router can advertise 172.16.1.0/24, the underlay is reachable.

[LIU Yulong (dragon889)]

(1) Add br-ex IP to L3 agent_config, maybe a potential solution. Then let the bgpspeaker process gateway routes from the DB query. The br-ex IP need network manager to add to physical router.
(2) For all the AZ, the physical router may assume that 172.16.1.0/24 IPs are all handled by themself, aka something called route black hole. And the BGP speaker in neutron now does not use the subnet gateway IP as the next_hop IP.

[Sahid Orentino (sahid-ferdjaoui)]

Let's close this report. there are still a discussion on-going around it and it has 6 year old. Please reopen with details if needed.

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]