[Backport] floating ip scheduled to wrong router

[Alexander Ignatov]

This is backport for fix merged in stable/juno:

https://bugs.launchpad.net/neutron/+bug/1422476

Original description:
================

I have a tenant network, two external networks, two routers (each one has gateway set to one of the external networks, and one port on the tenant network) and floating ip's on each external network.

In icehouse, this worked fine. the floating ip for each network was attached to the correct router. After upgrading to RDO Juno, I'm seeing both sets of floating ip's getting assigned to the same router:

[root@cloud ~]# ip netns exec qrouter-209158a6-ee00-405f-b929-7cb386460d94 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
44: qr-ffaaacc1-06: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:5c:e1:58 brd ff:ff:ff:ff:ff:ff
    inet 192.168.127.1/24 brd 192.168.127.255 scope global qr-ffaaacc1-06
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe5c:e158/64 scope link
       valid_lft forever preferred_lft forever
53: qg-1a260edc-41: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:81:dc:f7 brd ff:ff:ff:ff:ff:ff
    inet 192.101.107.185/25 brd 192.101.107.255 scope global qg-1a260edc-41
       valid_lft forever preferred_lft forever
    inet 192.168.122.179/32 brd 192.168.122.179 scope global qg-1a260edc-41
       valid_lft forever preferred_lft forever
    inet 192.168.122.128/32 brd 192.168.122.128 scope global qg-1a260edc-41
       valid_lft forever preferred_lft forever
    inet 192.101.107.171/32 brd 192.101.107.171 scope global qg-1a260edc-41
       valid_lft forever preferred_lft forever
    inet 192.101.107.181/32 brd 192.101.107.181 scope global qg-1a260edc-41
       valid_lft forever preferred_lft forever
    inet 192.101.107.180/32 brd 192.101.107.180 scope global qg-1a260edc-41
       valid_lft forever preferred_lft forever
    inet 192.101.107.179/32 brd 192.101.107.179 scope global qg-1a260edc-41
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe81:dcf7/64 scope link
       valid_lft forever preferred_lft forever

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Kevin Fox <email address hidden>
Review: https://review.fuel-infra.org/7221

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/7221
Submitter: Eugene Nikanorov <email address hidden>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: 726df3138d66f4eb8280d87c96aab776494d7b80
Author: Kevin Fox <email address hidden>
Date: Mon Jun 1 13:14:06 2015

Fixes floating IP regression with multiple routers

During the refactor here:
Change-Id: I09e8a694cdff7f64a642a39b45cbd12422132806
Too much code was removed and caused floating ips to get miss assigned when
multiple routers with external networks in the same tenant are present. The
first router in the tenant was always being chosen. This patch adds back
some of the original code as well as a unit test.

Change-Id: I6f663cb1ce3e4a1340c415d13787a9855c4dcac2
Closes-Bug: 1422476
Closes-Bug: #1460655
(cherry picked from review https://review.openstack.org/#/c/180934/3)

[Kristina Berezovskaia (kkuznetsova)]

Verify on:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "504"
  build_id: "2015-06-03_22-43-02"
  nailgun_sha: "0c53b2a744f8ed3e7fdd826df271dfdf02c071e8"
  python-fuelclient_sha: "4fc55db0265bbf39c369df398b9dc7d6469ba13b"
  astute_sha: "cbae24e9904be2ff8d1d49c0c48d1bdc33574228"
  fuel-library_sha: "e0ca19c44599d4dd44565ba246dda77340a99e7a"
  fuel-ostf_sha: "d272b162a4c6ec8f5907fc1bb4a03d4aab52aa88"
  fuelmain_sha: "bcc909ffc5dd5156ba54cae348b6a07c1b607b24"
3 controllers, 2 compute, Ubuntu+neutron+vlan

Steps to reproduce:
1) Create one more external network and subnet
2) In file /etc/neutron/l3_agent.ini on 2 nodes-controllers write gateway_external_network_id as id for ext_net04
3) In file /etc/neutron/l3_agent.ini on the last node-controller write gateway_external_network_id as id for new external network
4) pcs resource disable p_neutron-l3-agent (wait untill l3-agent become XXX)
5) pcs resource enable p_neutron-l3-agent
6) create new private net and subnet
7) create new router and connect it with new external and private nets
8) boot vm1 and vm2 in net04 and new private net
9) assosiate floating ips for them
10) check on what l3-agents are router04 and new router
11) run ip netns exec qrouter-* ip a on this nodes

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/9295
Submitter: mos-infra-ci <>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: b31ba09215e4335d0e3b2a55008203a0a5d47842
Author: Alexander Ignatov <email address hidden>
Date: Tue Jul 14 10:49:09 2015

Merge the latest state of stable/kilo

Closes-Bug: #1450142
Closes-Bug: #1457123
Closes-Bug: #1430171
Closes-Bug: #1460655
Closes-Bug: #1454421
Closes-Bug: #1442334
Closes-Bug: #1466490

Change-Id: Ie44d227cb6be9375f7ae2a157eadea6cc9976bb5

[Anna Babich (ababich)]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "187"
  build_id: "2015-08-18_03-05-20"
  nailgun_sha: "4710801a2f4a6d61d652f8f1e64215d9dde37d2e"
  python-fuelclient_sha: "4c74a60aa60c06c136d9197c7d09fa4f8c8e2863"
  fuel-agent_sha: "57145b1d8804389304cd04322ba0fb3dc9d30327"
  fuel-nailgun-agent_sha: "e01693992d7a0304d926b922b43f3b747c35964c"
  astute_sha: "e24ca066bf6160bc1e419aaa5d486cad1aaa937d"
  fuel-library_sha: "0062e69db17f8a63f85996039bdefa87aea498e1"
  fuel-ostf_sha: "17786b86b78e5b66d2b1c15500186648df10c63d"
  fuelmain_sha: "c9dad194e82a60bf33060eae635fff867116a9ce"

Verified on cluster: neutron+vxlan+l2pop, 3 controllers, 2 computes

Verification scenario
1. Create one more external network and subnet:
neutron net-create net01_ext -- --router:external=True
neutron subnet-create net01_ext 172.18.171.0/24 --name net01_ext__subnet --allocation-pool start=172.18.171.152,end=172.18.171.152 --gateway=172.18.171.1 --enable_dhcp=False
2. In file /etc/neutron/l3_agent.ini on 2 nodes-controllers write gateway_external_network_id as id for ext_net04
3. In file /etc/neutron/l3_agent.ini on the last node-controller write gateway_external_network_id as id for new external network
4. pcs resource disable p_neutron-l3-agent (wait untill l3-agent become XXX)
5. pcs resource enable p_neutron-l3-agent
6. create new private net and subnet
7. create new router and connect it with new external and private nets
8. boot vm1 and vm2 in net04 and new private net
9. assosiate floating ips for them
10. check on what l3-agents are router04 and new router:
root@node-3:~# router01_id=$(neutron router-show router01 | grep ' id ' | awk '{print $4}')
root@node-3:~# router04_id=$(neutron router-show router04 | grep ' id ' | awk '{print $4}')
root@node-3:~# neutron l3-agent-list-hosting-router $router01_id
+--------------------------------------+-------------------+----------------+-------+----------+
| id | host | admin_state_up | alive | ha_state |
+--------------------------------------+-------------------+----------------+-------+----------+
| 42b9e870-a983-40ed-b520-226a81a0b17b | node-3.domain.tld | True | :-) | |
+--------------------------------------+-------------------+----------------+-------+----------+
root@node-3:~# neutron l3-agent-list-hosting-router $router04_id
+--------------------------------------+-------------------+----------------+-------+----------+
| id | host | admin_state_up | alive | ha_state |
+--------------------------------------+-------------------+----------------+-------+----------+
| 5973c36b-308f-48e2-94e0-1d6b6ee26109 | node-1.domain.tld | True | :-) | |
+--------------------------------------+-------------------+----------------+-------+----------+
11. run ip netns exec qrouter-* ip a on this nodes:
root@node-3:~# ip netns list | grep qrouter-$router01_id
qrouter-584e65e1-ae4b-4d67-a580-0bbac4450c9d
root@node-3:~# ip netns e qrouter-584e65e1-ae4b-4d67-a580-0bbac4450c9d ip a
1: lo: <LOOPBACK,UP,LO...