DVR: Notify specific agent when dealing with floating ips

[Oleg Bondarev]

Upstream bug: https://bugs.launchpad.net/neutron/+bug/1486828

Problem Description
===================
Currently in DVR case neutron server will fanout the message to all L3 agents hosting the router when floating ip create/update/delete happens. This becomes a problem at scale.

Proposed Change
==============
In fact only l3 agent on compute node where associated fixed port is located needs to be notified. This should significally decrease server load and RPC communications.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/13611

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/13612

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/13613

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13611
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: e6ee4d6a00bf0574d0e8c4f6c0b92155c6a2e666
Author: Oleg Bondarev <email address hidden>
Date: Tue Nov 3 11:02:34 2015

DVR: notify specific agent when creating floating ip

Currently when floating ip is created, a lot of useless action
is happening: floating ip router is scheduled, all l3 agents where
router is scheduled are notified about router update, all agents
request full router info from server. All this becomes a big
performance problem at scale with lots of compute nodes.
In fact on (associated) Floating IP creation we really need
to notify specific l3 agent on compute node where associated
VM port is located and do not need to schedule router and
bother other agents where rourter is scheduled. This should
significally decrease unneeded load on neutron server at scale.

Partial-Bug: #1512635
Related-Bug: #1500823
Partial-Bug: #1486828

Change-Id: I0cbe8c51c3714e6cbdc48ca37135b783f8014905

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13612
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: d6bf787b12b283ba99c42ab2478ff950854e360a
Author: Oleg Bondarev <email address hidden>
Date: Mon Nov 23 13:16:55 2015

DVR: Notify specific agent when update floatingip

The L3 agent was determined when update floatingip.
So notify the specific agent rather than notify all agents.
This will save some RPC resources. This is only for DVR routers.
Legacy and HA routers notify only the relevant agents.
This reproposes commit 52e91f48f2327b47f126893f9cb12f153380a9a6
which was reverted by commit a2f7e0343a147a30a637af4e1cb9a866f557e87d
because of Ironic gate failures.
Now the patch preserves original behavior for legacy routers and
should not break Ironic tests.

Partial-Bug: #1512635
Related-Bug: #1500823
Partial-Bug: #1486828
Change-Id: I4ef7a69ad033b979ea0e29620a4febfe5e0c30dd

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/14140

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/14153

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/14154

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13613
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: c9dd831a4d1717bcd2566af6513baa5e5f5fabe1
Author: Oleg Bondarev <email address hidden>
Date: Tue Nov 24 12:01:43 2015

DVR: notify specific agent when deleting floating ip

In DVR case we only need to notify the l3 agent on compute node
where associated fixed port is located.

Closes-Bug: #1512635
Related-Bug: #1500823
Closes-Bug: #1486828
Change-Id: I644238ca295c4eb6df75a99a8ef6143a801b27cb

[Sergii Rizvan (srizvan)]

Oleg, please provide us with clear steps to reproduce this problem.

[Oleg Bondarev (obondarev)]

This bug was filed as kind of sub-item for resolving a complex DVR scalability issue, see https://bugs.launchpad.net/mos/+bug/1493732 https://bugs.launchpad.net/mos/+bug/1500823

In order to reproduce mos-neutron team was running rally tests on a 200 node lab untill we see problems with rabbit and long queues of rpc messages which neutron server was unable to handle in required time.

For this particular bug fixes I can give steps to verify:
 - have a cluster with at least one controller and one compute node
 - have a network connected to a DVR router with external gateway
 - spawn a VM on that network
 - assign floating IP to that VM
 - inspect l3 agents logs to see that notification was sent to only l3 agent on compute node
 - delete floating IP from that VM
 - inspect l3 agents logs to see that notification was sent to only l3 agent on compute node

By inspecting logs I mean looking for a particular log message ('Got routers updated notification') at certain moments (floating ip association/disassociation)

[Kristina Berezovskaia (kkuznetsova)]

Verify on:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "307"
  build_id: "307"
  fuel-nailgun_sha: "26c78e1c2913b8ca194126ca38161ff9593daef4"
  python-fuelclient_sha: "6810231ec6d333fc10f7020f9bdf5122e05e998a"
  fuel-agent_sha: "f4c22a341ce8a98cc9151bd928f1d97071288b3c"
  fuel-nailgun-agent_sha: "a33a58d378c117c0f509b0e7badc6f0910364154"
  astute_sha: "c56dfde2da034151a7e707b381c4cf9d213b4ba2"
  fuel-library_sha: "03558ce6627a5adb417b1f6bbe1c0b3a853e8d6b"
  fuel-ostf_sha: "9910a4726cbd038c257582b429527e40c4c3cb20"
  fuel-mirror_sha: "5d986722c7e911d1dd44800f2721bdbfcb7f3282"
  fuelmenu_sha: "5abc3c8fea8e578cf9250f40acf9df98adb214a3"
  shotgun_sha: "cacb93cbc28910ff0dc38f30a855efa9af50d8ce"
  network-checker_sha: "d443ef47abeda58d319bc8d33d5005dd09440a02"
  fuel-upgrade_sha: "718aa3d7021fee2970f0fa6791cf5188578cc516"
  fuelmain_sha: "713b2d7421f1ee660d17e406668f74550f96e18a"
(vxlan+l2+dvr+neutron, 3 controllers, 2 compute)

Repeated steps to verify, logs were not "active" on controllers and compute without vm. 'Got routers updated notification' appeared only in log on compute with vm

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/14140
Submitter: Denis V. Meltsaykin <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 868e711fa281180c645248d525e3b51495e96fb1
Author: Oleg Bondarev <email address hidden>
Date: Tue Nov 24 11:55:08 2015

DVR: notify specific agent when creating floating ip

Currently when floating ip is created, a lot of useless action
is happening: floating ip router is scheduled, all l3 agents where
router is scheduled are notified about router update, all agents
request full router info from server. All this becomes a big
performance problem at scale with lots of compute nodes.
In fact on (associated) Floating IP creation we really need
to notify specific l3 agent on compute node where associated
VM port is located and do not need to schedule router and
bother other agents where rourter is scheduled. This should
significally decrease unneeded load on neutron server at scale.

Partial-Bug: #1512635
Related-Bug: #1500823
Related-Bug: #1493732
Partial-Bug: #1486828

Change-Id: I0cbe8c51c3714e6cbdc48ca37135b783f8014905

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/14154
Submitter: Denis V. Meltsaykin <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: dbcc3d300c57a4919080f8c58f9061088d90f31f
Author: Oleg Bondarev <email address hidden>
Date: Mon Dec 28 20:49:27 2015

DVR: notify specific agent when deleting floating ip

In DVR case we only need to notify the l3 agent on compute node
where associated fixed port is located.

Closes-Bug: #1512635
Related-Bug: #1500823
Related-Bug: #1493732
Closes-Bug: #1486828

Conflicts:

 neutron/tests/functional/services/l3_router/test_l3_dvr_router_plugin.py

Change-Id: I644238ca295c4eb6df75a99a8ef6143a801b27cb

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/14153
Submitter: Denis V. Meltsaykin <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 0723acb923d9c9939943ead91ea5830130299ddd
Author: Oleg Bondarev <email address hidden>
Date: Mon Dec 28 20:49:18 2015

DVR: Notify specific agent when update floatingip

The L3 agent was determined when update floatingip.
So notify the specific agent rather than notify all agents.
This will save some RPC resources. This is only for DVR routers.
Legacy and HA routers notify only the relevant agents.
This reproposes commit 52e91f48f2327b47f126893f9cb12f153380a9a6
which was reverted by commit a2f7e0343a147a30a637af4e1cb9a866f557e87d
because of Ironic gate failures.
Now the patch preserves original behavior for legacy routers and
should not break Ironic tests.

Partial-Bug: #1512635
Related-Bug: #1500823
Related-Bug: #1493732
Partial-Bug: #1486828

Conflicts:
 neutron/tests/functional/services/l3_router/test_l3_dvr_router_plugin.py

Change-Id: I4ef7a69ad033b979ea0e29620a4febfe5e0c30dd

[Sergii Rizvan (srizvan)]

Verified on 7.0 on Ubuntu
Packages:
neutron-common 1:2015.1.1-1~u14.04+mos5348
neutron-dhcp-agent 1:2015.1.1-1~u14.04+mos5348
neutron-l3-agent 1:2015.1.1-1~u14.04+mos5348
neutron-metadata-agent 1:2015.1.1-1~u14.04+mos5348
neutron-plugin-ml2 1:2015.1.1-1~u14.04+mos5348
neutron-plugin-openvswitch-agent 1:2015.1.1-1~u14.04+mos5348
neutron-server 1:2015.1.1-1~u14.04+mos5348
python-neutron 1:2015.1.1-1~u14.04+mos5348

[TatyanaGladysheva (tgladysheva)]

Test case is automated: https://review.gerrithub.io/#/c/274626/