neutron metadata access takes a lot of time (>10 sec)

[Leontii Istomin]

During boot_runcommand_delete rally scenario we have found that neutron metadata responds too long:
/var/log/remote/node-5.domain.tld/neutron-metadata-agent.log:2015-09-29T00:23:03.977566+00:00 info: 2015-09-29 00:23:03.976 15601 INFO eventlet.wsgi.server [-] 10.1.0.18,<local> - - [29/Sep/2015 00:23:03] "GET /latest/meta-data/local-hostname HTTP/1.1" 200 150 9.955739

/var/log/remote/node-35.domain.tld/neutron-metadata-agent.log:2015-09-29T09:41:13.394359+00:00 info: 2015-09-29 09:41:13.397 11182 INFO eventlet.wsgi.server [-] 10.2.0.57,<local> - - [29/Sep/2015 09:41:13] "GET /2009-04-04/meta-data/public-keys HTTP/1.1" 200 169 9.883363

/var/log/remote/node-40.domain.tld/neutron-metadata-agent.log:2015-09-29T09:41:55.041256+00:00 info: 2015-09-29 09:41:55.043 10855 INFO eventlet.wsgi.server [-] 10.2.0.56,<local> - - [29/Sep/2015 09:41:55] "GET /latest/meta-data/block-device-mapping/ HTTP/1.1" 200 124 9.759650

/var/log/remote/node-46.domain.tld/neutron-metadata-agent.log:2015-09-29T01:38:16.248982+00:00 info: 2015-09-29 01:38:16.246 22365 INFO eventlet.wsgi.server [-] 10.2.0.17,<local> - - [29/Sep/2015 01:38:16] "GET /latest/meta-data/placement/ HTTP/1.1" 200 134 9.727450

Cloud init script waits metadata 10 seconds.
We need to figure out why getting of metadata keeps that time.

Cluster configuration:
Baremetal,Ubuntu,IBP,HA, Neutron-vxlan,DVR,Ceph-all,Nova-debug,Nova-quotas,7.0-301
Controllers:3 Computes+Ceph:45 LMA:2

api: '1.0'
astute_sha: 6c5b73f93e24cc781c809db9159927655ced5012
auth_required: true
build_id: '301'
build_number: '301'
feature_groups:
- mirantis
fuel-agent_sha: 50e90af6e3d560e9085ff71d2950cfbcca91af67
fuel-library_sha: 5d50055aeca1dd0dc53b43825dc4c8f7780be9dd
fuel-nailgun-agent_sha: d7027952870a35db8dc52f185bb1158cdd3d1ebd
fuel-ostf_sha: 2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c
fuelmain_sha: a65d453215edb0284a2e4761be7a156bb5627677
nailgun_sha: 4162b0c15adb425b37608c787944d1983f543aa8
openstack_version: 2015.1.0-7.0
production: docker
python-fuelclient_sha: 486bde57cda1badb68f915f66c61b544108606f3
release: '7.0'

Diagnostic Snapshot: http://mos-scale-share.mirantis.com/fuel-snapshot-2015-09-29_11-00-31.tar.xz

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/13611

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/13612

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/13613

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/13772

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13611
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: e6ee4d6a00bf0574d0e8c4f6c0b92155c6a2e666
Author: Oleg Bondarev <email address hidden>
Date: Tue Nov 3 11:02:34 2015

DVR: notify specific agent when creating floating ip

Currently when floating ip is created, a lot of useless action
is happening: floating ip router is scheduled, all l3 agents where
router is scheduled are notified about router update, all agents
request full router info from server. All this becomes a big
performance problem at scale with lots of compute nodes.
In fact on (associated) Floating IP creation we really need
to notify specific l3 agent on compute node where associated
VM port is located and do not need to schedule router and
bother other agents where rourter is scheduled. This should
significally decrease unneeded load on neutron server at scale.

Partial-Bug: #1512635
Related-Bug: #1500823
Partial-Bug: #1486828

Change-Id: I0cbe8c51c3714e6cbdc48ca37135b783f8014905

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13772
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: 60f90495cfac978f7cbde103c5868827247d3d26
Author: Oleg Bondarev <email address hidden>
Date: Tue Nov 10 13:54:46 2015

DVR: only notify needed agents on new VM port creation

When a new VM which should be serviced by a DVR router appears
on compute host, this router is scheduled to that host and
notification is sent. Before the patch it was a broad notification
while really we only need to notify agent on target host.
This should decrease the load on neutron server at scale.

Closes-Bug: #1514762
Related-Bug: #1500823
Closes-Bug: #1486795
Change-Id: Id48b6f6a71530c4f6092d2a07b2db1a5cd300c05

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13612
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: d6bf787b12b283ba99c42ab2478ff950854e360a
Author: Oleg Bondarev <email address hidden>
Date: Mon Nov 23 13:16:55 2015

DVR: Notify specific agent when update floatingip

The L3 agent was determined when update floatingip.
So notify the specific agent rather than notify all agents.
This will save some RPC resources. This is only for DVR routers.
Legacy and HA routers notify only the relevant agents.
This reproposes commit 52e91f48f2327b47f126893f9cb12f153380a9a6
which was reverted by commit a2f7e0343a147a30a637af4e1cb9a866f557e87d
because of Ironic gate failures.
Now the patch preserves original behavior for legacy routers and
should not break Ironic tests.

Partial-Bug: #1512635
Related-Bug: #1500823
Partial-Bug: #1486828
Change-Id: I4ef7a69ad033b979ea0e29620a4febfe5e0c30dd

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/14140

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/14147

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/14153

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/14154

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13613
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: c9dd831a4d1717bcd2566af6513baa5e5f5fabe1
Author: Oleg Bondarev <email address hidden>
Date: Tue Nov 24 12:01:43 2015

DVR: notify specific agent when deleting floating ip

In DVR case we only need to notify the l3 agent on compute node
where associated fixed port is located.

Closes-Bug: #1512635
Related-Bug: #1500823
Closes-Bug: #1486828
Change-Id: I644238ca295c4eb6df75a99a8ef6143a801b27cb

[Oleg Bondarev (obondarev)]

Fixes for 7.0 are waiting to be merged https://review.fuel-infra.org/14140 https://review.fuel-infra.org/14147 https://review.fuel-infra.org/14153 https://review.fuel-infra.org/14154

[Oleg Bondarev (obondarev)]

For 8.0 will be closed once tested on scale lab (fixes merged)

[Oleg Bondarev (obondarev)]

After discussion in team it was decided to put bug in Fix Committed as fixes were merged: https://review.fuel-infra.org/#/c/13613/ https://review.fuel-infra.org/#/c/13612/ https://review.fuel-infra.org/#/c/13772/ https://review.fuel-infra.org/#/c/13611/

Theses fixes resolved the bug for 7.0 which was tested on 200 nodes

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/14140
Submitter: Denis V. Meltsaykin <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 868e711fa281180c645248d525e3b51495e96fb1
Author: Oleg Bondarev <email address hidden>
Date: Tue Nov 24 11:55:08 2015

DVR: notify specific agent when creating floating ip

Currently when floating ip is created, a lot of useless action
is happening: floating ip router is scheduled, all l3 agents where
router is scheduled are notified about router update, all agents
request full router info from server. All this becomes a big
performance problem at scale with lots of compute nodes.
In fact on (associated) Floating IP creation we really need
to notify specific l3 agent on compute node where associated
VM port is located and do not need to schedule router and
bother other agents where rourter is scheduled. This should
significally decrease unneeded load on neutron server at scale.

Partial-Bug: #1512635
Related-Bug: #1500823
Related-Bug: #1493732
Partial-Bug: #1486828

Change-Id: I0cbe8c51c3714e6cbdc48ca37135b783f8014905

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/14147
Submitter: Denis V. Meltsaykin <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 00198bf9818f115d5acff9e6a5bef8080004c606
Author: Oleg Bondarev <email address hidden>
Date: Mon Dec 28 14:49:43 2015

DVR: only notify needed agents on new VM port creation

When a new VM which should be serviced by a DVR router appears
on compute host, this router is scheduled to that host and
notification is sent. Before the patch it was a broad notification
while really we only need to notify agent on target host.
This should decrease the load on neutron server at scale.

Closes-Bug: #1514762
Related-Bug: #1500823
Related-Bug: #1493732
Closes-Bug: #1486795

Conflicts:
 neutron/db/l3_dvrscheduler_db.py
 neutron/tests/functional/services/l3_router/test_l3_dvr_router_plugin.py
 neutron/tests/unit/scheduler/test_l3_agent_scheduler.py

Change-Id: Id48b6f6a71530c4f6092d2a07b2db1a5cd300c05

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/14154
Submitter: Denis V. Meltsaykin <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: dbcc3d300c57a4919080f8c58f9061088d90f31f
Author: Oleg Bondarev <email address hidden>
Date: Mon Dec 28 20:49:27 2015

DVR: notify specific agent when deleting floating ip

In DVR case we only need to notify the l3 agent on compute node
where associated fixed port is located.

Closes-Bug: #1512635
Related-Bug: #1500823
Related-Bug: #1493732
Closes-Bug: #1486828

Conflicts:

 neutron/tests/functional/services/l3_router/test_l3_dvr_router_plugin.py

Change-Id: I644238ca295c4eb6df75a99a8ef6143a801b27cb

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/14153
Submitter: Denis V. Meltsaykin <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 0723acb923d9c9939943ead91ea5830130299ddd
Author: Oleg Bondarev <email address hidden>
Date: Mon Dec 28 20:49:18 2015

DVR: Notify specific agent when update floatingip

The L3 agent was determined when update floatingip.
So notify the specific agent rather than notify all agents.
This will save some RPC resources. This is only for DVR routers.
Legacy and HA routers notify only the relevant agents.
This reproposes commit 52e91f48f2327b47f126893f9cb12f153380a9a6
which was reverted by commit a2f7e0343a147a30a637af4e1cb9a866f557e87d
because of Ironic gate failures.
Now the patch preserves original behavior for legacy routers and
should not break Ironic tests.

Partial-Bug: #1512635
Related-Bug: #1500823
Related-Bug: #1493732
Partial-Bug: #1486828

Conflicts:
 neutron/tests/functional/services/l3_router/test_l3_dvr_router_plugin.py

Change-Id: I4ef7a69ad033b979ea0e29620a4febfe5e0c30dd

[Vitaly Sedelnik (vsedelnik)]

All fixes for this issue in 7.0 landed in 7.0-mu-2. Setting to Fix Released and targetting to 7.0-mu-2. I will make sure that release notes will be updated accordingly