dvr router update does not scale for many fips

[Erik Colnick]

On an installation with 7 compute nodes and distributed routing enabled, the time it takes to process a sync_routers request on the controller grows linearly with the number of floating ips. With 120 floating ips associated to vm instances distributed across the the compute nodes (all attached to one router), the time to associate or disassociate a floating ip to an instance has been observed to take over 40 seconds (and this with 3 load balanced controller nodes and 10 rpc worker threads and 10 api worker threads configured on each controller node).

Tracing the logs, it is observed that the highest percentage of time spent as the number of floating ips associated to vms increases is in the '_process_floating_ips' method of the l3_dvr_db.py source, which makes multiple DB requests per floating ip. The second longest time is spent in the get_sync_data method itself prior to the call to _process_floating_ips where a call is made to the get_vm_port_hostid method for each floating ip.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/150110

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/150576
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f10f1f16777572ca22d43670095c6d73ae9dcce5
Submitter: Jenkins
Branch: master

commit f10f1f16777572ca22d43670095c6d73ae9dcce5
Author: Erik Colnick <email address hidden>
Date: Tue Jan 27 13:21:18 2015 -0700

    Refactor to facilitate DVR scale performance

    To facilitate work for DVR scale performance improvements, refactor
    l3_agentschedulers_db.list_active_sync_routers_on_active_l3_agent
    to pull out logic that will be overridden into the private method
    _get_active_l3_agent_routers_sync_data

    Partial-Bug: 1413314

    Change-Id: Ia454b037529c9b6f3750a67dc0eabb7e01419ea8

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/150110
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b96a22661290ce2ea747537512eab2fb767679e6
Submitter: Jenkins
Branch: master

commit b96a22661290ce2ea747537512eab2fb767679e6
Author: Erik Colnick <email address hidden>
Date: Fri Jan 23 12:16:28 2015 -0700

    Improve DVR scale performance

    Only process floating ips on a router that are relevant to the agent
    hosting the router (don't process floating ips assigned to a router
    if the associated vm is not hosted on the compute node requesting the
    router sync). In this way, the number of database calls made during
    the DVR router updates is optimized to eliminate unnecessary
    duplication of calls which return the same data or are made to get
    data for routers which are not relevant to the sync_routers request
    from the agent.

    Change-Id: I4e8477bb61ffff164d2f3bbebb94e95a25838ce0
    Partial-Bug: #1413314

[Armando Migliaccio (armando-migliaccio)]

It looks the two targeted bugs merged. If there are any loose ends, we should raise another, more specific bug and leave this one completed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: neutron-pecan
Review: https://review.openstack.org/185072

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/231950

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jakub Libosvar (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/231950
Reason: Abandoned as per reviewers comment - we don't have a tool in this branch to validate the change.