nova-manage floating commands are slow

[Joel Friedly]

nova-manage floating (list|create|delete) all execute very slowly. Creating/deleting realistic numbers of IPs can take hours, and simply listing them takes multiple seconds. These operations are essentially inserting/deleting/selecting from a mysql database, and should all be fast (< 1 second).

Reference values:
On a development cloud that we have, creating a /16 block of IPs (65,536 IPs) took around two hours, and simply doing the SQL SELECT to print them out with nova-manage floating list took 10 seconds.

Implementation details:
Looking at the nova-manage command, the create/delete commands spend the majority of their time in the db.floating_ip_create()/destroy() functions, spending roughly .2s to execute each on our machines.
The list command needs to be investigated more.
We suspect that the issues lies in using sqlalchemy inefficiently, most likely using a new transaction to update every single row. Assuming that sqlalchemy supports pipelining multiple row updates into a single transaction, we think this could improve execution speeds a lot.

[Thierry Carrez (ttx)]

Might be related to https://lists.launchpad.net/openstack/msg14461.html

[Michael Still (mikal)]

Some indexes got added to the floating IPs table in schema 125, and I was hoping that that would improve things. A quick check with devstack says not though. I suspect the problem here is that each IP is created with its own call to the database.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/12315

[Michael Still (mikal)]

Listing 65,000 floating IPs on my laptop takes 7.5 seconds of wall time, which seems reasonable to me. Its a single database operation unless the floating IP has a fixed IP associated with it for what its worth. Deletion suffers from similar problems to create and I'll take a pass at fixing that now.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/12315
Committed: http://github.com/openstack/nova/commit/0bd4a04afb35d7e60b34b8a1aaab45b7805cad15
Submitter: Jenkins
Branch: master

commit 0bd4a04afb35d7e60b34b8a1aaab45b7805cad15
Author: Michael Still <email address hidden>
Date: Mon Sep 3 22:23:24 2012 +1000

    Speed up creating floating ips.

    Previously, we would make two separate DB calls for each floating ip
    we created in floating_ip_bulk_create. This was just a little slow
    (several hours to create a /16). This patch speeds up floating ip
    creation just a little:

    $ time bin/nova-manage floating create --ip_range=10.250.0.0/16

    real 0m48.766s
    user 0m34.342s
    sys 0m1.984s

    Resolves bug 1023560.

    Change-Id: Ie3663440d414b102111178899db0c571769edba8

[Michael Still (mikal)]

I've changed this back to "in progress" because it still needs a delete speed up patch (which I will send today).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/12490

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/12490
Committed: http://github.com/openstack/nova/commit/5339141f04bf3ce470677e5fc3800ce4ac3fc0ae
Submitter: Jenkins
Branch: master

commit 5339141f04bf3ce470677e5fc3800ce4ac3fc0ae
Author: Michael Still <email address hidden>
Date: Tue Sep 4 20:46:00 2012 +1000

    Improve floating IP delete speed.

    This is similar to the change to improve bulk creation speed. Before:

    ./bin/nova-manage floating delete --ip_range 10.250.0.0/16
    337.20user 10.22system 44:07.10elapsed 13%CPU (0avgtext+0avgdata 119536maxresident)k
    400inputs+0outputs (4major+11704minor)pagefaults 0swaps

    After:

    ./bin/nova-manage floating delete --ip_range 10.250.0.0/16
    6.95user 0.16system 1:04.04elapsed 11%CPU (0avgtext+0avgdata 120016maxresident)k
    0inputs+0outputs (0major+11750minor)pagefaults 0swaps

    Resolves bug 1023560.

    Change-Id: If4f23222025cae88576f469022e0f7d907c9c4fa