Ironic nova driver spawn() makes too many redundant calls

[Chris Behrens]

The nova virt driver has a scale issue in that it makes way too many API calls, especially on behalf of spawn. I have submitted a review to remove at least 2 API calls by not having to re-query the node when plugging vifs. But after that, we are still left with for spawn:

spawn() gets node from Ironic
spawn() updates node
spawn() -> _add_driver_fields() updates node 'n' times where 'n' is # of required pxe fields
spawn() makes a node validate call
spawn() -> _plug_vifs() -> _unplug_vifs() lists ports.
spawn() -> _plug_vifs() -> _unplug_vifs() makes 'n' calls to update ports where 'n' is number of ports.
spawn() -> _plug_vifs() lists ports.
spawn() -> _plug_vifs() makes 'n' calls to update ports where 'n' is number of ports.
spawn() updates node to set 'active' provision state

We need to figure out a way to make batch calls for some of these, IMO.

[aeva black (tenbrae)]

Yes. The following is what it should look like:

- update node to set instance_uuid (this should return the node obj)
- set 'n' driver fields in one call (rather than 'n' separate calls)
- one call to validate
- one call to list ports
- one call per port that is plugged in (rather than one call for every port)
- one call for every port (since they all need to be plugged in)
- one call to set provision_state to active

This should result in a total of:
1 + 1 + 1 + 1 + p + P + 1
= 5 + p + P
calls, where p is "number of already plugged in ports" (often zero) and P is number of ports that need to be plugged in (often the actual number of physical ports).

[Chris Behrens (cbehrens)]

At minimum I'd like to see p and P reduced to 1 each as well. That would still be 7 API calls which seems a little crazy just to tell Ironic to provision an instance. I don't know if all of these must hit the conductor before the API returns, but each API call has some set of latency associated with it, and having to wait on RPC to conductor in Ironic API contributes more to that. If 'p' and 'P' absolutely cannot be reduced (though I don't see why they couldn't be), there's at least another solution to possible latency by making these calls in greenthreads. That really doesn't completely address my concerns, though.

A little background: We (Rackspace) have gotten bitten by this "API call explosion" issue before in other areas of OpenStack (in neutron plugins -- before it was called neutron), which is why I'd like to reduce this as much as possible. I see it becoming an issue and would rather fix this as early as we can.

I understand the desire to have API calls that do single specific tasks and I agree. But I start arguing with myself when I want things to perform the best. I guess balance is key.

[Chris Behrens (cbehrens)]

https://review.openstack.org/#/c/78760/ fixed what we can for now. The rest requires API changes of some sort, which we'll have to do after Juno opens or whatever.

[aeva black (tenbrae)]

Untargeting from Icehouse as any further improvements will require API changes, which should be deferred to Juno at this point.

[Dmitry Tantsur (divius)]

Hi Chris! Are you still working on it or intend to work? If not, please unassign youself.

[Dmitry Tantsur (divius)]

Hi! As there was no new activity on this bug for a while, I'm unassigning it. Feel free to assign yourself, if you're still working on this bug.

[Dmitry Tantsur (divius)]

This bug should now be fixed in Nova, as Ironic driver now lives there

[Michael Davies (mrda)]

This was raised in http://eavesdrop.openstack.org/meetings/ironic/2015/ironic.2015-06-15-17.02.log.html and the ironic team don't see this as a major concern for now. Updating the priority back to wishlist.