live migration generates several network-changed events which lock up refreshing the nw info cache
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
In Progress
|
Medium
|
Matt Riedemann | ||
Queens |
In Progress
|
Medium
|
Matt Riedemann | ||
neutron |
Fix Released
|
Undecided
|
Matt Riedemann |
Bug Description
Chris Friesen has reported that in Newton with a live migration that has ~16 ports per instance, the "network-changed" events generated from neutron when the vifs are unplugged from the source host can effectively block the network info cache refresh that's called at the end of the live migration operation. Details are in the IRC logs:
But this stands out:
cfriesen mriedem: so it looks like _build_
In Nova the 'network-changed' event is handled generically because there is no port id in the event, so nova just refreshes the entire nw info cache on the instance - which can be expensive and redundant since it's doing a lot of queries to Neutron to build up information about ports, fixed IPs, floating IPs, subnets and networks, and Neutron doesn't have bulk query APIs or allow OR filters in the API for bulk queries on things like floating IPs.
Looking in neutron's code that sends the network-changed event, there is a port in scope, it's just not sent like for network-vif-deleted events.
We should be able to scope the network-changed event to a specific port on the neutron side and check for that on the nova side so we don't have to refresh the entire network info cache, but just the vif that was updated.
tags: | added: neutron |
Changed in neutron: | |
assignee: | nobody → Matt Riedemann (mriedem) |
Changed in nova: | |
assignee: | nobody → Matt Riedemann (mriedem) |
tags: | added: neutron-proactive-backport-potential |
tags: | removed: neutron-proactive-backport-potential |
Changed in nova: | |
importance: | Undecided → Medium |
The messy thing with this is going to be the existing nw info cache refresh code in nova doesn't know how to handle a single port id, if it's given that it expects it's from attaching a port to an instance.