Upon rebuild instances might never get to Active state
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
neutron |
Fix Released
|
High
|
Salvatore Orlando | ||
Havana |
Fix Released
|
High
|
Ihar Hrachyshka | ||
Icehouse |
Fix Released
|
High
|
Ihar Hrachyshka |
Bug Description
VMware mine sweeper for Neutron (*) recently showed a 100% failure rate on tempest.
Logs for two instances of these failures are available at [1] and [2]
The failure manifested as an instance unable to go active after a rebuild.
A bit of instrumentation and log analysis revealed no obvious error on the neutron side - and also that the instance was actually in "running" state even if its task state was "rebuilding/
N-API logs [3] revealed that the instance spawn was timing out on a missed notification from neutron regarding VIF plug - however the same log showed such notification was received [4]
It turns out that, after rebuild, the instance network cache had still 'active': False for the instance's VIF, even if the status for the corresponding port was 'ACTIVE'. This happened because after the network-vif-plugged event was received, nothing triggered a refresh of the instance network info. For this reason, the VM, after a rebuild, kept waiting for an even which obviously was never sent from neutron.
While this manifested only on mine sweeper - this appears to be a nova bug - manifesting in vmware minesweeper only because of the way the plugin synchronizes with the backend for reporting the operational status of a port.
A simple solution for this problem would be to reload the instance network info cache when network-vif-plugged events are received by nova. (But as the reporter knows nothing about nova this might be a very bad idea as well)
[1] http://
[2] http://
[3] http://
[4] http://
(*) runs libvirt/KVM + NSX
tags: | added: icehouse-backport-potential |
Changed in neutron: | |
milestone: | none → juno-2 |
importance: | Undecided → High |
Changed in neutron: | |
status: | Fix Committed → Fix Released |
tags: | removed: icehouse-backport-potential in-stable-icehouse |
Changed in neutron: | |
milestone: | juno-2 → 2014.2 |
description: | updated |
I'm not sure why gerrit did not update it. Commit msg looks fine.
https:/ /review. openstack. org/#/c/ 99182/