Boot from volumes that fail in initialize_connection are not rescheduled

[Samuel Matzek]

Version: OpenStack Liberty

Boot from volumes that fail in volume initialize_connection are not rescheduled. Initialize connection failures can be very host-specific and in many cases the boot would succeed if the instance build was rescheduled to another host.

The instance is not rescheduled because the initialize_connection is being called down this stack:
nova.compute.manager _build_resources
nova.compute.manager _prep_block_device
nova.virt.block_device attach_block_devices
nova.virt.block_device.DriverVolumeBlockDevice.attach

When this fails an exception is thrown which lands in this block:
https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1740
and throws an InvalidBDM exception which is caught by this block:
https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2110

this in turn throws a BuildAbortException which causes the instance to not be rescheduled by landing the flow in this block:
https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2004

To fix this we likely need a different exception thrown from nova.virt.block_device.DriverVolumeBlockDevice.attach when the failure is in initialize_connection and then work back up the stack to ensure that when this different exception is thrown a BuildAbortException is not thrown so the reschedule can happen.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/246505

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/246505
Reason: This review is > 6 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Maciej Szankin (mszankin)]

Liberty has hit the EOL, so this one is invalid. Mitaka was removed from affected releases, so I am closing this one.

[Matt Riedemann (mriedem)]

I wouldn't say that we won't ever fix this, since I've wondered why we don't reschedule on volume failures like we do with networking failures, but it's not a high priority.