In case of compute reboot the evacuation of a SR-IOV VM can fail

[Anton Rodionov]

In case of compute reboot the evacuation of an SR-IOV VM with ha-offline metadata fails if:
- on the destination compute the PCI address used by the SR-IOV VM is already in use.
- on the destination compute the PCI address used by the SR-IOV VM does not exist.

I tried to run the same test executing via CLI the forcemove and migrate commands. In this case nova is able to write in the XML an available PCI address on the destination compute so the evacuation/migration is successfully completed.

At the same time, if a compute reboot is done, the VM will go in error state if one of the two conditions above are fulfilled. Extract from nova-show:
| fault | {"message": "Requested operation is not valid: PCI device 0000:3b:0a.5 is in use by driver QEMU, domain instance-000024c9", "code": 500, "details": " File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 375, in decorated_function |

I collected the shell log of these operations in the attached log file. It is a bit long but there is everything there.

from less /var/log/libvirt/libvirtd.log in case the PCI address does not exist on the destination compute:
2017-11-15 11:22:04.121+0000: 23297: error : virPCIDeviceNew:1596 : Device 0000:5e:02.4 not found: could not access /sys/bus/pci/devices/0000:5e:02.4/config: No such file or directory
2017-11-15 11:22:04.121+0000: 23297: error : virPCIDeviceNew:1596 : Device 0000:5e:02.4 not found: could not access /sys/bus/pci/devices/0000:5e:02.4/config: No such file or directory

Error in nova-compute if the PCI address is already in use:
Requested operation is not valid: PCI device 0000:3b:0a.5 is in use by driver QEMU, domain instance-000024c9

MOS version: MOS 9.2+ code from stable branch from September 2017
Steps to reproduce: attached as a file - all the executed commands are included.
The issue is reproducible all the time.

[Vladyslav Drok (vdrok)]

OK so what does the reboot has to do with this? Looking at the logs, evacuation fails prior to the reboot right?

[Anton Rodionov (arodionov)]

Here is an explanation from the customer:
In our nodes reboot itself means evcauate with target host. So, execution.log contains both scenarios.
Scenario-1: Evacuate without target host via "nova evacuate <VM name>"
Scenario-2: Reboot which means evacuate with targhet host via "nova evacuate <VM name> <Target host>"
This Bug is raised for addressing both the scenarios.

[Alexander Rubtsov (arubtsov)]

The customer has confirmed that the patch successfully passed the tests.
Can you merge the corresponding commit: https://review.fuel-infra.org/#/c/37448/?

[Vladimir Jigulin (vjigulin)]

Closing the bug because we have a confirmation (#4) that the path works