nova migration a vm with pci devices to a host don't have pci available

[Yongli He]

on nova master, multinodes with 2 compute nodes available. only one of nova compute had pci devices available.

1. booting vm with pci reqested, assume it's name: pci-vm
2. nova migrate pci-vm
3. the pci-vm been schedule to a host don't have pci devices available.

we see exception on nova compute(the host don't have pci devs and which nova schedule the pci-vm to):

exception type 1:
env/local/lib/python2.7/site-packages/nova/pci/request.py", line 157, in _translate_alias_to_requests
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server raise exception.PciRequestAliasNotDefined(alias=name)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server
PciRequestAliasNotDefined: PCI alias intelpci is not defined

exception type2: (define pci configurations but no pci device available )

2018-07-22 23:10:16.528 7 ERROR oslo_messaging.rpc.server overhead=overhead, limits=limits)
2018-07-22 23:10:16.528 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/claims.py", line 95, in __init__
2018-07-22 23:10:16.528 7 ERROR oslo_messaging.rpc.server self._claim_test(resources, limits)
2018-07-22 23:10:16.528 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/claims.py", line 162, in _claim_test
2018-07-22 23:10:16.528 7 ERROR oslo_messaging.rpc.server "; ".join(reasons))
2018-07-22 23:10:16.528 7 ERROR oslo_messaging.rpc.server ComputeResourcesUnavailable: Insufficient compute resources: Claim pci failed.

[Feng Wen (cnwenf)]

Have you configured PciPassthroughFilter in schduler.enabled_filters? The default is not configured this filter, if you need PCI scheduling, you should configure it.

[Yongli He (yongli-he)]

@Feng Wen, i did. but thanks for remind, i do forgot include pci configuration in the conductor service. the failure information updated.

the problem still exist. the problem is while do resize/migration, API layer not load PCI information into instance object. i'd gonna to load PCI information while do migration to resolve this problem, i think.

[Matt Riedemann (mriedem)]

> the problem is while do resize/migration, API layer not load PCI information into instance object.

I'm not sure what you mean here. The PciPassthroughFilter filter looks at the RequestSpec.pci_requests which are stored when the instance is originally created:

https://github.com/openstack/nova/blob/58aed34f520d872855d23254a30266ebc2ca9953/nova/compute/api.py#L886

Can you check the nova_api.request_specs table for the instance in question and verify the pci_requests field is stored in the request spec json blob?

The other thing would be to check the pci_stats for the compute node in the compute_nodes table which the filter is looking at:

https://github.com/openstack/nova/blob/58aed34f520d872855d23254a30266ebc2ca9953/nova/scheduler/filters/pci_passthrough_filter.py#L50

Is the compute reporting pci_stats and if so, do they match what's in the RequestSpec.pci_requests?

[Chris Friesen (cbf123)]

Does this fix the behaviour? http://paste.openstack.org/show/726559/

[Chris Friesen (cbf123)]

I should explain that the above is a fix from some time ago, so I'm curious if it's still applicable now.

[Yongli He (yongli-he)]

@Chris Friesen i think this would fix this. it's what i'm talk about.

@Matt Riedemann there is missing expected_attrs = ['pci_requests'], just like

http://paste.openstack.org/show/726559/ , i'm gonna verify this patch.

[Yongli He (yongli-he)]

this patch does fix the problem:
http://paste.openstack.org/show/726559/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/588455

[Matt Riedemann (mriedem)]

This still doesn't make sense to me. The API looks up the request spec which should have the pci_requests in it - does the request spec exists for this instance?

If the request spec doesn't exist, conductor will create a request spec from the instance which will lazy-load instance.pci_requests - is that load not happening and if not, why not? Is the data in the instance_extra table?

[Yongli He (yongli-he)]

>>pci_requests in it - does the request spec exists for this instance?
from migration API, it does not.

>>If the request spec doesn't exist, conductor will create a request spec from the instance which >>will lazy-load instance.pci_requests - is that load not happening and if not, why not? Is the
here is the problem, by default, instance.pci_requests did not loaded. i paste the code here:
objects/instance.py

    @staticmethod
    def _from_db_object(context, instance, db_inst, expected_attrs=None):
     .....
        if 'pci_devices' in expected_attrs:
            pci_devices = base.obj_make_list(
                    context, objects.PciDeviceList(context),
                    objects.PciDevice, db_inst['pci_devices'])

[Yongli He (yongli-he)]

maybe following 2 lines code need be change in some way:

https://github.com/openstack/nova/blob/681cb7f21f2ecab7b6c0451b746ab6f45c5a515c/nova/api/openstack/compute/migrate_server.py#L53

way before the call to the compute node's command, the migrate API create a instance from compute_api.get, which is not the conductor logic:

https://github.com/openstack/nova/blob/681cb7f21f2ecab7b6c0451b746ab6f45c5a515c/nova/compute/api.py#L2388

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.opendev.org/588455
Reason: No updates on the review in a long time so I'm going to abandon this.

[Matt Riedemann (mriedem)]

We don't have a clear reproducer of the issue so we don't really know what's going on here, and haven't had any replies in a long time so I'm going to mark this as invalid.

[Jacolex (jacolex)]

The same story - pci device on one host in the AZ. Create instance with flavor containing pci_passthrough property working good. But when resizing instance to that flavor - seems the pcipassthroughfilter not working in that case and some other host throws error nova.exception.PciRequestAliasNotDefined: PCI alias nvme is not defined.

[Ivan Borisov (ivan.borisov)]

I can confirm that the issue very similar to described above still persist in nova 20.0.0.

Steps to reproduce:
1. Configure openstack services as described in https://docs.openstack.org/nova/pike/admin/pci-passthrough.html
2. Configure multiple pci passthrough aliases for different pci devices, say GPU_A and GPU_B
3. Create flavors flavor_A and flavor_B, each requesting specific pci passthrough alias via flavor metadata.
3. Create instance from flavor_A (with GPU_A)
4. Try to resize instance to flavor_b (with GPU_B)

Expected result:
Resize happens as usual, can be finished successfully.

Actual result:
During instance scheduling nova-scheduler seems to ignore pci_passthrough:alias in flavor_B metadata and treat instance as if it didn't have pci passthrough request at all.
As result, instance gets scheduled to host that doesn't have GPU_B devices and resize fails with error:
Insufficient compute resources: Claim pci failed.

Not sure whether I should provide any additional info.

[Ivan Borisov (ivan.borisov)]

Since it looks like problem has not been resolved, and, I believe, has been confirmed by multiple operators, I'll change status to "Confirmed" for now.

[sean mooney (sean-k-mooney)]

setting this back to invalid.

there is a seperate bug for resize

https://bugs.launchpad.net/nova/+bug/1805969

this was previously marked as invalid because there was no clear repoduce or the issue and we have other bug tracking various edge case so let not keep this open as that has not been adressed

[Pablo (pescobar001)]

We are hitting this problem with the PciPassthroughFilter when we try to resize a VM from a flavor without pci_passthrough to a flavor containing pci_passthrough

When we create a VM from scratch using a flavor containing pci_passthrough we see this in the nova-scheduler logs and the machines is created without problems:

Filter PciPassthroughFilter returned 2 host(s) get_filtered_objects

If we create a VM using a flavor without pci passthrough and we resize to a flavor with pci passthrough we see this in the nova-scheduler logs

Filter PciPassthroughFilter returned 14 host(s) get_filtered_objects

The resized VM is scheduled to a wrong hypervisor without the required GPU card and the VM ends in error state "Insufficient compute resources: Claim pci failed."

Debugging the problem we tried to stop the nova-compute service in every hypervisor except those two having the right GPU cards and then the resize worked.

The problems seems to be that nova-scheduler doesn't apply the PciPassthroughFilter properly when resizing a VM.

We are running Wallaby with nova 23.1.1.dev6

[Pablo (pescobar001)]

https://review.opendev.org/c/openstack/nova/+/853274