Cannot specify subset of PCI devices for PCI passthrough

[Shesha Sreenivasamurthy]

I went through the recent code merge of PCI passthrough implementation. I see some drawbacks in the implementation and would be great improvement if the following can be accommodated.

1. Ability to specify a sub-set of PCI devices to be exported to Openstack via entries in nova.conf of the compute node.
Example: pci_passthrough_whitelist=[{"product_id":"1520", "vendor_id":"8086", "deviceids":"0000:06:00.0, 0000:06:00.1, 0000:06:00.2, 0000:06:00.3"}]

2. Group PCI devices, so that when creating flavor, we can choose, how many interfaces from each group needs to be presented to the guest. The group name can be specified in nova.conf of the control node if needed.

For example, if we have to SRIOV cards where each is connected physically to different networks and each has 32 Virtual functions, there is no way in current implementation to spin up a VM with one interface (VF) from each Group

Example:
pci_alias={"product_id":"1520", "name":"IN", "deviceids":"0000:06:00.0, 0000:06:00.2"}
pci_alias={"product_id":"1520", "name":"OUT", "deviceids":"0000:06:00.1, 0000:06:00.3"}

3. 'nova show' and 'nova hypervisor-stats' and 'nova hypervisor-show' shows (a) pci device associated with the VM, (b) shows how may groups are there and what is their current usage, and (c) show individual hypervisor PCI device usage respectively.

4. Some ordering involved which enables guest to predict which PCI device belongs to which network.
Example, in the alphabetical order of the network names ? With this, guest can predict that the first interface will be from IN-group and second will be from OUT-group.

Is there any chance that we can see these in Havana ?

We have an implementation that does these at: https://github.com/CiscoSystems/nova/tree/grizzly-multitool but is not based on current implementation.

Thanks,
Shesha

[Russell Bryant (russellb)]

It's too late for adding new features to Havana.

You may want to discuss these on openstack-dev. You'll probably get more attention to them that way.

[Yongli He (yongli-he)]

good ideas, seem we might had to discuss and push it in next cycle.

[Yongli He (yongli-he)]

i take it -:)

[Shesha Sreenivasamurthy (sheshas)]

Thanks for taking it.

[David Kang (dkang)]

It would be good to use "extra_info" field to store other generic information.
For example, some PCI device has its corresponding name in the file system(e.g. /dev/sda) or device name (e.g. eth0).
 LXC host device passthrough and PCI passthrough can have a lot in common.
Most of the work done for PCI passthrough can be used for LXC passthrough.
However, LXC host device passthrough needs device name.
Storing that information in "extra_info" field can make PCI passthrough scheme usable for LXC device passthrough.

[Yongli He (yongli-he)]

summit topic for pci passthrough next:

http://summit.openstack.org/cfp/details/129

[jiang, yunhong (yunhong-jiang)]

Hmm, LXC passthrough is something interesting. Will LXC passthrough will be PCI based? Currently create PCI specific table like BDF etc. Will it be ok for LXC passthrough?

[David Kang (dkang)]

 I'm not sure if I understand well what you mean by "Will LXC passthrough will be PCI based?".
What is "BDF"?

 I would say the PCI passthrough infrastructure can be used for LXC passthrough for PCI devices using extra_info field.
I believe that if extra_info field is flexible enough - such as json string -, supporting LXC host device passthrough for PCI device
is straightforward in most cases.

[Yi Liu (ryan-yi-liu)]

Hi, I also meet this issue...
I think to specify subset of PCI devices is a very common requirement. Is there time schedule for development of this feature?

[lvmxh (shaohef)]

For issue 1, I think it is already fixed.

I specify a sub-set of PCI devices and a single pci device in /etc/nova/nova.conf
$ cat /etc/nova/nova.conf |grep pci_passthrough_whitelist
pci_passthrough_whitelist= {"product_id":"10fb", "vendor_id":"8086", "deviceids":"0000:01:00.0, 0000:01:00.1"}
pci_passthrough_whitelist= {"product_id":"8c20", "vendor_id":"8086", "deviceids":"0000:00:1b.0"}

and I can see nova compute can parse the /etc/nova/nova.conf correctly.
mysql> SELECT * FROM nova.pci_devices;
+---------------------+------------+------------+---------+----+-----------------+--------------+------------+-----------+----------+------------------+-----------------+-----------+------------+---------------+------------+
| created_at | updated_at | deleted_at | deleted | id | compute_node_id | address | product_id | vendor_id | dev_type | dev_id | label | status | extra_info | instance_uuid | request_id |
+---------------------+------------+------------+---------+----+-----------------+--------------+------------+-----------+----------+------------------+-----------------+-----------+------------+---------------+------------+
| 2014-12-25 14:56:03 | NULL | NULL | 0 | 1 | 1 | 0000:01:00.0 | 10fb | 8086 | type-PCI | pci_0000_01_00_0 | label_8086_10fb | available | {} | NULL | NULL |
| 2014-12-25 14:56:03 | NULL | NULL | 0 | 2 | 1 | 0000:01:00.1 | 10fb | 8086 | type-PCI | pci_0000_01_00_1 | label_8086_10fb | available | {} | NULL | NULL |
| 2014-12-25 14:56:03 | NULL | NULL | 0 | 3 | 1 | 0000:00:1b.0 | 8c20 | 8086 | type-PCI | pci_0000_00_1b_0 | label_8086_8c20 | available | {} | NULL | NULL |
+---------------------+------------+------------+---------+----+-----------------+--------------+------------+-----------+----------+------------------+-----------------+-----------+------------+---------------+------------+
3 rows in set (0.00 sec)

here is the last commit of my nova source code.
$ git show
commit dcb3463f234ec1a6cc69de4d1ce6ec5ddfc0e698
Merge: 7441591 459ca56
Author: Jenkins <email address hidden>
Date: Fri Nov 28 12:02:09 2014 +0000

    Merge "Check server group policy on migrate/evacuate"

[lvmxh (shaohef)]

For issue one: you can see yongli's patch:
$ git show 5d383629571cf10fa94a879ba93e6d2ad9f4262a

it say:
The configuration option 'pci_passthrough_whitelist' can be defined multi times, each for one type of device.

As I difined above:
$ cat /etc/nova/nova.conf |grep pci_passthrough_whitelist
pci_passthrough_whitelist= {"product_id":"10fb", "vendor_id":"8086", "deviceids":"0000:01:00.0, 0000:01:00.1"}
pci_passthrough_whitelist= {"product_id":"8c20", "vendor_id":"8086", "deviceids":"0000:00:1b.0"}

but it does not support mutile items in one configuration option 'pci_passthrough_whitelist'.
pci_passthrough_whitelist=[{"product_id":"10fb", "vendor_id":"8086", "deviceids":"0000:01:00.0, 0000:01:00.1"}, {"product_id":"8c20", "vendor_id":"8086", "deviceids":"0000:00:1b.0"}]

I think you can define multi times. This is more readable than define all devices in one configuration option 'pci_passthrough_whitelist'.

Maybe extra tool will be needed to list pci device and to let users choose his pci device, and then configure the 'pci_passthrough_whitelist' option to cat /etc/nova/nova.conf automatically

[Markus Zoeller (markus_z) (mzoeller)]

This wishlist bug has been open a year without any activity. I'm going to move it to "Opinion / Wishlist", which is an easily-obtainable queue of older requests that have come on.

Feature requests for nova are done with blueprints [1] and with specs [2]. I'll recommend to read [3] if not yet done. The effort to implement the requested feature is then driven only by the blueprint (and spec).

References:
[1] https://blueprints.launchpad.net/nova/
[2] https://github.com/openstack/nova-specs
[3] https://wiki.openstack.org/wiki/Blueprints