nova: SR-IOV for VF doesn't work

[Big Ben]

Hi,

I'm trying to enable SR-IOV VF in my kolla deployment, but it doesn't work. nova compute report following error message:

PortBindingFailed: Binding failed for port 859e46b7-f864-4768-8001-8001987444c7, please check neutron logs for more information.

but in neutron agent log there's nothing reported there. here below is the detailed nova-compute log during the VM spawning on compute node. how should I do to find out why this port binding fail here?

2017-01-12 11:08:18.783 7 INFO nova.compute.resource_tracker [req-1061e954-97ff-45d4-88ce-8e23ec628978 - - - - -] Compute_service record updated for kolla-com1:kolla-com1
2017-01-12 11:09:19.957 7 INFO nova.compute.resource_tracker [req-1061e954-97ff-45d4-88ce-8e23ec628978 - - - - -] Auditing locally available compute resources for node kolla-com1
2017-01-12 11:09:20.643 7 INFO nova.compute.resource_tracker [req-1061e954-97ff-45d4-88ce-8e23ec628978 - - - - -] Total usable vcpus: 56, total allocated vcpus: 0
2017-01-12 11:09:20.643 7 INFO nova.compute.resource_tracker [req-1061e954-97ff-45d4-88ce-8e23ec628978 - - - - -] Final resource view: name=kolla-com1 phys_ram=128889MB used_ram=512MB phys_disk=607GB used_disk=0GB total_vcpus=56 used_
vcpus=0 pci_stats=[PciDevicePool(count=1,numa_node=0,product_id='10ed',tags={dev_type='type-VF',physical_network='physnet1'},vendor_id='8086')]
2017-01-12 11:09:20.745 7 INFO nova.compute.resource_tracker [req-1061e954-97ff-45d4-88ce-8e23ec628978 - - - - -] Compute_service record updated for kolla-com1:kolla-com1
2017-01-12 11:10:01.389 7 INFO nova.compute.claims [req-c53cec5b-31da-4ac8-b6fa-4e2722f6737c a88269010813419abd5531f58cdebaf0 9a07609d8f16443a92c98582f2e646cc - - -] [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] Attempting claim: m
emory 2048 MB, disk 10 GB, vcpus 2 CPU
2017-01-12 11:10:01.390 7 INFO nova.compute.claims [req-c53cec5b-31da-4ac8-b6fa-4e2722f6737c a88269010813419abd5531f58cdebaf0 9a07609d8f16443a92c98582f2e646cc - - -] [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] Total memory: 12888
9 MB, used: 512.00 MB
2017-01-12 11:10:01.391 7 INFO nova.compute.claims [req-c53cec5b-31da-4ac8-b6fa-4e2722f6737c a88269010813419abd5531f58cdebaf0 9a07609d8f16443a92c98582f2e646cc - - -] [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] memory limit: 19333
3.50 MB, free: 192821.50 MB
2017-01-12 11:10:01.392 7 INFO nova.compute.claims [req-c53cec5b-31da-4ac8-b6fa-4e2722f6737c a88269010813419abd5531f58cdebaf0 9a07609d8f16443a92c98582f2e646cc - - -] [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] Total disk: 607 GB,
 used: 0.00 GB
2017-01-12 11:10:01.392 7 INFO nova.compute.claims [req-c53cec5b-31da-4ac8-b6fa-4e2722f6737c a88269010813419abd5531f58cdebaf0 9a07609d8f16443a92c98582f2e646cc - - -] [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] disk limit not spec
ified, defaulting to unlimited
2017-01-12 11:10:01.393 7 INFO nova.compute.claims [req-c53cec5b-31da-4ac8-b6fa-4e2722f6737c a88269010813419abd5531f58cdebaf0 9a07609d8f16443a92c98582f2e646cc - - -] [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] Total vcpu: 56 VCPU
, used: 0.00 VCPU
2017-01-12 11:10:01.394 7 INFO nova.compute.claims [req-c53cec5b-31da-4ac8-b6fa-4e2722f6737c a88269010813419abd5531f58cdebaf0 9a07609d8f16443a92c98582f2e646cc - - -] [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] vcpu limit: 896.00
VCPU, free: 896.00 VCPU
2017-01-12 11:10:01.398 7 INFO nova.compute.claims [req-c53cec5b-31da-4ac8-b6fa-4e2722f6737c a88269010813419abd5531f58cdebaf0 9a07609d8f16443a92c98582f2e646cc - - -] [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] Claim successful
2017-01-12 11:10:02.806 7 INFO nova.virt.libvirt.driver [req-c53cec5b-31da-4ac8-b6fa-4e2722f6737c a88269010813419abd5531f58cdebaf0 9a07609d8f16443a92c98582f2e646cc - - -] [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] Creating image
2017-01-12 11:10:02.866 7 ERROR nova.compute.manager [req-c53cec5b-31da-4ac8-b6fa-4e2722f6737c a88269010813419abd5531f58cdebaf0 9a07609d8f16443a92c98582f2e646cc - - -] Instance failed network setup after 1 attempt(s)
2017-01-12 11:10:02.866 7 ERROR nova.compute.manager Traceback (most recent call last):
2017-01-12 11:10:02.866 7 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1397, in _allocate_network_async
2017-01-12 11:10:02.866 7 ERROR nova.compute.manager bind_host_id=bind_host_id)
2017-01-12 11:10:02.866 7 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 881, in allocate_for_instance
2017-01-12 11:10:02.866 7 ERROR nova.compute.manager neutron, admin_client, requests_and_created_ports, nets,
2017-01-12 11:10:02.866 7 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1000, in _update_ports_for_instance
2017-01-12 11:10:02.866 7 ERROR nova.compute.manager for vif in created_vifs:
2017-01-12 11:10:02.866 7 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-12 11:10:02.866 7 ERROR nova.compute.manager self.force_reraise()
2017-01-12 11:10:02.866 7 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-12 11:10:02.866 7 ERROR nova.compute.manager six.reraise(self.type_, self.value, self.tb)
2017-01-12 11:10:02.866 7 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 970, in _update_ports_for_instance
2017-01-12 11:10:02.866 7 ERROR nova.compute.manager updated_port = self._update_port(
2017-01-12 11:10:02.866 7 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 437, in _update_port
2017-01-12 11:10:02.866 7 ERROR nova.compute.manager port = port_response['port']
2017-01-12 11:10:02.866 7 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 175, in _ensure_no_port_binding_failure
2017-01-12 11:10:02.866 7 ERROR nova.compute.manager raise exception.PortBindingFailed(port_id=port['id'])
2017-01-12 11:10:02.866 7 ERROR nova.compute.manager PortBindingFailed: Binding failed for port 859e46b7-f864-4768-8001-8001987444c7, please check neutron logs for more information.
2017-01-12 11:10:02.866 7 ERROR nova.compute.manager
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [req-c53cec5b-31da-4ac8-b6fa-4e2722f6737c a88269010813419abd5531f58cdebaf0 9a07609d8f16443a92c98582f2e646cc - - -] [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] Instance failed t
o spawn
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] Traceback (most recent call last):
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2078, in _build_resources
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] yield resources
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1920, in _build_and_run_instance
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] block_device_info=block_device_info)
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2579, in spawn
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] write_to_disk=True)
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4611, in _get_guest_xml
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] network_info_str = str(network_info)
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 525, in __str__
017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] return self._sync_wrapper(fn, *args, **kwargs)
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 508, in _sync_wrapper
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] self.wait()
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 540, in wait
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] self[:] = self._gt.wait()
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] return self._exit_event.wait()
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 125, in wait
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] current.throw(*self._exc)
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] result = function(*args, **kwargs)
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 1066, in context_wrapper
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] return func(*args, **kwargs)
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1414, in _allocate_network_async
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] six.reraise(*exc_info)
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1397, in _allocate_network_async
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] bind_host_id=bind_host_id)
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 881, in allocate_for_instance
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] neutron, admin_client, requests_and_created_ports, nets,
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1000, in _update_ports_for_instance
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] for vif in created_vifs:
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] self.force_reraise()
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] six.reraise(self.type_, self.value, self.tb)
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 970, in _update_ports_for_instance
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] updated_port = self._update_port(
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 437, in _update_port
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] port = port_response['port']
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 175, in _ensure_no_port_binding_failure
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] raise exception.PortBindingFailed(port_id=port['id'])
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] PortBindingFailed: Binding failed for port 859e46b7-f864-4768-8001-8001987444c7, please check neutron logs for more information.
2017-01-12 11:10:03.768 7 ERROR nova.compute.manager [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976]
2017-01-12 11:10:03.772 7 INFO nova.compute.manager [req-c53cec5b-31da-4ac8-b6fa-4e2722f6737c a88269010813419abd5531f58cdebaf0 9a07609d8f16443a92c98582f2e646cc - - -] [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] Terminating instan
ce
2017-01-12 11:10:03.778 7 INFO nova.virt.libvirt.driver [-] [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] During wait destroy, instance disappeared.
2017-01-12 11:10:03.808 7 INFO nova.virt.libvirt.driver [req-c53cec5b-31da-4ac8-b6fa-4e2722f6737c a88269010813419abd5531f58cdebaf0 9a07609d8f16443a92c98582f2e646cc - - -] [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] Deleting insta
nce files /var/lib/nova/instances/834e5d29-552f-4af2-86a1-3e93d4f06976_del
2017-01-12 11:10:03.810 7 INFO nova.virt.libvirt.driver [req-c53cec5b-31da-4ac8-b6fa-4e2722f6737c a88269010813419abd5531f58cdebaf0 9a07609d8f16443a92c98582f2e646cc - - -] [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] Deletion of /v
ar/lib/nova/instances/834e5d29-552f-4af2-86a1-3e93d4f06976_del complete
2017-01-12 11:10:04.109 7 INFO nova.compute.manager [req-c53cec5b-31da-4ac8-b6fa-4e2722f6737c a88269010813419abd5531f58cdebaf0 9a07609d8f16443a92c98582f2e646cc - - -] [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] Took 0.33 seconds
to destroy the instance on the hypervisor.
2017-01-12 11:10:04.498 7 INFO nova.compute.manager [req-c53cec5b-31da-4ac8-b6fa-4e2722f6737c a88269010813419abd5531f58cdebaf0 9a07609d8f16443a92c98582f2e646cc - - -] [instance: 834e5d29-552f-4af2-86a1-3e93d4f06976] Took 0.39 seconds
to deallocate network for instance.

And here is the configuration files on controller node:

(neutron-server)[neutron@kolla-cic neutron]$ cat /etc/neutron/plugins/ml2/ml2_conf.ini
[ml2]
type_drivers = flat,vlan,vxlan
tenant_network_types = vlan
mechanism_drivers = openvswitch,l2population, sriovnicswitch

[ml2_type_vlan]
network_vlan_ranges = physnet1:2000:2029

[ml2_type_flat]
flat_networks = physnet1

[ml2_type_vxlan]
vni_ranges = 1:1000
vxlan_group = 239.1.1.1

[securitygroup]
firewall_driver = neutron.agent.firewall.NoopFirewallDriver

[agent]
tunnel_types = vxlan
l2_population = true
arp_responder = true

[ovs]
bridge_mappings = physnet1:br-ex
..............................

(neutron-server)[neutron@kolla-cic neutron]$ cat /etc/neutron/plugins/ml2/ml2_conf_sriov.ini
[ml2_sriov]
agent_required = False
supported_pci_vendor_devs = 8086:10ed

(nova-api)[nova@kolla-cic /]$ cat /etc/nova/nova.conf
[DEFAULT]
..............
scheduler_available_filters = nova.scheduler.filters.all_filters
scheduler_default_filters = RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,PciPassthroughFilter

(neutron-server)[neutron@kolla-cic /]$ ps -feww
UID PID PPID C STIME TTY TIME CMD
neutron 1 0 0 10:26 ? 00:00:00 /usr/local/bin/dumb-init /bin/bash /usr/local/bin/kolla_start
neutron 6 1 0 10:26 ? 00:00:02 /usr/bin/python /usr/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini --config-file /etc/neutron/neutron_lbaas.conf --config-file /etc/neutron/neutron_vpnaas.conf
neutron 18 6 1 10:26 ? 00:01:01 /usr/bin/python /usr/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini --config-file /etc/neutron/neutron_lbaas.conf --config-file /etc/neutron/neutron_vpnaas.conf
neutron 19 6 1 10:26 ? 00:01:12 /usr/bin/python /usr/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini --config-file /etc/neutron/neutron_lbaas.conf --config-file /etc/neutron/neutron_vpnaas.conf
neutron 20 6 1 10:26 ? 00:01:05 /usr/bin/python /usr/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini --config-file /etc/neutron/neutron_lbaas.conf --config-file /etc/neutron/neutron_vpnaas.conf
neutron 21 6 0 10:26 ? 00:00:01 /usr/bin/python /usr/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini --config-file /etc/neutron/neutron_lbaas.conf --config-file /etc/neutron/neutron_vpnaas.conf
neutron 22 6 1 10:26 ? 00:01:09 /usr/bin/python /usr/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini --config-file /etc/neutron/neutron_lbaas.conf --config-file /etc/neutron/neutron_vpnaas.conf
neutron 23 6 1 10:26 ? 00:01:24 /usr/bin/python /usr/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini --config-file /etc/neutron/neutron_lbaas.conf --config-file /etc/neutron/neutron_vpnaas.conf
neutron 24 6 2 10:26 ? 00:01:25 /usr/bin/python /usr/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini --config-file /etc/neutron/neutron_lbaas.conf --config-file /etc/neutron/neutron_vpnaas.conf
neutron 25 6 1 10:26 ? 00:01:23 /usr/bin/python /usr/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini --config-file /etc/neutron/neutron_lbaas.conf --config-file /etc/neutron/neutron_vpnaas.conf

configuration files on compute node:

(nova-compute)[root@kolla-com1 neutron]# cat /etc/nova/nova.conf
[DEFAULT]
......................
firewall_driver = nova.virt.firewall.NoopFirewallDriver
scheduler_max_attempts = 10
linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver
allow_resize_to_same_host = true
compute_driver = libvirt.LibvirtDriver
....................................................
pci_passthrough_whitelist = [{"address":"*:03:10.0","physical_network":"physnet1"}]

(neutron-openvswitch-agent)[root@kolla-com1 /]# cat /etc/neutron/plugins/ml2/ml2_conf.ini
[ml2]
type_drivers = flat,vlan,vxlan
tenant_network_types = vlan
mechanism_drivers = openvswitch,l2population, sriovnicswitch

[ml2_type_vlan]
network_vlan_ranges = physnet1:2000:2029

[ml2_type_flat]
flat_networks = physnet1

[ml2_type_vxlan]
vni_ranges = 1:1000
vxlan_group = 239.1.1.1

[securitygroup]
firewall_driver = neutron.agent.firewall.NoopFirewallDriver

[agent]
tunnel_types = vxlan
l2_population = true
arp_responder = true

[ovs]
bridge_mappings = physnet1:br-ex
..............................................

(neutron-openvswitch-agent)[root@kolla-com1 /]# cat /etc/neutron/plugins/ml2/ml2_conf.ini
[ml2]
type_drivers = flat,vlan,vxlan
tenant_network_types = vlan
mechanism_drivers = openvswitch,l2population, sriovnicswitch

[ml2_type_vlan]
network_vlan_ranges = physnet1:2000:2029

[ml2_type_flat]
flat_networks = physnet1

[ml2_type_vxlan]
vni_ranges = 1:1000
vxlan_group = 239.1.1.1

[securitygroup]
firewall_driver = neutron.agent.firewall.NoopFirewallDriver

[agent]
tunnel_types = vxlan
l2_population = true
arp_responder = true

[ovs]
bridge_mappings = physnet1:br-ex
........................................

(neutron-openvswitch-agent)[root@kolla-com1 /]# ps -ef
UID PID PPID C STIME TTY TIME CMD
neutron 1 0 0 10:26 ? 00:00:00 /usr/local/bin/dumb-init /bin/bash /usr/local/bin/kolla_start
neutron 6 1 2 10:26 ? 00:01:54 /usr/bin/python /usr/bin/neutron-openvswitch-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini
root 20 6 0 10:26 ? 00:00:00 sudo neutron-rootwrap /etc/neutron/rootwrap.conf ovsdb-client monitor Interface name,ofport,external_ids --format=json
root 22 20 0 10:26 ? 00:00:00 /usr/bin/python /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovsdb-client monitor Interface name,ofport,external_ids --format=json
root 52 22 0 10:26 ? 00:00:00 /usr/bin/ovsdb-client monitor Interface name,ofport,external_ids --format=json
root 4047 0 0 11:32 ? 00:00:00 bash
root 4107 4047 0 11:33 ? 00:00:00 ps -ef

root@kolla-com1:~/kolla# lspci -nnvvv | grep -i "virtual func"
03:10.0 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
        Subsystem: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:7b11]
03:10.1 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
        Subsystem: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:7b11]

[Eduardo Gonzalez (egonzalez90)]

At the first glance, I'm seeing that some options in both nova and neutron are missing.
For example, configure scheduler filters and sr_iov config in ml2.
Please review configuration options and host setup for sr-iov [0].
As kolla does not currently support sr-iov capabilities, will need to use customized files and also modify how neutron services start to make use of sr_iov config file.

http://docs.openstack.org/newton/networking-guide/config-sriov.html

[Big Ben (bigben80)]

Dear Eduardo,

Thank you for looking into this report.

The scheduler filters I put in the nova.conf file on the controller, as described in the openstack sriov document recommended.

(nova-api)[nova@kolla-cic /]$ cat /etc/nova/nova.conf
[DEFAULT]
..............
scheduler_available_filters = nova.scheduler.filters.all_filters
scheduler_default_filters = RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,PciPassthroughFilter

I used ml2_conf_sriov.conf file as the openstack sriov document recommended, it looks like below

(neutron-server)[neutron@kolla-cic neutron]$ cat /etc/neutron/plugins/ml2/ml2_conf_sriov.ini
[ml2_sriov]
agent_required = False
supported_pci_vendor_devs = 8086:10ed

And kolla defaultly will not start neutron-server with ml2_conf_sriov.ini, so I modified the json file for kolla_start and make this file be included when neutron-server start. on the neutron-server docker processes are like these below:

neutron-server)[neutron@kolla-cic /]$ ps -feww
UID PID PPID C STIME TTY TIME CMD
neutron 1 0 0 10:26 ? 00:00:00 /usr/local/bin/dumb-init /bin/bash /usr/local/bin/kolla_start
neutron 6 1 0 10:26 ? 00:00:02 /usr/bin/python /usr/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini --config-file /etc/neutron/neutron_lbaas.conf --config-file /etc/neutron/neutron_vpnaas.conf
..................................................

[Big Ben (bigben80)]

Hi,

I'm wondering if the information in this ticket is clear...

so in general, I configured the nova and neutron according to document>

http://docs.openstack.org/newton/networking-guide/config-sriov.html

to make neutron server start with configuration file "ml2_conf_sriov.ini", I changed the json file for neutron server kolla_start, and include this "ml2_conf_sriov.ini" in the start up command line.

after the modification, neutron server start with option "--config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini". but I still see this PortBindingFailed problem when trying to boot a VM with port which have "direct" type.

[Chris Nipper (cnipp)]

Neutron SRIOV requirements have changed since this bug was created. The Neutron SRIOV agent is now a requirement.

Attached is a patch for kolla-ansible that adds support for the SRIOV agent container (this was built not too long ago but support has not been added yet). Along with the current Ocata SRIOV documentation (https://docs.openstack.org/ocata/networking-guide/config-sriov.html), I was able to get Neutron SRIOV working correctly with Kolla.

[Big Ben (bigben80)]

Very good job! Thank you
Chris <email address hidden>于2017年7月27日 周四上午12:26写道：

> Neutron SRIOV requirements have changed since this bug was created. The
> Neutron SRIOV agent is now a requirement.
>
> Attached is a patch for kolla-ansible that adds support for the SRIOV
> agent container (this was built not too long ago but support has not
> been added yet). Along with the current Ocata SRIOV documentation
> (https://docs.openstack.org/ocata/networking-guide/config-sriov.html), I
> was able to get Neutron SRIOV working correctly with Kolla.
>
>
> ** Patch added: "sriov.patch"
>
> https://bugs.launchpad.net/kolla/+bug/1655921/+attachment/4922033/+files/sriov.patch
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1655921
>
> Title:
> nova: SR-IOV for VF doesn't work
>
> Status in kolla:
> New
>
> Bug description:
> Hi,
>
> I'm trying to enable SR-IOV VF in my kolla deployment, but it doesn't
> work. nova compute report following error message:
>
> PortBindingFailed: Binding failed for port
> 859e46b7-f864-4768-8001-8001987444c7, please check neutron logs for
> more information.
>
> but in neutron agent log there's nothing reported there. here below is
> the detailed nova-compute log during the VM spawning on compute node.
> how should I do to find out why this port binding fail here?
>
>
> 2017-01-12 11:08:18.783 7 INFO nova.compute.resource_tracker
> [req-1061e954-97ff-45d4-88ce-8e23ec628978 - - - - -] Compute_service record
> updated for kolla-com1:kolla-com1
> 2017-01-12 11:09:19.957 7 INFO nova.compute.resource_tracker
> [req-1061e954-97ff-45d4-88ce-8e23ec628978 - - - - -] Auditing locally
> available compute resources for node kolla-com1
> 2017-01-12 11:09:20.643 7 INFO nova.compute.resource_tracker
> [req-1061e954-97ff-45d4-88ce-8e23ec628978 - - - - -] Total usable vcpus:
> 56, total allocated vcpus: 0
> 2017-01-12 11:09:20.643 7 INFO nova.compute.resource_tracker
> [req-1061e954-97ff-45d4-88ce-8e23ec628978 - - - - -] Final resource view:
> name=kolla-com1 phys_ram=128889MB used_ram=512MB phys_disk=607GB
> used_disk=0GB total_vcpus=56 used_
> vcpus=0
> pci_stats=[PciDevicePool(count=1,numa_node=0,product_id='10ed',tags={dev_type='type-VF',physical_network='physnet1'},vendor_id='8086')]
> 2017-01-12 11:09:20.745 7 INFO nova.compute.resource_tracker
> [req-1061e954-97ff-45d4-88ce-8e23ec628978 - - - - -] Compute_service record
> updated for kolla-com1:kolla-com1
> 2017-01-12 11:10:01.389 7 INFO nova.compute.claims
> [req-c53cec5b-31da-4ac8-b6fa-4e2722f6737c a88269010813419abd5531f58cdebaf0
> 9a07609d8f16443a92c98582f2e646cc - - -] [instance:
> 834e5d29-552f-4af2-86a1-3e93d4f06976] Attempting claim: m
> emory 2048 MB, disk 10 GB, vcpus 2 CPU
> 2017-01-12 11:10:01.390 7 INFO nova.compute.claims
> [req-c53cec5b-31da-4ac8-b6fa-4e2722f6737c a88269010813419abd5531f58cdebaf0
> 9a07609d8f16443a92c98582f2e646cc - - -] [instance:
> 834e5d29-552f-4af2-86a1-3e93d4f06976] Total memory: 12888
> 9 MB, used: 512.00 MB
> 2017-01-12 11:10:01.391 7 INFO nova.compute.claims
> [req-c53cec5b-31da-4ac8-b6fa-4e2722f6737c a88269010813419abd5...

[Charlie Kang (charlie-kang)]

What are the next steps to do to see if we can get this merged in?

Thanks

[Charlie Kang (charlie-kang)]

https://review.openstack.org/#/c/487610/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/487610
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=6bb27ac06f10241121380748d3097bc9a27ad36f
Submitter: Zuul
Branch: master

commit 6bb27ac06f10241121380748d3097bc9a27ad36f
Author: Christopher Nipper <email address hidden>
Date: Wed Jul 26 15:00:20 2017 -0700

    Add support for Neutron SRIOV agent container

    Agent is required for Neutron SRIOV post Mitaka
    Closes-Bug: #1655921

    Addressed comments
    Added release notes

    Change-Id: Ib8181c4da65fccd0c5f8487217060e5682977665
    Signed-off-by: Christopher Nipper <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 6.0.0.0b3 development milestone.