os-vif

get_ifname_by_pci_address failed

Bug #1921631 reported by Erik Ilavsky
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
os-vif
New
Undecided
Unassigned

Bug Description

Hi, ubuntu 20.04 amd64 5.4.0-70-generic,
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt probe_vf=0 apparmor=0 selinux=0

root@hp1111:/home/ubuntu# ls -l /sys/class/net | grep 05:10.0
lrwxrwxrwx 1 root root 0 Mar 28 09:42 enp5s0f0v126 -> ../../devices/pci0000:00/0000:00:02.0/0000:05:10.0/net/enp5s0f0v126

root@hp1111:/home/ubuntu# ip link show enp5s0f0v126
132: enp5s0f0v126: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 66:2a:39:9e:e5:16 brd ff:ff:ff:ff:ff:ff

  nova-compute:
    charm: cs:nova-compute
    bindings:
      "": *internal-space
      internal: *internal-space
    options:
      config-flags: default_ephemeral_format=ext4
      enable-live-migration: true
      enable-resize: true
      migration-auth-type: ssh
      openstack-origin: *openstack-origin
      ceph-osd-replication-count: *ceph-osd-replication-count
      libvirt-image-backend: *libvirt-image-backend
      verbose: *verbose
      use-internal-endpoints: True
      restrict-ceph-pools: False
      flat-interface: physnet1 physnet2 prov_sriov_1
      virt-type: kvm
      hugepages: 60%
      reserved-host-memory: 16384
      pci-passthrough-whitelist: '{ "vendor_id": "15b3", "product_id": "1016", "physical_network": "prov_sriov_1" }'

2021-03-28 15:13:58.256 465334 DEBUG nova.network.os_vif_util [req-306e2d84-8d39-4e08-8df2-ba87baa323ae aa48b560d18541489031f10410fe33c7 80088d4e3eea4f90958175e58eb7ddbe - c35b9b534cf14b36802207ec524932e0 c35b9b534cf14b36802207ec524932e0] Converting VIF {"id": "9f7aae43-8e80-44fb-b626-f021423e6063", "address": "fa:16:3e:10:ae:3a", "network": {"id": "c5ffad64-b4aa-4925-8c6a-31c35a8b8b5c", "bridge": "br-int", "label": "ext_net2", "subnets": [{"cidr": "192.0.3.0/24", "dns": [], "gateway": {"address": "192.0.3.1", "type": "gateway", "version": 4, "meta": {}}, "ips": [{"address": "192.0.3.50", "type": "fixed", "version": 4, "meta": {}, "floating_ips": []}], "routes": [], "version": 4, "meta": {"dhcp_server": "192.0.3.1"}}], "meta": {"injected": false, "tenant_id": "80088d4e3eea4f90958175e58eb7ddbe", "mtu": 1500, "physical_network": "prov_sriov_1", "tunneled": false}}, "type": "ovs", "details": {"port_filter": true}, "devname": "tap9f7aae43-8e", "ovs_interfaceid": "9f7aae43-8e80-44fb-b626-f021423e6063", "qbh_params": null, "qbg_params": null, "active": false, "vnic_type": "direct", "profile": {"capabilities": ["switchdev"], "physical_network": "prov_sriov_1", "pci_vendor_info": "15b3:1016", "pci_slot": "0000:05:10.0"}, "preserve_on_delete": true, "meta": {}} nova_to_osvif_vif /usr/lib/python3/dist-packages/nova/network/os_vif_util.py:501
2021-03-28 15:13:58.257 465334 DEBUG nova.network.os_vif_util [req-306e2d84-8d39-4e08-8df2-ba87baa323ae aa48b560d18541489031f10410fe33c7 80088d4e3eea4f90958175e58eb7ddbe - c35b9b534cf14b36802207ec524932e0 c35b9b534cf14b36802207ec524932e0] Converted object VIFHostDevice(active=False,address=fa:16:3e:10:ae:3a,dev_address=0000:05:10.0,dev_type='ethernet',has_traffic_filtering=True,id=9f7aae43-8e80-44fb-b626-f021423e6063,network=Network(c5ffad64-b4aa-4925-8c6a-31c35a8b8b5c),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True) nova_to_osvif_vif /usr/lib/python3/dist-packages/nova/network/os_vif_util.py:538
2021-03-28 15:13:58.258 465334 DEBUG os_vif [req-306e2d84-8d39-4e08-8df2-ba87baa323ae aa48b560d18541489031f10410fe33c7 80088d4e3eea4f90958175e58eb7ddbe - c35b9b534cf14b36802207ec524932e0 c35b9b534cf14b36802207ec524932e0] Unplugging vif VIFHostDevice(active=False,address=fa:16:3e:10:ae:3a,dev_address=0000:05:10.0,dev_type='ethernet',has_traffic_filtering=True,id=9f7aae43-8e80-44fb-b626-f021423e6063,network=Network(c5ffad64-b4aa-4925-8c6a-31c35a8b8b5c),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True) unplug /usr/lib/python3/dist-packages/os_vif/__init__.py:109
2021-03-28 15:13:58.258 465334 ERROR os_vif [req-306e2d84-8d39-4e08-8df2-ba87baa323ae aa48b560d18541489031f10410fe33c7 80088d4e3eea4f90958175e58eb7ddbe - c35b9b534cf14b36802207ec524932e0 c35b9b534cf14b36802207ec524932e0] Failed to unplug vif VIFHostDevice(active=False,address=fa:16:3e:10:ae:3a,dev_address=0000:05:10.0,dev_type='ethernet',has_traffic_filtering=True,id=9f7aae43-8e80-44fb-b626-f021423e6063,network=Network(c5ffad64-b4aa-4925-8c6a-31c35a8b8b5c),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True): vif_plug_ovs.exception.PciDeviceNotFoundById: PCI device 0000:05:10.0 not found
2021-03-28 15:13:58.258 465334 ERROR os_vif Traceback (most recent call last):
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 77, in plug
2021-03-28 15:13:58.258 465334 ERROR os_vif plugin.plug(vif, instance_info)
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 305, in plug
2021-03-28 15:13:58.258 465334 ERROR os_vif self._plug_vf(vif, instance_info)
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 269, in _plug_vf
2021-03-28 15:13:58.258 465334 ERROR os_vif pf_ifname = linux_net.get_ifname_by_pci_address(
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/linux_net.py", line 355, in get_ifname_by_pci_address
2021-03-28 15:13:58.258 465334 ERROR os_vif raise exception.PciDeviceNotFoundById(id=pci_addr)
2021-03-28 15:13:58.258 465334 ERROR os_vif vif_plug_ovs.exception.PciDeviceNotFoundById: PCI device 0000:05:10.0 not found
2021-03-28 15:13:58.258 465334 ERROR os_vif
2021-03-28 15:13:58.258 465334 ERROR os_vif During handling of the above exception, another exception occurred:
2021-03-28 15:13:58.258 465334 ERROR os_vif
2021-03-28 15:13:58.258 465334 ERROR os_vif Traceback (most recent call last):
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 680, in _plug_os_vif
2021-03-28 15:13:58.258 465334 ERROR os_vif os_vif.plug(vif, instance_info)
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 82, in plug
2021-03-28 15:13:58.258 465334 ERROR os_vif raise os_vif.exception.PlugException(vif=vif, err=err)
2021-03-28 15:13:58.258 465334 ERROR os_vif os_vif.exception.PlugException: Failed to plug VIF VIFHostDevice(active=False,address=fa:16:3e:10:ae:3a,dev_address=0000:05:10.0,dev_type='ethernet',has_traffic_filtering=True,id=9f7aae43-8e80-44fb-b626-f021423e6063,network=Network(c5ffad64-b4aa-4925-8c6a-31c35a8b8b5c),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: PCI device 0000:05:10.0 not found

2021-03-28 15:13:58.258 465334 ERROR os_vif
2021-03-28 15:13:58.258 465334 ERROR os_vif During handling of the above exception, another exception occurred:
2021-03-28 15:13:58.258 465334 ERROR os_vif
2021-03-28 15:13:58.258 465334 ERROR os_vif Traceback (most recent call last):
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2392, in _build_and_run_instance
2021-03-28 15:13:58.258 465334 ERROR os_vif self.driver.spawn(context, instance, image_meta,
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3743, in spawn
2021-03-28 15:13:58.258 465334 ERROR os_vif self._create_guest_with_network(
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6735, in _create_guest_with_network
2021-03-28 15:13:58.258 465334 ERROR os_vif self._cleanup_failed_start(
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-28 15:13:58.258 465334 ERROR os_vif self.force_reraise()
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-28 15:13:58.258 465334 ERROR os_vif six.reraise(self.type_, self.value, self.tb)
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-03-28 15:13:58.258 465334 ERROR os_vif raise value
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6700, in _create_guest_with_network
2021-03-28 15:13:58.258 465334 ERROR os_vif self.plug_vifs(instance, network_info)
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1169, in plug_vifs
2021-03-28 15:13:58.258 465334 ERROR os_vif self.vif_driver.plug(instance, vif)
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 704, in plug
2021-03-28 15:13:58.258 465334 ERROR os_vif self._plug_os_vif(instance, vif_obj)
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 684, in _plug_os_vif
2021-03-28 15:13:58.258 465334 ERROR os_vif raise exception.InternalError(msg)
2021-03-28 15:13:58.258 465334 ERROR os_vif nova.exception.InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFHostDevice(active=False,address=fa:16:3e:10:ae:3a,dev_address=0000:05:10.0,dev_type='ethernet',has_traffic_filtering=True,id=9f7aae43-8e80-44fb-b626-f021423e6063,network=Network(c5ffad64-b4aa-4925-8c6a-31c35a8b8b5c),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: PCI device 0000:05:10.0 not found
2021-03-28 15:13:58.258 465334 ERROR os_vif
2021-03-28 15:13:58.258 465334 ERROR os_vif During handling of the above exception, another exception occurred:
2021-03-28 15:13:58.258 465334 ERROR os_vif
2021-03-28 15:13:58.258 465334 ERROR os_vif Traceback (most recent call last):
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2218, in _do_build_and_run_instance
2021-03-28 15:13:58.258 465334 ERROR os_vif self._build_and_run_instance(context, instance, image,
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2488, in _build_and_run_instance
2021-03-28 15:13:58.258 465334 ERROR os_vif raise exception.RescheduledException(
2021-03-28 15:13:58.258 465334 ERROR os_vif nova.exception.RescheduledException: Build of instance bd1a7d09-cf67-406c-8d0a-92256eb1daf8 was re-scheduled: Failure running os_vif plugin plug method: Failed to plug VIF VIFHostDevice(active=False,address=fa:16:3e:10:ae:3a,dev_address=0000:05:10.0,dev_type='ethernet',has_traffic_filtering=True,id=9f7aae43-8e80-44fb-b626-f021423e6063,network=Network(c5ffad64-b4aa-4925-8c6a-31c35a8b8b5c),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: PCI device 0000:05:10.0 not found
2021-03-28 15:13:58.258 465334 ERROR os_vif
2021-03-28 15:13:58.258 465334 ERROR os_vif During handling of the above exception, another exception occurred:
2021-03-28 15:13:58.258 465334 ERROR os_vif
2021-03-28 15:13:58.258 465334 ERROR os_vif Traceback (most recent call last):
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 110, in unplug
2021-03-28 15:13:58.258 465334 ERROR os_vif plugin.unplug(vif, instance_info)
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 380, in unplug
2021-03-28 15:13:58.258 465334 ERROR os_vif self._unplug_vf(vif)
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 343, in _unplug_vf
2021-03-28 15:13:58.258 465334 ERROR os_vif pf_ifname = linux_net.get_ifname_by_pci_address(
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/linux_net.py", line 355, in get_ifname_by_pci_address
2021-03-28 15:13:58.258 465334 ERROR os_vif raise exception.PciDeviceNotFoundById(id=pci_addr)
2021-03-28 15:13:58.258 465334 ERROR os_vif vif_plug_ovs.exception.PciDeviceNotFoundById: PCI device 0000:05:10.0 not found
2021-03-28 15:13:58.258 465334 ERROR os_vif
2021-03-28 15:13:58.260 465334 WARNING nova.compute.manager [req-306e2d84-8d39-4e08-8df2-ba87baa323ae aa48b560d18541489031f10410fe33c7 80088d4e3eea4f90958175e58eb7ddbe - c35b9b534cf14b36802207ec524932e0 c35b9b534cf14b36802207ec524932e0] [instance: bd1a7d09-cf67-406c-8d0a-92256eb1daf8] Cleaning up VIFs failed for instance. Error: Failure running os_vif plugin unplug method: Failed to unplug VIF VIFHostDevice(active=False,address=fa:16:3e:10:ae:3a,dev_address=0000:05:10.0,dev_type='ethernet',has_traffic_filtering=True,id=9f7aae43-8e80-44fb-b626-f021423e6063,network=Network(c5ffad64-b4aa-4925-8c6a-31c35a8b8b5c),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: PCI device 0000:05:10.0 not found: nova.exception.InternalError: Failure running os_vif plugin unplug method: Failed to unplug VIF VIFHostDevice(active=False,address=fa:16:3e:10:ae:3a,dev_address=0000:05:10.0,dev_type='ethernet',has_traffic_filtering=True,id=9f7aae43-8e80-44fb-b626-f021423e6063,network=Network(c5ffad64-b4aa-4925-8c6a-31c35a8b8b5c),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: PCI device 0000:05:10.0 not found
2021-03-28 15:13:58.260 465334 DEBUG nova.compute.manager [req-306e2d84-8d39-4e08-8df2-ba87baa323ae aa48b560d18541489031f10410fe33c7 80088d4e3eea4f90958175e58eb7ddbe - c35b9b534cf14b36802207ec524932e0 c35b9b534cf14b36802207ec524932e0] [instance: bd1a7d09-cf67-406c-8d0a-92256eb1daf8] Deallocating network for instance _deallocate_network /usr/lib/python3/dist-packages/nova/compute/manager.py:2052

Revision history for this message
Erik Ilavsky (erik.ilavsky) wrote :

  ovn-chassis:
    charm: /root/ovn-chassis (only changed ovn-encap-type=geneve to bindings:ovn-encap-type=vxlan )
      "": *internal-space
      ovsdb: *internal-space
      data: *overlay-space
    options:
      ovn-bridge-mappings: physnet1:br-ex physnet2:br-ex prov_sriov_1:br-ex
      bridge-interface-mappings: br-ex:50:6b:4b:35:bc:27 br-ex:b8:59:9f:2a:56:99 br-ex:b8:59:9f:2a:65:c9
      enable-sriov: true
      enable-hardware-offload: false
      enable-dpdk: false
      sriov-device-mappings: prov_sriov_1:enp5s0f0
      sriov-numvfs: enp5s0f0:127

ovn-chassis 20.03.1 #11
nova-compute 22.1.0 #325

Revision history for this message
Erik Ilavsky (erik.ilavsky) wrote :

this is principal problem because library is running neutron-api in lxd container and in this container isn't have information about interface names, because is in separate network namespace.

root@juju-053afe-163-lxd-7:/home/ubuntu# lshw -quiet -class network -businfo | grep 0000:05:10.0
pci@0000:05:10.0 network MT27710 Family [ConnectX-4 Lx Virtual Function]

Revision history for this message
Erik Ilavsky (erik.ilavsky) wrote :

ignore last comment, it's running with nova.

Revision history for this message
Erik Ilavsky (erik.ilavsky) wrote :

working combination is kernel 5.8 and os-vif 2.4

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.