| 2020-08-20 13:11:35 |
Peter Sabaini |
bug |
|
|
added bug |
| 2020-08-20 13:24:43 |
Dan Ackerson |
bug |
|
|
added subscriber Canonical IS BootStack |
| 2020-08-20 13:26:59 |
Peter Sabaini |
bug |
|
|
added subscriber Canonical Field High |
| 2020-08-20 13:35:14 |
Peter Sabaini |
description |
When spawning an SR-IOV enabled instance on a newly deployed host, nova attempts to spawn it with an type-PF pci device. This fails with the below stack trace.
After restarting neutron-sriov-agent and nova-compute services on the compute node and spawning an SR-IOV instance again, a type-VF pci device is selected, and instance spawning succeeds.
Stack trace:
2020-08-20 08:29:09.558 7624 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 6db8011e6ecd4fd0aaa53c8f89f08b1b __call__ /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:400
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [req-e3e49d07-24c6-4c62-916e-f830f70983a2 ddcfb3640535428798aa3c8545362bd4 dd99e7950a5b46b5b924ccd1720b6257 - 015e4fd7db304665ab5378caa691bb8b 015e4fd7db304665ab5378caa691bb8b] [insta
nce: 9498ea75-fe88-4020-9a9e-f4c437c6de11] Instance failed to spawn: libvirtError: unsupported configuration: Interface type hostdev is currently supported on SR-IOV Virtual Functions only
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] Traceback (most recent call last):
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2274, in _build_resources
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] yield resources
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2054, in _build_and_run_instance
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] block_device_info=block_device_info)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3147, in spawn
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] destroy_disks_on_failure=True)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5651, in _create_domain_and_network
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] destroy_disks_on_failure)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] self.force_reraise()
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] six.reraise(self.type_, self.value, self.tb)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5620, in _create_domain_and_network
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] post_xml_callback=post_xml_callback)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5555, in _create_domain
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] guest.launch(pause=pause)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 144, in launch
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] self._encoded_xml, errors='ignore')
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] self.force_reraise()
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] six.reraise(self.type_, self.value, self.tb)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 139, in launch
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] return self._domain.createWithFlags(flags)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] rv = execute(f, *args, **kwargs)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] six.reraise(c, e, tb)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] rv = meth(*args, **kwargs)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1092, in createWithFlags
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] libvirtError: unsupported configuration: Interface type hostdev is currently supported on SR-IOV Virtual Functions only
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11]
2020-08-20 08:29:09.599 7624 INFO nova.compute.manager [req-e3e49d07-24c6-4c62-916e-f830f70983a2 ddcfb3640535428798aa3c8545362bd4 dd99e7950a5b46b5b924ccd1720b6257 - 015e4fd7db304665ab5378caa691bb8b 015e4fd7db304665ab5378caa691bb8b] [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] Terminating instance
To reproduce, bring up an instance with an SR-IOV port on a freshly deployed compute:
+ openstack port create -f value -c id --network testinstance_net --vnic-type=direct --binding-profile type=dict --binding-profile physical_network=physnet2 testinstance_net-port
+ openstack server create --flavor ce6da933-adc3-4e5f-a688-63b037705729 --image a3580f59-a6c6-41f6-85fa-2fc7277492a1 --nic port-id=547cd89a-3f91-4646-84d9-c9559b497526 --availability-zone nova:foo-compute-host testinstance_vanilla_66016d81-bc32-4def-a7b3-a3a164ca5164
Observe that a PF is getting selected for the sriov nic.
From nova-compute.log:
<interface type='hostdev' managed='yes'>
<mac address='98:03:9b:61:22:e9'/>
<source>
<address type='pci' domain='0x0000' bus='0xd8' slot='0x00' function='0x1'/>
</source>
<vlan>
<tag id='48'/>
</vlan>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</interface>
...
2020-08-20 08:29:09.056 7624 DEBUG nova.virt.libvirt.vif [req-e3e49d07-24c6-4c62-916e-f830f70983a2 ddcfb3640535428798aa3c8545362bd4 dd99e7950a5b46b5b924ccd1720b6257 - 015e4fd7db304665ab5378caa691bb8b 015e4fd7db304665ab5378caa691bb8b]
vif_type=hw_veb ...
vif={"profile":
{"pci_slot": "0000:d8:00.1", "physical_network": "physnet2", "pci_vendor_info": "15b3:1015"},
"ovs_interfaceid": null, "preserve_on_delete": true, "network": {"bridge": null, "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [],
"address": "192.168.0.5"}], "version": 4, "meta": {"dhcp_server": "192.168.0.2"}, "dns": [], "routes": [], "cidr": "192.168.0.0/29",
"gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "192.168.0.1"}}], "meta": {"injected": false, "tenant_id": "dd99e7950a5b46b5b924ccd1720b6257",
"physical_network": "physnet2", "mtu": 9000},
"id": "60b3001e-21c1-4947-8996-314449f614c060b3001e-21c1-4947-8996-314449f614c0", "label": "net_20Aug-1"}, "devname": "tapf3953098-98", "vnic_type": "direct", "qbh_params": null, "meta": {},
"details": {"port_filter": false, "vlan": "48"}, "address": "98:03:9b:61:22:e9", "active": false, "type": "hw_veb", "id": "f3953098-98f7-4dd1-8b31-11f51a5a760f", "qbg_params": null}
virt_type=kvm get_config /usr/lib/python2.7/dist-packages/nova/virt/libvirt/vif.py:572
Device is a PF:
# lspci | grep d8:00.1
d8:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
Also the nova pci_devices table has it's dev_type correctly listed:
mysql> select compute_nodes.host, pci_devices.created_at, compute_node_id, address, dev_type, status, pci_devices.dev_id from pci_devices join compute_nodes ON (compute_nodes.id = pci_devices.compute_node_id) where compute_nodes.host = 'foo-compute-host' and pci_devices.dev_type = 'type-PF';
+------------------+---------------------+-----------------+--------------+----------+-----------+------------------+
| host | created_at | compute_node_id | address | dev_type | status | dev_id |
+------------------+---------------------+-----------------+--------------+----------+-----------+------------------+
| foo-compute-host | 2020-08-12 17:10:19 | 95 | 0000:19:00.1 | type-PF | available | pci_0000_19_00_1 |
| foo-compute-host | 2020-08-12 17:10:19 | 95 | 0000:d8:00.1 | type-PF | available | pci_0000_d8_00_1 |
+------------------+---------------------+-----------------+--------------+----------+-----------+------------------+
Restarting services:
# systemctl status neutron-sriov-agent.service
# systemctl restart neutron-sriov-agent.service
Spawning an instance again, it gets allocated a type-VF port (and spawning succeeds):
<interface type='hostdev' managed='yes'>
<mac address='fa:16:3e:34:d2:99'/>
<source>
<address type='pci' domain='0x0000' bus='0xd8' slot='0x05' function='0x1'/>
</source>
<vlan>
<tag id='4'/>
</vlan>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</interface>
# lspci | grep d8:05.1
d8:05.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
Software versions:
# dpkg -l | grep nova-common
ii nova-common 2:17.0.12-0ubuntu1 all OpenStack Compute - common files
# dpkg -l | grep libvirt0
ii libvirt0:amd64 4.0.0-1ubuntu8.17 amd64 library for interfacing with different virtualization systems
# lsb_release -r
Release: 18.04 |
When spawning an SR-IOV enabled instance on a newly deployed host, nova attempts to spawn it with an type-PF pci device. This fails with the below stack trace.
After restarting neutron-sriov-agent and nova-compute services on the compute node and spawning an SR-IOV instance again, a type-VF pci device is selected, and instance spawning succeeds.
Stack trace:
2020-08-20 08:29:09.558 7624 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 6db8011e6ecd4fd0aaa53c8f89f08b1b __call__ /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:400
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [req-e3e49d07-24c6-4c62-916e-f830f70983a2 ddcfb3640535428798aa3c8545362bd4 dd99e7950a5b46b5b924ccd1720b6257 - 015e4fd7db304665ab5378caa691bb8b 015e4fd7db304665ab5378caa691bb8b] [insta
nce: 9498ea75-fe88-4020-9a9e-f4c437c6de11] Instance failed to spawn: libvirtError: unsupported configuration: Interface type hostdev is currently supported on SR-IOV Virtual Functions only
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] Traceback (most recent call last):
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2274, in _build_resources
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] yield resources
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2054, in _build_and_run_instance
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] block_device_info=block_device_info)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3147, in spawn
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] destroy_disks_on_failure=True)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5651, in _create_domain_and_network
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] destroy_disks_on_failure)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] self.force_reraise()
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] six.reraise(self.type_, self.value, self.tb)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5620, in _create_domain_and_network
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] post_xml_callback=post_xml_callback)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5555, in _create_domain
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] guest.launch(pause=pause)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 144, in launch
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] self._encoded_xml, errors='ignore')
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] self.force_reraise()
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] six.reraise(self.type_, self.value, self.tb)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 139, in launch
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] return self._domain.createWithFlags(flags)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] rv = execute(f, *args, **kwargs)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] six.reraise(c, e, tb)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] rv = meth(*args, **kwargs)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1092, in createWithFlags
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] libvirtError: unsupported configuration: Interface type hostdev is currently supported on SR-IOV Virtual Functions only
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11]
2020-08-20 08:29:09.599 7624 INFO nova.compute.manager [req-e3e49d07-24c6-4c62-916e-f830f70983a2 ddcfb3640535428798aa3c8545362bd4 dd99e7950a5b46b5b924ccd1720b6257 - 015e4fd7db304665ab5378caa691bb8b 015e4fd7db304665ab5378caa691bb8b] [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] Terminating instance
To reproduce, bring up an instance with an SR-IOV port on a freshly deployed compute:
+ openstack port create -f value -c id --network testinstance_net --vnic-type=direct --binding-profile type=dict --binding-profile physical_network=physnet2 testinstance_net-port
+ openstack server create --flavor ce6da933-adc3-4e5f-a688-63b037705729 --image a3580f59-a6c6-41f6-85fa-2fc7277492a1 --nic port-id=547cd89a-3f91-4646-84d9-c9559b497526 --availability-zone nova:foo-compute-host testinstance_vanilla_66016d81-bc32-4def-a7b3-a3a164ca5164
Observe that a PF is getting selected for the sriov nic.
From nova-compute.log:
<interface type='hostdev' managed='yes'>
<mac address='98:03:9b:61:22:e9'/>
<source>
<address type='pci' domain='0x0000' bus='0xd8' slot='0x00' function='0x1'/>
</source>
<vlan>
<tag id='48'/>
</vlan>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</interface>
...
2020-08-20 08:29:09.056 7624 DEBUG nova.virt.libvirt.vif [req-e3e49d07-24c6-4c62-916e-f830f70983a2 ddcfb3640535428798aa3c8545362bd4 dd99e7950a5b46b5b924ccd1720b6257 - 015e4fd7db304665ab5378caa691bb8b 015e4fd7db304665ab5378caa691bb8b]
vif_type=hw_veb ...
vif={"profile":
{"pci_slot": "0000:d8:00.1", "physical_network": "physnet2", "pci_vendor_info": "15b3:1015"},
"ovs_interfaceid": null, "preserve_on_delete": true, "network": {"bridge": null, "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [],
"address": "192.168.0.5"}], "version": 4, "meta": {"dhcp_server": "192.168.0.2"}, "dns": [], "routes": [], "cidr": "192.168.0.0/29",
"gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "192.168.0.1"}}], "meta": {"injected": false, "tenant_id": "dd99e7950a5b46b5b924ccd1720b6257",
"physical_network": "physnet2", "mtu": 9000},
"id": "60b3001e-21c1-4947-8996-314449f614c060b3001e-21c1-4947-8996-314449f614c0", "label": "net_20Aug-1"}, "devname": "tapf3953098-98", "vnic_type": "direct", "qbh_params": null, "meta": {},
"details": {"port_filter": false, "vlan": "48"}, "address": "98:03:9b:61:22:e9", "active": false, "type": "hw_veb", "id": "f3953098-98f7-4dd1-8b31-11f51a5a760f", "qbg_params": null}
virt_type=kvm get_config /usr/lib/python2.7/dist-packages/nova/virt/libvirt/vif.py:572
Device is a PF:
# lspci | grep d8:00.1
d8:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
Also the nova pci_devices table has it's dev_type correctly listed:
mysql> select compute_nodes.host, pci_devices.created_at, compute_node_id, address, dev_type, status, pci_devices.dev_id from pci_devices join compute_nodes ON (compute_nodes.id = pci_devices.compute_node_id) where compute_nodes.host = 'foo-compute-host' and pci_devices.dev_type = 'type-PF';
+------------------+---------------------+-----------------+--------------+----------+-----------+------------------+
| host | created_at | compute_node_id | address | dev_type | status | dev_id |
+------------------+---------------------+-----------------+--------------+----------+-----------+------------------+
| foo-compute-host | 2020-08-12 17:10:19 | 95 | 0000:19:00.1 | type-PF | available | pci_0000_19_00_1 |
| foo-compute-host | 2020-08-12 17:10:19 | 95 | 0000:d8:00.1 | type-PF | available | pci_0000_d8_00_1 |
+------------------+---------------------+-----------------+--------------+----------+-----------+------------------+
Restarting services:
# systemctl status neutron-sriov-agent.service
# systemctl restart neutron-sriov-agent.service
Spawning an instance again, it gets allocated a type-VF port (and spawning succeeds):
<interface type='hostdev' managed='yes'>
<mac address='fa:16:3e:34:d2:99'/>
<source>
<address type='pci' domain='0x0000' bus='0xd8' slot='0x05' function='0x1'/>
</source>
<vlan>
<tag id='4'/>
</vlan>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</interface>
# lspci | grep d8:05.1
d8:05.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
After spawning an instance, the PF get marked as "unavailable" in the nova db:
+------------------+---------------------+---------------------+---------------+-----------------+--------------+----------+-------------+------------------+
| host | created_at | updated_at | instance_uuid | compute_node_id | address | dev_type | status | dev_id |
+------------------+---------------------+---------------------+---------------+-----------------+--------------+----------+-------------+------------------+
| foo-compute-host | 2020-08-12 17:10:19 | 2020-08-20 11:45:07 | NULL | 95 | 0000:19:00.1 | type-PF | available | pci_0000_19_00_1 |
| foo-compute-host | 2020-08-12 17:10:19 | 2020-08-20 11:46:30 | NULL | 95 | 0000:d8:00.1 | type-PF | unavailable | pci_0000_d8_00_1 |
+------------------+---------------------+---------------------+---------------+-----------------+--------------+----------+-------------+------------------+
Software versions:
# dpkg -l | grep nova-common
ii nova-common 2:17.0.12-0ubuntu1 all OpenStack Compute - common files
# dpkg -l | grep libvirt0
ii libvirt0:amd64 4.0.0-1ubuntu8.17 amd64 library for interfacing with different virtualization systems
# lsb_release -r
Release: 18.04 |
|
| 2020-08-31 00:43:56 |
Dominique Poulain |
bug |
|
|
added subscriber Dominique Poulain |
| 2020-08-31 12:16:19 |
Hemanth Nakkina |
nova: assignee |
|
Hemanth Nakkina (hemanth-n) |
|
| 2020-09-01 04:10:30 |
OpenStack Infra |
nova: status |
New |
In Progress |
|
| 2020-09-04 15:25:53 |
sean mooney |
nova: importance |
Undecided |
Medium |
|
| 2020-09-04 15:27:11 |
sean mooney |
tags |
|
compute pci resource-tracker |
|
| 2020-09-11 11:29:37 |
sean mooney |
nominated for series |
|
nova/queens |
|
| 2020-09-11 11:29:37 |
sean mooney |
bug task added |
|
nova/queens |
|
| 2020-09-11 11:29:37 |
sean mooney |
nominated for series |
|
nova/ussuri |
|
| 2020-09-11 11:29:37 |
sean mooney |
bug task added |
|
nova/ussuri |
|
| 2020-09-11 11:29:37 |
sean mooney |
nominated for series |
|
nova/rocky |
|
| 2020-09-11 11:29:37 |
sean mooney |
bug task added |
|
nova/rocky |
|
| 2020-09-11 11:29:37 |
sean mooney |
nominated for series |
|
nova/train |
|
| 2020-09-11 11:29:37 |
sean mooney |
bug task added |
|
nova/train |
|
| 2020-09-11 11:29:37 |
sean mooney |
nominated for series |
|
nova/stein |
|
| 2020-09-11 11:29:37 |
sean mooney |
bug task added |
|
nova/stein |
|
| 2020-09-11 11:29:47 |
sean mooney |
nova/queens: status |
New |
Confirmed |
|
| 2020-09-11 11:29:51 |
sean mooney |
nova/queens: importance |
Undecided |
Medium |
|
| 2020-09-11 11:30:04 |
sean mooney |
nova/rocky: importance |
Undecided |
Medium |
|
| 2020-09-11 11:30:04 |
sean mooney |
nova/rocky: status |
New |
Triaged |
|
| 2020-09-11 11:30:10 |
sean mooney |
nova/queens: status |
Confirmed |
Triaged |
|
| 2020-09-11 11:30:18 |
sean mooney |
nova/stein: importance |
Undecided |
Medium |
|
| 2020-09-11 11:30:18 |
sean mooney |
nova/stein: status |
New |
Triaged |
|
| 2020-09-11 11:30:31 |
sean mooney |
nova/train: importance |
Undecided |
Medium |
|
| 2020-09-11 11:30:31 |
sean mooney |
nova/train: status |
New |
Triaged |
|
| 2020-09-11 11:30:47 |
sean mooney |
nova/ussuri: importance |
Undecided |
Medium |
|
| 2020-09-11 11:30:47 |
sean mooney |
nova/ussuri: status |
New |
Triaged |
|
| 2020-11-03 03:59:36 |
Hua Zhang |
bug |
|
|
added subscriber Hua Zhang |
| 2020-11-06 05:23:51 |
OpenStack Infra |
nova: status |
In Progress |
Fix Released |
|
| 2020-11-06 06:23:37 |
OpenStack Infra |
nova/ussuri: status |
Triaged |
In Progress |
|
| 2020-11-06 06:23:37 |
OpenStack Infra |
nova/ussuri: assignee |
|
Hemanth Nakkina (hemanth-n) |
|
| 2020-11-06 11:10:19 |
OpenStack Infra |
nova/train: status |
Triaged |
In Progress |
|
| 2020-11-06 11:10:19 |
OpenStack Infra |
nova/train: assignee |
|
Hemanth Nakkina (hemanth-n) |
|
| 2020-11-06 11:25:32 |
OpenStack Infra |
nova/stein: status |
Triaged |
In Progress |
|
| 2020-11-06 11:25:32 |
OpenStack Infra |
nova/stein: assignee |
|
Hemanth Nakkina (hemanth-n) |
|
| 2020-11-08 01:22:10 |
OpenStack Infra |
nova/rocky: status |
Triaged |
In Progress |
|
| 2020-11-08 01:22:10 |
OpenStack Infra |
nova/rocky: assignee |
|
Hemanth Nakkina (hemanth-n) |
|
| 2020-11-08 01:30:04 |
OpenStack Infra |
nova/queens: status |
Triaged |
In Progress |
|
| 2020-11-08 01:30:04 |
OpenStack Infra |
nova/queens: assignee |
|
Hemanth Nakkina (hemanth-n) |
|
| 2020-11-09 23:01:17 |
melanie witt |
nominated for series |
|
nova/victoria |
|
| 2020-11-09 23:01:17 |
melanie witt |
bug task added |
|
nova/victoria |
|
| 2020-11-10 01:09:48 |
OpenStack Infra |
nova/victoria: status |
New |
Fix Committed |
|
| 2020-11-20 18:21:12 |
Elod Illes |
nova/victoria: status |
Fix Committed |
Fix Released |
|
| 2020-11-20 18:21:21 |
Elod Illes |
nova/victoria: importance |
Undecided |
Medium |
|
| 2021-01-12 10:33:52 |
Hemanth Nakkina |
nova/victoria: assignee |
|
Hemanth Nakkina (hemanth-n) |
|
| 2021-01-12 11:00:23 |
Hemanth Nakkina |
bug task added |
|
nova (Ubuntu) |
|
| 2021-01-12 11:01:02 |
Hemanth Nakkina |
nominated for series |
|
Ubuntu Groovy |
|
| 2021-01-12 11:01:02 |
Hemanth Nakkina |
bug task added |
|
nova (Ubuntu Groovy) |
|
| 2021-01-12 11:01:02 |
Hemanth Nakkina |
nominated for series |
|
Ubuntu Hirsute |
|
| 2021-01-12 11:01:02 |
Hemanth Nakkina |
bug task added |
|
nova (Ubuntu Hirsute) |
|
| 2021-01-12 11:01:02 |
Hemanth Nakkina |
nominated for series |
|
Ubuntu Bionic |
|
| 2021-01-12 11:01:02 |
Hemanth Nakkina |
bug task added |
|
nova (Ubuntu Bionic) |
|
| 2021-01-12 11:01:02 |
Hemanth Nakkina |
nominated for series |
|
Ubuntu Focal |
|
| 2021-01-12 11:01:02 |
Hemanth Nakkina |
bug task added |
|
nova (Ubuntu Focal) |
|
| 2021-01-12 11:04:25 |
Hemanth Nakkina |
bug task added |
|
cloud-archive |
|
| 2021-01-12 11:06:38 |
Hemanth Nakkina |
nominated for series |
|
cloud-archive/rocky |
|
| 2021-01-12 11:06:38 |
Hemanth Nakkina |
bug task added |
|
cloud-archive/rocky |
|
| 2021-01-12 11:06:38 |
Hemanth Nakkina |
nominated for series |
|
cloud-archive/train |
|
| 2021-01-12 11:06:38 |
Hemanth Nakkina |
bug task added |
|
cloud-archive/train |
|
| 2021-01-12 11:06:38 |
Hemanth Nakkina |
nominated for series |
|
cloud-archive/queens |
|
| 2021-01-12 11:06:38 |
Hemanth Nakkina |
bug task added |
|
cloud-archive/queens |
|
| 2021-01-12 11:06:38 |
Hemanth Nakkina |
nominated for series |
|
cloud-archive/stein |
|
| 2021-01-12 11:06:38 |
Hemanth Nakkina |
bug task added |
|
cloud-archive/stein |
|
| 2021-01-12 11:06:38 |
Hemanth Nakkina |
nominated for series |
|
cloud-archive/ussuri |
|
| 2021-01-12 11:06:38 |
Hemanth Nakkina |
bug task added |
|
cloud-archive/ussuri |
|
| 2021-01-12 11:06:38 |
Hemanth Nakkina |
nominated for series |
|
cloud-archive/victoria |
|
| 2021-01-12 11:06:38 |
Hemanth Nakkina |
bug task added |
|
cloud-archive/victoria |
|
| 2021-01-12 11:52:51 |
Hemanth Nakkina |
nova (Ubuntu Hirsute): status |
New |
Fix Released |
|
| 2021-01-12 11:54:02 |
Hemanth Nakkina |
nova (Ubuntu Groovy): status |
New |
Fix Released |
|
| 2021-01-12 11:54:23 |
Hemanth Nakkina |
cloud-archive/victoria: importance |
Undecided |
Medium |
|
| 2021-01-12 11:54:23 |
Hemanth Nakkina |
cloud-archive/victoria: status |
New |
Fix Released |
|
| 2021-01-12 11:54:41 |
Hemanth Nakkina |
nova (Ubuntu Hirsute): importance |
Undecided |
Medium |
|
| 2021-01-12 11:54:55 |
Hemanth Nakkina |
nova (Ubuntu Groovy): importance |
Undecided |
Medium |
|
| 2021-01-13 08:47:36 |
Hemanth Nakkina |
description |
When spawning an SR-IOV enabled instance on a newly deployed host, nova attempts to spawn it with an type-PF pci device. This fails with the below stack trace.
After restarting neutron-sriov-agent and nova-compute services on the compute node and spawning an SR-IOV instance again, a type-VF pci device is selected, and instance spawning succeeds.
Stack trace:
2020-08-20 08:29:09.558 7624 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 6db8011e6ecd4fd0aaa53c8f89f08b1b __call__ /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:400
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [req-e3e49d07-24c6-4c62-916e-f830f70983a2 ddcfb3640535428798aa3c8545362bd4 dd99e7950a5b46b5b924ccd1720b6257 - 015e4fd7db304665ab5378caa691bb8b 015e4fd7db304665ab5378caa691bb8b] [insta
nce: 9498ea75-fe88-4020-9a9e-f4c437c6de11] Instance failed to spawn: libvirtError: unsupported configuration: Interface type hostdev is currently supported on SR-IOV Virtual Functions only
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] Traceback (most recent call last):
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2274, in _build_resources
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] yield resources
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2054, in _build_and_run_instance
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] block_device_info=block_device_info)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3147, in spawn
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] destroy_disks_on_failure=True)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5651, in _create_domain_and_network
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] destroy_disks_on_failure)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] self.force_reraise()
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] six.reraise(self.type_, self.value, self.tb)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5620, in _create_domain_and_network
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] post_xml_callback=post_xml_callback)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5555, in _create_domain
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] guest.launch(pause=pause)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 144, in launch
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] self._encoded_xml, errors='ignore')
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] self.force_reraise()
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] six.reraise(self.type_, self.value, self.tb)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 139, in launch
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] return self._domain.createWithFlags(flags)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] rv = execute(f, *args, **kwargs)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] six.reraise(c, e, tb)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] rv = meth(*args, **kwargs)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1092, in createWithFlags
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] libvirtError: unsupported configuration: Interface type hostdev is currently supported on SR-IOV Virtual Functions only
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11]
2020-08-20 08:29:09.599 7624 INFO nova.compute.manager [req-e3e49d07-24c6-4c62-916e-f830f70983a2 ddcfb3640535428798aa3c8545362bd4 dd99e7950a5b46b5b924ccd1720b6257 - 015e4fd7db304665ab5378caa691bb8b 015e4fd7db304665ab5378caa691bb8b] [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] Terminating instance
To reproduce, bring up an instance with an SR-IOV port on a freshly deployed compute:
+ openstack port create -f value -c id --network testinstance_net --vnic-type=direct --binding-profile type=dict --binding-profile physical_network=physnet2 testinstance_net-port
+ openstack server create --flavor ce6da933-adc3-4e5f-a688-63b037705729 --image a3580f59-a6c6-41f6-85fa-2fc7277492a1 --nic port-id=547cd89a-3f91-4646-84d9-c9559b497526 --availability-zone nova:foo-compute-host testinstance_vanilla_66016d81-bc32-4def-a7b3-a3a164ca5164
Observe that a PF is getting selected for the sriov nic.
From nova-compute.log:
<interface type='hostdev' managed='yes'>
<mac address='98:03:9b:61:22:e9'/>
<source>
<address type='pci' domain='0x0000' bus='0xd8' slot='0x00' function='0x1'/>
</source>
<vlan>
<tag id='48'/>
</vlan>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</interface>
...
2020-08-20 08:29:09.056 7624 DEBUG nova.virt.libvirt.vif [req-e3e49d07-24c6-4c62-916e-f830f70983a2 ddcfb3640535428798aa3c8545362bd4 dd99e7950a5b46b5b924ccd1720b6257 - 015e4fd7db304665ab5378caa691bb8b 015e4fd7db304665ab5378caa691bb8b]
vif_type=hw_veb ...
vif={"profile":
{"pci_slot": "0000:d8:00.1", "physical_network": "physnet2", "pci_vendor_info": "15b3:1015"},
"ovs_interfaceid": null, "preserve_on_delete": true, "network": {"bridge": null, "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [],
"address": "192.168.0.5"}], "version": 4, "meta": {"dhcp_server": "192.168.0.2"}, "dns": [], "routes": [], "cidr": "192.168.0.0/29",
"gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "192.168.0.1"}}], "meta": {"injected": false, "tenant_id": "dd99e7950a5b46b5b924ccd1720b6257",
"physical_network": "physnet2", "mtu": 9000},
"id": "60b3001e-21c1-4947-8996-314449f614c060b3001e-21c1-4947-8996-314449f614c0", "label": "net_20Aug-1"}, "devname": "tapf3953098-98", "vnic_type": "direct", "qbh_params": null, "meta": {},
"details": {"port_filter": false, "vlan": "48"}, "address": "98:03:9b:61:22:e9", "active": false, "type": "hw_veb", "id": "f3953098-98f7-4dd1-8b31-11f51a5a760f", "qbg_params": null}
virt_type=kvm get_config /usr/lib/python2.7/dist-packages/nova/virt/libvirt/vif.py:572
Device is a PF:
# lspci | grep d8:00.1
d8:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
Also the nova pci_devices table has it's dev_type correctly listed:
mysql> select compute_nodes.host, pci_devices.created_at, compute_node_id, address, dev_type, status, pci_devices.dev_id from pci_devices join compute_nodes ON (compute_nodes.id = pci_devices.compute_node_id) where compute_nodes.host = 'foo-compute-host' and pci_devices.dev_type = 'type-PF';
+------------------+---------------------+-----------------+--------------+----------+-----------+------------------+
| host | created_at | compute_node_id | address | dev_type | status | dev_id |
+------------------+---------------------+-----------------+--------------+----------+-----------+------------------+
| foo-compute-host | 2020-08-12 17:10:19 | 95 | 0000:19:00.1 | type-PF | available | pci_0000_19_00_1 |
| foo-compute-host | 2020-08-12 17:10:19 | 95 | 0000:d8:00.1 | type-PF | available | pci_0000_d8_00_1 |
+------------------+---------------------+-----------------+--------------+----------+-----------+------------------+
Restarting services:
# systemctl status neutron-sriov-agent.service
# systemctl restart neutron-sriov-agent.service
Spawning an instance again, it gets allocated a type-VF port (and spawning succeeds):
<interface type='hostdev' managed='yes'>
<mac address='fa:16:3e:34:d2:99'/>
<source>
<address type='pci' domain='0x0000' bus='0xd8' slot='0x05' function='0x1'/>
</source>
<vlan>
<tag id='4'/>
</vlan>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</interface>
# lspci | grep d8:05.1
d8:05.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
After spawning an instance, the PF get marked as "unavailable" in the nova db:
+------------------+---------------------+---------------------+---------------+-----------------+--------------+----------+-------------+------------------+
| host | created_at | updated_at | instance_uuid | compute_node_id | address | dev_type | status | dev_id |
+------------------+---------------------+---------------------+---------------+-----------------+--------------+----------+-------------+------------------+
| foo-compute-host | 2020-08-12 17:10:19 | 2020-08-20 11:45:07 | NULL | 95 | 0000:19:00.1 | type-PF | available | pci_0000_19_00_1 |
| foo-compute-host | 2020-08-12 17:10:19 | 2020-08-20 11:46:30 | NULL | 95 | 0000:d8:00.1 | type-PF | unavailable | pci_0000_d8_00_1 |
+------------------+---------------------+---------------------+---------------+-----------------+--------------+----------+-------------+------------------+
Software versions:
# dpkg -l | grep nova-common
ii nova-common 2:17.0.12-0ubuntu1 all OpenStack Compute - common files
# dpkg -l | grep libvirt0
ii libvirt0:amd64 4.0.0-1ubuntu8.17 amd64 library for interfacing with different virtualization systems
# lsb_release -r
Release: 18.04 |
When spawning an SR-IOV enabled instance on a newly deployed host, nova attempts to spawn it with an type-PF pci device. This fails with the below stack trace.
After restarting neutron-sriov-agent and nova-compute services on the compute node and spawning an SR-IOV instance again, a type-VF pci device is selected, and instance spawning succeeds.
Stack trace:
2020-08-20 08:29:09.558 7624 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 6db8011e6ecd4fd0aaa53c8f89f08b1b __call__ /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:400
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [req-e3e49d07-24c6-4c62-916e-f830f70983a2 ddcfb3640535428798aa3c8545362bd4 dd99e7950a5b46b5b924ccd1720b6257 - 015e4fd7db304665ab5378caa691bb8b 015e4fd7db304665ab5378caa691bb8b] [insta
nce: 9498ea75-fe88-4020-9a9e-f4c437c6de11] Instance failed to spawn: libvirtError: unsupported configuration: Interface type hostdev is currently supported on SR-IOV Virtual Functions only
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] Traceback (most recent call last):
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2274, in _build_resources
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] yield resources
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2054, in _build_and_run_instance
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] block_device_info=block_device_info)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3147, in spawn
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] destroy_disks_on_failure=True)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5651, in _create_domain_and_network
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] destroy_disks_on_failure)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] self.force_reraise()
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] six.reraise(self.type_, self.value, self.tb)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5620, in _create_domain_and_network
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] post_xml_callback=post_xml_callback)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5555, in _create_domain
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] guest.launch(pause=pause)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 144, in launch
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] self._encoded_xml, errors='ignore')
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] self.force_reraise()
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] six.reraise(self.type_, self.value, self.tb)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 139, in launch
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] return self._domain.createWithFlags(flags)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] rv = execute(f, *args, **kwargs)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] six.reraise(c, e, tb)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] rv = meth(*args, **kwargs)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1092, in createWithFlags
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] libvirtError: unsupported configuration: Interface type hostdev is currently supported on SR-IOV Virtual Functions only
2020-08-20 08:29:09.561 7624 ERROR nova.compute.manager [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11]
2020-08-20 08:29:09.599 7624 INFO nova.compute.manager [req-e3e49d07-24c6-4c62-916e-f830f70983a2 ddcfb3640535428798aa3c8545362bd4 dd99e7950a5b46b5b924ccd1720b6257 - 015e4fd7db304665ab5378caa691bb8b 015e4fd7db304665ab5378caa691bb8b] [instance: 9498ea75-fe88-4020-9a9e-f4c437c6de11] Terminating instance
To reproduce, bring up an instance with an SR-IOV port on a freshly deployed compute:
+ openstack port create -f value -c id --network testinstance_net --vnic-type=direct --binding-profile type=dict --binding-profile physical_network=physnet2 testinstance_net-port
+ openstack server create --flavor ce6da933-adc3-4e5f-a688-63b037705729 --image a3580f59-a6c6-41f6-85fa-2fc7277492a1 --nic port-id=547cd89a-3f91-4646-84d9-c9559b497526 --availability-zone nova:foo-compute-host testinstance_vanilla_66016d81-bc32-4def-a7b3-a3a164ca5164
Observe that a PF is getting selected for the sriov nic.
From nova-compute.log:
<interface type='hostdev' managed='yes'>
<mac address='98:03:9b:61:22:e9'/>
<source>
<address type='pci' domain='0x0000' bus='0xd8' slot='0x00' function='0x1'/>
</source>
<vlan>
<tag id='48'/>
</vlan>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</interface>
...
2020-08-20 08:29:09.056 7624 DEBUG nova.virt.libvirt.vif [req-e3e49d07-24c6-4c62-916e-f830f70983a2 ddcfb3640535428798aa3c8545362bd4 dd99e7950a5b46b5b924ccd1720b6257 - 015e4fd7db304665ab5378caa691bb8b 015e4fd7db304665ab5378caa691bb8b]
vif_type=hw_veb ...
vif={"profile":
{"pci_slot": "0000:d8:00.1", "physical_network": "physnet2", "pci_vendor_info": "15b3:1015"},
"ovs_interfaceid": null, "preserve_on_delete": true, "network": {"bridge": null, "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [],
"address": "192.168.0.5"}], "version": 4, "meta": {"dhcp_server": "192.168.0.2"}, "dns": [], "routes": [], "cidr": "192.168.0.0/29",
"gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "192.168.0.1"}}], "meta": {"injected": false, "tenant_id": "dd99e7950a5b46b5b924ccd1720b6257",
"physical_network": "physnet2", "mtu": 9000},
"id": "60b3001e-21c1-4947-8996-314449f614c060b3001e-21c1-4947-8996-314449f614c0", "label": "net_20Aug-1"}, "devname": "tapf3953098-98", "vnic_type": "direct", "qbh_params": null, "meta": {},
"details": {"port_filter": false, "vlan": "48"}, "address": "98:03:9b:61:22:e9", "active": false, "type": "hw_veb", "id": "f3953098-98f7-4dd1-8b31-11f51a5a760f", "qbg_params": null}
virt_type=kvm get_config /usr/lib/python2.7/dist-packages/nova/virt/libvirt/vif.py:572
Device is a PF:
# lspci | grep d8:00.1
d8:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
Also the nova pci_devices table has it's dev_type correctly listed:
mysql> select compute_nodes.host, pci_devices.created_at, compute_node_id, address, dev_type, status, pci_devices.dev_id from pci_devices join compute_nodes ON (compute_nodes.id = pci_devices.compute_node_id) where compute_nodes.host = 'foo-compute-host' and pci_devices.dev_type = 'type-PF';
+------------------+---------------------+-----------------+--------------+----------+-----------+------------------+
| host | created_at | compute_node_id | address | dev_type | status | dev_id |
+------------------+---------------------+-----------------+--------------+----------+-----------+------------------+
| foo-compute-host | 2020-08-12 17:10:19 | 95 | 0000:19:00.1 | type-PF | available | pci_0000_19_00_1 |
| foo-compute-host | 2020-08-12 17:10:19 | 95 | 0000:d8:00.1 | type-PF | available | pci_0000_d8_00_1 |
+------------------+---------------------+-----------------+--------------+----------+-----------+------------------+
Restarting services:
# systemctl status neutron-sriov-agent.service
# systemctl restart neutron-sriov-agent.service
Spawning an instance again, it gets allocated a type-VF port (and spawning succeeds):
<interface type='hostdev' managed='yes'>
<mac address='fa:16:3e:34:d2:99'/>
<source>
<address type='pci' domain='0x0000' bus='0xd8' slot='0x05' function='0x1'/>
</source>
<vlan>
<tag id='4'/>
</vlan>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</interface>
# lspci | grep d8:05.1
d8:05.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
After spawning an instance, the PF get marked as "unavailable" in the nova db:
+------------------+---------------------+---------------------+---------------+-----------------+--------------+----------+-------------+------------------+
| host | created_at | updated_at | instance_uuid | compute_node_id | address | dev_type | status | dev_id |
+------------------+---------------------+---------------------+---------------+-----------------+--------------+----------+-------------+------------------+
| foo-compute-host | 2020-08-12 17:10:19 | 2020-08-20 11:45:07 | NULL | 95 | 0000:19:00.1 | type-PF | available | pci_0000_19_00_1 |
| foo-compute-host | 2020-08-12 17:10:19 | 2020-08-20 11:46:30 | NULL | 95 | 0000:d8:00.1 | type-PF | unavailable | pci_0000_d8_00_1 |
+------------------+---------------------+---------------------+---------------+-----------------+--------------+----------+-------------+------------------+
Software versions:
# dpkg -l | grep nova-common
ii nova-common 2:17.0.12-0ubuntu1 all OpenStack Compute - common files
# dpkg -l | grep libvirt0
ii libvirt0:amd64 4.0.0-1ubuntu8.17 amd64 library for interfacing with different virtualization systems
# lsb_release -r
Release: 18.04
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[Impact]
Spawning an SR-IOV instance fails on a newly deployed compute.
Nova attempts to spawn a PCI device of type type-PCI instead of type-VF.
This was happened in OpenStack Queens deployment.
[Test case]
1. Issue can be reproduced by following steps in comment #3
https://bugs.launchpad.net/nova/+bug/1892361/comments/3
2. Install the package with fixed code
3. Confirm bug have been fixed
Repeat the steps mentioned in comment #3 and check if the instance with sriov port is created successfully.
[Where problems could occur]
Upstream CI ran all the functional test cases that triggers this scenario.
Installation of new package will result in restart of nova-compute service. |
|
| 2021-01-13 08:50:18 |
Hemanth Nakkina |
attachment added |
|
Debdiff for focal https://bugs.launchpad.net/ubuntu/focal/+source/nova/+bug/1892361/+attachment/5452615/+files/lp892361_focal.debdiff |
|
| 2021-01-13 08:50:47 |
Hemanth Nakkina |
attachment added |
|
Debdiff for UCA train https://bugs.launchpad.net/ubuntu/focal/+source/nova/+bug/1892361/+attachment/5452616/+files/lp1892361_train.debdiff |
|
| 2021-01-13 08:51:09 |
Hemanth Nakkina |
attachment added |
|
Debdiff for UCA stein https://bugs.launchpad.net/ubuntu/focal/+source/nova/+bug/1892361/+attachment/5452617/+files/lp1892361_stein.debdiff |
|
| 2021-01-13 08:51:37 |
Hemanth Nakkina |
attachment added |
|
Debdiff for UCA rocky https://bugs.launchpad.net/ubuntu/focal/+source/nova/+bug/1892361/+attachment/5452618/+files/lp1892361_rocky.debdiff |
|
| 2021-01-13 08:52:02 |
Hemanth Nakkina |
attachment added |
|
Debdiff for bionic https://bugs.launchpad.net/ubuntu/focal/+source/nova/+bug/1892361/+attachment/5452619/+files/lp1892361_bionic.debdiff |
|
| 2021-01-18 15:40:04 |
Chris MacNaughton |
nova (Ubuntu Focal): assignee |
|
Chris MacNaughton (chris.macnaughton) |
|
| 2021-01-20 09:47:47 |
Robie Basak |
nova (Ubuntu Focal): status |
New |
Fix Committed |
|
| 2021-01-20 09:47:50 |
Robie Basak |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2021-01-20 09:47:52 |
Robie Basak |
bug |
|
|
added subscriber SRU Verification |
| 2021-01-20 09:47:58 |
Robie Basak |
tags |
compute pci resource-tracker |
compute pci resource-tracker verification-needed verification-needed-focal |
|
| 2021-01-22 13:32:37 |
Chris MacNaughton |
cloud-archive/ussuri: status |
New |
Fix Committed |
|
| 2021-01-22 13:32:40 |
Chris MacNaughton |
tags |
compute pci resource-tracker verification-needed verification-needed-focal |
compute pci resource-tracker verification-needed verification-needed-focal verification-ussuri-needed |
|
| 2021-01-23 05:47:04 |
Hemanth Nakkina |
nova/ussuri: status |
In Progress |
Fix Released |
|
| 2021-01-23 05:47:47 |
Hemanth Nakkina |
nova/train: status |
In Progress |
Fix Released |
|
| 2021-02-01 03:16:48 |
Hemanth Nakkina |
tags |
compute pci resource-tracker verification-needed verification-needed-focal verification-ussuri-needed |
compute pci resource-tracker verification-done-focal verification-needed verification-ussuri-done |
|
| 2021-02-01 20:40:33 |
melanie witt |
nova/ussuri: status |
Fix Released |
Fix Committed |
|
| 2021-02-01 20:42:57 |
melanie witt |
nova/train: status |
Fix Released |
Fix Committed |
|
| 2021-02-05 12:24:13 |
Elod Illes |
nova/train: status |
Fix Committed |
Fix Released |
|
| 2021-02-05 12:32:47 |
Elod Illes |
nova/ussuri: status |
Fix Committed |
Fix Released |
|
| 2021-02-08 05:20:54 |
Hemanth Nakkina |
attachment added |
|
lp1892361_stein.debdiff https://bugs.launchpad.net/nova/queens/+bug/1892361/+attachment/5461168/+files/lp1892361_stein.debdiff |
|
| 2021-02-15 14:54:50 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2021-02-15 15:04:55 |
Launchpad Janitor |
nova (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2021-02-15 15:12:24 |
Chris MacNaughton |
cloud-archive/ussuri: status |
Fix Committed |
Fix Released |
|
| 2021-02-17 13:08:35 |
Corey Bryant |
cloud-archive/train: status |
New |
Fix Committed |
|
| 2021-02-17 13:08:38 |
Corey Bryant |
tags |
compute pci resource-tracker verification-done-focal verification-needed verification-ussuri-done |
compute pci resource-tracker verification-done-focal verification-needed verification-train-needed verification-ussuri-done |
|
| 2021-02-22 02:03:13 |
Hemanth Nakkina |
nova/stein: status |
In Progress |
Fix Committed |
|
| 2021-03-01 07:08:11 |
Hemanth Nakkina |
tags |
compute pci resource-tracker verification-done-focal verification-needed verification-train-needed verification-ussuri-done |
compute pci resource-tracker verification-done-focal verification-needed verification-train-done verification-ussuri-done |
|
| 2021-03-01 16:06:25 |
Chris MacNaughton |
cloud-archive/train: status |
Fix Committed |
Fix Released |
|
| 2021-05-24 18:20:41 |
Corey Bryant |
cloud-archive/stein: status |
New |
Fix Committed |
|
| 2021-05-24 18:20:45 |
Corey Bryant |
tags |
compute pci resource-tracker verification-done-focal verification-needed verification-train-done verification-ussuri-done |
compute pci resource-tracker verification-done-focal verification-needed verification-stein-needed verification-train-done verification-ussuri-done |
|
| 2021-05-25 10:24:35 |
OpenStack Infra |
cloud-archive/rocky: status |
New |
Fix Committed |
|
| 2021-05-25 10:44:57 |
OpenStack Infra |
cloud-archive/queens: status |
New |
In Progress |
|
| 2021-06-03 14:01:00 |
OpenStack Infra |
cloud-archive/queens: status |
In Progress |
Fix Committed |
|
| 2021-06-07 11:25:50 |
Hemanth Nakkina |
nova/rocky: status |
In Progress |
Fix Committed |
|
| 2021-06-07 11:26:07 |
Hemanth Nakkina |
nova/queens: status |
In Progress |
Fix Committed |
|
| 2021-06-07 13:37:47 |
Edward Hope-Morley |
nova/rocky: status |
Fix Committed |
New |
|
| 2021-06-07 13:38:01 |
Edward Hope-Morley |
nova/queens: status |
Fix Committed |
New |
|
| 2021-06-14 06:20:36 |
Hemanth Nakkina |
tags |
compute pci resource-tracker verification-done-focal verification-needed verification-stein-needed verification-train-done verification-ussuri-done |
compute pci resource-tracker verification-done-focal verification-needed verification-stein-done verification-train-done verification-ussuri-done |
|
| 2021-06-16 11:03:31 |
Robie Basak |
nova (Ubuntu Bionic): status |
New |
Fix Committed |
|
| 2021-06-16 11:03:35 |
Robie Basak |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2021-06-16 11:03:43 |
Robie Basak |
tags |
compute pci resource-tracker verification-done-focal verification-needed verification-stein-done verification-train-done verification-ussuri-done |
compute pci resource-tracker verification-done-focal verification-needed verification-needed-bionic verification-stein-done verification-train-done verification-ussuri-done |
|
| 2021-06-16 12:20:45 |
Chris MacNaughton |
tags |
compute pci resource-tracker verification-done-focal verification-needed verification-needed-bionic verification-stein-done verification-train-done verification-ussuri-done |
compute pci resource-tracker verification-done-focal verification-needed verification-needed-bionic verification-rocky-needed verification-stein-done verification-train-done verification-ussuri-done |
|
| 2021-06-17 10:38:14 |
Hemanth Nakkina |
tags |
compute pci resource-tracker verification-done-focal verification-needed verification-needed-bionic verification-rocky-needed verification-stein-done verification-train-done verification-ussuri-done |
compute pci resource-tracker verification-done-focal verification-needed verification-needed-bionic verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done |
|
| 2021-06-17 11:58:38 |
Hemanth Nakkina |
tags |
compute pci resource-tracker verification-done-focal verification-needed verification-needed-bionic verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done |
compute pci resource-tracker verification-done-bionic verification-done-focal verification-needed verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done |
|
| 2021-06-18 08:14:49 |
Chris MacNaughton |
tags |
compute pci resource-tracker verification-done-bionic verification-done-focal verification-needed verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done |
compute pci resource-tracker verification-done-bionic verification-done-focal verification-needed verification-queens-needed verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done |
|
| 2021-06-21 05:57:58 |
Hemanth Nakkina |
tags |
compute pci resource-tracker verification-done-bionic verification-done-focal verification-needed verification-queens-needed verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done |
compute pci resource-tracker verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done |
|
| 2021-06-24 17:46:02 |
Launchpad Janitor |
nova (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2021-06-28 13:26:42 |
Chris MacNaughton |
cloud-archive/stein: status |
Fix Committed |
Fix Released |
|
| 2021-06-28 13:27:43 |
Chris MacNaughton |
cloud-archive/rocky: status |
Fix Committed |
Fix Released |
|
| 2021-06-28 13:30:51 |
Chris MacNaughton |
cloud-archive/queens: status |
Fix Committed |
Fix Released |
|
| 2021-07-18 03:31:42 |
Billy Olsen |
nova/rocky: status |
New |
Fix Released |
|
| 2021-07-18 03:31:57 |
Billy Olsen |
nova/queens: status |
New |
Fix Released |
|
| 2021-09-13 10:56:31 |
James Page |
nova/stein: status |
Fix Committed |
Fix Released |
|
