StarlingX

  1. Bug #1821938
  2. Activity log

Activity log for bug #1821938

Date Who What changed Old value New value Message
2019-03-27 15:12:25 Yang Liu bug added bug
2019-03-29 19:03:07 Ghada Khalil starlingx: importance Undecided High
2019-03-29 19:03:10 Ghada Khalil starlingx: status New Triaged
2019-03-29 19:03:34 Ghada Khalil tags stx.2019.05 stx.distro.openstack stx.helpwanted
2019-04-02 19:03:45 sean mooney bug task added nova
2019-04-02 19:04:07 sean mooney nova: importance Undecided High
2019-04-02 19:04:12 sean mooney nova: assignee sean mooney (sean-k-mooney)
2019-04-02 19:04:31 sean mooney nova: status New In Progress
2019-04-02 19:06:19 sean mooney tags stx.2019.05 stx.distro.openstack stx.helpwanted stein-rc-potential stx.2019.05 stx.distro.openstack stx.helpwanted
2019-04-02 19:36:01 Ghada Khalil tags stein-rc-potential stx.2019.05 stx.distro.openstack stx.helpwanted stein-rc-potential stx.2019.05 stx.distro.openstack
2019-04-02 19:36:28 Ghada Khalil starlingx: assignee Chris Friesen (cbf123)
2019-04-02 19:37:27 Ghada Khalil starlingx: assignee Chris Friesen (cbf123) Jim Gauld (jgauld)
2019-04-02 19:37:40 Ghada Khalil starlingx: status Triaged In Progress
2019-04-02 19:43:18 Ghada Khalil description Brief Description ----------------- Unable to enable a host as nova hypervisor due to pci device cannot be found if the host has QAT devices (C62x or DH895XCC) configured. Severity -------- Major Steps to Reproduce ------------------ - Install and configure a system where worker nodes have QAT devices configured. e.g., [wrsroot@controller-0 ~(keystone_admin)]$ system host-device-list compute-0 +------------------+--------------+----------+-----------+-----------+---------------------------+---------------------------------+----------------------------------------+-----------+---------+ | name | address | class id | vendor id | device id | class name | vendor name | device name | numa_node | enabled | +------------------+--------------+----------+-----------+-----------+---------------------------+---------------------------------+----------------------------------------+-----------+---------+ | pci_0000_09_00_0 | 0000:09:00.0 | 0b4000 | 8086 | 0435 | Co-processor | Intel Corporation | DH895XCC Series QAT | 0 | True | | pci_0000_0c_00_0 | 0000:0c:00.0 | 030000 | 102b | 0522 | VGA compatible controller | Matrox Electronics Systems Ltd. | MGA G200e [Pilot] ServerEngines (SEP1) | 0 | True | +------------------+--------------+----------+-----------+-----------+---------------------------+---------------------------------+----------------------------------------+-----------+---------+ compute-0:~$ lspci | grep QAT 09:00.0 Co-processor: Intel Corporation DH895XCC Series QAT 09:01.0 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Function 09:01.1 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Function ... - check nova hypervisor-list Expected Behavior ------------------ - Nova hypervisors exist on system Actual Behavior ---------------- [wrsroot@controller-0 ~(keystone_admin)]$ nova hypervisor-list +----+---------------------+-------+--------+ | ID | Hypervisor hostname | State | Status | +----+---------------------+-------+--------+ +----+---------------------+-------+--------+ Reproducibility --------------- Reproducible System Configuration -------------------- Any system type with QAT devices configured on worker node Branch/Pull Time/Commit ----------------------- master as of 2019-03-18 Last Pass -------------- on f/stein branch in early feb Timestamp/Logs -------------- # nova-compute pods are spewing errors so they can't register themselves properly as hypervisors: 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager [req-4f652d4c-da7e-4516-9baa-915265c3fdda - - - - -] Error updating resources for node compute-0.: PciDeviceNotFoundById: PCI device 0000:09:02.3 not found 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager Traceback (most recent call last): 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 7956, in _update_available_resource_for_node 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager startup=startup) 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 727, in update_available_resource 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager resources = self.driver.get_available_resource(nodename) 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager File "/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7098, in get_available_resource 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager self._get_pci_passthrough_devices() 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager File "/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6102, in _get_pci_passthrough_devices 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager pci_info.append(self._get_pcidev_info(name)) 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager File "/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6062, in _get_pcidev_info 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager device.update(_get_device_type(cfgdev, address)) 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager File "/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6021, in _get_device_type 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager pci_address, pf_interface=True), 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager File "/var/lib/openstack/lib/python2.7/site-packages/nova/pci/utils.py", line 159, in get_ifname_by_pci_address 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager raise exception.PciDeviceNotFoundById(id=pci_addr) 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager PciDeviceNotFoundById: PCI device 0000:09:02.3 not found 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager Brief Description ----------------- Unable to enable a host as nova hypervisor due to pci device cannot be found if the host has QAT devices (C62x or DH895XCC) configured. Severity -------- Major Steps to Reproduce ------------------ - Install and configure a system where worker nodes have QAT devices configured. e.g., [wrsroot@controller-0 ~(keystone_admin)]$ system host-device-list compute-0 +------------------+--------------+----------+-----------+-----------+---------------------------+---------------------------------+----------------------------------------+-----------+---------+ | name | address | class id | vendor id | device id | class name | vendor name | device name | numa_node | enabled | +------------------+--------------+----------+-----------+-----------+---------------------------+---------------------------------+----------------------------------------+-----------+---------+ | pci_0000_09_00_0 | 0000:09:00.0 | 0b4000 | 8086 | 0435 | Co-processor | Intel Corporation | DH895XCC Series QAT | 0 | True | | pci_0000_0c_00_0 | 0000:0c:00.0 | 030000 | 102b | 0522 | VGA compatible controller | Matrox Electronics Systems Ltd. | MGA G200e [Pilot] ServerEngines (SEP1) | 0 | True | +------------------+--------------+----------+-----------+-----------+---------------------------+---------------------------------+----------------------------------------+-----------+---------+ compute-0:~$ lspci | grep QAT 09:00.0 Co-processor: Intel Corporation DH895XCC Series QAT 09:01.0 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Function 09:01.1 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Function ... - check nova hypervisor-list Expected Behavior ------------------ - Nova hypervisors exist on system Actual Behavior ---------------- [wrsroot@controller-0 ~(keystone_admin)]$ nova hypervisor-list +----+---------------------+-------+--------+ | ID | Hypervisor hostname | State | Status | +----+---------------------+-------+--------+ +----+---------------------+-------+--------+ Reproducibility --------------- Reproducible System Configuration -------------------- Any system type with QAT devices configured on worker node Branch/Pull Time/Commit ----------------------- stx master as of 2019-03-18 Last Pass -------------- on f/stein branch in early feb Timestamp/Logs -------------- # nova-compute pods are spewing errors so they can't register themselves properly as hypervisors: 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager [req-4f652d4c-da7e-4516-9baa-915265c3fdda - - - - -] Error updating resources for node compute-0.: PciDeviceNotFoundById: PCI device 0000:09:02.3 not found 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager Traceback (most recent call last): 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 7956, in _update_available_resource_for_node 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager startup=startup) 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 727, in update_available_resource 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager resources = self.driver.get_available_resource(nodename) 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager File "/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7098, in get_available_resource 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager self._get_pci_passthrough_devices() 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager File "/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6102, in _get_pci_passthrough_devices 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager pci_info.append(self._get_pcidev_info(name)) 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager File "/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6062, in _get_pcidev_info 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager device.update(_get_device_type(cfgdev, address)) 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager File "/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6021, in _get_device_type 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager pci_address, pf_interface=True), 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager File "/var/lib/openstack/lib/python2.7/site-packages/nova/pci/utils.py", line 159, in get_ifname_by_pci_address 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager raise exception.PciDeviceNotFoundById(id=pci_addr) 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager PciDeviceNotFoundById: PCI device 0000:09:02.3 not found 2019-03-25 18:46:49,899.899 62394 ERROR nova.compute.manager
2019-04-02 19:52:04 Ghada Khalil bug added subscriber Ghada Khalil
2019-04-03 20:51:42 OpenStack Infra nova: status In Progress Fix Released
2019-04-05 01:01:21 OpenStack Infra tags stein-rc-potential stx.2019.05 stx.distro.openstack in-stable-stein stein-rc-potential stx.2019.05 stx.distro.openstack
2019-04-05 14:48:03 Ghada Khalil starlingx: status In Progress Fix Committed
2019-04-05 20:32:49 Ken Young tags in-stable-stein stein-rc-potential stx.2019.05 stx.distro.openstack in-stable-stein stein-rc-potential stx.2.0 stx.distro.openstack
2019-04-09 18:56:58 Ghada Khalil tags in-stable-stein stein-rc-potential stx.2.0 stx.distro.openstack in-stable-stein stein-rc-potential stx.2.0 stx.distro.openstack stx.retestneeded
2019-04-09 23:18:36 Maria Guadalupe Perez Ibara attachment added ALL_NODES_20190409.153749.tar https://bugs.launchpad.net/starlingx/+bug/1821938/+attachment/5254557/+files/ALL_NODES_20190409.153749.tar
2019-04-10 13:18:38 Ghada Khalil starlingx: status Fix Committed Fix Released
2019-04-10 23:36:19 Yang Liu tags in-stable-stein stein-rc-potential stx.2.0 stx.distro.openstack stx.retestneeded in-stable-stein stein-rc-potential stx.2.0 stx.distro.openstack