Activity log for bug #1829652

Date Who What changed Old value New value Message
2019-05-19 12:29:57 Damien Claisse bug added bug
2019-05-19 12:30:53 Damien Claisse attachment added Panic stack trace https://bugs.launchpad.net/ubuntu/+source/kernel-package/+bug/1829652/+attachment/5264773/+files/panic_stack_trace.txt
2019-05-19 12:32:25 Damien Claisse description Hi, I've recently upgraded my system to ubuntu 19.04, and kernel consistently panics when I boot a virtual machine using an SR-IOV VFIO virtual function. Could reproduce with 5.0.0-13-generic and 5.0.0-15-generic, doesn't happen with 4.18.0-20-generic. I'm using libvirt and VM is using KVM, XML snippet to delegate virtual function to VM is: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x10' function='0x0'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </hostdev> As soon as I start the VM using "virsh start vm", kernel panics. I'm using X550T intel NIC (lspci output: Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)) on an Supermicro X11SSH-CTF motherboard, with up to date bios and 4 virtual functions (class/net/em1/device/sriov_numvfs = 4). Given panic stack trace (see attachment), I've narrowed down the bug to ixgbe_ipsec_vf_clear function in upstream ixgbe driver: https://github.com/torvalds/linux/blob/72cf0b07418a9c8349aa9137194b1ccba6e54a9d/drivers/net/ethernet/intel/ixgbe/ixgbe_ipsec.c#L840 When I comment its usage (https://github.com/torvalds/linux/blob/72cf0b07418a9c8349aa9137194b1ccba6e54a9d/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c#L734), everything works as expected (I'm not using any ipsec feature). Didn't try to fix the ixgbe_ipsec_vf_clear function though. Please find attached a stack trace collected using IPMI serial-over-lan. Hi, I've recently upgraded my system to ubuntu 19.04, and kernel consistently panics when I boot a virtual machine using an SR-IOV VFIO virtual function. Could reproduce with 5.0.0-13-generic and 5.0.0-15-generic, doesn't happen with 4.18.0-20-generic. I'm using libvirt and VM is using KVM, XML snippet to delegate virtual function to VM is:     <hostdev mode='subsystem' type='pci' managed='yes'>       <driver name='vfio'/>       <source>         <address domain='0x0000' bus='0x05' slot='0x10' function='0x0'/>       </source>       <alias name='hostdev0'/>       <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>     </hostdev> As soon as I start the VM using "virsh start vm", kernel panics. I'm using X550T intel NIC (lspci output: Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)) on a Supermicro X11SSH-CTF motherboard, with up to date bios and 4 virtual functions (class/net/em1/device/sriov_numvfs = 4). Given panic stack trace (see attachment), I've narrowed down the bug to ixgbe_ipsec_vf_clear function in upstream ixgbe driver: https://github.com/torvalds/linux/blob/72cf0b07418a9c8349aa9137194b1ccba6e54a9d/drivers/net/ethernet/intel/ixgbe/ixgbe_ipsec.c#L840 When I comment its usage (https://github.com/torvalds/linux/blob/72cf0b07418a9c8349aa9137194b1ccba6e54a9d/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c#L734), everything works as expected (I'm not using any ipsec feature). Didn't try to fix the ixgbe_ipsec_vf_clear function though. Please find attached a stack trace collected using IPMI serial-over-lan.
2019-05-19 12:33:48 Damien Claisse affects kernel-package (Ubuntu) linux (Ubuntu)
2019-05-19 12:37:28 Damien Claisse attachment added lspci-vvnn.log https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1829652/+attachment/5264783/+files/lspci-vvnn.log
2019-05-19 13:00:09 Ubuntu Kernel Bot linux (Ubuntu): status New Incomplete
2019-05-19 13:00:12 Ubuntu Kernel Bot tags disco
2019-05-19 13:41:29 Damien Claisse linux (Ubuntu): status Incomplete Confirmed
2019-05-20 22:23:05 Terry Rudd bug added subscriber Terry Rudd
2019-05-22 20:46:39 dann frazier linux (Ubuntu): assignee dann frazier (dannf)
2019-05-22 20:46:57 dann frazier nominated for series Ubuntu Eoan
2019-05-22 20:46:57 dann frazier bug task added linux (Ubuntu Eoan)
2019-05-22 20:46:57 dann frazier nominated for series Ubuntu Disco
2019-05-22 20:46:57 dann frazier bug task added linux (Ubuntu Disco)
2019-05-22 20:47:09 dann frazier linux (Ubuntu Disco): status New In Progress
2019-05-22 20:47:11 dann frazier linux (Ubuntu Eoan): status Confirmed In Progress
2019-05-22 20:47:12 dann frazier linux (Ubuntu Disco): assignee dann frazier (dannf)
2019-05-23 19:08:15 dann frazier bug added subscriber dann frazier
2019-06-24 18:02:56 dann frazier description Hi, I've recently upgraded my system to ubuntu 19.04, and kernel consistently panics when I boot a virtual machine using an SR-IOV VFIO virtual function. Could reproduce with 5.0.0-13-generic and 5.0.0-15-generic, doesn't happen with 4.18.0-20-generic. I'm using libvirt and VM is using KVM, XML snippet to delegate virtual function to VM is:     <hostdev mode='subsystem' type='pci' managed='yes'>       <driver name='vfio'/>       <source>         <address domain='0x0000' bus='0x05' slot='0x10' function='0x0'/>       </source>       <alias name='hostdev0'/>       <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>     </hostdev> As soon as I start the VM using "virsh start vm", kernel panics. I'm using X550T intel NIC (lspci output: Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)) on a Supermicro X11SSH-CTF motherboard, with up to date bios and 4 virtual functions (class/net/em1/device/sriov_numvfs = 4). Given panic stack trace (see attachment), I've narrowed down the bug to ixgbe_ipsec_vf_clear function in upstream ixgbe driver: https://github.com/torvalds/linux/blob/72cf0b07418a9c8349aa9137194b1ccba6e54a9d/drivers/net/ethernet/intel/ixgbe/ixgbe_ipsec.c#L840 When I comment its usage (https://github.com/torvalds/linux/blob/72cf0b07418a9c8349aa9137194b1ccba6e54a9d/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c#L734), everything works as expected (I'm not using any ipsec feature). Didn't try to fix the ixgbe_ipsec_vf_clear function though. Please find attached a stack trace collected using IPMI serial-over-lan. [Impact] If you pass a vf of an intel 10G nic - such as the X550T - into a VM, and that nic does not support IPsec, the system will oops. [Test Case] Enable virtual functions (e.g. (class/net/em1/device/sriov_numvfs = 4). Use an XML snippet to delegate virtual function to VM, e.g.:     <hostdev mode='subsystem' type='pci' managed='yes'>       <driver name='vfio'/>       <source>         <address domain='0x0000' bus='0x05' slot='0x10' function='0x0'/>       </source>       <alias name='hostdev0'/>       <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>     </hostdev> Start VM. [Fix] https://git.kernel.org/pub/scm/linux/kernel/git/jkirsher/next-queue.git/commit/?h=dev-queue&id=a2243a4faae5a3c210995ade2cbca369b20daa0f [Regression Risk] Restricted to the ixgbe driver.
2019-06-27 05:39:02 Khaled El Mously linux (Ubuntu Disco): status In Progress Fix Committed
2019-07-03 11:03:14 Ubuntu Kernel Bot tags disco disco verification-needed-disco
2019-07-09 00:20:40 Damien Claisse tags disco verification-needed-disco verification-done-disco
2019-07-15 22:40:05 Launchpad Janitor linux (Ubuntu Eoan): status In Progress Fix Released
2019-07-23 05:25:24 Launchpad Janitor linux (Ubuntu Disco): status Fix Committed Fix Released
2019-07-23 05:25:24 Launchpad Janitor cve linked 2018-12126
2019-07-23 05:25:24 Launchpad Janitor cve linked 2018-12127
2019-07-23 05:25:24 Launchpad Janitor cve linked 2018-12130
2019-07-23 05:25:24 Launchpad Janitor cve linked 2019-11091
2019-07-23 05:25:24 Launchpad Janitor cve linked 2019-11833
2019-07-23 05:25:24 Launchpad Janitor cve linked 2019-11884