Jon, Looks like a qemu issue for the Server Team to take a look at. Michael On 03/31/2017 02:59 PM, Launchpad Bug Tracker wrote: > bugproxy (bugproxy) has assigned this bug to you for Ubuntu: > > ---Problem Description--- > I am trying to do hotplug attach with Mellanox CX3 card to a guest but I get failure. > virsh attach-device powerio-le12-ubuntu-17.04 ./add_cx3.xml > error: Failed to attach device from ./add_cx3.xml > error: internal error: unable to execute QEMU command 'device_add': vfio error: 0044:01:00.0: failed to setup container for group 6: RAM memory listener initialization failed for container > > > from the log file from qemu I see this: > 2017-02-14T22:55:40.721108Z qemu-system-ppc64: backend does not support BE vnet headers; falling back on use rspace virtio > > This is with kernel 4.9.0-15-generic and qemu level: > dpkg --list| grep qemu > ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu2 all PXE boot firmware - ROM images for qemu > ii qemu 1:2.8+dfsg-2ubuntu1 ppc64el fast processor emulator > ii qemu-block-extra:ppc64el 1:2.8+dfsg-2ubuntu1 ppc64el extra block backend modules for qemu-system and qemu-utils > ii qemu-kvm 1:2.8+dfsg-2ubuntu1 ppc64el QEMU Full virtualization > ii qemu-slof 20161019+dfsg-1 all Slimline Open Firmware -- QEMU PowerPC version > ii qemu-system 1:2.8+dfsg-2ubuntu1 ppc64el QEMU full system emulation binaries > ii qemu-system-arm 1:2.8+dfsg-2ubuntu1 ppc64el QEMU full system emulation binaries (arm) > ii qemu-system-common 1:2.8+dfsg-2ubuntu1 ppc64el QEMU full system emulation binaries (common files) > ii qemu-system-mips 1:2.8+dfsg-2ubuntu1 ppc64el QEMU full system emulation binaries (mips) > ii qemu-system-misc 1:2.8+dfsg-2ubuntu1 ppc64el QEMU full system emulation binaries (miscellaneous) > ii qemu-system-ppc 1:2.8+dfsg-2ubuntu1 ppc64el QEMU full system emulation binaries (ppc) > ii qemu-system-sparc 1:2.8+dfsg-2ubuntu1 ppc64el QEMU full system emulation binaries (sparc) > ii qemu-system-x86 1:2.8+dfsg-2ubuntu1 ppc64el QEMU full system emulation binaries (x86) > ii qemu-user 1:2.8+dfsg-2ubuntu1 ppc64el QEMU user mode emulation binaries > ii qemu-user-binfmt 1:2.8+dfsg-2ubuntu1 ppc64el QEMU user mode binfmt registration for qemu-user > ii qemu-utils 1:2.8+dfsg-2ubuntu1 ppc64el QEMU utilities > > > ---uname output--- > 4.9.0-15-generic #16-Ubuntu SMP Fri Jan 20 15:28:49 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux > > Machine Type = P8 > > ---Steps to Reproduce--- > bring up a guest and then try to attach device like this: > virsh attach-device powerio-le12-ubuntu-17.04 ./add_cx3.xml --live > error: Failed to attach device from ./add_cx3.xml > error: internal error: unable to execute QEMU command 'device_add': vfio error: 0044:01:00.0: failed to setup container for group 6: RAM memory listener initialization failed for container > > When I retried the steps for add_cx3.xml on the same machine I noticed > the following in the host logs: > > [ 1374.276210] KVM guest htab at c000001e56000000 (order 26), LPID 1 > [ 1383.824281] hrtimer: interrupt took 923 ns > [ 1447.479194] audit_printk_skb: 15 callbacks suppressed > [ 1447.479198] audit: type=1400 audit(1487194729.006:17): apparmor="DENIED" operation="setrlimit" profile="/usr/sbin/libvirtd" pid=6853 comm="libvirtd" rlimit=memlock value=8694792192 > [ 1447.481927] pci 0044:01 : [PE# 002] Disabling 64-bit DMA bypass > [ 1447.481935] pci 0044:01 : [PE# 002] Removing DMA window #0 > [ 1447.481978] pci 0044:01 : [PE# 002] Removing DMA window #0 > [ 1447.481980] pci 0044:01 : [PE# 002] Removing DMA window #1 > [ 1447.485667] pci 0044:01 : [PE# 002] Setting up window#0 0..7fffffff pg=1000 > [ 1447.485670] pci 0044:01 : [PE# 002] Enabling 64-bit DMA bypass > [ 1517.030701] audit: type=1400 audit(1487194798.559:18): apparmor="DENIED" operation="setrlimit" profile="/usr/sbin/libvirtd" pid=6853 comm="libvirtd" rlimit=memlock value=8694792192 > [ 1517.033286] pci 0044:01 : [PE# 002] Disabling 64-bit DMA bypass > [ 1517.033290] pci 0044:01 : [PE# 002] Removing DMA window #0 > [ 1517.033322] pci 0044:01 : [PE# 002] Removing DMA window #0 > [ 1517.033325] pci 0044:01 : [PE# 002] Removing DMA window #1 > [ 1517.036971] pci 0044:01 : [PE# 002] Setting up window#0 0..7fffffff pg=1000 > [ 1517.036974] pci 0044:01 : [PE# 002] Enabling 64-bit DMA bypass > > I'm not sure if the apparmor issues are affecting functionality or not. > That may be worth looking into a separate bug, or a dupe of > https://bugzilla.linux.ibm.com/show_bug.cgi?id=146192 > > As noted there I did the following to work around it: > > sudo aa-complain /usr/sbin/libvirtd > sudo aa-complain /etc/apparmor.d/libvirt/libvirt-????????-????-????-????-???????????? > > I still got the VFIO memory listener error however. If I install QEMU > 2.7.0 I no longer see the VFIO error and things seems to succeed from a > host perspective: > > root@powerio-le11:/etc/libvirt/qemu# virsh attach-device powerio-le12-ubuntu-17.04 ./add_cx3.xml --live > Device attached successfully > > root@powerio-le11:/etc/libvirt/qemu# dmesg | tail -6 > [ 3880.813971] KVM guest htab at c000001e56000000 (order 26), LPID 1 > [ 3917.656384] audit: type=1400 audit(1487197199.210:26): apparmor="ALLOWED" operation="setrlimit" profile="/usr/sbin/libvirtd" pid=6853 comm="libvirtd" rlimit=memlock value=8694792192 > [ 3917.659276] pci 0044:01 : [PE# 002] Disabling 64-bit DMA bypass > [ 3917.659284] pci 0044:01 : [PE# 002] Removing DMA window #0 > [ 3917.688803] vfio-pci 0044:01:00.0: enabling device (0400 -> 0402) > [ 3917.800106] vfio_ecap_init: 0044:01:00.0 hiding ecap 0x19@0x18c > > In the guest things look okay initially: > > [ 28.797667] RTAS: event: 1, Type: Unknown, Severity: 1 > [ 29.062821] pci 0000:00:05.0: [15b3:1007] type 00 class 0x020000 > [ 29.063118] pci 0000:00:05.0: reg 0x10: [mem 0x100a0000000-0x100a00fffff 64bit] > [ 29.063341] pci 0000:00:05.0: reg 0x18: [mem 0x2c0200000000-0x2c0201ffffff 64bit pref] > [ 29.063701] pci 0000:00:05.0: reg 0x30: [mem 0x00000000-0x000fffff pref] > [ 29.065237] iommu: Adding device 0000:00:05.0 to group 0 > [ 29.065332] pci 0000:00:05.0: BAR 2: assigned [mem 0x10122000000-0x10123ffffff 64bit pref] > [ 29.065675] pci 0000:00:05.0: BAR 0: assigned [mem 0x10121800000-0x101218fffff 64bit] > [ 29.066010] pci 0000:00:05.0: BAR 6: assigned [mem 0x100a0000000-0x100a00fffff pref] > [ 29.066105] mlx4_core: Mellanox ConnectX core driver v4.0-1.0.1 (29 Jan 2017) > [ 29.066127] mlx4_core: Initializing 0000:00:05.0 > [ 29.066210] mlx4_core 0000:00:05.0: enabling device (0000 -> 0002) > [ 29.076273] mlx4_core 0000:00:05.0: Using 64-bit direct DMA at offset 800000000000000 > > > but eventually I see the following error: > > > [ 89.925954] mlx4_core 0000:00:05.0: device is going to be reset > [ 99.923755] mlx4_core 0000:00:05.0: Failed to obtain HW semaphore, aborting > [ 99.924052] mlx4_core 0000:00:05.0: Fail to reset HCA > [ 99.924305] kernel BUG at /var/lib/dkms/mlnx-ofed-kernel/4.0/build/drivers/net/ethernet/mellanox/mlx4/catas.c:193! > [ 99.924643] Oops: Exception in kernel mode, sig: 5 [#1] > [ 99.924811] SMP NR_CPUS=2048 [ 99.924889] NUMA > [ 99.924968] pSeries > [ 99.925048] Modules linked in: rdma_ucm(OE) ib_ucm(OE) ib_ipoib(OE) ib_uverbs(OE) ib_umad(OE) mlx5_ib(OE) mlx5_core(OE) mlx4_ib(OE) mlx4_en(OE) mlx4_core(OE) devlink vmx_crypto ib_iser rdma_cm(OE) iw_cm(OE) ib_cm(OE) ib_core(OE) mlx_compat(OE) configfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi knem(OE) ip_tables x_tables autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ibmvscsi crc32c_vpmsum virtio_net virtio_blk > [ 99.927029] CPU: 10 PID: 4600 Comm: drmgr Tainted: G OE 4.9.0-12-generic #13-Ubuntu > [ 99.927316] task: c0000001dfc27e00 task.stack: c0000001dd630000 > [ 99.927515] NIP: d000000003c62794 LR: d000000003c6277c CTR: c0000000006c4a80 > [ 99.927752] REGS: c0000001dd6332a0 TRAP: 0700 Tainted: G OE (4.9.0-12-generic) > [ 99.928029] MSR: 800000010282b033 [ 99.928645] CR: 48022222 XER: 20000000 > [ 99.928764] CFAR: c000000000710c28 SOFTE: 1 > GPR00: d000000003c6277c c0000001dd633520 d000000003cbca7c 0000000000000029 > GPR04: 0000000000000001 000000000000014d 6573657420484341 0d0a6c20746f2072 > GPR08: 0000000000000007 0000000000000001 c0000001e2a94300 0000000000000006 > GPR12: 0000000000002200 c00000000fb85a00 c0000001dd9a0060 0000000000000000 > GPR16: 00000000024080c0 00000000024202c2 0000000000000000 d000000003cb7418 > GPR20: 0000000000000000 d0000800802c0680 c0000001dd9a04e8 c0000001dd9a0518 > GPR24: 000000000000ea60 0000000000000000 c000000001443a00 c0000001dd6336f0 > GPR28: 0000000000000000 0000000000000004 c0000001e2a94360 c0000001dd9a0060 > NIP [d000000003c62794] mlx4_enter_error_state.part.0+0x35c/0x460 [mlx4_core] > [ 99.931952] LR [d000000003c6277c] mlx4_enter_error_state.part.0+0x344/0x460 [mlx4_core] > [ 99.932190] Call Trace: > [ 99.932278] [c0000001dd633520] [d000000003c6277c] mlx4_enter_error_state.part.0+0x344/0x460 [mlx4_core] (unreliable) > [ 99.932647] [c0000001dd6335b0] [d000000003c66df8] __mlx4_cmd+0x720/0x970 [mlx4_core] > [ 99.932946] [c0000001dd633680] [d000000003c73d88] mlx4_QUERY_FW+0x90/0x420 [mlx4_core] > [ 99.933238] [c0000001dd633730] [d000000003c7fd28] mlx4_load_one+0x440/0x1ac0 [mlx4_core] > [ 99.933520] [c0000001dd633850] [d000000003c81a40] mlx4_init_one+0x698/0x7c0 [mlx4_core] > [ 99.933922] [c0000001dd633960] [c00000000063049c] local_pci_probe+0x6c/0x140 > [ 99.934171] [c0000001dd6339f0] [c0000000006312e8] pci_device_probe+0x178/0x200 > [ 99.934430] [c0000001dd633a50] [c000000000716970] driver_probe_device+0x240/0x540 > [ 99.934657] [c0000001dd633ae0] [c00000000071344c] bus_for_each_drv+0x8c/0xf0 > [ 99.934848] [c0000001dd633b30] [c0000000007164f0] __device_attach+0x140/0x210 > [ 99.935057] [c0000001dd633bc0] [c000000000621d38] pci_bus_add_device+0x78/0x100 > [ 99.935270] [c0000001dd633c30] [c000000000621e20] pci_bus_add_devices+0x60/0xe0 > [ 99.935488] [c0000001dd633c70] [c000000000625b44] pci_rescan_bus+0x44/0x70 > [ 99.935666] [c0000001dd633ca0] [c000000000631ee4] bus_rescan_store+0x84/0xb0 > [ 99.935840] [c0000001dd633ce0] [c000000000712fb4] bus_attr_store+0x44/0x70 > [ 99.936039] [c0000001dd633d00] [c0000000003d52b8] sysfs_kf_write+0x68/0xa0 > [ 99.936210] [c0000001dd633d20] [c0000000003d417c] kernfs_fop_write+0x17c/0x250 > [ 99.936407] [c0000001dd633d70] [c00000000031924c] __vfs_write+0x3c/0x70 > [ 99.936583] [c0000001dd633d90] [c00000000031a4b4] vfs_write+0xd4/0x240 > [ 99.936760] [c0000001dd633de0] [c00000000031c018] SyS_write+0x68/0x110 > [ 99.936934] [c0000001dd633e30] [c00000000000bd84] system_call+0x38/0xe0 > [ 99.937102] Instruction dump: > [ 99.937188] e93f0000 3d020000 e8888078 e8690000 386300a0 4803f8f1 e8410018 e95f0000 > [ 99.937472] e92a0000 81290098 2f890001 409efea0 <0fe00000> 60000000 60420000 e93f0000 > [ 99.937726] ---[ end trace 66826e43e8c8b7ba ]--- > [ 99.937832] > > It's not clear to me if this new guest issue is specific to QEMU 2.7, or > something that would also be present on 2.8 if not for the VFIO issue > originally noted in this bug. First step I think will be to root-cause > the VFIO issue, fix it, and see if the guest issue remains afterward. If > it does we can track that as a separate bug (or perhaps we already seen > this somewhere? seems vaguely familiar). > > Need to hop of machine for today, but can look at it more tomorrow. > > (In reply to comment #10) > >> [ 1517.030701] audit: type=1400 audit(1487194798.559:18): apparmor="DENIED" >> operation="setrlimit" profile="/usr/sbin/libvirtd" pid=6853 comm="libvirtd" >> rlimit=memlock value=8694792192 >> I'm not sure if the apparmor issues are affecting functionality or not. That >> may be worth looking into a separate bug, or a dupe of >> https://bugzilla.linux.ibm.com/show_bug.cgi?id=146192 >> > Let me check again the Ubuntu 16.10 system because I did the same steps to update the /etc/libvirt/qemu.conf in Ubuntu 17.04 like I did in 16.10 but still see it. Not sure if I did something else. > >> It's not clear to me if this new guest issue is specific to QEMU 2.7, or >> something that would also be present on 2.8 if not for the VFIO issue >> originally noted in this bug. First step I think will be to root-cause the >> VFIO issue, fix it, and see if the guest issue remains afterward. If it does >> we can track that as a separate bug (or perhaps we already seen this >> somewhere? seems vaguely familiar). >> >> Need to hop of machine for today, but can look at it more tomorrow. > For this I see it with Ubuntu 16.10 KVM and the issue is the command are timing out like the dmas are not getting to the HW. I can see this with any Mellanox card I had tried. I can open separate bug more specific to 16.10 if you want. > > == Comment: #15 - MICHAEL D. ROTH