I think the issue is caused by pcieport_drv by which the MSI is enabled on 0021:02:01.0. Behind the PLX downstream port, there is surprise hotpluggable slot. When pnv-php.ko is loaded, it tries to enable MSI and the backtrace is thrown this time. Two things can be done to avoid this awkward situation:
(1) Prohibit pcieport_drv. It does nothing on PowerNV platform. pnv-php.ko doesn't depend on any functionalities exported by it.
(2) I will add some code in pnv-php.ko to skip enabling MSIx/MSI if the PLX downstream port has associated driver. With it, we won't see the backtrace hopefully. However, the surprise hotplug functionality is lost.
root@ltc84-pkvm1:~# lspci -vvvs 0021:02:01.0 | grep "Kernel driver in use"
Kernel driver in use: pcieport
root@ltc84-pkvm1:~# echo 0021:02:01.0 > /sys/bus/pci/drivers/pcieport/unbind
root@ltc84-pkvm1:~# lspci -vvvs 0021:02:01.0 | grep "Kernel driver in use"
root@ltc84-pkvm1:~# insmod /lib/modules/4.9.0-15-generic/kernel/drivers/pci/hotplug/pnv-php.ko
root@ltc84-pkvm1:~# dmesg | tail -30
[ 8.677661] audit: type=1400 audit(1487122689.888:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/libvirtd" pid=4220 comm="apparmor_parser"
[ 8.677671] audit: type=1400 audit(1487122689.888:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/libvirtd//qemu_bridge_helper" pid=4220 comm="apparmor_parser"
[ 8.679095] audit: type=1400 audit(1487122689.888:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="virt-aa-helper" pid=4219 comm="apparmor_parser"
[ 8.751964] audit: type=1400 audit(1487122689.960:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/sbin/dhclient" pid=4218 comm="apparmor_parser"
[ 8.751974] audit: type=1400 audit(1487122689.960:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=4218 comm="apparmor_parser"
[ 8.751979] audit: type=1400 audit(1487122689.960:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=4218 comm="apparmor_parser"
[ 8.751984] audit: type=1400 audit(1487122689.960:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=4218 comm="apparmor_parser"
[ 8.781147] audit: type=1400 audit(1487122689.992:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/tcpdump" pid=4222 comm="apparmor_parser"
[ 12.169618] ip6_tables: (C) 2000-2006 Netfilter Core Team
[ 12.196455] Ebtables v2.0 registered
[ 12.229787] systemd[1]: apt-daily.timer: Adding 8h 33min 26.836791s random time.
[ 12.238212] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[ 12.367033] IPv6: ADDRCONF(NETDEV_UP): enP34p1s0f0: link is not ready
[ 12.443774] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[ 12.460617] systemd[1]: apt-daily.timer: Adding 1h 54min 42.960931s random time.
[ 12.483745] Netfilter messages via NETLINK v0.30.
[ 12.539679] ip_set: protocol 6
[ 13.610428] Rounding down aligned max_sectors from 4294967295 to 4294967168
[ 14.424319] qla2xxx [0000:00:00.0]-0005: : QLogic Fibre Channel HBA Driver: 8.07.00.38-k.
[ 15.175489] tg3 0022:01:00.0 enP34p1s0f0: Link is up at 1000 Mbps, full duplex
[ 15.175502] tg3 0022:01:00.0 enP34p1s0f0: Flow control is off for TX and off for RX
[ 15.175509] tg3 0022:01:00.0 enP34p1s0f0: EEE is disabled
[ 15.175532] IPv6: ADDRCONF(NETDEV_CHANGE): enP34p1s0f0: link becomes ready
[ 17.712986] virbr0: port 1(virbr0-nic) entered blocking state
[ 17.712990] virbr0: port 1(virbr0-nic) entered disabled state
[ 17.713205] device virbr0-nic entered promiscuous mode
[ 17.901095] virbr0: port 1(virbr0-nic) entered blocking state
[ 17.901099] virbr0: port 1(virbr0-nic) entered listening state
[ 18.035009] virbr0: port 1(virbr0-nic) entered disabled state
[ 154.054440] PowerPC PowerNV PCI Hotplug Driver version: 0.1
I think the issue is caused by pcieport_drv by which the MSI is enabled on 0021:02:01.0. Behind the PLX downstream port, there is surprise hotpluggable slot. When pnv-php.ko is loaded, it tries to enable MSI and the backtrace is thrown this time. Two things can be done to avoid this awkward situation:
(1) Prohibit pcieport_drv. It does nothing on PowerNV platform. pnv-php.ko doesn't depend on any functionalities exported by it.
(2) I will add some code in pnv-php.ko to skip enabling MSIx/MSI if the PLX downstream port has associated driver. With it, we won't see the backtrace hopefully. However, the surprise hotplug functionality is lost.
root@ltc84-pkvm1:~# lspci -vvvs 0021:02:01.0 | grep "Kernel driver in use" pci/drivers/ pcieport/ unbind 4.9.0-15- generic/ kernel/ drivers/ pci/hotplug/ pnv-php. ko 9.888:2) : apparmor="STATUS" operation= "profile_ load" profile= "unconfined" name="/ usr/sbin/ libvirtd" pid=4220 comm="apparmor_ parser" 9.888:3) : apparmor="STATUS" operation= "profile_ load" profile= "unconfined" name="/ usr/sbin/ libvirtd/ /qemu_bridge_ helper" pid=4220 comm="apparmor_ parser" 9.888:4) : apparmor="STATUS" operation= "profile_ load" profile= "unconfined" name="virt- aa-helper" pid=4219 comm="apparmor_ parser" 9.960:5) : apparmor="STATUS" operation= "profile_ load" profile= "unconfined" name="/ sbin/dhclient" pid=4218 comm="apparmor_ parser" 9.960:6) : apparmor="STATUS" operation= "profile_ load" profile= "unconfined" name="/ usr/lib/ NetworkManager/ nm-dhcp- client. action" pid=4218 comm="apparmor_ parser" 9.960:7) : apparmor="STATUS" operation= "profile_ load" profile= "unconfined" name="/ usr/lib/ NetworkManager/ nm-dhcp- helper" pid=4218 comm="apparmor_ parser" 9.960:8) : apparmor="STATUS" operation= "profile_ load" profile= "unconfined" name="/ usr/lib/ connman/ scripts/ dhclient- script" pid=4218 comm="apparmor_ parser" 9.992:9) : apparmor="STATUS" operation= "profile_ load" profile= "unconfined" name="/ usr/sbin/ tcpdump" pid=4222 comm="apparmor_ parser" NETDEV_ UP): enP34p1s0f0: link is not ready 00.0]-0005: : QLogic Fibre Channel HBA Driver: 8.07.00.38-k. NETDEV_ CHANGE) : enP34p1s0f0: link becomes ready
Kernel driver in use: pcieport
root@ltc84-pkvm1:~# echo 0021:02:01.0 > /sys/bus/
root@ltc84-pkvm1:~# lspci -vvvs 0021:02:01.0 | grep "Kernel driver in use"
root@ltc84-pkvm1:~# insmod /lib/modules/
root@ltc84-pkvm1:~# dmesg | tail -30
[ 8.677661] audit: type=1400 audit(148712268
[ 8.677671] audit: type=1400 audit(148712268
[ 8.679095] audit: type=1400 audit(148712268
[ 8.751964] audit: type=1400 audit(148712268
[ 8.751974] audit: type=1400 audit(148712268
[ 8.751979] audit: type=1400 audit(148712268
[ 8.751984] audit: type=1400 audit(148712268
[ 8.781147] audit: type=1400 audit(148712268
[ 12.169618] ip6_tables: (C) 2000-2006 Netfilter Core Team
[ 12.196455] Ebtables v2.0 registered
[ 12.229787] systemd[1]: apt-daily.timer: Adding 8h 33min 26.836791s random time.
[ 12.238212] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[ 12.367033] IPv6: ADDRCONF(
[ 12.443774] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[ 12.460617] systemd[1]: apt-daily.timer: Adding 1h 54min 42.960931s random time.
[ 12.483745] Netfilter messages via NETLINK v0.30.
[ 12.539679] ip_set: protocol 6
[ 13.610428] Rounding down aligned max_sectors from 4294967295 to 4294967168
[ 14.424319] qla2xxx [0000:00:
[ 15.175489] tg3 0022:01:00.0 enP34p1s0f0: Link is up at 1000 Mbps, full duplex
[ 15.175502] tg3 0022:01:00.0 enP34p1s0f0: Flow control is off for TX and off for RX
[ 15.175509] tg3 0022:01:00.0 enP34p1s0f0: EEE is disabled
[ 15.175532] IPv6: ADDRCONF(
[ 17.712986] virbr0: port 1(virbr0-nic) entered blocking state
[ 17.712990] virbr0: port 1(virbr0-nic) entered disabled state
[ 17.713205] device virbr0-nic entered promiscuous mode
[ 17.901095] virbr0: port 1(virbr0-nic) entered blocking state
[ 17.901099] virbr0: port 1(virbr0-nic) entered listening state
[ 18.035009] virbr0: port 1(virbr0-nic) entered disabled state
[ 154.054440] PowerPC PowerNV PCI Hotplug Driver version: 0.1
root@ltc84-pkvm1:~# ls /sys/bus/pci/slots/ pci/slots/ Slot3/address
Slot1 Slot2 Slot3 Slot4 Slot5
root@ltc84-pkvm1:~# cat /sys/bus/
0021:03