root@ltcalpine-lp4:/home/ubuntu# lspci -v 01:00.0 Serial controller: Digi International Device 00f6 (prog-if 06 [16950]) Subsystem: IBM Device 0422 Flags: fast devsel, IRQ 504 Memory at 3fc200400000 (32-bit, non-prefetchable) [size=16K] Memory at 3fc200200000 (32-bit, non-prefetchable) [size=2M] Memory at 3fc200000000 (32-bit, non-prefetchable) [size=2M] Capabilities: [40] Power Management version 3 Capabilities: [70] Express Endpoint, MSI 00 Capabilities: [b0] MSI-X: Enable- Count=16 Masked- Capabilities: [100] Device Serial Number 00-30-e0-11-11-00-01-50 Capabilities: [110] Power Budgeting Kernel driver in use: serial root@ltcalpine-lp4:/home/ubuntu/ibm-lop-utils/src/errinjct# ./errinjct eeh -v -f 0 -p U78C7.001.RCH0040-P1-C12 Injecting an ioa-bus-error with the following data: BUS ADDR: 00000000 ADDR MASK: 00000000 CONFIG ADDR: 10000 PHB UNIT_ID: 800000020000019 FUNCTION: 0 Load to PCI Memory Address Space - inject an Address Parity Error Call to RTAS errinjct succeeded! If the correct information was provided and there is activity on the bus, the hardware should hit the error However, if incorrect information was provided or there is no bus activity, you may not get a hit. root@ltcalpine-lp4:/home/ubuntu/ibm-lop-utils/src/errinjct# dmesg [ 0.000000] Allocated 4980736 bytes for 2048 pacas at c00000000e7e0000 [ 0.000000] Using pSeries machine description [ 0.000000] Page sizes from device-tree: [ 0.000000] base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=0 [ 0.000000] base_shift=12: shift=16, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=7 [ 0.000000] base_shift=12: shift=24, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=56 [ 0.000000] base_shift=16: shift=16, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=1 [ 0.000000] base_shift=16: shift=24, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=8 [ 0.000000] base_shift=24: shift=24, sllp=0x0100, avpnm=0x00000001, tlbiel=0, penc=0 [ 0.000000] base_shift=34: shift=34, sllp=0x0120, avpnm=0x000007ff, tlbiel=0, penc=3 [ 0.000000] Page orders: linear mapping = 24, virtual = 16, io = 16, vmemmap = 24 [ 0.000000] Using 1TB segments [ 0.000000] Found initrd at 0xc00000000a000000:0xc00000000b48b1cc [ 0.000000] bootconsole [udbg0] enabled [ 0.000000] Partition configured for 32 cpus. [ 0.000000] CPU maps initialized for 8 threads per core [ 0.000000] (thread shift is 3) [ 0.000000] Freed 4849664 bytes for unused pacas [ 0.000000] Starting Linux ppc64le #33~14.04.1-Ubuntu SMP Thu Feb 4 12:26:42 UTC 2016 [ 0.000000] ----------------------------------------------------- [ 0.000000] ppc64_pft_size = 0x1d [ 0.000000] phys_mem_size = 0xc80000000 [ 0.000000] cpu_features = 0x07fc7aec18500249 [ 0.000000] possible = 0x1fffffef18500649 [ 0.000000] always = 0x0000000018100040 [ 0.000000] cpu_user_features = 0xdc0065c7 0xef000000 [ 0.000000] mmu_features = 0x7c000003 [ 0.000000] firmware_features = 0x00000001c45ffc5f [ 0.000000] htab_hash_mask = 0x3fffff [ 0.000000] ----------------------------------------------------- [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 4.2.0-28-generic (buildd@bos01-ppc64el-011) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #33~14.04.1-Ubuntu SMP Thu Feb 4 12:26:42 UTC 2016 (Ubuntu 4.2.0-28.33~14.04.1-generic 4.2.8-ckt3) [ 0.000000] Node 0 Memory: [ 0.000000] Node 1 Memory: 0x0-0xc80000000 [ 0.000000] numa: Initmem setup node 0 [ 0.000000] numa: NODE_DATA [mem 0xc7fedd000-0xc7fee7fff] [ 0.000000] numa: NODE_DATA(0) on node 1 [ 0.000000] numa: Initmem setup node 1 [mem 0x00000000-0xc7fffffff] [ 0.000000] numa: NODE_DATA [mem 0xc7fed2000-0xc7fedcfff] [ 0.000000] Section 3197 and 3199 (node 1) have a circular dependency on usemap and pgdat allocations [ 0.000000] PCI host bridge /pci@800000020000019 ranges: [ 0.000000] MEM 0x00003fc200000000..0x00003fc27effffff -> 0x0000000080000000 [ 0.000000] MEM 0x0000304000000000..0x0000304fffffffff -> 0x0003d04000000000 [ 0.000000] PPC64 nvram contains 15360 bytes [ 0.000000] Top of RAM: 0xc80000000, Total RAM: 0xc80000000 [ 0.000000] Memory hole size: 0MB [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000000000-0x0000000c7fffffff] [ 0.000000] DMA32 empty [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 1: [mem 0x0000000000000000-0x0000000c7fffffff] [ 0.000000] Could not find start_pfn for node 0 [ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0xffffffffffffffff] [ 0.000000] On node 0 totalpages: 0 [ 0.000000] Initmem setup node 1 [mem 0x0000000000000000-0x0000000c7fffffff] [ 0.000000] On node 1 totalpages: 819200 [ 0.000000] DMA zone: 800 pages used for memmap [ 0.000000] DMA zone: 0 pages reserved [ 0.000000] DMA zone: 819200 pages, LIFO batch:1 [ 0.000000] PERCPU: Embedded 3 pages/cpu @c000000c7f600000 s123800 r0 d72808 u262144 [ 0.000000] pcpu-alloc: s123800 r0 d72808 u262144 alloc=1*1048576 [ 0.000000] pcpu-alloc: [0] 00 01 02 03 [0] 04 05 06 07 [ 0.000000] pcpu-alloc: [0] 08 09 10 11 [0] 12 13 14 15 [ 0.000000] pcpu-alloc: [0] 16 17 18 19 [0] 20 21 22 23 [ 0.000000] pcpu-alloc: [0] 24 25 26 27 [0] 28 29 30 31 [ 0.000000] Built 2 zonelists in Node order, mobility grouping on. Total pages: 818400 [ 0.000000] Policy zone: DMA [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinux-4.2.0-28-generic root=UUID=e1f93728-1fdc-418f-bd81-c074f7d55f29 ro splash quiet vt.handoff=7 [ 0.000000] log_buf_len individual max cpu contribution: 4096 bytes [ 0.000000] log_buf_len total cpu_extra contributions: 126976 bytes [ 0.000000] log_buf_len min size: 131072 bytes [ 0.000000] log_buf_len: 262144 bytes [ 0.000000] early log buf free: 126128(96%) [ 0.000000] PID hash table entries: 4096 (order: -1, 32768 bytes) [ 0.000000] Sorting __ex_table... [ 0.000000] Memory: 52221888K/52428800K available (10816K kernel code, 1344K rwdata, 3548K rodata, 6016K init, 2816K bss, 206912K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=32, Nodes=2 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] Build-time adjustment of leaf fanout to 64. [ 0.000000] RCU restricting CPUs from NR_CPUS=2048 to nr_cpu_ids=32. [ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=32 [ 0.000000] NR_IRQS:512 nr_irqs:512 16 [ 0.000000] pic: no ISA interrupt controller [ 0.000000] time_init: decrementer frequency = 512.000000 MHz [ 0.000000] time_init: processor frequency = 3026.000000 MHz [ 0.000001] clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x761537d007, max_idle_ns: 440795202126 ns [ 0.000003] clocksource: timebase mult[1f40000] shift[24] registered [ 0.000005] clockevent: decrementer mult[83126e98] shift[32] cpu[0] [ 0.000054] vt handoff: transparent VT on vt#7 [ 0.000063] Console: colour dummy device 80x25 [ 0.000075] console [hvc0] enabled [ 0.000077] bootconsole [udbg0] disabled [ 0.000098] mempolicy: Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl [ 0.000102] pid_max: default: 32768 minimum: 301 [ 0.000139] Security Framework initialized [ 0.000152] AppArmor: AppArmor initialized [ 0.000153] Yama: becoming mindful. [ 0.000625] Dentry cache hash table entries: 8388608 (order: 10, 67108864 bytes) [ 0.015465] Inode-cache hash table entries: 4194304 (order: 9, 33554432 bytes) [ 0.022564] Mount-cache hash table entries: 131072 (order: 4, 1048576 bytes) [ 0.022573] Mountpoint-cache hash table entries: 131072 (order: 4, 1048576 bytes) [ 0.023247] Initializing cgroup subsys blkio [ 0.023250] Initializing cgroup subsys memory [ 0.023262] Initializing cgroup subsys devices [ 0.023264] Initializing cgroup subsys freezer [ 0.023267] Initializing cgroup subsys net_cls [ 0.023269] Initializing cgroup subsys perf_event [ 0.023272] Initializing cgroup subsys net_prio [ 0.023274] Initializing cgroup subsys hugetlb [ 0.023286] ftrace: allocating 28202 entries in 11 pages [ 0.034688] EEH: pSeries platform initialized [ 0.034699] POWER8 performance monitor hardware support registered [ 0.038966] Brought up 16 CPUs [ 0.038971] Node 0 CPUs: 0-15 [ 0.038974] Node 1 CPUs: [ 0.039703] devtmpfs: initialized [ 0.053812] evm: security.selinux [ 0.053814] evm: security.SMACK64 [ 0.053815] evm: security.SMACK64EXEC [ 0.053816] evm: security.SMACK64TRANSMUTE [ 0.053817] evm: security.SMACK64MMAP [ 0.053818] evm: security.ima [ 0.053819] evm: security.capability [ 0.053930] EEH: devices created [ 0.054078] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns [ 0.054491] NET: Registered protocol family 16 [ 0.056949] EEH: PCI Enhanced I/O Error Handling Enabled [ 0.056999] IBM eBus Device Driver [ 0.066728] cpuidle: using governor ladder [ 0.078787] cpuidle: using governor menu [ 0.078942] pstore: Registered nvram as persistent store backend [ 0.082111] PCI: Probing PCI hardware [ 0.082114] PCI: I/O resource not set for host bridge /pci@800000020000019 (domain 0) [ 0.082150] PCI host bridge to bus 0000:01 [ 0.082154] pci_bus 0000:01: root bus resource [mem 0x3fc200000000-0x3fc27effffff] (bus address [0x80000000-0xfeffffff]) [ 0.082157] pci_bus 0000:01: root bus resource [mem 0x304000000000-0x304fffffffff] (bus address [0x3d04000000000-0x3d04fffffffff]) [ 0.082159] pci_bus 0000:01: root bus resource [bus 01-ff] [ 0.084836] pci 0000:01:00.0: supports D1 D2 [ 0.084838] pci 0000:01:00.0: PME# supported from D1 D2 D3hot D3cold [ 0.175042] IOMMU table initialized, virtual merging enabled [ 0.175060] iommu: Adding device 0000:01:00.0 to group 0 [ 0.175660] PCI: Probing PCI hardware done [ 0.175773] pseries-rng: Registering arch random hook. [ 0.188063] vgaarb: loaded [ 0.188369] SCSI subsystem initialized [ 0.188453] libata version 3.00 loaded. [ 0.188498] usbcore: registered new interface driver usbfs [ 0.188508] usbcore: registered new interface driver hub [ 0.188599] usbcore: registered new device driver usb [ 0.188633] pps_core: LinuxPPS API ver. 1 registered [ 0.188635] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.188640] PTP clock support registered [ 0.188991] NetLabel: Initializing [ 0.188993] NetLabel: domain hash size = 128 [ 0.188994] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.189006] NetLabel: unlabeled traffic allowed by default [ 0.189176] clocksource: Switched to clocksource timebase [ 0.196865] AppArmor: AppArmor Filesystem Enabled [ 0.201358] NET: Registered protocol family 2 [ 0.201596] TCP established hash table entries: 524288 (order: 6, 4194304 bytes) [ 0.202636] TCP bind hash table entries: 65536 (order: 4, 1048576 bytes) [ 0.202768] TCP: Hash tables configured (established 524288 bind 65536) [ 0.202790] UDP hash table entries: 32768 (order: 4, 1048576 bytes) [ 0.202952] UDP-Lite hash table entries: 32768 (order: 4, 1048576 bytes) [ 0.203192] NET: Registered protocol family 1 [ 0.203311] PCI: CLS 128 bytes, default 128 [ 0.203357] Trying to unpack rootfs image as initramfs... [ 0.525409] Freeing initrd memory: 20992K (c00000000a000000 - c00000000b480000) [ 0.525610] RTAS daemon started [ 0.526319] RTAS: event: 51, Type: Platform Error, Severity: 2 [ 0.547312] hv-24x7: read 1330 catalog entries, created 1349 event attrs (0 failures), 169 descs [ 0.547979] futex hash table entries: 8192 (order: 4, 1048576 bytes) [ 0.548076] Initialise system trusted keyring [ 0.548095] audit: initializing netlink subsys (disabled) [ 0.548115] audit: type=2000 audit(1455531296.540:1): initialized [ 0.550091] HugeTLB registered 16 MB page size, pre-allocated 0 pages [ 0.550092] HugeTLB registered 16 GB page size, pre-allocated 0 pages [ 0.552226] zpool: loaded [ 0.552228] zbud: loaded [ 0.552454] VFS: Disk quotas dquot_6.6.0 [ 0.552512] VFS: Dquot-cache hash table entries: 8192 (order 0, 65536 bytes) [ 0.553100] fuse init (API version 7.23) [ 0.553261] Key type big_key registered [ 0.553891] Key type asymmetric registered [ 0.553895] Asymmetric key parser 'x509' registered [ 0.553909] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 0.554014] io scheduler noop registered [ 0.554018] io scheduler deadline registered (default) [ 0.554055] io scheduler cfq registered [ 0.554113] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.554518] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled [ 0.556183] serial 0000:01:00.0: enabling device (0140 -> 0142) [ 0.558441] ttyS0: detected caps 00000700 should be 00000500 [ 0.558444] 0000:01:00.0: ttyS0 at MMIO 0x3fc200401000 (irq = 504, base_baud = 4000000) is a 16C950/954 [ 0.558572] ttyS1: detected caps 00000700 should be 00000500 [ 0.558574] 0000:01:00.0: ttyS1 at MMIO 0x3fc200401200 (irq = 504, base_baud = 4000000) is a 16C950/954 [ 0.558734] Linux agpgart interface v0.103 [ 0.561080] brd: module loaded [ 0.562577] loop: module loaded [ 0.562908] ibmvscsi 3000000f: SRP_VERSION: 16.a [ 0.563247] scsi host0: IBM POWER Virtual SCSI Adapter 1.5.9 [ 0.563422] ibmvscsi 3000000f: partner initialization complete [ 0.563482] ibmvscsi 3000000f: host srp version: 16.a, host partition ltcalpine-vios (100), OS 3, max io 262144 [ 0.563579] ibmvscsi 3000000f: Client reserve enabled [ 0.563587] ibmvscsi 3000000f: sent SRP login [ 0.563637] ibmvscsi 3000000f: SRP_LOGIN succeeded [ 0.577376] libphy: Fixed MDIO Bus: probed [ 0.577379] tun: Universal TUN/TAP device driver, 1.6 [ 0.577380] tun: (C) 1999-2004 Max Krasnyansky [ 0.577460] ibmveth: IBM Power Virtual Ethernet Driver 1.05 [ 0.577554] scsi 0:0:1:0: Direct-Access AIX VDASD 0001 PQ: 0 ANSI: 3 [ 0.577604] ehea: IBM eHEA ethernet device driver (Release EHEA_0107) [ 0.577621] PPP generic driver version 2.4.2 [ 0.577690] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.577712] ehci-pci: EHCI PCI platform driver [ 0.577723] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.577727] ohci-pci: OHCI PCI platform driver [ 0.577737] uhci_hcd: USB Universal Host Controller Interface driver [ 0.577849] mousedev: PS/2 mouse device common for all mice [ 0.577882] scsi 0:0:2:0: CD-ROM AIX VOPTA PQ: 0 ANSI: 4 [ 0.577931] i2c /dev entries driver [ 0.578127] device-mapper: uevent: version 1.0.3 [ 0.578215] device-mapper: ioctl: 4.33.0-ioctl (2015-8-18) initialised: dm-devel@redhat.com [ 0.578667] pseries_idle_driver registered [ 0.578711] ledtrig-cpu: registered to indicate activity on CPUs [ 0.579026] NET: Registered protocol family 10 [ 0.579377] NET: Registered protocol family 17 [ 0.579390] Key type dns_resolver registered [ 0.579600] Loading compiled-in X.509 certificates [ 0.580382] Loaded X.509 cert 'Build time autogenerated kernel key: c391955df76f7c224024334dce9279d328f9072f' [ 0.580393] registered taskstats version 1 [ 0.580411] zswap: loading zswap [ 0.580413] zswap: using zbud pool [ 0.580417] zswap: using lzo compressor [ 0.583003] Key type trusted registered [ 0.586522] Key type encrypted registered [ 0.586528] AppArmor: AppArmor sha1 policy hashing enabled [ 0.586532] ima: No TPM chip found, activating TPM-bypass! [ 0.586551] evm: HMAC attrs: 0x1 [ 0.587343] hctosys: unable to open rtc device (rtc0) [ 0.601885] sd 0:0:1:0: Attached scsi generic sg0 type 0 [ 0.601886] sd 0:0:1:0: [sda] 104857600 512-byte logical blocks: (53.6 GB/50.0 GiB) [ 0.601951] sd 0:0:1:0: [sda] Write Protect is off [ 0.601955] sd 0:0:1:0: [sda] Mode Sense: 17 00 00 08 [ 0.602005] sd 0:0:1:0: [sda] Cache data unavailable [ 0.602008] sd 0:0:1:0: [sda] Assuming drive cache: write through [ 0.602142] sr 0:0:2:0: [sr0] scsi-1 drive [ 0.602145] cdrom: Uniform CD-ROM driver Revision: 3.20 [ 0.602300] sr 0:0:2:0: Attached scsi CD-ROM sr0 [ 0.602383] sr 0:0:2:0: Attached scsi generic sg1 type 5 [ 0.609434] sda: sda1 sda2 sda3 [ 0.609998] sd 0:0:1:0: [sda] Attached SCSI disk [ 0.610317] Freeing unused kernel memory: 6016K (c000000000e10000 - c0000000013f0000) [ 0.622250] systemd-udevd[183]: starting version 204 [ 0.748333] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null) [ 0.988995] random: init urandom read with 54 bits of entropy available [ 1.434471] Adding 2187200k swap on /dev/sda3. Priority:-1 extents:1 across:2187200k FS [ 1.508521] init: console-setup main process (332) terminated with status 1 [ 1.610642] systemd-udevd[353]: starting version 204 [ 1.699744] EXT4-fs (sda2): re-mounted. Opts: errors=remount-ro [ 2.196703] pseries_rng: Registering IBM pSeries RNG driver [ 2.198038] rtc-generic rtc-generic: rtc core: registered rtc-generic as rtc0 [ 2.233481] init: urfkill main process (355) killed by TRAP signal [ 2.233492] init: urfkill main process ended, respawning [ 2.243977] init: urfkill main process (480) killed by TRAP signal [ 2.243987] init: urfkill main process ended, respawning [ 2.254996] init: urfkill main process (491) killed by TRAP signal [ 2.255006] init: urfkill main process ended, respawning [ 2.264183] init: urfkill main process (506) killed by TRAP signal [ 2.264193] init: urfkill main process ended, respawning [ 2.269729] random: nonblocking pool is initialized [ 2.292590] init: urfkill main process (510) killed by TRAP signal [ 2.292603] init: urfkill main process ended, respawning [ 2.301074] init: urfkill main process (517) killed by TRAP signal [ 2.301084] init: urfkill main process ended, respawning [ 2.310798] init: urfkill main process (523) killed by TRAP signal [ 2.310813] init: urfkill main process ended, respawning [ 2.320191] init: urfkill main process (529) killed by TRAP signal [ 2.320201] init: urfkill main process ended, respawning [ 2.328845] init: urfkill main process (533) killed by TRAP signal [ 2.328854] init: urfkill main process ended, respawning [ 2.337132] init: urfkill main process (537) killed by TRAP signal [ 2.337141] init: urfkill main process ended, respawning [ 2.345748] init: urfkill main process (541) killed by TRAP signal [ 2.345758] init: urfkill respawning too fast, stopped [ 2.567323] audit: type=1400 audit(1455531298.557:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/sbin/dhclient" pid=516 comm="apparmor_parser" [ 2.567338] audit: type=1400 audit(1455531298.557:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=516 comm="apparmor_parser" [ 2.567351] audit: type=1400 audit(1455531298.557:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=516 comm="apparmor_parser" [ 2.567362] audit: type=1400 audit(1455531298.557:5): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/sbin/dhclient" pid=528 comm="apparmor_parser" [ 2.567368] audit: type=1400 audit(1455531298.557:6): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=528 comm="apparmor_parser" [ 2.567373] audit: type=1400 audit(1455531298.557:7): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=528 comm="apparmor_parser" [ 2.567934] audit: type=1400 audit(1455531298.557:8): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=516 comm="apparmor_parser" [ 2.567940] audit: type=1400 audit(1455531298.557:9): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=516 comm="apparmor_parser" [ 2.567958] audit: type=1400 audit(1455531298.557:10): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=528 comm="apparmor_parser" [ 2.792626] init: failsafe main process (579) killed by TERM signal [ 3.303642] init: plymouth-upstart-bridge main process ended, respawning [ 2297.567163] EEH: Frozen PHB#0-PE#10000 detected [ 2297.567185] EEH: PE location: U78C7.001.RCH0040-P2, PHB location: U78C7.001.RCH0040-P2 [ 2297.567192] CPU: 8 PID: 0 Comm: swapper/8 Not tainted 4.2.0-28-generic #33~14.04.1-Ubuntu [ 2297.567194] Call Trace: [ 2297.567202] [c000000c7ffa3c10] [c000000000a82900] dump_stack+0x90/0xbc (unreliable) [ 2297.567208] [c000000c7ffa3c40] [c00000000003733c] eeh_dev_check_failure+0x52c/0x580 [ 2297.567211] [c000000c7ffa3ce0] [c000000000037414] eeh_check_failure+0x84/0xe0 [ 2297.567216] [c000000c7ffa3d20] [c000000000638e80] mem_serial_in+0xa0/0xb0 [ 2297.567219] [c000000c7ffa3d50] [c00000000063d854] serial8250_default_handle_irq+0x54/0xe0 [ 2297.567223] [c000000c7ffa3d80] [c000000000637db8] serial8250_interrupt+0x78/0x170 [ 2297.567227] [c000000c7ffa3e10] [c0000000001271d0] handle_irq_event_percpu+0x90/0x2b0 [ 2297.567230] [c000000c7ffa3ed0] [c00000000012745c] handle_irq_event+0x6c/0xd0 [ 2297.567234] [c000000c7ffa3f00] [c00000000012c1ac] handle_fasteoi_irq+0xec/0x250 [ 2297.567237] [c000000c7ffa3f30] [c0000000001263a8] generic_handle_irq+0x58/0x90 [ 2297.567240] [c000000c7ffa3f60] [c0000000000111e0] __do_irq+0x80/0x190 [ 2297.567244] [c000000c7ffa3f90] [c0000000000242d8] call_do_irq+0x14/0x24 [ 2297.567247] [c000000c727479b0] [c000000000011390] do_IRQ+0xa0/0x120 [ 2297.567250] [c000000c72747a10] [c000000000002594] hardware_interrupt_common+0x114/0x180 [ 2297.567258] --- interrupt: 501 at plpar_hcall_norets+0x1c/0x28 [ 2297.567258] LR = check_and_cede_processor+0x34/0x50 [ 2297.567263] [c000000c72747d00] [c0000000008a9dd0] check_and_cede_processor+0x20/0x50 (unreliable) [ 2297.567267] [c000000c72747d60] [c0000000008a9ff8] shared_cede_loop+0x68/0x170 [ 2297.567270] [c000000c72747da0] [c0000000008a719c] cpuidle_enter_state+0xbc/0x350 [ 2297.567273] [c000000c72747e00] [c0000000001110bc] call_cpuidle+0x7c/0xd0 [ 2297.567276] [c000000c72747e40] [c000000000111460] cpu_startup_entry+0x350/0x450 [ 2297.567280] [c000000c72747f10] [c000000000044bb4] start_secondary+0x364/0x3a0 [ 2297.567284] [c000000c72747f90] [c000000000008b6c] start_secondary_prolog+0x10/0x14 [ 2297.567316] EEH: Detected PCI bus error on PHB#0-PE#10000 [ 2297.567349] EEH: This PCI device has failed 1 times in the last hour [ 2297.567355] EEH: Notify device drivers to shutdown [ 2297.567785] EEH: Collect temporary log [ 2297.569474] EEH: of node=0000:01:00:0 [ 2297.569572] EEH: PCI device/vendor: 00f6114f [ 2297.569670] EEH: PCI cmd/status register: 00180142 [ 2297.569672] EEH: PCI-E capabilities and status follow: [ 2297.570155] EEH: PCI-E 00: 0001b010 00008240 0010501e 00073c11 [ 2297.570541] EEH: PCI-E 10: 10110000 00000000 00000000 00000000 [ 2297.570543] EEH: PCI-E 20: 00000000 [ 2297.572382] RTAS: event: 52, Type: Platform Error, Severity: 2 [ 2297.572384] EEH: Reset without hotplug activity [ 2299.628948] eeh_reset_pe: Failure 0 resetting PHB#0-PE#10000 [ 2299.628948] (1) [ 2301.688944] eeh_reset_pe: Failure 0 resetting PHB#0-PE#10000 [ 2301.688944] (2) [ 2303.748938] eeh_reset_pe: Failure 0 resetting PHB#0-PE#10000 [ 2303.748938] (3) [ 2303.748942] eeh_handle_normal_event: Cannot reset, err=-5 [ 2303.748944] EEH: Unable to recover from failure from PHB#0-PE#10000. [ 2303.748944] Please try reseating or replacing it [ 2303.804072] RTAS: event: 53, Type: Platform Error, Severity: 2 [ 2303.804076] pseries_eeh_configure_bridge: Unable to configure bridge PHB#0-PE#10000 (-1) [ 2303.805265] EEH: of node=0000:01:00:0 [ 2303.805363] EEH: PCI device/vendor: 00f6114f [ 2303.805462] EEH: PCI cmd/status register: 00100140 [ 2303.805463] EEH: PCI-E capabilities and status follow: [ 2303.805950] EEH: PCI-E 00: 0001b010 00008240 0010501e 00073c11 [ 2303.806334] EEH: PCI-E 10: 10110000 00000000 00000000 00000000 [ 2303.806336] EEH: PCI-E 20: 00000000 [ 2303.807958] RTAS: event: 54, Type: Platform Error, Severity: 2 [ 2303.808880] serial 0000:01:00.0: disabling already-disabled device [ 2303.808915] ------------[ cut here ]------------ [ 2303.808926] WARNING: at /build/linux-lts-wily-AY0TLL/linux-lts-wily-4.2.0/drivers/pci/pci.c:1550 [ 2303.808932] Modules linked in: rtc_generic pseries_rng binfmt_misc [ 2303.808952] CPU: 12 PID: 90 Comm: eehd Not tainted 4.2.0-28-generic #33~14.04.1-Ubuntu [ 2303.808960] task: c000000c713c54c0 ti: c000000c71410000 task.ti: c000000c71410000 [ 2303.808968] NIP: c0000000005902c4 LR: c0000000005902c0 CTR: c00000000053ff60 [ 2303.808974] REGS: c000000c714136e0 TRAP: 0700 Not tainted (4.2.0-28-generic) [ 2303.808979] MSR: 8000000100029033 CR: 28000082 XER: 20000000 [ 2303.809000] CFAR: c000000000a7af14 SOFTE: 1 [ 2303.809000] GPR00: c0000000005902c0 c000000c71413960 c0000000014fdf00 0000000000000036 [ 2303.809000] GPR04: 0000000000000001 0000000000000000 0000000000000045 736964203a302e30 [ 2303.809000] GPR08: c0000000013fdf00 0000000000000000 0000000000000000 0000000000000001 [ 2303.809000] GPR12: 0000000000000000 c00000000e7e7200 c0000000000e0a78 c000000c7e157580 [ 2303.809000] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 2303.809000] GPR20: 0000000000000000 0000000000000000 0000000000000000 c000000000cbd210 [ 2303.809000] GPR24: c000000000cbd1e8 0000000000100100 c00000000146cdf8 c000000000cbf138 [ 2303.809000] GPR28: c00000000146cdd0 c000000c716c1000 c000000c716c1740 c000000c716c1000 [ 2303.809093] NIP [c0000000005902c4] pci_disable_device+0xe4/0x100 [ 2303.809103] LR [c0000000005902c0] pci_disable_device+0xe0/0x100 [ 2303.809108] Call Trace: [ 2303.809118] [c000000c71413960] [c0000000005902c0] pci_disable_device+0xe0/0x100 (unreliable) [ 2303.809131] [c000000c714139d0] [c000000000644d64] pciserial_remove_one+0x34/0x50 [ 2303.809143] [c000000c71413a00] [c000000000593db0] pci_device_remove+0x70/0x110 [ 2303.809154] [c000000c71413a40] [c0000000006779dc] __device_release_driver+0xbc/0x180 [ 2303.809165] [c000000c71413a70] [c000000000677ae0] device_release_driver+0x40/0x70 [ 2303.809173] [c000000c71413aa0] [c00000000058a710] pci_stop_bus_device+0xe0/0x100 [ 2303.809182] [c000000c71413ae0] [c00000000058a8bc] pci_stop_and_remove_bus_device+0x2c/0x50 [ 2303.809193] [c000000c71413b10] [c0000000000478e8] pcibios_remove_pci_devices+0xc8/0x160 [ 2303.809200] [c000000c71413ba0] [c00000000003bb18] eeh_handle_normal_event+0xe8/0x410 [ 2303.809209] [c000000c71413c20] [c00000000003c014] eeh_handle_event+0x54/0x340 [ 2303.809216] [c000000c71413cd0] [c00000000003c424] eeh_event_handler+0x124/0x1d0 [ 2303.809224] [c000000c71413d80] [c0000000000e0b84] kthread+0x114/0x140 [ 2303.809234] [c000000c71413e30] [c000000000009538] ret_from_kernel_thread+0x5c/0xa4 [ 2303.809237] Instruction dump: [ 2303.809241] 387f0098 480e05c1 60000000 e8bf00e8 2fa50000 7c641b78 409e0008 e8bf00a8 [ 2303.809255] 3c62ff82 38637060 484eabf1 60000000 <0fe00000> 39200001 3d42fff6 992a2f18 [ 2303.809267] ---[ end trace b1a7cb75bba81f0d ]--- [ 2303.809277] iommu: Removing device 0000:01:00.0 from group 0 [ 2303.809432] ------------[ cut here ]------------ [ 2303.809439] WARNING: at /build/linux-lts-wily-AY0TLL/linux-lts-wily-4.2.0/include/linux/kref.h:47 [ 2303.809443] Modules linked in: rtc_generic pseries_rng binfmt_misc [ 2303.809454] CPU: 1 PID: 2393 Comm: bash Tainted: G W 4.2.0-28-generic #33~14.04.1-Ubuntu [ 2303.809461] task: c000000c6c227ba0 ti: c000000c6c2d8000 task.ti: c000000c6c2d8000 [ 2303.809466] NIP: c000000000532f84 LR: c0000000002c4890 CTR: c0000000002c48e0 [ 2303.809471] REGS: c000000c6c2db6d0 TRAP: 0700 Tainted: G W (4.2.0-28-generic) [ 2303.809476] MSR: 8000000000029033 CR: 24222422 XER: 00000000 [ 2303.809491] CFAR: c000000000532f3c SOFTE: 1 [ 2303.809491] GPR00: c0000000002c4890 c000000c6c2db950 c0000000014fdf00 c000000c6efc0000 [ 2303.809491] GPR04: c000000c6efc0000 c000000c6c2dbab0 0000000000000001 0000000000000000 [ 2303.809491] GPR08: 0000000000000000 0000000000000001 c00000000145df00 c000000c78776cc0 [ 2303.809491] GPR12: c0000000002c48e0 c00000000e7e0980 0000000022000000 0000000010172000 [ 2303.809491] GPR16: 000000001013ab40 000001003b0660c8 0000000000000000 0000000000000001 [ 2303.809491] GPR20: 0000000000000000 0000000000000000 0000000000000022 0000000000000000 [ 2303.809491] GPR24: 0000000000000020 c000000c6c2dbab0 c0000000002c4060 c000000c6efc0000 [ 2303.809491] GPR28: c000000c7e0a1800 0000000000000000 0000000000000000 c000000c6efc0000 [ 2303.809562] NIP [c000000000532f84] kobject_get+0x94/0xb0 [ 2303.809568] LR [c0000000002c4890] cdev_get+0x50/0xa0 [ 2303.809571] Call Trace: [ 2303.809578] [c000000c6c2db950] [0000000010172a74] 0x10172a74 (unreliable) [ 2303.809591] [c000000c6c2db9c0] [c0000000002c4890] cdev_get+0x50/0xa0 [ 2303.809599] [c000000c6c2db9f0] [c0000000002c490c] exact_lock+0x2c/0x50 [ 2303.809606] [c000000c6c2dba20] [c00000000067c460] kobj_lookup+0x110/0x200 [ 2303.809612] [c000000c6c2dba90] [c0000000002c50b0] chrdev_open+0x190/0x270 [ 2303.809618] [c000000c6c2dbaf0] [c0000000002baa40] do_dentry_open+0x290/0x420 [ 2303.809626] [c000000c6c2dbb50] [c0000000002d11dc] do_last+0x19c/0xf90 [ 2303.809633] [c000000c6c2dbc10] [c0000000002d2074] path_openat+0xa4/0x3c0 [ 2303.809641] [c000000c6c2dbc90] [c0000000002d4228] do_filp_open+0x88/0x110 [ 2303.809650] [c000000c6c2dbdb0] [c0000000002bc7e8] do_sys_open+0x1c8/0x390 [ 2303.809660] [c000000c6c2dbe30] [c000000000009204] system_call+0x38/0xb4 [ 2303.809665] Instruction dump: [ 2303.809669] 4e800020 e89f0000 3c62ff82 7fe5fb78 38632a58 48547f45 60000000 0fe00000 [ 2303.809679] 4bffffa8 3d42fff6 892a2f04 69290001 <0b090000> 2fa90000 41feffb4 39200001 [ 2303.809690] ---[ end trace b1a7cb75bba81f0e ]--- root@ltcalpine-lp4:/home/ubuntu# lspci -nn 01:00.0 Serial controller [0700]: Digi International Device [114f:00f6] root@ltcalpine-lp4:/home/ubuntu# [ 2297.567163] EEH: Frozen PHB#0-PE#10000 detected [ 2297.567185] EEH: PE location: U78C7.001.RCH0040-P2, PHB location: U78C7.001.RCH0040-P2 [ 2303.748944] EEH: Unable to recover from failure from PHB#0-PE#10000. [ 2303.748944] Please try reseating or replacing it Linux ltcalpine-lp4 4.2.0-28-generic #33~14.04.1-Ubuntu SMP Thu Feb 4 12:26:42 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux root@ltcalpine-lp4:/home/ubuntu#