Script started on Wed 21 Dec 2016 11:54:34 AM CST root@10:~# virsh console dotg9 Connected to domain dotg9 Escape character is ^] root@dotg9:~# df -h Filesystem Size Used Avail Use% Mounted on udev 2.7G 0 2.7G 0% /dev tmpfs 561M 16M 545M 3% /run /dev/mapper/mpatha-part2 19G 3.5G 15G 20% / tmpfs 2.8G 0 2.8G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 2.8G 0 2.8G 0% /sys/fs/cgroup tmpfs 561M 0 561M 0% /run/user/0 root@dotg9:~# root@dotg9:~# muli tipath -l mpatha (36005076802818bda300000000000049f) dm-0 IBM,2145 size=20G features='1 queue_if_no_path' hwhandler='0' wp=rw |-+- policy='round-robin 0' prio=0 status=active | `- 1:0:0:0 sdb 8:16 active undef running `-+- policy='round-robin 0' prio=0 status=enabled `- 0:0:0:1 sda 8:0 active undef running root@dotg9:~# root@dotg9:~# root@dotg9:~# root@dotg9:~# echo c > /proc/sysrq/ -trigger [ 72.969550] sysrq: SysRq : Trigger a crash [ 72.969629] Unable to handle kernel paging request for data at address 0x00000000 [ 72.969710] Faulting instruction address: 0xc0000000006ccb18 [ 72.969779] Oops: Kernel access of bad area, sig: 11 [#1] [ 72.969831] SMP NR_CPUS=2048 NUMA pSeries [ 72.969890] Modules linked in: vmx_crypto sunrpc autofs4 dm_round_robin virtio_net crc32c_vpmsum virtio_scsi scsi_dh_emc scsi_dh_rdac scsi_dh_alua dm_multipath [ 72.970098] CPU: 3 PID: 1754 Comm: bash Not tainted 4.8.0-32-generic #34~16.04.1-Ubuntu [ 72.970176] task: c000000003b00000 task.stack: c000000003b4c000 [ 72.970242] NIP: c0000000006ccb18 LR: c0000000006cdc3c CTR: c0000000006ccae0 [ 72.970319] REGS: c000000003b4f9d0 TRAP: 0300 Not tainted (4.8.0-32-generic) [ 72.970397] MSR: 8000000000009033 CR: 28242222 XER: 20000000 [ 72.970585] CFAR: c000000000014ae8 DAR: 0000000000000000 DSISR: 42000000 SOFTE: 1 GPR00: c0000000006cdc3c c000000003b4fc50 c0000000014a6600 0000000000000063 GPR04: c00000017fecaca0 c00000017fedfb40 0000000000000114 0000000000004f10 GPR08: 0000000000000007 0000000000000001 0000000000000000 0000000000000001 GPR12: c0000000006ccae0 c000000007b81b00 ffffffffffffffff 0000000022000000 GPR16: 0000000010170dd0 0000010034cf03a8 00000000101405c8 00000000100c70a0 GPR20: 0000000000000000 000000001017dd78 0000000010140460 0000000000000000 GPR24: 0000000010153320 000000001017b628 c0000000013be7a0 0000000000000004 GPR28: c0000000013beb60 0000000000000063 c000000001372590 0000000000000000 [ 72.971700] NIP [c0000000006ccb18] sysrq_handle_crash+0x38/0x50 [ 72.971768] LR [c0000000006cdc3c] __handle_sysrq+0xec/0x280 [ 72.971821] Call Trace: [ 72.971861] [c000000003b4fc50] [c000000000b52f58] printk+0x5c/0x70 (unreliable) [ 72.971956] [c000000003b4fc70] [c0000000006cdc3c] __handle_sysrq+0xec/0x280 [ 72.972036] [c000000003b4fd10] [c0000000006ce3e0] write_sysrq_trigger+0x70/0x90 [ 72.972138] [c000000003b4fd40] [c0000000003c499c] proc_reg_write+0x8c/0xd0 [ 72.972220] [c000000003b4fd70] [c000000000322ae0] __vfs_write+0x40/0x80 [ 72.972300] [c000000003b4fd90] [c000000000323e84] vfs_write+0xd4/0x270 [ 72.972381] [c000000003b4fde0] [c000000000325a8c] SyS_write+0x6c/0x110 [ 72.972462] [c000000003b4fe30] [c000000000009584] system_call+0x38/0xec [ 72.972540] Instruction dump: [ 72.972581] 7c0802a6 f8010010 60000000 7c0802a6 f8010010 f821ffe1 3d22001a 3949c9e0 [ 72.972718] 39200001 912a0000 7c0004ac 39400000 <992a0000> 38210020 e8010010 7c0803a6 [ 72.972861] ---[ end trace d15d78c8e220cadf ]--- [ 72.976240] [ 72.976292] Sending IPI to other CPUs [ 72.977340] IPI complete I'm in purgatory [ 0.000000] Allocated 65536 bytes for 8 pacas at c00000000fff0000 [ 0.000000] Adjusted logical boot cpu id: logical 0 physical 24 [ 0.000000] Page sizes from device-tree: [ 0.000000] base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=0 [ 0.000000] base_shift=16: shift=16, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=1 [ 0.000000] Page orders: linear mapping = 16, virtual = 16, io = 16, vmemmap = 16 [ 0.000000] Using 1TB segments [ 0.000000] Initializing hash mmu with SLB [ 0.000000] Linux version 4.8.0-32-generic (buildd@bos01-ppc64el-001) (gcc version 5.4.0 20160609 (Ubuntu/IBM 5.4.0-6ubuntu1~16.04.4) ) #34~16.04.1-Ubuntu SMP Tue Dec 13 17:01:57 UTC 2016 (Ubuntu 4.8.0-32.34~16.04.1-generic 4.8.11) [ 0.000000] Found initrd at 0xc000000009810000:0xc00000000a6bc7ae [ 0.000000] Using pSeries machine description [ 0.000000] bootconsole [udbg0] enabled [ 0.000000] Partition configured for 4 cpus, operating system maximum is 1. [ 0.000000] CPU maps initialized for 1 thread per core [ 0.000000] (thread shift is 0) -> smp_release_cpus() spinning_secondaries = 3 <- smp_release_cpus() [ 0.000000] ----------------------------------------------------- [ 0.000000] ppc64_pft_size = 0x1a [ 0.000000] phys_mem_size = 0x28010000 [ 0.000000] dcache_bsize = 0x80 [ 0.000000] icache_bsize = 0x80 [ 0.000000] cpu_features = 0x37fc7a6c18500249 [ 0.000000] possible = 0x3fffffff18500649 [ 0.000000] always = 0x0000000018100040 [ 0.000000] cpu_user_features = 0xdc0065c2 0xef000000 [ 0.000000] mmu_features = 0x78000001 [ 0.000000] firmware_features = 0x00000001405a440b [ 0.000000] htab_hash_mask = 0x7ffff [ 0.000000] physical_start = 0x8000000 [ 0.000000] ----------------------------------------------------- [ 0.000000] Node 0 Memory: 0x0-0x30000000 [ 0.000000] numa: Initmem setup node 0 [mem 0x00000000-0x2fffffff] [ 0.000000] numa: NODE_DATA [mem 0x27fce300-0x27fd7fff] [ 0.000000] PCI host bridge /pci@800000020000000 ranges: [ 0.000000] IO 0x0000010080000000..0x000001008000ffff -> 0x0000000000000000 [ 0.000000] MEM 0x00000100a0000000..0x000001101fffffff -> 0x0000000080000000 [ 0.000000] PPC64 nvram contains 65536 bytes [ 0.000000] Top of RAM: 0x30000000, Total RAM: 0x28010000 [ 0.000000] Memory hole size: 127MB [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000000000-0x000000002fffffff] [ 0.000000] DMA32 empty [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000000000-0x0000000027ffffff] [ 0.000000] node 0: [mem 0x000000002fff0000-0x000000002fffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000002fffffff] [ 0.000000] On node 0 totalpages: 10241 [ 0.000000] DMA zone: 11 pages used for memmap [ 0.000000] DMA zone: 0 pages reserved [ 0.000000] DMA zone: 10241 pages, LIFO batch:1 [ 0.000000] percpu: Embedded 3 pages/cpu @c000000027e00000 s148888 r0 d47720 u1048576 [ 0.000000] pcpu-alloc: s148888 r0 d47720 u1048576 alloc=1*1048576 [ 0.000000] pcpu-alloc: [0] 0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 10230 [ 0.000000] Policy zone: DMA [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinux-4.8.0-32-generic root=UUID=a557092e-9d7f-496e-a371-7a043e192ec9 ro loglevel=8 irqpoll nr_cpus=1 nousb systemd.unit=kdump-tools.service elfcorehdr=155584K [ 0.000000] Misrouted IRQ fixup and polling support enabled [ 0.000000] This may significantly impact system performance [ 0.000000] PID hash table entries: 4096 (order: -1, 32768 bytes) [ 0.000000] Sorting __ex_table... [ 0.000000] Memory: 417088K/655424K available (11648K kernel code, 1600K rwdata, 3500K rodata, 4608K init, 2961K bss, 238336K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] Build-time adjustment of leaf fanout to 64. [ 0.000000] RCU restricting CPUs from NR_CPUS=2048 to nr_cpu_ids=1. [ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=1 [ 0.000000] NR_IRQS:512 nr_irqs:512 16 [ 0.000000] pic: no ISA interrupt controller [ 0.000000] time_init: decrementer frequency = 512.000000 MHz [ 0.000000] time_init: processor frequency = 3425.000000 MHz [ 0.000002] clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x761537d007, max_idle_ns: 440795202126 ns [ 0.000865] clocksource: timebase mult[1f40000] shift[24] registered [ 0.001396] clockevent: decrementer mult[83126e98] shift[32] cpu[0] [ 0.002026] Console: colour dummy device 80x25 [ 0.002407] console [hvc0] enabled [ 0.002407] console [hvc0] enabled [ 0.002719] bootconsole [udbg0] disabled [ 0.002719] bootconsole [udbg0] disabled [ 0.003110] pid_max: default: 32768 minimum: 301 [ 0.003267] Security Framework initialized [ 0.003292] Yama: becoming mindful. [ 0.003400] AppArmor: AppArmor initialized [ 0.003541] Dentry cache hash table entries: 131072 (order: 4, 1048576 bytes) [ 0.003790] Inode-cache hash table entries: 65536 (order: 3, 524288 bytes) [ 0.003932] Mount-cache hash table entries: 8192 (order: 0, 65536 bytes) [ 0.003974] Mountpoint-cache hash table entries: 8192 (order: 0, 65536 bytes) [ 0.005510] ftrace: allocating 30128 entries in 12 pages [ 0.017568] EEH: pSeries platform initialized [ 0.017634] POWER8 performance monitor hardware support registered [ 0.017676] power8-pmu: PMAO restore workaround active. [ 0.018398] Brought up 1 CPUs [ 0.018439] Node 0 CPUs: 0 [ 0.018652] devtmpfs: initialized [ 0.029590] evm: security.selinux [ 0.029616] evm: security.SMACK64 [ 0.029639] evm: security.SMACK64EXEC [ 0.029662] evm: security.SMACK64TRANSMUTE [ 0.029685] evm: security.SMACK64MMAP [ 0.029707] evm: security.ima [ 0.029731] evm: security.capability [ 0.029840] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns [ 0.030130] NET: Registered protocol family 16 [ 0.030916] EEH: No capable adapters found [ 0.030984] IBM eBus Device Driver [ 0.031114] cpuidle: using governor ladder [ 0.031140] cpuidle: using governor menu [ 0.031229] RTAS daemon started [ 0.031309] pstore: using zlib compression [ 0.031334] pstore: Registered nvram as persistent store backend Linux ppc64le #34~16.04.1-Ubun[ 0.031928] PCI: Probing PCI hardware [ 0.031984] PCI host bridge to bus 0000:00 [ 0.032010] pci_bus 0000:00: root bus resource [io 0x10000-0x1ffff] (bus address [0x0000-0xffff]) [ 0.032064] pci_bus 0000:00: root bus resource [mem 0x100a0000000-0x1101fffffff] (bus address [0x80000000-0xfffffffff]) [ 0.032124] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.062588] IOMMU table initialized, virtual merging enabled [ 0.062647] iommu: Adding device 0000:00:01.0 to group 0 [ 0.062722] iommu: Adding device 0000:00:02.0 to group 0 [ 0.062777] iommu: Adding device 0000:00:03.0 to group 0 [ 0.062832] iommu: Adding device 0000:00:04.0 to group 0 [ 0.062975] PCI: Probing PCI hardware done [ 0.064337] SCSI subsystem initialized [ 0.064382] libata version 3.00 loaded. [ 0.064430] vgaarb: loaded [ 0.064476] usbcore: registered new interface driver usbfs [ 0.064515] usbcore: registered new interface driver hub [ 0.064552] usbcore: registered new device driver usb [ 0.064596] pps_core: LinuxPPS API ver. 1 registered [ 0.064627] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.064681] PTP clock support registered [ 0.064841] NetLabel: Initializing [ 0.064866] NetLabel: domain hash size = 128 [ 0.064896] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.064939] NetLabel: unlabeled traffic allowed by default [ 0.065040] clocksource: Switched to clocksource timebase [ 0.071845] VFS: Disk quotas dquot_6.6.0 [ 0.071930] VFS: Dquot-cache hash table entries: 8192 (order 0, 65536 bytes) [ 0.072093] hugetlbfs: disabling because there are no supported hugepage sizes [ 0.072185] AppArmor: AppArmor Filesystem Enabled [ 0.073446] NET: Registered protocol family 2 [ 0.073623] TCP established hash table entries: 8192 (order: 0, 65536 bytes) [ 0.073688] TCP bind hash table entries: 8192 (order: 1, 131072 bytes) [ 0.073741] TCP: Hash tables configured (established 8192 bind 8192) [ 0.073787] UDP hash table entries: 2048 (order: 0, 65536 bytes) [ 0.073835] UDP-Lite hash table entries: 2048 (order: 0, 65536 bytes) [ 0.073906] NET: Registered protocol family 1 [ 0.133362] PCI: CLS 0 bytes, default 128 [ 0.133421] Trying to unpack rootfs image as initramfs... [ 0.333877] Freeing initrd memory: 14976K (c000000009810000 - c00000000a6b0000) [ 0.334357] futex hash table entries: 256 (order: -1, 32768 bytes) [ 0.334426] audit: initializing netlink subsys (disabled) [ 0.334484] audit: type=2000 audit(1482342896.324:1): initialized [ 0.334650] Initialise system trusted keyrings [ 0.334751] workingset: timestamp_bits=38 max_order=13 bucket_order=0 [ 0.336341] zbud: loaded [ 0.336926] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 0.337116] fuse init (API version 7.25) [ 0.337262] Allocating IMA blacklist keyring. [ 0.338085] Key type asymmetric registered [ 0.338111] Asymmetric key parser 'x509' registered [ 0.338172] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246) [ 0.338233] io scheduler noop registered [ 0.338257] io scheduler deadline registered (default) [ 0.338293] io scheduler cfq registered [ 0.338387] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.338530] virtio-pci 0000:00:01.0: virtio_pci: leaving for legacy driver [ 0.354965] virtio-pci 0000:00:02.0: virtio_pci: leaving for legacy driver [ 0.390897] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver [ 0.418988] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled [ 0.420000] Linux agpgart interface v0.103 [ 0.422176] brd: module loaded [ 0.423010] loop: module loaded [ 0.423177] libphy: Fixed MDIO Bus: probed [ 0.423202] tun: Universal TUN/TAP device driver, 1.6 [ 0.423233] tun: (C) 1999-2004 Max Krasnyansky [ 0.423293] ibmveth: IBM Power Virtual Ethernet Driver 1.05 [ 0.423331] ehea: IBM eHEA ethernet device driver (Release EHEA_0107) [ 0.423378] PPP generic driver version 2.4.2 [ 0.423450] VFIO - User Level meta-driver version: 0.3 [ 0.425549] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.425602] ehci-pci: EHCI PCI platform driver [ 0.425643] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.425707] ohci-pci: OHCI PCI platform driver [ 0.426045] ohci-pci 0000:00:04.0: OHCI PCI host controller [ 0.426083] ohci-pci 0000:00:04.0: new USB bus registered, assigned bus number 1 [ 0.426230] ohci-pci 0000:00:04.0: irq 20, io mem 0x100e0000000 [ 0.485330] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 0.485372] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.485418] usb usb1: Product: OHCI PCI host controller [ 0.485449] usb usb1: Manufacturer: Linux 4.8.0-32-generic ohci_hcd [ 0.485488] usb usb1: SerialNumber: 0000:00:04.0 [ 0.485606] hub 1-0:1.0: USB hub found [ 0.485649] hub 1-0:1.0: 3 ports detected [ 0.485827] uhci_hcd: USB Universal Host Controller Interface driver [ 0.485954] mousedev: PS/2 mouse device common for all mice [ 0.486070] rtc-generic rtc-generic: rtc core: registered rtc-generic as rtc0 [ 0.486119] i2c /dev entries driver [ 0.486236] device-mapper: uevent: version 1.0.3 [ 0.486317] device-mapper: ioctl: 4.35.0-ioctl (2016-06-23) initialised: dm-devel@redhat.com [ 0.486385] pseries_idle_driver registered [ 0.486420] ledtrig-cpu: registered to indicate activity on CPUs [ 0.486604] NET: Registered protocol family 10 [ 0.486821] NET: Registered protocol family 17 [ 0.486860] Key type dns_resolver registered [ 0.487019] registered taskstats version 1 [ 0.487048] Loading compiled-in X.509 certificates [ 0.489327] Loaded X.509 cert 'Build time autogenerated kernel key: beb24bb1779de8415f01800ee18c1ac8e810856b' [ 0.489429] zswap: loaded using pool lzo/zbud [ 0.508807] Key type big_key registered [ 0.513890] Key type trusted registered [ 0.515436] Key type encrypted registered [ 0.515468] AppArmor: AppArmor sha1 policy hashing enabled [ 0.515500] ima: No TPM chip found, activating TPM-bypass! [ 0.515566] evm: HMAC attrs: 0x1 [ 0.515764] rtc-generic rtc-generic: setting system clock to 2016-12-21 17:54:57 UTC (1482342897) [ 0.517526] Freeing unused kernel memory: 4608K (c000000008ee0000 - c000000009360000) [ 0.517601] This architecture does not have kernel memory protection. Loading, please wait... Begin: Loading multipath modules ... Success: loaded module dm-multipath. done. Begin: Loading multipath hardware handlers ... [ 0.531929] alua: device handler registered Success: loaded module scsi_dh_alua. [ 0.533883] rdac: device handler registered Success: loaded module scsi_dh_rdac. [ 0.536898] emc: device handler registered Success: loaded module scsi_dh_emc. done. Begin: Starting multipathd ... done. [ 0.542909] random: multipathd: uninitialized urandom read (16 bytes read) [ 0.543238] random: multipathd: uninitialized urandom read (16 bytes read) [ 0.543288] random: multipathd: uninitialized urandom read (16 bytes read) [ 0.543426] random: multipathd: uninitialized urandom read (16 bytes read) [ 0.543564] random: multipathd: uninitialized urandom read (16 bytes read) [ 0.543688] random: multipathd: uninitialized urandom read (16 bytes read) [ 0.543804] random: multipathd: uninitialized urandom read (16 bytes read) [ 0.543927] random: multipathd: uninitialized urandom read (16 bytes read) [ 0.544044] random: multipathd: uninitialized urandom read (16 bytes read) [ 0.544139] random: multipathd: uninitialized urandom read (16 bytes read) starting version 229 [ 0.591490] scsi host0: Virtio SCSI HBA [ 0.723223] scsi host1: Virtio SCSI HBA [ 22.005063] scsi 1:0:0:0: tag#0 abort [ 22.005160] scsi 0:0:0:0: tag#0 abort Begin: Loading essential drivers ... done. Begin: Running /scripts/init-premount ... done. Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done. Begin: Running /scripts/local-premount ... Begin: Waiting for udev to settle (multipath) ... done. [ 242.665075] INFO: task kworker/u2:0:6 blocked for more than 120 seconds. [ 242.665159] Not tainted 4.8.0-32-generic #34~16.04.1-Ubuntu [ 242.665205] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 242.665265] kworker/u2:0 D 0000000000000000 0 6 2 0x00000800 [ 242.665326] Workqueue: scsi_tmf_0 scmd_eh_abort_handler [ 242.665366] Call Trace: [ 242.665387] [c00000002607f6a0] [c00000000964c930] logbuf_lock+0x0/0x4 (unreliable) [ 242.665443] [c00000002607f870] [c00000000801a6f0] __switch_to+0x2a0/0x420 [ 242.665491] [c00000002607f8d0] [c000000008b44fe4] __schedule+0x2f4/0x990 [ 242.665540] [c00000002607f9b0] [c000000008b456cc] schedule+0x4c/0xc0 [ 242.665588] [c00000002607f9e0] [c000000008b49ce4] schedule_timeout+0x284/0x480 [ 242.665643] [c00000002607fad0] [c000000008b4669c] wait_for_common+0xec/0x240 [ 242.665699] [c00000002607fb50] [d00000000075212c] virtscsi_tmf.constprop.4+0xcc/0x160 [virtio_scsi] [ 242.665762] [c00000002607fbc0] [c0000000087bf04c] scmd_eh_abort_handler+0x14c/0x390 [ 242.665817] [c00000002607fc50] [c0000000080ef5e8] process_one_work+0x1e8/0x5b0 [ 242.665872] [c00000002607fce0] [c0000000080efa58] worker_thread+0xa8/0x650 [ 242.665921] [c00000002607fd80] [c0000000080f8224] kthread+0x114/0x140 [ 242.665969] [c00000002607fe30] [c0000000080098f0] ret_from_kernel_thread+0x5c/0x6c [ 242.666035] INFO: task systemd-udevd:170 blocked for more than 120 seconds. [ 242.666074] Not tainted 4.8.0-32-generic #34~16.04.1-Ubuntu [ 242.666112] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 242.666157] systemd-udevd D 00003fffaf6eaed4 0 170 162 0x00040002 [ 242.666206] Call Trace: [ 242.666224] [c00000001e0b78a0] [c00000000801a6f0] __switch_to+0x2a0/0x420 [ 242.666270] [c00000001e0b7900] [c000000008b44fe4] __schedule+0x2f4/0x990 [ 242.666317] [c00000001e0b79e0] [c000000008b456cc] schedule+0x4c/0xc0 [ 242.666364] [c00000001e0b7a10] [c0000000080fcea4] async_synchronize_cookie_domain+0xb4/0x220 [ 242.666420] [c00000001e0b7ad0] [c000000008b53c00] do_init_module+0x118/0x264 [ 242.666467] [c00000001e0b7b60] [c0000000081896e8] load_module+0x16a8/0x1870 [ 242.666516] [c00000001e0b7d30] [c000000008189bf0] SyS_finit_module+0xf0/0x170 [ 242.666563] [c00000001e0b7e30] [c000000008009584] system_call+0x38/0xec [ 242.666613] INFO: task kworker/u2:3:209 blocked for more than 120 seconds. [ 242.666652] Not tainted 4.8.0-32-generic #34~16.04.1-Ubuntu [ 242.666690] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 242.666736] kworker/u2:3 D 0000000000000000 0 209 2 0x00000800 [ 242.666786] Workqueue: scsi_tmf_1 scmd_eh_abort_handler [ 242.666824] Call Trace: [ 242.666842] [c00000001e3bb6a0] [c00000000964c930] logbuf_lock+0x0/0x4 (unreliable) [ 242.666897] [c00000001e3bb870] [c00000000801a6f0] __switch_to+0x2a0/0x420 [ 242.666944] [c00000001e3bb8d0] [c000000008b44fe4] __schedule+0x2f4/0x990 [ 242.666992] [c00000001e3bb9b0] [c000000008b456cc] schedule+0x4c/0xc0 [ 242.667038] [c00000001e3bb9e0] [c000000008b49ce4] schedule_timeout+0x284/0x480 [ 242.667094] [c00000001e3bbad0] [c000000008b4669c] wait_for_common+0xec/0x240 [ 242.667141] [c00000001e3bbb50] [d00000000075212c] virtscsi_tmf.constprop.4+0xcc/0x160 [virtio_scsi] [ 242.667204] [c00000001e3bbbc0] [c0000000087bf04c] scmd_eh_abort_handler+0x14c/0x390 [ 242.667258] [c00000001e3bbc50] [c0000000080ef5e8] process_one_work+0x1e8/0x5b0 [ 242.667312] [c00000001e3bbce0] [c0000000080efa58] worker_thread+0xa8/0x650 [ 242.667359] [c00000001e3bbd80] [c0000000080f8224] kthread+0x114/0x140 [ 242.667407] [c00000001e3bbe30] [c0000000080098f0] ret_from_kernel_thread+0x5c/0x6c done. Begin: Waiting for root file system ... Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. [ 363.497084] INFO: task kworker/u2:0:6 blocked for more than 120 seconds. [ 363.497183] Not tainted 4.8.0-32-generic #34~16.04.1-Ubuntu [ 363.497229] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 363.497285] kworker/u2:0 D 0000000000000000 0 6 2 0x00000800 [ 363.497383] Workqueue: scsi_tmf_0 scmd_eh_abort_handler [ 363.497437] Call Trace: [ 363.497481] [c00000002607f6a0] [c00000000964c930] logbuf_lock+0x0/0x4 (unreliable) [ 363.497556] [c00000002607f870] [c00000000801a6f0] __switch_to+0x2a0/0x420 [ 363.497622] [c00000002607f8d0] [c000000008b44fe4] __schedule+0x2f4/0x990 [ 363.497679] [c00000002607f9b0] [c000000008b456cc] schedule+0x4c/0xc0 [ 363.497735] [c00000002607f9e0] [c000000008b49ce4] schedule_timeout+0x284/0x480 [ 363.497801] [c00000002607fad0] [c000000008b4669c] wait_for_common+0xec/0x240 [ 363.497890] [c00000002607fb50] [d00000000075212c] virtscsi_tmf.constprop.4+0xcc/0x160 [virtio_scsi] [ 363.497966] [c00000002607fbc0] [c0000000087bf04c] scmd_eh_abort_handler+0x14c/0x390 [ 363.498039] [c00000002607fc50] [c0000000080ef5e8] process_one_work+0x1e8/0x5b0 [ 363.498104] [c00000002607fce0] [c0000000080efa58] worker_thread+0xa8/0x650 [ 363.498160] [c00000002607fd80] [c0000000080f8224] kthread+0x114/0x140 [ 363.498217] [c00000002607fe30] [c0000000080098f0] ret_from_kernel_thread+0x5c/0x6c [ 363.498321] INFO: task systemd-udevd:170 blocked for more than 120 seconds. [ 363.498368] Not tainted 4.8.0-32-generic #34~16.04.1-Ubuntu [ 363.498413] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 363.498469] systemd-udevd D 00003fffaf6eaed4 0 170 162 0x00040002 [ 363.498532] Call Trace: [ 363.498553] [c00000001e0b78a0] [c00000000801a6f0] __switch_to+0x2a0/0x420 [ 363.498609] [c00000001e0b7900] [c000000008b44fe4] __schedule+0x2f4/0x990 [ 363.498665] [c00000001e0b79e0] [c000000008b456cc] schedule+0x4c/0xc0 [ 363.498722] [c00000001e0b7a10] [c0000000080fcea4] async_synchronize_cookie_domain+0xb4/0x220 [ 363.498786] [c00000001e0b7ad0] [c000000008b53c00] do_init_module+0x118/0x264 [ 363.498844] [c00000001e0b7b60] [c0000000081896e8] load_module+0x16a8/0x1870 [ 363.498901] [c00000001e0b7d30] [c000000008189bf0] SyS_finit_module+0xf0/0x170 [ 363.498958] [c00000001e0b7e30] [c000000008009584] system_call+0x38/0xec [ 363.499024] INFO: task kworker/u2:3:209 blocked for more than 120 seconds. [ 363.499070] Not tainted 4.8.0-32-generic #34~16.04.1-Ubuntu [ 363.499116] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 363.499173] kworker/u2:3 D 0000000000000000 0 209 2 0x00000800 [ 363.499244] Workqueue: scsi_tmf_1 scmd_eh_abort_handler [ 363.499294] Call Trace: [ 363.499314] [c00000001e3bb6a0] [c00000000964c930] logbuf_lock+0x0/0x4 (unreliable) [ 363.499380] [c00000001e3bb870] [c00000000801a6f0] __switch_to+0x2a0/0x420 [ 363.499436] [c00000001e3bb8d0] [c000000008b44fe4] __schedule+0x2f4/0x990 [ 363.499492] [c00000001e3bb9b0] [c000000008b456cc] schedule+0x4c/0xc0 [ 363.499549] [c00000001e3bb9e0] [c000000008b49ce4] schedule_timeout+0x284/0x480 [ 363.499614] [c00000001e3bbad0] [c000000008b4669c] wait_for_common+0xec/0x240 [ 363.499671] [c00000001e3bbb50] [d00000000075212c] virtscsi_tmf.constprop.4+0xcc/0x160 [virtio_scsi] [ 363.499746] [c00000001e3bbbc0] [c0000000087bf04c] scmd_eh_abort_handler+0x14c/0x390 [ 363.499810] [c00000001e3bbc50] [c0000000080ef5e8] process_one_work+0x1e8/0x5b0 [ 363.499875] [c00000001e3bbce0] [c0000000080efa58] worker_thread+0xa8/0x650 [ 363.499931] [c00000001e3bbd80] [c0000000080f8224] kthread+0x114/0x140 [ 363.499989] [c00000001e3bbe30] [c0000000080098f0] ret_from_kernel_thread+0x5c/0x6c Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. done. Gave up waiting for root device. Common problems: - Boot args (cat /proc/cmdline) - Check rootdelay= (did the system wait long enough?) - Check root= (did the system wait for the right device?) - Missing modules (cat /proc/modules; ls /dev) ALERT! UUID=a557092e-9d7f-496e-a371-7a043e192ec9 does not exist. Dropping to a shell! BusyBox v1.22.1 (Ubuntu 1:1.22.0-15ubuntu1) built-in shell (ash) Enter 'help' for a list of built-in commands. (initramfs)  root@10:~# exit Script done on Wed 21 Dec 2016 12:02:44 PM CST