| 2015-09-14 10:10:32 |
B. |
bug |
|
|
added bug |
| 2015-09-14 10:10:32 |
B. |
attachment added |
|
kern.log https://bugs.launchpad.net/bugs/1495442/+attachment/4463892/+files/kern.log |
|
| 2015-09-14 10:30:08 |
Brad Figg |
linux (Ubuntu): status |
New |
Incomplete |
|
| 2015-09-14 10:30:10 |
Brad Figg |
tags |
|
utopic |
|
| 2015-09-14 11:21:43 |
B. |
tags |
utopic |
apport-collected staging trusty utopic |
|
| 2015-09-14 11:21:45 |
B. |
description |
Ubuntu 14.04.3 LTS with Kernel 3.19.0-26-generic
One of our servers running rsnapshot and nfs-kernel-server on XFS partitions
crashed or hanged two sundays in a raw.
Kernel 3.19.0-26-generic hangs multiple times (same time) in xlog_cil_force_lsn (XFS).
I assume this bug is not related with bug report 979498.
I had also this issue with kernel 3.16.0-46-generic or 3.13.0-57-generic but I have
no log during the crash to determine which of the above kernel was running.
I switched to kernel 3.19.0-26-generic because 3.16.0-48-generic was not
working well with Intel Gigabit -- tx hang (bug 1492146)
INFO: task kthreadd:2 blocked for more than 120 seconds.
Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kthreadd D ffff8802362eb3a8 0 2 0 0x00000000
ffff8802362eb3a8 ffff8802362c09d0 0000000000013e80 ffff8802362ebfd8
0000000000013e80 ffff880235c28000 ffff8802362c09d0 ffff880200000001
ffff8802362eb500 7fffffffffffffff ffff8802362eb4f8 ffff8802362c09d0
Call Trace:
[<ffffffff817b24f9>] schedule+0x29/0x70
[<ffffffff817b55bc>] schedule_timeout+0x20c/0x280
[<ffffffff810aa4a4>] ? update_curr+0xe4/0x180
[<ffffffff817b3224>] wait_for_completion+0xa4/0x170
[<ffffffff810a0a70>] ? wake_up_state+0x20/0x20
[<ffffffff8108e7ed>] flush_work+0xed/0x1c0
[<ffffffff8108ac40>] ? destroy_worker+0x90/0x90
[<ffffffffc027451e>] xlog_cil_force_lsn+0x7e/0x1f0 [xfs]
[<ffffffff8117d667>] ? __free_pages+0x27/0x30
[<ffffffff811cd1a0>] ? __free_slab+0xd0/0x1f0
[<ffffffffc0272c7e>] _xfs_log_force_lsn+0x5e/0x2d0 [xfs]
[<ffffffffc0272f1e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
[<ffffffffc0264e79>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[<ffffffffc026198d>] __xfs_iunpin_wait+0x8d/0x120 [xfs]
[<ffffffff810b4e50>] ? autoremove_wake_function+0x40/0x40
[<ffffffffc0264e79>] xfs_iunpin_wait+0x19/0x20 [xfs]
[<ffffffffc025a18c>] xfs_reclaim_inode+0x7c/0x340 [xfs]
[<ffffffffc025a6a7>] xfs_reclaim_inodes_ag+0x257/0x370 [xfs]
[<ffffffffc025b1e3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[<ffffffffc0269f25>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[<ffffffff811efc79>] super_cache_scan+0x169/0x170
[<ffffffff81188346>] shrink_node_slabs+0x1d6/0x370
[<ffffffff8118af9a>] shrink_zone+0x20a/0x240
[<ffffffff8118b335>] do_try_to_free_pages+0x155/0x440
[<ffffffff8117b74f>] ? zone_watermark_ok+0x1f/0x30
[<ffffffff8118b6da>] try_to_free_pages+0xba/0x150
[<ffffffff8117f21b>] __alloc_pages_nodemask+0x61b/0xa60
[<ffffffff8117f72a>] alloc_kmem_pages_node+0x6a/0x130
[<ffffffff81072263>] ? copy_process.part.26+0xf3/0x1c00
[<ffffffff81072283>] copy_process.part.26+0x113/0x1c00
[<ffffffff810ab2ae>] ? dequeue_task_fair+0x44e/0x660
[<ffffffff810ac001>] ? put_prev_entity+0x31/0x3f0
[<ffffffff810af20c>] ? pick_next_task_fair+0x19c/0x880
[<ffffffff81093730>] ? kthread_create_on_node+0x1c0/0x1c0
[<ffffffff810ac3ef>] ? put_prev_task_fair+0x2f/0x50
[<ffffffff81073f25>] do_fork+0xd5/0x340
[<ffffffff810741b6>] kernel_thread+0x26/0x30
[<ffffffff810941ea>] kthreadd+0x15a/0x1c0
[<ffffffff81094090>] ? kthread_create_on_cpu+0x60/0x60
[<ffffffff817b67d8>] ret_from_fork+0x58/0x90
[<ffffffff81094090>] ? kthread_create_on_cpu+0x60/0x60
10:17:01 xyz CRON[59020]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
11:17:01 xyz CRON[59177]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
12:00:01 xyz CRON[59282]: (root) CMD (/usr/bin/rsnapshot -c /etc/rsnapshot.conf hourly)
12:03:05 xyz kernel: [616881.456247] INFO: task kthreadd:2 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.456587] Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:03:05 xyz kernel: [616881.457934] INFO: task kswapd0:57 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.458271] Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:03:05 xyz kernel: [616881.485629] INFO: task nfsd:2002 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.492761] Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:03:05 xyz kernel: [616881.523050] INFO: task snmpd:2155 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.530783] Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:03:05 xyz kernel: [616881.562789] INFO: task kworker/1:0:59030 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.571082] Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:03:05 xyz kernel: [616881.605747] INFO: task kworker/3:1H:59110 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.615112] Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:03:05 xyz kernel: [616881.652107] INFO: task rm:59284 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.661441] Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:05:05 xyz kernel: [617001.782228] INFO: task kswapd0:57 blocked for more than 120 seconds.
12:05:05 xyz kernel: [617001.791580] Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:05:05 xyz kernel: [617001.828636] INFO: task irqbalance:1353 blocked for more than 120 seconds.
12:05:05 xyz kernel: [617001.838019] Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:05:05 xyz kernel: [617001.875425] INFO: task nfsd:2002 blocked for more than 120 seconds.
12:05:05 xyz kernel: [617001.884799] Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
...
REFERENCES
http://marc.info/?l=linux-xfs&w=2&r=1&s=xlog_cil_force_lsn%2B0x7e%2F0x1f0&q=b
http://marc.info/?l=linux-xfs&w=2&r=1&s=xlog_cil_force_lsn&q=b
http://oss.sgi.com/cgi-bin/namazu.cgi?query=xlog_cil_force_lsn&submit=Search!&idxname=xfs&max=100&result=short&sort=date%3Alate
http://oss.sgi.com/archives/xfs/2015-03/msg00174.html
http://oss.sgi.com/archives/xfs/2014-09/msg00102.html
http://marc.info/?t=133455915600002&r=1&w=2
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/979498 |
Ubuntu 14.04.3 LTS with Kernel 3.19.0-26-generic
One of our servers running rsnapshot and nfs-kernel-server on XFS partitions
crashed or hanged two sundays in a raw.
Kernel 3.19.0-26-generic hangs multiple times (same time) in xlog_cil_force_lsn (XFS).
I assume this bug is not related with bug report 979498.
I had also this issue with kernel 3.16.0-46-generic or 3.13.0-57-generic but I have
no log during the crash to determine which of the above kernel was running.
I switched to kernel 3.19.0-26-generic because 3.16.0-48-generic was not
working well with Intel Gigabit -- tx hang (bug 1492146)
INFO: task kthreadd:2 blocked for more than 120 seconds.
Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kthreadd D ffff8802362eb3a8 0 2 0 0x00000000
ffff8802362eb3a8 ffff8802362c09d0 0000000000013e80 ffff8802362ebfd8
0000000000013e80 ffff880235c28000 ffff8802362c09d0 ffff880200000001
ffff8802362eb500 7fffffffffffffff ffff8802362eb4f8 ffff8802362c09d0
Call Trace:
[<ffffffff817b24f9>] schedule+0x29/0x70
[<ffffffff817b55bc>] schedule_timeout+0x20c/0x280
[<ffffffff810aa4a4>] ? update_curr+0xe4/0x180
[<ffffffff817b3224>] wait_for_completion+0xa4/0x170
[<ffffffff810a0a70>] ? wake_up_state+0x20/0x20
[<ffffffff8108e7ed>] flush_work+0xed/0x1c0
[<ffffffff8108ac40>] ? destroy_worker+0x90/0x90
[<ffffffffc027451e>] xlog_cil_force_lsn+0x7e/0x1f0 [xfs]
[<ffffffff8117d667>] ? __free_pages+0x27/0x30
[<ffffffff811cd1a0>] ? __free_slab+0xd0/0x1f0
[<ffffffffc0272c7e>] _xfs_log_force_lsn+0x5e/0x2d0 [xfs]
[<ffffffffc0272f1e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
[<ffffffffc0264e79>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[<ffffffffc026198d>] __xfs_iunpin_wait+0x8d/0x120 [xfs]
[<ffffffff810b4e50>] ? autoremove_wake_function+0x40/0x40
[<ffffffffc0264e79>] xfs_iunpin_wait+0x19/0x20 [xfs]
[<ffffffffc025a18c>] xfs_reclaim_inode+0x7c/0x340 [xfs]
[<ffffffffc025a6a7>] xfs_reclaim_inodes_ag+0x257/0x370 [xfs]
[<ffffffffc025b1e3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[<ffffffffc0269f25>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[<ffffffff811efc79>] super_cache_scan+0x169/0x170
[<ffffffff81188346>] shrink_node_slabs+0x1d6/0x370
[<ffffffff8118af9a>] shrink_zone+0x20a/0x240
[<ffffffff8118b335>] do_try_to_free_pages+0x155/0x440
[<ffffffff8117b74f>] ? zone_watermark_ok+0x1f/0x30
[<ffffffff8118b6da>] try_to_free_pages+0xba/0x150
[<ffffffff8117f21b>] __alloc_pages_nodemask+0x61b/0xa60
[<ffffffff8117f72a>] alloc_kmem_pages_node+0x6a/0x130
[<ffffffff81072263>] ? copy_process.part.26+0xf3/0x1c00
[<ffffffff81072283>] copy_process.part.26+0x113/0x1c00
[<ffffffff810ab2ae>] ? dequeue_task_fair+0x44e/0x660
[<ffffffff810ac001>] ? put_prev_entity+0x31/0x3f0
[<ffffffff810af20c>] ? pick_next_task_fair+0x19c/0x880
[<ffffffff81093730>] ? kthread_create_on_node+0x1c0/0x1c0
[<ffffffff810ac3ef>] ? put_prev_task_fair+0x2f/0x50
[<ffffffff81073f25>] do_fork+0xd5/0x340
[<ffffffff810741b6>] kernel_thread+0x26/0x30
[<ffffffff810941ea>] kthreadd+0x15a/0x1c0
[<ffffffff81094090>] ? kthread_create_on_cpu+0x60/0x60
[<ffffffff817b67d8>] ret_from_fork+0x58/0x90
[<ffffffff81094090>] ? kthread_create_on_cpu+0x60/0x60
10:17:01 xyz CRON[59020]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
11:17:01 xyz CRON[59177]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
12:00:01 xyz CRON[59282]: (root) CMD (/usr/bin/rsnapshot -c /etc/rsnapshot.conf hourly)
12:03:05 xyz kernel: [616881.456247] INFO: task kthreadd:2 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.456587] Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:03:05 xyz kernel: [616881.457934] INFO: task kswapd0:57 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.458271] Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:03:05 xyz kernel: [616881.485629] INFO: task nfsd:2002 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.492761] Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:03:05 xyz kernel: [616881.523050] INFO: task snmpd:2155 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.530783] Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:03:05 xyz kernel: [616881.562789] INFO: task kworker/1:0:59030 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.571082] Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:03:05 xyz kernel: [616881.605747] INFO: task kworker/3:1H:59110 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.615112] Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:03:05 xyz kernel: [616881.652107] INFO: task rm:59284 blocked for more than 120 seconds.
12:03:05 xyz kernel: [616881.661441] Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:05:05 xyz kernel: [617001.782228] INFO: task kswapd0:57 blocked for more than 120 seconds.
12:05:05 xyz kernel: [617001.791580] Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:05:05 xyz kernel: [617001.828636] INFO: task irqbalance:1353 blocked for more than 120 seconds.
12:05:05 xyz kernel: [617001.838019] Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
...
12:05:05 xyz kernel: [617001.875425] INFO: task nfsd:2002 blocked for more than 120 seconds.
12:05:05 xyz kernel: [617001.884799] Tainted: G C 3.19.0-26-generic #28~14.04.1-Ubuntu
...
REFERENCES
http://marc.info/?l=linux-xfs&w=2&r=1&s=xlog_cil_force_lsn%2B0x7e%2F0x1f0&q=b
http://marc.info/?l=linux-xfs&w=2&r=1&s=xlog_cil_force_lsn&q=b
http://oss.sgi.com/cgi-bin/namazu.cgi?query=xlog_cil_force_lsn&submit=Search!&idxname=xfs&max=100&result=short&sort=date%3Alate
http://oss.sgi.com/archives/xfs/2015-03/msg00174.html
http://oss.sgi.com/archives/xfs/2014-09/msg00102.html
http://marc.info/?t=133455915600002&r=1&w=2
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/979498
---
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Sep 14 07:30 seq
crw-rw---- 1 root audio 116, 33 Sep 14 07:30 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.13
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 14.04
HibernationDevice: RESUME=/dev/mapper/mainvg-swap
IwConfig: Error: [Errno 2] No such file or directory
MachineType: HP ProLiant DL380e Gen8
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
LANGUAGE=en_US:en
TERM=xterm
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.19.0-28-generic root=/dev/mapper/mainvg-root ro elevator=deadline nohz=off transparent_hugepage=always intel_iommu=on acpi_irq_nobalance cgroup_enable=memory swapaccount=1 nomdmonddf nomdmonisw
ProcVersionSignature: Ubuntu 3.19.0-28.30~14.04.1-generic 3.19.8-ckt5
RelatedPackageVersions:
linux-restricted-modules-3.19.0-28-generic N/A
linux-backports-modules-3.19.0-28-generic N/A
linux-firmware 1.127.15
RfKill: Error: [Errno 2] No such file or directory
StagingDrivers: visorutil
Tags: trusty staging
Uname: Linux 3.19.0-28-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
_MarkForUpload: True
dmi.bios.date: 02/10/2014
dmi.bios.vendor: HP
dmi.bios.version: P73
dmi.chassis.type: 23
dmi.chassis.vendor: HP
dmi.modalias: dmi:bvnHP:bvrP73:bd02/10/2014:svnHP:pnProLiantDL380eGen8:pvr:cvnHP:ct23:cvr:
dmi.product.name: ProLiant DL380e Gen8
dmi.sys.vendor: HP |
|
| 2015-09-14 11:21:46 |
B. |
attachment added |
|
BootDmesg.txt https://bugs.launchpad.net/bugs/1495442/+attachment/4463949/+files/BootDmesg.txt |
|
| 2015-09-14 11:21:47 |
B. |
attachment added |
|
CurrentDmesg.txt https://bugs.launchpad.net/bugs/1495442/+attachment/4463950/+files/CurrentDmesg.txt |
|
| 2015-09-14 11:21:49 |
B. |
attachment added |
|
Lspci.txt https://bugs.launchpad.net/bugs/1495442/+attachment/4463951/+files/Lspci.txt |
|
| 2015-09-14 11:21:51 |
B. |
attachment added |
|
Lsusb.txt https://bugs.launchpad.net/bugs/1495442/+attachment/4463952/+files/Lsusb.txt |
|
| 2015-09-14 11:21:52 |
B. |
attachment added |
|
ProcCpuinfo.txt https://bugs.launchpad.net/bugs/1495442/+attachment/4463953/+files/ProcCpuinfo.txt |
|
| 2015-09-14 11:21:54 |
B. |
attachment added |
|
ProcInterrupts.txt https://bugs.launchpad.net/bugs/1495442/+attachment/4463954/+files/ProcInterrupts.txt |
|
| 2015-09-14 11:21:56 |
B. |
attachment added |
|
ProcModules.txt https://bugs.launchpad.net/bugs/1495442/+attachment/4463955/+files/ProcModules.txt |
|
| 2015-09-14 11:21:58 |
B. |
attachment added |
|
UdevDb.txt https://bugs.launchpad.net/bugs/1495442/+attachment/4463956/+files/UdevDb.txt |
|
| 2015-09-14 11:22:00 |
B. |
attachment added |
|
UdevLog.txt https://bugs.launchpad.net/bugs/1495442/+attachment/4463957/+files/UdevLog.txt |
|
| 2015-09-14 11:22:01 |
B. |
attachment added |
|
WifiSyslog.txt https://bugs.launchpad.net/bugs/1495442/+attachment/4463958/+files/WifiSyslog.txt |
|
| 2015-09-14 14:42:54 |
Joseph Salisbury |
linux (Ubuntu): importance |
Undecided |
High |
|
| 2015-09-14 20:46:36 |
Scottix |
bug |
|
|
added subscriber Scottix |
| 2015-11-22 04:18:30 |
Launchpad Janitor |
linux (Ubuntu): status |
Incomplete |
Expired |
|
