Random crash

Bug #1368245 reported by Leonid Samokhine
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Incomplete
Undecided
Unassigned

Bug Description

We are experiencing a random crashes (about ones in 2 days) after upgrading to Ubuntu 14.04.1 LTS.
The server starts killing processes due to lack of memory. If you wait long enough it hangs and only hard reboot can fix it.

Here is I believe the first record in syslog when it starts:

Sep 11 11:21:08 pricing kernel: [158208.874463] ------------[ cut here ]------------
Sep 11 11:21:08 pricing kernel: [158208.874473] WARNING: CPU: 1 PID: 24257 at /build/buildd/linux-3.13.0/fs/dcache.c:362 d_shrink_del+0x71/0x80()
Sep 11 11:21:08 pricing kernel: [158208.874475] Modules linked in: xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_addrtype xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 x
t_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables isofs crct10dif_pclmul crc32_
pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd xfs libcrc32c raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid10 raid6_pq ra
id1 raid0 multipath linear
Sep 11 11:21:08 pricing kernel: [158208.874515] CPU: 1 PID: 24257 Comm: sh Not tainted 3.13.0-35-generic #62-Ubuntu
Sep 11 11:21:08 pricing kernel: [158208.874518] 0000000000000009 ffff880011055c78 ffffffff8171e320 0000000000000000
Sep 11 11:21:08 pricing kernel: [158208.874522] ffff880011055cb0 ffffffff8106775d ffff88000b6a0900 ffff880011055d30
Sep 11 11:21:08 pricing kernel: [158208.874525] ffff88000b6a0958 ffff88000b6a0900 ffff88000b6a0900 ffff880011055cc0
Sep 11 11:21:08 pricing kernel: [158208.874528] Call Trace:
Sep 11 11:21:08 pricing kernel: [158208.874534] [<ffffffff8171e320>] dump_stack+0x45/0x56
Sep 11 11:21:08 pricing kernel: [158208.874539] [<ffffffff8106775d>] warn_slowpath_common+0x7d/0xa0
Sep 11 11:21:08 pricing kernel: [158208.874542] [<ffffffff8106783a>] warn_slowpath_null+0x1a/0x20
Sep 11 11:21:08 pricing kernel: [158208.874545] [<ffffffff811d2af1>] d_shrink_del+0x71/0x80
Sep 11 11:21:08 pricing kernel: [158208.874548] [<ffffffff811d3f64>] shrink_dentry_list+0x64/0xe0
Sep 11 11:21:08 pricing kernel: [158208.874551] [<ffffffff811d4d78>] shrink_dcache_parent+0x28/0x70
Sep 11 11:21:08 pricing kernel: [158208.874556] [<ffffffff81228686>] proc_flush_task+0xb6/0x1b0
Sep 11 11:21:08 pricing kernel: [158208.874560] [<ffffffff810685e0>] release_task+0x30/0x460
Sep 11 11:21:08 pricing kernel: [158208.874564] [<ffffffff8109e2f1>] ? thread_group_cputime_adjusted+0x41/0x50
Sep 11 11:21:08 pricing kernel: [158208.874567] [<ffffffff810692ef>] wait_consider_task+0x8df/0xb20
Sep 11 11:21:08 pricing kernel: [158208.874570] [<ffffffff81069630>] do_wait+0x100/0x240
Sep 11 11:21:08 pricing kernel: [158208.874573] [<ffffffff8106a764>] SyS_wait4+0x64/0xe0
Sep 11 11:21:08 pricing kernel: [158208.874577] [<ffffffff81068340>] ? task_stopped_code+0x60/0x60
Sep 11 11:21:08 pricing kernel: [158208.874580] [<ffffffff8172ed6d>] system_call_fastpath+0x1a/0x1f
Sep 11 11:21:08 pricing kernel: [158208.874582] ---[ end trace 5777f83b2241c020 ]---
Sep 11 11:21:35 pricing kernel: [158236.096055] BUG: soft lockup - CPU#1 stuck for 22s! [sh:24257]
Sep 11 11:21:35 pricing kernel: [158236.096055] Modules linked in: xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_addrtype xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 x
t_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables isofs crct10dif_pclmul crc32_
pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd xfs libcrc32c raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid10 raid6_pq ra
id1 raid0 multipath linear
Sep 11 11:21:35 pricing kernel: [158236.096055] CPU: 1 PID: 24257 Comm: sh Tainted: G W 3.13.0-35-generic #62-Ubuntu
Sep 11 11:21:35 pricing kernel: [158236.096055] task: ffff8800de5f8000 ti: ffff880011054000 task.ti: ffff880011054000
Sep 11 11:21:35 pricing kernel: [158236.096055] RIP: e030:[<ffffffff811d2a80>] [<ffffffff811d2a80>] d_shrink_del+0x0/0x80
Sep 11 11:21:35 pricing kernel: [158236.096055] RSP: e02b:ffff880011055ce0 EFLAGS: 00000246
Sep 11 11:21:35 pricing kernel: [158236.096055] RAX: ffff88000b6a0980 RBX: ffff88000b6a0980 RCX: ffff88000b6a0980
Sep 11 11:21:35 pricing kernel: [158236.096055] RDX: 000000000000fbe4 RSI: 00000000fbe4fbe2 RDI: ffff88000b6a0900
Sep 11 11:21:35 pricing kernel: [158236.096055] RBP: ffff880011055d18 R08: 0000000000000000 R09: 00000000000001c5
Sep 11 11:21:35 pricing kernel: [158236.096055] R10: 0000000000000000 R11: ffff8800110559a6 R12: ffff880011055d30
Sep 11 11:21:35 pricing kernel: [158236.096055] R13: ffff88000b6a0958 R14: ffff88000b6a0900 R15: ffff88000b6a0900
Sep 11 11:21:35 pricing kernel: [158236.096055] FS: 00007f1611926740(0000) GS:ffff8803bdc80000(0000) knlGS:0000000000000000
Sep 11 11:21:35 pricing kernel: [158236.096055] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Sep 11 11:21:35 pricing kernel: [158236.096055] CR2: 00007f079beb1d50 CR3: 000000000cfcf000 CR4: 0000000000002660
Sep 11 11:21:35 pricing kernel: [158236.096055] Stack:
Sep 11 11:21:35 pricing kernel: [158236.096055] ffffffff811d3f64 ffff88000b6a0980 ffff880011055d30 ffff88010764f3c0
Sep 11 11:21:35 pricing kernel: [158236.096055] ffff8803a6d603e0 ffff88010764f3c0 0000000000005ec2 ffff880011055d58
Sep 11 11:21:35 pricing kernel: [158236.096055] ffffffff811d4d78 ffff88010764f3c0 ffff88000b6a0b00 ffff88000b6a0980
Sep 11 11:21:35 pricing kernel: [158236.096055] Call Trace:
Sep 11 11:21:35 pricing kernel: [158236.096055] [<ffffffff811d3f64>] ? shrink_dentry_list+0x64/0xe0
Sep 11 11:21:35 pricing kernel: [158236.096055] [<ffffffff811d4d78>] shrink_dcache_parent+0x28/0x70
Sep 11 11:21:35 pricing kernel: [158236.096055] [<ffffffff81228686>] proc_flush_task+0xb6/0x1b0

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-3.13.0-35-generic 3.13.0-35.62
ProcVersionSignature: Ubuntu 3.13.0-35.62-generic 3.13.11.6
Uname: Linux 3.13.0-35-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Sep 11 12:29 seq
 crw-rw---- 1 root audio 116, 33 Sep 11 12:29 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.14.1-0ubuntu3.3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
CurrentDmesg: [ 45.163030] init: plymouth-upstart-bridge main process ended, respawning
Date: Thu Sep 11 13:41:04 2014
Ec2AMI: ami-3fec7956
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1c
Ec2InstanceType: m3.xlarge
Ec2Kernel: aki-88aa75e1
Ec2Ramdisk: unavailable
IwConfig:
 eth0 no wireless extensions.

 lo no wireless extensions.
Lspci:

Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-35-generic N/A
 linux-backports-modules-3.13.0-35-generic N/A
 linux-firmware 1.127.5
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: Upgraded to trusty on 2014-09-07 (4 days ago)

Revision history for this message
Leonid Samokhine (samokhine) wrote :
Revision history for this message
Brad Figg (brad-figg) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1368245

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Dave Chiluk (chiluk) wrote :

Thank you for your opening of this bug, but this issue has already been resolved by 3.13.0-36 please see bug 1368245 for more information.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.