| 2015-02-19 19:26:25 |
Simon Déziel |
bug |
|
|
added bug |
| 2015-02-19 19:26:25 |
Simon Déziel |
attachment added |
|
dmesg -T https://bugs.launchpad.net/bugs/1423672/+attachment/4322691/+files/dmesg-T.log |
|
| 2015-02-19 19:26:49 |
Simon Déziel |
attachment added |
|
/proc/version_signature https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1423672/+attachment/4322692/+files/version.log |
|
| 2015-02-19 19:30:10 |
Brad Figg |
linux (Ubuntu): status |
New |
Incomplete |
|
| 2015-02-19 19:30:11 |
Brad Figg |
tags |
|
trusty |
|
| 2015-02-19 19:49:20 |
Simon Déziel |
tags |
trusty |
apport-collected trusty |
|
| 2015-02-19 19:49:21 |
Simon Déziel |
description |
I noticed that one of my VM had this "dmesg -T" output:
[Tue Feb 17 09:53:27 2015] systemd-udevd[5433]: starting version 204
[Thu Feb 19 06:25:08 2015] EXT4-fs error (device vda1): ext4_mb_generate_buddy:756: group 5, block bitmap and bg descriptor inconsistent: 16446 vs 16445 free clusters
[Thu Feb 19 06:25:09 2015] Aborting journal on device vda1-8.
[Thu Feb 19 06:25:09 2015] EXT4-fs (vda1): Remounting filesystem read-only
[Thu Feb 19 06:25:09 2015] ------------[ cut here ]------------
[Thu Feb 19 06:25:09 2015] WARNING: CPU: 0 PID: 9946 at /build/buildd/linux-3.13.0/fs/ext4/ext4_jbd2.c:259 __ext4_handle_dirty_metadata+0x1a2/0x1c0()
[Thu Feb 19 06:25:09 2015] Modules linked in: nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner xt_conntrack nf_conntrack iptable_filter ip_tables x_tables serio_raw psmouse floppy
[Thu Feb 19 06:25:09 2015] CPU: 0 PID: 9946 Comm: logrotate Not tainted 3.13.0-45-generic #74-Ubuntu
[Thu Feb 19 06:25:09 2015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[Thu Feb 19 06:25:09 2015] 0000000000000009 ffff880003a11aa0 ffffffff81720eb6 0000000000000000
[Thu Feb 19 06:25:09 2015] ffff880003a11ad8 ffffffff810677cd ffff880000c41340 0000000000000000
[Thu Feb 19 06:25:09 2015] ffff88000a58e000 ffffffff81835280 0000000000001302 ffff880003a11ae8
[Thu Feb 19 06:25:09 2015] Call Trace:
[Thu Feb 19 06:25:09 2015] [<ffffffff81720eb6>] dump_stack+0x45/0x56
[Thu Feb 19 06:25:09 2015] [<ffffffff810677cd>] warn_slowpath_common+0x7d/0xa0
[Thu Feb 19 06:25:09 2015] [<ffffffff810678aa>] warn_slowpath_null+0x1a/0x20
[Thu Feb 19 06:25:09 2015] [<ffffffff8126e862>] __ext4_handle_dirty_metadata+0x1a2/0x1c0
[Thu Feb 19 06:25:09 2015] [<ffffffff81246a5a>] ? ext4_dirty_inode+0x2a/0x60
[Thu Feb 19 06:25:09 2015] [<ffffffff81277086>] ext4_free_blocks+0x646/0xbf0
[Thu Feb 19 06:25:09 2015] [<ffffffff810aacc5>] ? wake_up_bit+0x25/0x30
[Thu Feb 19 06:25:09 2015] [<ffffffff812685b5>] ext4_ext_rm_leaf+0x505/0x8f0
[Thu Feb 19 06:25:09 2015] [<ffffffff81267527>] ? __ext4_ext_check+0x197/0x370
[Thu Feb 19 06:25:09 2015] [<ffffffff8126ad00>] ? ext4_ext_remove_space+0xc0/0x7e0
[Thu Feb 19 06:25:09 2015] [<ffffffff8126af5c>] ext4_ext_remove_space+0x31c/0x7e0
[Thu Feb 19 06:25:09 2015] [<ffffffff8126d300>] ext4_ext_truncate+0xb0/0xe0
[Thu Feb 19 06:25:09 2015] [<ffffffff81244eb9>] ext4_truncate+0x379/0x3c0
[Thu Feb 19 06:25:09 2015] [<ffffffff81245a18>] ext4_evict_inode+0x408/0x4d0
[Thu Feb 19 06:25:09 2015] [<ffffffff811d8f60>] evict+0xb0/0x1b0
[Thu Feb 19 06:25:09 2015] [<ffffffff811d9775>] iput+0xf5/0x180
[Thu Feb 19 06:25:09 2015] [<ffffffff811d4698>] __dentry_kill+0x1a8/0x200
[Thu Feb 19 06:25:09 2015] [<ffffffff811d4795>] dput+0xa5/0x180
[Thu Feb 19 06:25:09 2015] [<ffffffff811bf7e6>] __fput+0x176/0x260
[Thu Feb 19 06:25:09 2015] [<ffffffff811bf91e>] ____fput+0xe/0x10
[Thu Feb 19 06:25:09 2015] [<ffffffff810882f7>] task_work_run+0xa7/0xe0
[Thu Feb 19 06:25:09 2015] [<ffffffff81013ed7>] do_notify_resume+0x97/0xb0
[Thu Feb 19 06:25:09 2015] [<ffffffff81731c2a>] int_signal+0x12/0x17
[Thu Feb 19 06:25:09 2015] ---[ end trace ebff9843d81b5c41 ]---
[Thu Feb 19 06:25:09 2015] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
[Thu Feb 19 06:25:09 2015] IP: [<ffffffff8125d4c1>] __ext4_error_inode+0x31/0x160
[Thu Feb 19 06:25:09 2015] PGD 167067 PUD 161067 PMD 0
[Thu Feb 19 06:25:09 2015] Oops: 0000 [#1] SMP
[Thu Feb 19 06:25:09 2015] Modules linked in: nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner xt_conntrack nf_conntrack iptable_filter ip_tables x_tables serio_raw psmouse floppy
[Thu Feb 19 06:25:09 2015] CPU: 0 PID: 9946 Comm: logrotate Tainted: G W 3.13.0-45-generic #74-Ubuntu
[Thu Feb 19 06:25:09 2015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[Thu Feb 19 06:25:09 2015] task: ffff880009ac4800 ti: ffff880003a10000 task.ti: ffff880003a10000
[Thu Feb 19 06:25:09 2015] RIP: 0010:[<ffffffff8125d4c1>] [<ffffffff8125d4c1>] __ext4_error_inode+0x31/0x160
[Thu Feb 19 06:25:09 2015] RSP: 0000:ffff880003a11a58 EFLAGS: 00010292
[Thu Feb 19 06:25:09 2015] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000086
[Thu Feb 19 06:25:09 2015] RDX: 0000000000001302 RSI: ffffffff81a6e81f RDI: 0000000000000000
[Thu Feb 19 06:25:09 2015] RBP: ffff880003a11ae8 R08: ffffffff81a78568 R09: 0000000000000005
[Thu Feb 19 06:25:09 2015] R10: 00000000ffffffe2 R11: ffff880003a117ce R12: 0000000000000086
[Thu Feb 19 06:25:09 2015] R13: ffffffff81835280 R14: 0000000000001302 R15: ffffffff81a78568
[Thu Feb 19 06:25:09 2015] FS: 00007f74eaca4840(0000) GS:ffff88000b800000(0000) knlGS:0000000000000000
[Thu Feb 19 06:25:09 2015] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[Thu Feb 19 06:25:09 2015] CR2: 0000000001de4000 CR3: 0000000009b45000 CR4: 00000000000006f0
[Thu Feb 19 06:25:09 2015] Stack:
[Thu Feb 19 06:25:09 2015] ffff880003a11a60 0000000000000103 ebff9843d81b5c41 000000000000321e
[Thu Feb 19 06:25:09 2015] 00000000000014d8 0000000000000092 000000000000020e ffff88000a58e000
[Thu Feb 19 06:25:09 2015] ffff880003a11ae8 ffffffff8126e372 ffffffff810677df ffff880000c41340
[Thu Feb 19 06:25:09 2015] Call Trace:
[Thu Feb 19 06:25:09 2015] [<ffffffff8126e372>] ? ext4_journal_abort_handle+0x42/0xc0
[Thu Feb 19 06:25:09 2015] [<ffffffff810677df>] ? warn_slowpath_common+0x8f/0xa0
[Thu Feb 19 06:25:09 2015] [<ffffffff8126e7cf>] __ext4_handle_dirty_metadata+0x10f/0x1c0
[Thu Feb 19 06:25:09 2015] [<ffffffff81277086>] ext4_free_blocks+0x646/0xbf0
[Thu Feb 19 06:25:09 2015] [<ffffffff810aacc5>] ? wake_up_bit+0x25/0x30
[Thu Feb 19 06:25:09 2015] [<ffffffff812685b5>] ext4_ext_rm_leaf+0x505/0x8f0
[Thu Feb 19 06:25:09 2015] [<ffffffff81267527>] ? __ext4_ext_check+0x197/0x370
[Thu Feb 19 06:25:09 2015] [<ffffffff8126ad00>] ? ext4_ext_remove_space+0xc0/0x7e0
[Thu Feb 19 06:25:09 2015] [<ffffffff8126af5c>] ext4_ext_remove_space+0x31c/0x7e0
[Thu Feb 19 06:25:09 2015] [<ffffffff8126d300>] ext4_ext_truncate+0xb0/0xe0
[Thu Feb 19 06:25:09 2015] [<ffffffff81244eb9>] ext4_truncate+0x379/0x3c0
[Thu Feb 19 06:25:09 2015] [<ffffffff81245a18>] ext4_evict_inode+0x408/0x4d0
[Thu Feb 19 06:25:09 2015] [<ffffffff811d8f60>] evict+0xb0/0x1b0
[Thu Feb 19 06:25:09 2015] [<ffffffff811d9775>] iput+0xf5/0x180
[Thu Feb 19 06:25:09 2015] [<ffffffff811d4698>] __dentry_kill+0x1a8/0x200
[Thu Feb 19 06:25:09 2015] [<ffffffff811d4795>] dput+0xa5/0x180
[Thu Feb 19 06:25:09 2015] [<ffffffff811bf7e6>] __fput+0x176/0x260
[Thu Feb 19 06:25:09 2015] [<ffffffff811bf91e>] ____fput+0xe/0x10
[Thu Feb 19 06:25:09 2015] [<ffffffff810882f7>] task_work_run+0xa7/0xe0
[Thu Feb 19 06:25:09 2015] [<ffffffff81013ed7>] do_notify_resume+0x97/0xb0
[Thu Feb 19 06:25:09 2015] [<ffffffff81731c2a>] int_signal+0x12/0x17
[Thu Feb 19 06:25:09 2015] Code: 48 89 e5 41 57 4d 89 c7 41 56 41 89 d6 41 55 49 89 f5 48 c7 c6 1f e8 a6 81 41 54 49 89 cc 53 48 89 fb 48 83 ec 68 4c 89 4c 24 60 <48> 8b 47 28 48 8b 57 40 48 8b 80 f8 02 00 00 48 8b 40 68 89 90
[Thu Feb 19 06:25:09 2015] RIP [<ffffffff8125d4c1>] __ext4_error_inode+0x31/0x160
[Thu Feb 19 06:25:09 2015] RSP <ffff880003a11a58>
[Thu Feb 19 06:25:09 2015] CR2: 0000000000000028
[Thu Feb 19 06:25:10 2015] ---[ end trace ebff9843d81b5c42 ]---
cron.daily jobs fired at 6:25:01 apparently:
# tail -n2 /var/log/syslog
Feb 19 06:17:01 git CRON[9848]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Feb 19 06:25:01 git CRON[9853]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
# run-parts --test /etc/cron.daily
/etc/cron.daily/apt
/etc/cron.daily/autoremove
/etc/cron.daily/dpkg
/etc/cron.daily/hdd-backup
/etc/cron.daily/logrotate
/etc/cron.daily/passwd
/etc/cron.daily/upstart
It seems like all the jobs ran and the upstart one somehow triggered the crash:
# ls -alt /var/log/upstart/ | head
total 272
drwxrwxr-x 5 root syslog 4096 Feb 19 06:25 ..
drwxr-xr-x 2 root root 4096 Feb 14 06:25 .
-rw-r----- 1 root root 591 Feb 12 12:26 ureadahead.log.1.gz
-rw-r----- 1 root root 178 Feb 12 12:25 mountall.log.1.gz
Now that I have collected some information (sorry, I don't have ubuntu-bug installed on the VM) I'll reboot it and see how it goes.
More information on the VM:
# lsb_release -rd
Description: Ubuntu 14.04.2 LTS
Release: 14.04
# apt-cache policy linux-image-3.13.0-45-generic
linux-image-3.13.0-45-generic:
Installed: 3.13.0-45.74
Candidate: 3.13.0-45.74
Version table:
*** 3.13.0-45.74 0
500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
100 /var/lib/dpkg/status |
I noticed that one of my VM had this "dmesg -T" output:
[Tue Feb 17 09:53:27 2015] systemd-udevd[5433]: starting version 204
[Thu Feb 19 06:25:08 2015] EXT4-fs error (device vda1): ext4_mb_generate_buddy:756: group 5, block bitmap and bg descriptor inconsistent: 16446 vs 16445 free clusters
[Thu Feb 19 06:25:09 2015] Aborting journal on device vda1-8.
[Thu Feb 19 06:25:09 2015] EXT4-fs (vda1): Remounting filesystem read-only
[Thu Feb 19 06:25:09 2015] ------------[ cut here ]------------
[Thu Feb 19 06:25:09 2015] WARNING: CPU: 0 PID: 9946 at /build/buildd/linux-3.13.0/fs/ext4/ext4_jbd2.c:259 __ext4_handle_dirty_metadata+0x1a2/0x1c0()
[Thu Feb 19 06:25:09 2015] Modules linked in: nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner xt_conntrack nf_conntrack iptable_filter ip_tables x_tables serio_raw psmouse floppy
[Thu Feb 19 06:25:09 2015] CPU: 0 PID: 9946 Comm: logrotate Not tainted 3.13.0-45-generic #74-Ubuntu
[Thu Feb 19 06:25:09 2015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[Thu Feb 19 06:25:09 2015] 0000000000000009 ffff880003a11aa0 ffffffff81720eb6 0000000000000000
[Thu Feb 19 06:25:09 2015] ffff880003a11ad8 ffffffff810677cd ffff880000c41340 0000000000000000
[Thu Feb 19 06:25:09 2015] ffff88000a58e000 ffffffff81835280 0000000000001302 ffff880003a11ae8
[Thu Feb 19 06:25:09 2015] Call Trace:
[Thu Feb 19 06:25:09 2015] [<ffffffff81720eb6>] dump_stack+0x45/0x56
[Thu Feb 19 06:25:09 2015] [<ffffffff810677cd>] warn_slowpath_common+0x7d/0xa0
[Thu Feb 19 06:25:09 2015] [<ffffffff810678aa>] warn_slowpath_null+0x1a/0x20
[Thu Feb 19 06:25:09 2015] [<ffffffff8126e862>] __ext4_handle_dirty_metadata+0x1a2/0x1c0
[Thu Feb 19 06:25:09 2015] [<ffffffff81246a5a>] ? ext4_dirty_inode+0x2a/0x60
[Thu Feb 19 06:25:09 2015] [<ffffffff81277086>] ext4_free_blocks+0x646/0xbf0
[Thu Feb 19 06:25:09 2015] [<ffffffff810aacc5>] ? wake_up_bit+0x25/0x30
[Thu Feb 19 06:25:09 2015] [<ffffffff812685b5>] ext4_ext_rm_leaf+0x505/0x8f0
[Thu Feb 19 06:25:09 2015] [<ffffffff81267527>] ? __ext4_ext_check+0x197/0x370
[Thu Feb 19 06:25:09 2015] [<ffffffff8126ad00>] ? ext4_ext_remove_space+0xc0/0x7e0
[Thu Feb 19 06:25:09 2015] [<ffffffff8126af5c>] ext4_ext_remove_space+0x31c/0x7e0
[Thu Feb 19 06:25:09 2015] [<ffffffff8126d300>] ext4_ext_truncate+0xb0/0xe0
[Thu Feb 19 06:25:09 2015] [<ffffffff81244eb9>] ext4_truncate+0x379/0x3c0
[Thu Feb 19 06:25:09 2015] [<ffffffff81245a18>] ext4_evict_inode+0x408/0x4d0
[Thu Feb 19 06:25:09 2015] [<ffffffff811d8f60>] evict+0xb0/0x1b0
[Thu Feb 19 06:25:09 2015] [<ffffffff811d9775>] iput+0xf5/0x180
[Thu Feb 19 06:25:09 2015] [<ffffffff811d4698>] __dentry_kill+0x1a8/0x200
[Thu Feb 19 06:25:09 2015] [<ffffffff811d4795>] dput+0xa5/0x180
[Thu Feb 19 06:25:09 2015] [<ffffffff811bf7e6>] __fput+0x176/0x260
[Thu Feb 19 06:25:09 2015] [<ffffffff811bf91e>] ____fput+0xe/0x10
[Thu Feb 19 06:25:09 2015] [<ffffffff810882f7>] task_work_run+0xa7/0xe0
[Thu Feb 19 06:25:09 2015] [<ffffffff81013ed7>] do_notify_resume+0x97/0xb0
[Thu Feb 19 06:25:09 2015] [<ffffffff81731c2a>] int_signal+0x12/0x17
[Thu Feb 19 06:25:09 2015] ---[ end trace ebff9843d81b5c41 ]---
[Thu Feb 19 06:25:09 2015] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
[Thu Feb 19 06:25:09 2015] IP: [<ffffffff8125d4c1>] __ext4_error_inode+0x31/0x160
[Thu Feb 19 06:25:09 2015] PGD 167067 PUD 161067 PMD 0
[Thu Feb 19 06:25:09 2015] Oops: 0000 [#1] SMP
[Thu Feb 19 06:25:09 2015] Modules linked in: nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner xt_conntrack nf_conntrack iptable_filter ip_tables x_tables serio_raw psmouse floppy
[Thu Feb 19 06:25:09 2015] CPU: 0 PID: 9946 Comm: logrotate Tainted: G W 3.13.0-45-generic #74-Ubuntu
[Thu Feb 19 06:25:09 2015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[Thu Feb 19 06:25:09 2015] task: ffff880009ac4800 ti: ffff880003a10000 task.ti: ffff880003a10000
[Thu Feb 19 06:25:09 2015] RIP: 0010:[<ffffffff8125d4c1>] [<ffffffff8125d4c1>] __ext4_error_inode+0x31/0x160
[Thu Feb 19 06:25:09 2015] RSP: 0000:ffff880003a11a58 EFLAGS: 00010292
[Thu Feb 19 06:25:09 2015] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000086
[Thu Feb 19 06:25:09 2015] RDX: 0000000000001302 RSI: ffffffff81a6e81f RDI: 0000000000000000
[Thu Feb 19 06:25:09 2015] RBP: ffff880003a11ae8 R08: ffffffff81a78568 R09: 0000000000000005
[Thu Feb 19 06:25:09 2015] R10: 00000000ffffffe2 R11: ffff880003a117ce R12: 0000000000000086
[Thu Feb 19 06:25:09 2015] R13: ffffffff81835280 R14: 0000000000001302 R15: ffffffff81a78568
[Thu Feb 19 06:25:09 2015] FS: 00007f74eaca4840(0000) GS:ffff88000b800000(0000) knlGS:0000000000000000
[Thu Feb 19 06:25:09 2015] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[Thu Feb 19 06:25:09 2015] CR2: 0000000001de4000 CR3: 0000000009b45000 CR4: 00000000000006f0
[Thu Feb 19 06:25:09 2015] Stack:
[Thu Feb 19 06:25:09 2015] ffff880003a11a60 0000000000000103 ebff9843d81b5c41 000000000000321e
[Thu Feb 19 06:25:09 2015] 00000000000014d8 0000000000000092 000000000000020e ffff88000a58e000
[Thu Feb 19 06:25:09 2015] ffff880003a11ae8 ffffffff8126e372 ffffffff810677df ffff880000c41340
[Thu Feb 19 06:25:09 2015] Call Trace:
[Thu Feb 19 06:25:09 2015] [<ffffffff8126e372>] ? ext4_journal_abort_handle+0x42/0xc0
[Thu Feb 19 06:25:09 2015] [<ffffffff810677df>] ? warn_slowpath_common+0x8f/0xa0
[Thu Feb 19 06:25:09 2015] [<ffffffff8126e7cf>] __ext4_handle_dirty_metadata+0x10f/0x1c0
[Thu Feb 19 06:25:09 2015] [<ffffffff81277086>] ext4_free_blocks+0x646/0xbf0
[Thu Feb 19 06:25:09 2015] [<ffffffff810aacc5>] ? wake_up_bit+0x25/0x30
[Thu Feb 19 06:25:09 2015] [<ffffffff812685b5>] ext4_ext_rm_leaf+0x505/0x8f0
[Thu Feb 19 06:25:09 2015] [<ffffffff81267527>] ? __ext4_ext_check+0x197/0x370
[Thu Feb 19 06:25:09 2015] [<ffffffff8126ad00>] ? ext4_ext_remove_space+0xc0/0x7e0
[Thu Feb 19 06:25:09 2015] [<ffffffff8126af5c>] ext4_ext_remove_space+0x31c/0x7e0
[Thu Feb 19 06:25:09 2015] [<ffffffff8126d300>] ext4_ext_truncate+0xb0/0xe0
[Thu Feb 19 06:25:09 2015] [<ffffffff81244eb9>] ext4_truncate+0x379/0x3c0
[Thu Feb 19 06:25:09 2015] [<ffffffff81245a18>] ext4_evict_inode+0x408/0x4d0
[Thu Feb 19 06:25:09 2015] [<ffffffff811d8f60>] evict+0xb0/0x1b0
[Thu Feb 19 06:25:09 2015] [<ffffffff811d9775>] iput+0xf5/0x180
[Thu Feb 19 06:25:09 2015] [<ffffffff811d4698>] __dentry_kill+0x1a8/0x200
[Thu Feb 19 06:25:09 2015] [<ffffffff811d4795>] dput+0xa5/0x180
[Thu Feb 19 06:25:09 2015] [<ffffffff811bf7e6>] __fput+0x176/0x260
[Thu Feb 19 06:25:09 2015] [<ffffffff811bf91e>] ____fput+0xe/0x10
[Thu Feb 19 06:25:09 2015] [<ffffffff810882f7>] task_work_run+0xa7/0xe0
[Thu Feb 19 06:25:09 2015] [<ffffffff81013ed7>] do_notify_resume+0x97/0xb0
[Thu Feb 19 06:25:09 2015] [<ffffffff81731c2a>] int_signal+0x12/0x17
[Thu Feb 19 06:25:09 2015] Code: 48 89 e5 41 57 4d 89 c7 41 56 41 89 d6 41 55 49 89 f5 48 c7 c6 1f e8 a6 81 41 54 49 89 cc 53 48 89 fb 48 83 ec 68 4c 89 4c 24 60 <48> 8b 47 28 48 8b 57 40 48 8b 80 f8 02 00 00 48 8b 40 68 89 90
[Thu Feb 19 06:25:09 2015] RIP [<ffffffff8125d4c1>] __ext4_error_inode+0x31/0x160
[Thu Feb 19 06:25:09 2015] RSP <ffff880003a11a58>
[Thu Feb 19 06:25:09 2015] CR2: 0000000000000028
[Thu Feb 19 06:25:10 2015] ---[ end trace ebff9843d81b5c42 ]---
cron.daily jobs fired at 6:25:01 apparently:
# tail -n2 /var/log/syslog
Feb 19 06:17:01 git CRON[9848]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Feb 19 06:25:01 git CRON[9853]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
# run-parts --test /etc/cron.daily
/etc/cron.daily/apt
/etc/cron.daily/autoremove
/etc/cron.daily/dpkg
/etc/cron.daily/hdd-backup
/etc/cron.daily/logrotate
/etc/cron.daily/passwd
/etc/cron.daily/upstart
It seems like all the jobs ran and the upstart one somehow triggered the crash:
# ls -alt /var/log/upstart/ | head
total 272
drwxrwxr-x 5 root syslog 4096 Feb 19 06:25 ..
drwxr-xr-x 2 root root 4096 Feb 14 06:25 .
-rw-r----- 1 root root 591 Feb 12 12:26 ureadahead.log.1.gz
-rw-r----- 1 root root 178 Feb 12 12:25 mountall.log.1.gz
Now that I have collected some information (sorry, I don't have ubuntu-bug installed on the VM) I'll reboot it and see how it goes.
More information on the VM:
# lsb_release -rd
Description: Ubuntu 14.04.2 LTS
Release: 14.04
# apt-cache policy linux-image-3.13.0-45-generic
linux-image-3.13.0-45-generic:
Installed: 3.13.0-45.74
Candidate: 3.13.0-45.74
Version table:
*** 3.13.0-45.74 0
500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
100 /var/lib/dpkg/status
---
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Feb 19 14:34 seq
crw-rw---- 1 root audio 116, 33 Feb 19 14:34 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.7
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: [Errno 2] No such file or directory
CRDA: Error: [Errno 2] No such file or directory
CurrentDmesg:
[ 13.891047] init: console-font main process (855) terminated with status 71
[ 13.952825] init: plymouth-splash main process (870) terminated with status 1
[ 217.853139] random: nonblocking pool is initialized
DistroRelease: Ubuntu 14.04
IwConfig: Error: [Errno 2] No such file or directory
Lspci: Error: [Errno 2] No such file or directory
Lsusb: Error: [Errno 2] No such file or directory
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: root=UUID=cb9cbdad-c668-4503-85db-fcf9b02f3495 ro console=tty0 console=ttyS0,38400
ProcVersionSignature: Ubuntu 3.13.0-45.74-generic 3.13.11-ckt13
RelatedPackageVersions:
linux-restricted-modules-3.13.0-45-generic N/A
linux-backports-modules-3.13.0-45-generic N/A
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty
Uname: Linux 3.13.0-45-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
_MarkForUpload: True
dmi.bios.date: 01/01/2011
dmi.bios.vendor: Bochs
dmi.bios.version: Bochs
dmi.chassis.type: 1
dmi.chassis.vendor: Bochs
dmi.modalias: dmi:bvnBochs:bvrBochs:bd01/01/2011:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-2.0:cvnBochs:ct1:cvr:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.version: pc-i440fx-2.0
dmi.sys.vendor: QEMU |
|
| 2015-02-19 19:49:22 |
Simon Déziel |
attachment added |
|
BootDmesg.txt https://bugs.launchpad.net/bugs/1423672/+attachment/4322702/+files/BootDmesg.txt |
|
| 2015-02-19 19:49:23 |
Simon Déziel |
attachment added |
|
ProcCpuinfo.txt https://bugs.launchpad.net/bugs/1423672/+attachment/4322703/+files/ProcCpuinfo.txt |
|
| 2015-02-19 19:49:24 |
Simon Déziel |
attachment added |
|
ProcInterrupts.txt https://bugs.launchpad.net/bugs/1423672/+attachment/4322704/+files/ProcInterrupts.txt |
|
| 2015-02-19 19:49:26 |
Simon Déziel |
attachment added |
|
ProcModules.txt https://bugs.launchpad.net/bugs/1423672/+attachment/4322705/+files/ProcModules.txt |
|
| 2015-02-19 19:49:27 |
Simon Déziel |
attachment added |
|
UdevDb.txt https://bugs.launchpad.net/bugs/1423672/+attachment/4322706/+files/UdevDb.txt |
|
| 2015-02-19 19:49:29 |
Simon Déziel |
attachment added |
|
UdevLog.txt https://bugs.launchpad.net/bugs/1423672/+attachment/4322707/+files/UdevLog.txt |
|
| 2015-02-19 19:49:31 |
Simon Déziel |
attachment added |
|
WifiSyslog.txt https://bugs.launchpad.net/bugs/1423672/+attachment/4322708/+files/WifiSyslog.txt |
|
| 2015-02-19 21:00:02 |
Joseph Salisbury |
linux (Ubuntu): importance |
Undecided |
Medium |
|
| 2015-02-19 21:00:14 |
Joseph Salisbury |
tags |
apport-collected trusty |
apport-collected kernel-da-key trusty |
|
| 2015-02-19 21:24:52 |
Simon Déziel |
linux (Ubuntu): status |
Incomplete |
Confirmed |
|
| 2015-03-04 20:18:14 |
Simon Déziel |
summary |
ext4 turned read-only following a cronjob run |
ext4 turned read-only during logrotate daily run |
|
| 2015-03-07 10:17:14 |
Oliver Brakmann |
bug |
|
|
added subscriber Oliver Brakmann |
| 2015-03-09 19:28:09 |
Joseph Salisbury |
linux (Ubuntu): importance |
Medium |
High |
|
| 2015-03-09 19:28:16 |
Joseph Salisbury |
nominated for series |
|
Ubuntu Trusty |
|
| 2015-03-09 19:28:16 |
Joseph Salisbury |
bug task added |
|
linux (Ubuntu Trusty) |
|
| 2015-03-09 19:28:22 |
Joseph Salisbury |
linux (Ubuntu Trusty): importance |
Undecided |
High |
|
| 2015-03-09 19:28:25 |
Joseph Salisbury |
linux (Ubuntu Trusty): status |
New |
Confirmed |
|
| 2015-03-10 00:19:08 |
Nobuto Murata |
bug |
|
|
added subscriber Nobuto Murata |
| 2015-05-19 07:41:40 |
ben |
bug |
|
|
added subscriber ben |
| 2015-06-03 17:45:24 |
Joseph Salisbury |
linux (Ubuntu Trusty): status |
Confirmed |
Incomplete |
|
| 2015-06-03 17:45:27 |
Joseph Salisbury |
linux (Ubuntu): status |
Confirmed |
Incomplete |
|
| 2015-06-15 16:54:10 |
Simon Déziel |
linux (Ubuntu Trusty): status |
Incomplete |
Confirmed |
|
| 2015-06-15 16:55:21 |
Simon Déziel |
summary |
ext4 turned read-only during logrotate daily run |
ext4_mb_generate_buddy:756: group N, block bitmap and bg descriptor inconsistent: X vs Y |
|
| 2015-09-07 06:31:30 |
Peter |
bug |
|
|
added subscriber Peter |
| 2015-09-14 18:10:02 |
LouieGosselin |
bug |
|
|
added subscriber LouieGosselin |
| 2015-09-15 06:37:32 |
Matteo Sgalaberni |
bug |
|
|
added subscriber Matteo Sgalaberni |
| 2015-09-17 13:59:07 |
Emmanuel Lacour |
attachment added |
|
kern.log with backtrace https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1423672/+attachment/4466843/+files/kern.log |
|
| 2015-09-24 10:13:39 |
Emmanuel Lacour |
bug task added |
|
linux (Debian) |
|
| 2015-10-13 15:29:48 |
Jan Wagner |
bug |
|
|
added subscriber Jan Wagner |
| 2015-10-13 15:50:29 |
Jan Wagner |
bug watch added |
|
http://bugzilla.kernel.org/show_bug.cgi?id=104571 |
|
| 2015-10-13 15:50:29 |
Jan Wagner |
attachment added |
|
Screenshot_service_2015-10-13_17:25:18.png https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1423672/+attachment/4493620/+files/Screenshot_service_2015-10-13_17%3A25%3A18.png |
|
| 2015-10-29 17:26:21 |
Chris J Arges |
bug |
|
|
added subscriber Chris J Arges |
| 2015-11-04 18:35:41 |
BlueT - Matthew Lien - 練喆明 |
bug |
|
|
added subscriber BlueT - Matthew Lien - 練喆明 |
| 2015-11-12 07:27:58 |
Jan Wagner |
bug watch added |
|
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=772848 |
|
| 2015-11-12 07:27:58 |
Jan Wagner |
linux (Debian): importance |
Undecided |
Unknown |
|
| 2015-11-12 07:27:58 |
Jan Wagner |
linux (Debian): status |
New |
Unknown |
|
| 2015-11-12 07:27:58 |
Jan Wagner |
linux (Debian): remote watch |
|
Debian Bug tracker #772848 |
|
| 2015-11-12 07:28:40 |
Jan Wagner |
bug task added |
|
linux-kernel-headers |
|
| 2015-11-12 07:29:21 |
Jan Wagner |
affects |
linux-kernel-headers |
linux |
|
| 2015-11-12 07:48:17 |
Jan Wagner |
bug watch added |
|
http://bugzilla.kernel.org/show_bug.cgi?id=102731 |
|
| 2015-11-12 07:48:17 |
Jan Wagner |
bug watch added |
|
http://bugzilla.kernel.org/show_bug.cgi?id=89621 |
|
| 2015-11-12 08:47:02 |
Bug Watch Updater |
linux (Debian): status |
Unknown |
Confirmed |
|
| 2015-11-13 21:05:28 |
LouieGosselin |
attachment added |
|
/boot/config-3.16.0-4-amd64 for the VM in question https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1423672/+attachment/4518472/+files/config.txt |
|
| 2016-02-03 23:10:51 |
Jeff Strunk |
bug |
|
|
added subscriber Jeff Strunk |
| 2016-03-21 06:55:34 |
Václav Ovsík |
bug watch added |
|
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=818502 |
|
| 2016-03-21 14:13:38 |
Simon Déziel |
linux (Ubuntu): status |
Incomplete |
Confirmed |
|
| 2016-03-24 14:36:46 |
Dominique Poulain |
bug |
|
|
added subscriber Dominique Poulain |
| 2016-04-12 14:10:04 |
Leonardo Borda |
bug |
|
|
added subscriber Leonardo Borda |
| 2016-04-20 14:36:36 |
Chris J Arges |
bug task added |
|
linux-lts-utopic (Ubuntu) |
|
| 2016-04-20 14:36:46 |
Chris J Arges |
linux-lts-utopic (Ubuntu): status |
New |
Invalid |
|
| 2016-04-20 18:00:56 |
Chris J Arges |
linux (Ubuntu Trusty): assignee |
|
Chris J Arges (arges) |
|
| 2016-04-20 18:00:58 |
Chris J Arges |
linux-lts-utopic (Ubuntu Trusty): assignee |
|
Chris J Arges (arges) |
|
| 2016-04-21 18:58:42 |
Chris J Arges |
description |
I noticed that one of my VM had this "dmesg -T" output:
[Tue Feb 17 09:53:27 2015] systemd-udevd[5433]: starting version 204
[Thu Feb 19 06:25:08 2015] EXT4-fs error (device vda1): ext4_mb_generate_buddy:756: group 5, block bitmap and bg descriptor inconsistent: 16446 vs 16445 free clusters
[Thu Feb 19 06:25:09 2015] Aborting journal on device vda1-8.
[Thu Feb 19 06:25:09 2015] EXT4-fs (vda1): Remounting filesystem read-only
[Thu Feb 19 06:25:09 2015] ------------[ cut here ]------------
[Thu Feb 19 06:25:09 2015] WARNING: CPU: 0 PID: 9946 at /build/buildd/linux-3.13.0/fs/ext4/ext4_jbd2.c:259 __ext4_handle_dirty_metadata+0x1a2/0x1c0()
[Thu Feb 19 06:25:09 2015] Modules linked in: nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner xt_conntrack nf_conntrack iptable_filter ip_tables x_tables serio_raw psmouse floppy
[Thu Feb 19 06:25:09 2015] CPU: 0 PID: 9946 Comm: logrotate Not tainted 3.13.0-45-generic #74-Ubuntu
[Thu Feb 19 06:25:09 2015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[Thu Feb 19 06:25:09 2015] 0000000000000009 ffff880003a11aa0 ffffffff81720eb6 0000000000000000
[Thu Feb 19 06:25:09 2015] ffff880003a11ad8 ffffffff810677cd ffff880000c41340 0000000000000000
[Thu Feb 19 06:25:09 2015] ffff88000a58e000 ffffffff81835280 0000000000001302 ffff880003a11ae8
[Thu Feb 19 06:25:09 2015] Call Trace:
[Thu Feb 19 06:25:09 2015] [<ffffffff81720eb6>] dump_stack+0x45/0x56
[Thu Feb 19 06:25:09 2015] [<ffffffff810677cd>] warn_slowpath_common+0x7d/0xa0
[Thu Feb 19 06:25:09 2015] [<ffffffff810678aa>] warn_slowpath_null+0x1a/0x20
[Thu Feb 19 06:25:09 2015] [<ffffffff8126e862>] __ext4_handle_dirty_metadata+0x1a2/0x1c0
[Thu Feb 19 06:25:09 2015] [<ffffffff81246a5a>] ? ext4_dirty_inode+0x2a/0x60
[Thu Feb 19 06:25:09 2015] [<ffffffff81277086>] ext4_free_blocks+0x646/0xbf0
[Thu Feb 19 06:25:09 2015] [<ffffffff810aacc5>] ? wake_up_bit+0x25/0x30
[Thu Feb 19 06:25:09 2015] [<ffffffff812685b5>] ext4_ext_rm_leaf+0x505/0x8f0
[Thu Feb 19 06:25:09 2015] [<ffffffff81267527>] ? __ext4_ext_check+0x197/0x370
[Thu Feb 19 06:25:09 2015] [<ffffffff8126ad00>] ? ext4_ext_remove_space+0xc0/0x7e0
[Thu Feb 19 06:25:09 2015] [<ffffffff8126af5c>] ext4_ext_remove_space+0x31c/0x7e0
[Thu Feb 19 06:25:09 2015] [<ffffffff8126d300>] ext4_ext_truncate+0xb0/0xe0
[Thu Feb 19 06:25:09 2015] [<ffffffff81244eb9>] ext4_truncate+0x379/0x3c0
[Thu Feb 19 06:25:09 2015] [<ffffffff81245a18>] ext4_evict_inode+0x408/0x4d0
[Thu Feb 19 06:25:09 2015] [<ffffffff811d8f60>] evict+0xb0/0x1b0
[Thu Feb 19 06:25:09 2015] [<ffffffff811d9775>] iput+0xf5/0x180
[Thu Feb 19 06:25:09 2015] [<ffffffff811d4698>] __dentry_kill+0x1a8/0x200
[Thu Feb 19 06:25:09 2015] [<ffffffff811d4795>] dput+0xa5/0x180
[Thu Feb 19 06:25:09 2015] [<ffffffff811bf7e6>] __fput+0x176/0x260
[Thu Feb 19 06:25:09 2015] [<ffffffff811bf91e>] ____fput+0xe/0x10
[Thu Feb 19 06:25:09 2015] [<ffffffff810882f7>] task_work_run+0xa7/0xe0
[Thu Feb 19 06:25:09 2015] [<ffffffff81013ed7>] do_notify_resume+0x97/0xb0
[Thu Feb 19 06:25:09 2015] [<ffffffff81731c2a>] int_signal+0x12/0x17
[Thu Feb 19 06:25:09 2015] ---[ end trace ebff9843d81b5c41 ]---
[Thu Feb 19 06:25:09 2015] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
[Thu Feb 19 06:25:09 2015] IP: [<ffffffff8125d4c1>] __ext4_error_inode+0x31/0x160
[Thu Feb 19 06:25:09 2015] PGD 167067 PUD 161067 PMD 0
[Thu Feb 19 06:25:09 2015] Oops: 0000 [#1] SMP
[Thu Feb 19 06:25:09 2015] Modules linked in: nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner xt_conntrack nf_conntrack iptable_filter ip_tables x_tables serio_raw psmouse floppy
[Thu Feb 19 06:25:09 2015] CPU: 0 PID: 9946 Comm: logrotate Tainted: G W 3.13.0-45-generic #74-Ubuntu
[Thu Feb 19 06:25:09 2015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[Thu Feb 19 06:25:09 2015] task: ffff880009ac4800 ti: ffff880003a10000 task.ti: ffff880003a10000
[Thu Feb 19 06:25:09 2015] RIP: 0010:[<ffffffff8125d4c1>] [<ffffffff8125d4c1>] __ext4_error_inode+0x31/0x160
[Thu Feb 19 06:25:09 2015] RSP: 0000:ffff880003a11a58 EFLAGS: 00010292
[Thu Feb 19 06:25:09 2015] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000086
[Thu Feb 19 06:25:09 2015] RDX: 0000000000001302 RSI: ffffffff81a6e81f RDI: 0000000000000000
[Thu Feb 19 06:25:09 2015] RBP: ffff880003a11ae8 R08: ffffffff81a78568 R09: 0000000000000005
[Thu Feb 19 06:25:09 2015] R10: 00000000ffffffe2 R11: ffff880003a117ce R12: 0000000000000086
[Thu Feb 19 06:25:09 2015] R13: ffffffff81835280 R14: 0000000000001302 R15: ffffffff81a78568
[Thu Feb 19 06:25:09 2015] FS: 00007f74eaca4840(0000) GS:ffff88000b800000(0000) knlGS:0000000000000000
[Thu Feb 19 06:25:09 2015] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[Thu Feb 19 06:25:09 2015] CR2: 0000000001de4000 CR3: 0000000009b45000 CR4: 00000000000006f0
[Thu Feb 19 06:25:09 2015] Stack:
[Thu Feb 19 06:25:09 2015] ffff880003a11a60 0000000000000103 ebff9843d81b5c41 000000000000321e
[Thu Feb 19 06:25:09 2015] 00000000000014d8 0000000000000092 000000000000020e ffff88000a58e000
[Thu Feb 19 06:25:09 2015] ffff880003a11ae8 ffffffff8126e372 ffffffff810677df ffff880000c41340
[Thu Feb 19 06:25:09 2015] Call Trace:
[Thu Feb 19 06:25:09 2015] [<ffffffff8126e372>] ? ext4_journal_abort_handle+0x42/0xc0
[Thu Feb 19 06:25:09 2015] [<ffffffff810677df>] ? warn_slowpath_common+0x8f/0xa0
[Thu Feb 19 06:25:09 2015] [<ffffffff8126e7cf>] __ext4_handle_dirty_metadata+0x10f/0x1c0
[Thu Feb 19 06:25:09 2015] [<ffffffff81277086>] ext4_free_blocks+0x646/0xbf0
[Thu Feb 19 06:25:09 2015] [<ffffffff810aacc5>] ? wake_up_bit+0x25/0x30
[Thu Feb 19 06:25:09 2015] [<ffffffff812685b5>] ext4_ext_rm_leaf+0x505/0x8f0
[Thu Feb 19 06:25:09 2015] [<ffffffff81267527>] ? __ext4_ext_check+0x197/0x370
[Thu Feb 19 06:25:09 2015] [<ffffffff8126ad00>] ? ext4_ext_remove_space+0xc0/0x7e0
[Thu Feb 19 06:25:09 2015] [<ffffffff8126af5c>] ext4_ext_remove_space+0x31c/0x7e0
[Thu Feb 19 06:25:09 2015] [<ffffffff8126d300>] ext4_ext_truncate+0xb0/0xe0
[Thu Feb 19 06:25:09 2015] [<ffffffff81244eb9>] ext4_truncate+0x379/0x3c0
[Thu Feb 19 06:25:09 2015] [<ffffffff81245a18>] ext4_evict_inode+0x408/0x4d0
[Thu Feb 19 06:25:09 2015] [<ffffffff811d8f60>] evict+0xb0/0x1b0
[Thu Feb 19 06:25:09 2015] [<ffffffff811d9775>] iput+0xf5/0x180
[Thu Feb 19 06:25:09 2015] [<ffffffff811d4698>] __dentry_kill+0x1a8/0x200
[Thu Feb 19 06:25:09 2015] [<ffffffff811d4795>] dput+0xa5/0x180
[Thu Feb 19 06:25:09 2015] [<ffffffff811bf7e6>] __fput+0x176/0x260
[Thu Feb 19 06:25:09 2015] [<ffffffff811bf91e>] ____fput+0xe/0x10
[Thu Feb 19 06:25:09 2015] [<ffffffff810882f7>] task_work_run+0xa7/0xe0
[Thu Feb 19 06:25:09 2015] [<ffffffff81013ed7>] do_notify_resume+0x97/0xb0
[Thu Feb 19 06:25:09 2015] [<ffffffff81731c2a>] int_signal+0x12/0x17
[Thu Feb 19 06:25:09 2015] Code: 48 89 e5 41 57 4d 89 c7 41 56 41 89 d6 41 55 49 89 f5 48 c7 c6 1f e8 a6 81 41 54 49 89 cc 53 48 89 fb 48 83 ec 68 4c 89 4c 24 60 <48> 8b 47 28 48 8b 57 40 48 8b 80 f8 02 00 00 48 8b 40 68 89 90
[Thu Feb 19 06:25:09 2015] RIP [<ffffffff8125d4c1>] __ext4_error_inode+0x31/0x160
[Thu Feb 19 06:25:09 2015] RSP <ffff880003a11a58>
[Thu Feb 19 06:25:09 2015] CR2: 0000000000000028
[Thu Feb 19 06:25:10 2015] ---[ end trace ebff9843d81b5c42 ]---
cron.daily jobs fired at 6:25:01 apparently:
# tail -n2 /var/log/syslog
Feb 19 06:17:01 git CRON[9848]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Feb 19 06:25:01 git CRON[9853]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
# run-parts --test /etc/cron.daily
/etc/cron.daily/apt
/etc/cron.daily/autoremove
/etc/cron.daily/dpkg
/etc/cron.daily/hdd-backup
/etc/cron.daily/logrotate
/etc/cron.daily/passwd
/etc/cron.daily/upstart
It seems like all the jobs ran and the upstart one somehow triggered the crash:
# ls -alt /var/log/upstart/ | head
total 272
drwxrwxr-x 5 root syslog 4096 Feb 19 06:25 ..
drwxr-xr-x 2 root root 4096 Feb 14 06:25 .
-rw-r----- 1 root root 591 Feb 12 12:26 ureadahead.log.1.gz
-rw-r----- 1 root root 178 Feb 12 12:25 mountall.log.1.gz
Now that I have collected some information (sorry, I don't have ubuntu-bug installed on the VM) I'll reboot it and see how it goes.
More information on the VM:
# lsb_release -rd
Description: Ubuntu 14.04.2 LTS
Release: 14.04
# apt-cache policy linux-image-3.13.0-45-generic
linux-image-3.13.0-45-generic:
Installed: 3.13.0-45.74
Candidate: 3.13.0-45.74
Version table:
*** 3.13.0-45.74 0
500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
100 /var/lib/dpkg/status
---
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Feb 19 14:34 seq
crw-rw---- 1 root audio 116, 33 Feb 19 14:34 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.7
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: [Errno 2] No such file or directory
CRDA: Error: [Errno 2] No such file or directory
CurrentDmesg:
[ 13.891047] init: console-font main process (855) terminated with status 71
[ 13.952825] init: plymouth-splash main process (870) terminated with status 1
[ 217.853139] random: nonblocking pool is initialized
DistroRelease: Ubuntu 14.04
IwConfig: Error: [Errno 2] No such file or directory
Lspci: Error: [Errno 2] No such file or directory
Lsusb: Error: [Errno 2] No such file or directory
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: root=UUID=cb9cbdad-c668-4503-85db-fcf9b02f3495 ro console=tty0 console=ttyS0,38400
ProcVersionSignature: Ubuntu 3.13.0-45.74-generic 3.13.11-ckt13
RelatedPackageVersions:
linux-restricted-modules-3.13.0-45-generic N/A
linux-backports-modules-3.13.0-45-generic N/A
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty
Uname: Linux 3.13.0-45-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
_MarkForUpload: True
dmi.bios.date: 01/01/2011
dmi.bios.vendor: Bochs
dmi.bios.version: Bochs
dmi.chassis.type: 1
dmi.chassis.vendor: Bochs
dmi.modalias: dmi:bvnBochs:bvrBochs:bd01/01/2011:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-2.0:cvnBochs:ct1:cvr:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.version: pc-i440fx-2.0
dmi.sys.vendor: QEMU |
SRU Justification:
Impact: Users of VMs running 3.13/3.16 and ext4 can experience data corruption in the guest.
Fix: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?h=7dec5603b6b8dc4c3e1c65d318bd2a5a8c62a424
Testcase: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=818502#22
--
I noticed that one of my VM had this "dmesg -T" output:
[Tue Feb 17 09:53:27 2015] systemd-udevd[5433]: starting version 204
[Thu Feb 19 06:25:08 2015] EXT4-fs error (device vda1): ext4_mb_generate_buddy:756: group 5, block bitmap and bg descriptor inconsistent: 16446 vs 16445 free clusters
[Thu Feb 19 06:25:09 2015] Aborting journal on device vda1-8.
[Thu Feb 19 06:25:09 2015] EXT4-fs (vda1): Remounting filesystem read-only
[Thu Feb 19 06:25:09 2015] ------------[ cut here ]------------
[Thu Feb 19 06:25:09 2015] WARNING: CPU: 0 PID: 9946 at /build/buildd/linux-3.13.0/fs/ext4/ext4_jbd2.c:259 __ext4_handle_dirty_metadata+0x1a2/0x1c0()
[Thu Feb 19 06:25:09 2015] Modules linked in: nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner xt_conntrack nf_conntrack iptable_filter ip_tables x_tables serio_raw psmouse floppy
[Thu Feb 19 06:25:09 2015] CPU: 0 PID: 9946 Comm: logrotate Not tainted 3.13.0-45-generic #74-Ubuntu
[Thu Feb 19 06:25:09 2015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[Thu Feb 19 06:25:09 2015] 0000000000000009 ffff880003a11aa0 ffffffff81720eb6 0000000000000000
[Thu Feb 19 06:25:09 2015] ffff880003a11ad8 ffffffff810677cd ffff880000c41340 0000000000000000
[Thu Feb 19 06:25:09 2015] ffff88000a58e000 ffffffff81835280 0000000000001302 ffff880003a11ae8
[Thu Feb 19 06:25:09 2015] Call Trace:
[Thu Feb 19 06:25:09 2015] [<ffffffff81720eb6>] dump_stack+0x45/0x56
[Thu Feb 19 06:25:09 2015] [<ffffffff810677cd>] warn_slowpath_common+0x7d/0xa0
[Thu Feb 19 06:25:09 2015] [<ffffffff810678aa>] warn_slowpath_null+0x1a/0x20
[Thu Feb 19 06:25:09 2015] [<ffffffff8126e862>] __ext4_handle_dirty_metadata+0x1a2/0x1c0
[Thu Feb 19 06:25:09 2015] [<ffffffff81246a5a>] ? ext4_dirty_inode+0x2a/0x60
[Thu Feb 19 06:25:09 2015] [<ffffffff81277086>] ext4_free_blocks+0x646/0xbf0
[Thu Feb 19 06:25:09 2015] [<ffffffff810aacc5>] ? wake_up_bit+0x25/0x30
[Thu Feb 19 06:25:09 2015] [<ffffffff812685b5>] ext4_ext_rm_leaf+0x505/0x8f0
[Thu Feb 19 06:25:09 2015] [<ffffffff81267527>] ? __ext4_ext_check+0x197/0x370
[Thu Feb 19 06:25:09 2015] [<ffffffff8126ad00>] ? ext4_ext_remove_space+0xc0/0x7e0
[Thu Feb 19 06:25:09 2015] [<ffffffff8126af5c>] ext4_ext_remove_space+0x31c/0x7e0
[Thu Feb 19 06:25:09 2015] [<ffffffff8126d300>] ext4_ext_truncate+0xb0/0xe0
[Thu Feb 19 06:25:09 2015] [<ffffffff81244eb9>] ext4_truncate+0x379/0x3c0
[Thu Feb 19 06:25:09 2015] [<ffffffff81245a18>] ext4_evict_inode+0x408/0x4d0
[Thu Feb 19 06:25:09 2015] [<ffffffff811d8f60>] evict+0xb0/0x1b0
[Thu Feb 19 06:25:09 2015] [<ffffffff811d9775>] iput+0xf5/0x180
[Thu Feb 19 06:25:09 2015] [<ffffffff811d4698>] __dentry_kill+0x1a8/0x200
[Thu Feb 19 06:25:09 2015] [<ffffffff811d4795>] dput+0xa5/0x180
[Thu Feb 19 06:25:09 2015] [<ffffffff811bf7e6>] __fput+0x176/0x260
[Thu Feb 19 06:25:09 2015] [<ffffffff811bf91e>] ____fput+0xe/0x10
[Thu Feb 19 06:25:09 2015] [<ffffffff810882f7>] task_work_run+0xa7/0xe0
[Thu Feb 19 06:25:09 2015] [<ffffffff81013ed7>] do_notify_resume+0x97/0xb0
[Thu Feb 19 06:25:09 2015] [<ffffffff81731c2a>] int_signal+0x12/0x17
[Thu Feb 19 06:25:09 2015] ---[ end trace ebff9843d81b5c41 ]---
[Thu Feb 19 06:25:09 2015] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
[Thu Feb 19 06:25:09 2015] IP: [<ffffffff8125d4c1>] __ext4_error_inode+0x31/0x160
[Thu Feb 19 06:25:09 2015] PGD 167067 PUD 161067 PMD 0
[Thu Feb 19 06:25:09 2015] Oops: 0000 [#1] SMP
[Thu Feb 19 06:25:09 2015] Modules linked in: nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner xt_conntrack nf_conntrack iptable_filter ip_tables x_tables serio_raw psmouse floppy
[Thu Feb 19 06:25:09 2015] CPU: 0 PID: 9946 Comm: logrotate Tainted: G W 3.13.0-45-generic #74-Ubuntu
[Thu Feb 19 06:25:09 2015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[Thu Feb 19 06:25:09 2015] task: ffff880009ac4800 ti: ffff880003a10000 task.ti: ffff880003a10000
[Thu Feb 19 06:25:09 2015] RIP: 0010:[<ffffffff8125d4c1>] [<ffffffff8125d4c1>] __ext4_error_inode+0x31/0x160
[Thu Feb 19 06:25:09 2015] RSP: 0000:ffff880003a11a58 EFLAGS: 00010292
[Thu Feb 19 06:25:09 2015] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000086
[Thu Feb 19 06:25:09 2015] RDX: 0000000000001302 RSI: ffffffff81a6e81f RDI: 0000000000000000
[Thu Feb 19 06:25:09 2015] RBP: ffff880003a11ae8 R08: ffffffff81a78568 R09: 0000000000000005
[Thu Feb 19 06:25:09 2015] R10: 00000000ffffffe2 R11: ffff880003a117ce R12: 0000000000000086
[Thu Feb 19 06:25:09 2015] R13: ffffffff81835280 R14: 0000000000001302 R15: ffffffff81a78568
[Thu Feb 19 06:25:09 2015] FS: 00007f74eaca4840(0000) GS:ffff88000b800000(0000) knlGS:0000000000000000
[Thu Feb 19 06:25:09 2015] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[Thu Feb 19 06:25:09 2015] CR2: 0000000001de4000 CR3: 0000000009b45000 CR4: 00000000000006f0
[Thu Feb 19 06:25:09 2015] Stack:
[Thu Feb 19 06:25:09 2015] ffff880003a11a60 0000000000000103 ebff9843d81b5c41 000000000000321e
[Thu Feb 19 06:25:09 2015] 00000000000014d8 0000000000000092 000000000000020e ffff88000a58e000
[Thu Feb 19 06:25:09 2015] ffff880003a11ae8 ffffffff8126e372 ffffffff810677df ffff880000c41340
[Thu Feb 19 06:25:09 2015] Call Trace:
[Thu Feb 19 06:25:09 2015] [<ffffffff8126e372>] ? ext4_journal_abort_handle+0x42/0xc0
[Thu Feb 19 06:25:09 2015] [<ffffffff810677df>] ? warn_slowpath_common+0x8f/0xa0
[Thu Feb 19 06:25:09 2015] [<ffffffff8126e7cf>] __ext4_handle_dirty_metadata+0x10f/0x1c0
[Thu Feb 19 06:25:09 2015] [<ffffffff81277086>] ext4_free_blocks+0x646/0xbf0
[Thu Feb 19 06:25:09 2015] [<ffffffff810aacc5>] ? wake_up_bit+0x25/0x30
[Thu Feb 19 06:25:09 2015] [<ffffffff812685b5>] ext4_ext_rm_leaf+0x505/0x8f0
[Thu Feb 19 06:25:09 2015] [<ffffffff81267527>] ? __ext4_ext_check+0x197/0x370
[Thu Feb 19 06:25:09 2015] [<ffffffff8126ad00>] ? ext4_ext_remove_space+0xc0/0x7e0
[Thu Feb 19 06:25:09 2015] [<ffffffff8126af5c>] ext4_ext_remove_space+0x31c/0x7e0
[Thu Feb 19 06:25:09 2015] [<ffffffff8126d300>] ext4_ext_truncate+0xb0/0xe0
[Thu Feb 19 06:25:09 2015] [<ffffffff81244eb9>] ext4_truncate+0x379/0x3c0
[Thu Feb 19 06:25:09 2015] [<ffffffff81245a18>] ext4_evict_inode+0x408/0x4d0
[Thu Feb 19 06:25:09 2015] [<ffffffff811d8f60>] evict+0xb0/0x1b0
[Thu Feb 19 06:25:09 2015] [<ffffffff811d9775>] iput+0xf5/0x180
[Thu Feb 19 06:25:09 2015] [<ffffffff811d4698>] __dentry_kill+0x1a8/0x200
[Thu Feb 19 06:25:09 2015] [<ffffffff811d4795>] dput+0xa5/0x180
[Thu Feb 19 06:25:09 2015] [<ffffffff811bf7e6>] __fput+0x176/0x260
[Thu Feb 19 06:25:09 2015] [<ffffffff811bf91e>] ____fput+0xe/0x10
[Thu Feb 19 06:25:09 2015] [<ffffffff810882f7>] task_work_run+0xa7/0xe0
[Thu Feb 19 06:25:09 2015] [<ffffffff81013ed7>] do_notify_resume+0x97/0xb0
[Thu Feb 19 06:25:09 2015] [<ffffffff81731c2a>] int_signal+0x12/0x17
[Thu Feb 19 06:25:09 2015] Code: 48 89 e5 41 57 4d 89 c7 41 56 41 89 d6 41 55 49 89 f5 48 c7 c6 1f e8 a6 81 41 54 49 89 cc 53 48 89 fb 48 83 ec 68 4c 89 4c 24 60 <48> 8b 47 28 48 8b 57 40 48 8b 80 f8 02 00 00 48 8b 40 68 89 90
[Thu Feb 19 06:25:09 2015] RIP [<ffffffff8125d4c1>] __ext4_error_inode+0x31/0x160
[Thu Feb 19 06:25:09 2015] RSP <ffff880003a11a58>
[Thu Feb 19 06:25:09 2015] CR2: 0000000000000028
[Thu Feb 19 06:25:10 2015] ---[ end trace ebff9843d81b5c42 ]---
cron.daily jobs fired at 6:25:01 apparently:
# tail -n2 /var/log/syslog
Feb 19 06:17:01 git CRON[9848]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Feb 19 06:25:01 git CRON[9853]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
# run-parts --test /etc/cron.daily
/etc/cron.daily/apt
/etc/cron.daily/autoremove
/etc/cron.daily/dpkg
/etc/cron.daily/hdd-backup
/etc/cron.daily/logrotate
/etc/cron.daily/passwd
/etc/cron.daily/upstart
It seems like all the jobs ran and the upstart one somehow triggered the crash:
# ls -alt /var/log/upstart/ | head
total 272
drwxrwxr-x 5 root syslog 4096 Feb 19 06:25 ..
drwxr-xr-x 2 root root 4096 Feb 14 06:25 .
-rw-r----- 1 root root 591 Feb 12 12:26 ureadahead.log.1.gz
-rw-r----- 1 root root 178 Feb 12 12:25 mountall.log.1.gz
Now that I have collected some information (sorry, I don't have ubuntu-bug installed on the VM) I'll reboot it and see how it goes.
More information on the VM:
# lsb_release -rd
Description: Ubuntu 14.04.2 LTS
Release: 14.04
# apt-cache policy linux-image-3.13.0-45-generic
linux-image-3.13.0-45-generic:
Installed: 3.13.0-45.74
Candidate: 3.13.0-45.74
Version table:
*** 3.13.0-45.74 0
500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
100 /var/lib/dpkg/status
---
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Feb 19 14:34 seq
crw-rw---- 1 root audio 116, 33 Feb 19 14:34 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.7
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: [Errno 2] No such file or directory
CRDA: Error: [Errno 2] No such file or directory
CurrentDmesg:
[ 13.891047] init: console-font main process (855) terminated with status 71
[ 13.952825] init: plymouth-splash main process (870) terminated with status 1
[ 217.853139] random: nonblocking pool is initialized
DistroRelease: Ubuntu 14.04
IwConfig: Error: [Errno 2] No such file or directory
Lspci: Error: [Errno 2] No such file or directory
Lsusb: Error: [Errno 2] No such file or directory
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: root=UUID=cb9cbdad-c668-4503-85db-fcf9b02f3495 ro console=tty0 console=ttyS0,38400
ProcVersionSignature: Ubuntu 3.13.0-45.74-generic 3.13.11-ckt13
RelatedPackageVersions:
linux-restricted-modules-3.13.0-45-generic N/A
linux-backports-modules-3.13.0-45-generic N/A
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty
Uname: Linux 3.13.0-45-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
_MarkForUpload: True
dmi.bios.date: 01/01/2011
dmi.bios.vendor: Bochs
dmi.bios.version: Bochs
dmi.chassis.type: 1
dmi.chassis.vendor: Bochs
dmi.modalias: dmi:bvnBochs:bvrBochs:bd01/01/2011:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-2.0:cvnBochs:ct1:cvr:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.version: pc-i440fx-2.0
dmi.sys.vendor: QEMU |
|
| 2016-04-21 20:15:27 |
Tim Gardner |
linux (Ubuntu Trusty): status |
Confirmed |
Fix Committed |
|
| 2016-04-21 20:15:34 |
Tim Gardner |
linux-lts-utopic (Ubuntu Trusty): status |
New |
Fix Committed |
|
| 2016-05-19 14:53:51 |
Kamal Mostafa |
tags |
apport-collected kernel-da-key trusty |
apport-collected kernel-da-key trusty verification-needed-trusty |
|
| 2016-05-25 12:14:43 |
Per Cederqvist |
bug |
|
|
added subscriber Per Cederqvist |
| 2016-05-31 14:39:53 |
Launchpad Janitor |
linux (Ubuntu Trusty): status |
Fix Committed |
Fix Released |
|
| 2016-05-31 14:39:53 |
Launchpad Janitor |
cve linked |
|
2015-4004 |
|
| 2016-05-31 14:39:53 |
Launchpad Janitor |
cve linked |
|
2016-2069 |
|
| 2016-05-31 14:39:53 |
Launchpad Janitor |
cve linked |
|
2016-2117 |
|
| 2016-05-31 14:39:53 |
Launchpad Janitor |
cve linked |
|
2016-2187 |
|
| 2016-05-31 14:39:53 |
Launchpad Janitor |
cve linked |
|
2016-3672 |
|
| 2016-05-31 14:39:53 |
Launchpad Janitor |
cve linked |
|
2016-3951 |
|
| 2016-05-31 14:39:53 |
Launchpad Janitor |
cve linked |
|
2016-3955 |
|
| 2016-05-31 14:39:53 |
Launchpad Janitor |
cve linked |
|
2016-4485 |
|
| 2016-05-31 14:39:53 |
Launchpad Janitor |
cve linked |
|
2016-4486 |
|
| 2016-05-31 14:39:54 |
Launchpad Janitor |
linux (Ubuntu Trusty): status |
Fix Committed |
Fix Released |
|
| 2016-06-09 21:51:51 |
Launchpad Janitor |
linux-lts-utopic (Ubuntu Trusty): status |
Fix Committed |
Fix Released |
|
| 2016-06-09 21:51:51 |
Launchpad Janitor |
cve linked |
|
2016-1583 |
|
| 2016-07-08 00:18:58 |
Bug Watch Updater |
linux (Debian): status |
Confirmed |
Fix Released |
|
| 2016-09-30 16:28:09 |
Joseph Salisbury |
tags |
apport-collected kernel-da-key trusty verification-needed-trusty |
apport-collected kernel-da-key trusty verification-done-trusty |
|
| 2018-11-14 13:12:15 |
Marco Gaiarin |
bug |
|
|
added subscriber Marco Gaiarin |
| 2019-04-10 02:15:46 |
Bug Watch Updater |
linux: status |
Unknown |
Confirmed |
|
| 2019-04-10 02:15:46 |
Bug Watch Updater |
linux: importance |
Unknown |
Medium |
|
| 2019-07-24 20:55:36 |
Brad Figg |
tags |
apport-collected kernel-da-key trusty verification-done-trusty |
apport-collected cscc kernel-da-key trusty verification-done-trusty |
|
