EXT4 file system corruption on ubuntu server 14.04 LTS

[Sami Pietila]

There was extensive file system corruption triggered by a user (non-root) program consuming all memory. Dmesg contained following messages:

[2678426.843981] Out of memory: Kill process 31328 (python) score 983 or sacrifice child
[2678426.844151] Killed process 31328 (python) total-vm:60801412kB, anon-rss:60694516kB, file-rss:0kB
[2893469.391974] EXT4-fs warning (device dm-2): ext4_end_bio:317: I/O error -5 writing to inode 7865240 (offset 0 size 0 starting block 49231)
[2893469.391982] Buffer I/O error on device dm-2, logical block 49231
[2893469.392032] Buffer I/O error on device dm-2, logical block 49232
[2893469.395494] Buffer I/O error on device dm-2, logical block 49233
[2893469.569757] EXT4-fs warning (device dm-2): ext4_end_bio:317: I/O error -5 writing to inode 7864978 (offset 0 size 0 starting block 20808467)
[2893469.569762] Buffer I/O error on device dm-2, logical block 20808467
[2893469.569896] EXT4-fs warning (device dm-2): ext4_end_bio:317: I/O error -5 writing to inode 7866610 (offset 0 size 0 starting block 79732436)
[2893469.569899] Buffer I/O error on device dm-2, logical block 79732436
[2893469.570176] EXT4-fs warning (device dm-2): ext4_end_bio:317: I/O error -5 writing to inode 7866610 (offset 8192 size 4096 starting block 109098591)
[2893469.570180] Buffer I/O error on device dm-2, logical block 109098591
[2893469.707534] EXT4-fs warning (device dm-2): ext4_end_bio:317: I/O error -5 writing to inode 7865240 (offset 0 size 0 starting block 49217)
[2893469.707539] Buffer I/O error on device dm-2, logical block 49217
[2893469.707578] Buffer I/O error on device dm-2, logical block 49218
[2893469.707674] EXT4-fs warning (device dm-2): ext4_end_bio:317: I/O error -5 writing to inode 7865240 (offset 0 size 0 starting block 49223)
[2893469.707679] Buffer I/O error on device dm-2, logical block 49223
[2893469.707768] EXT4-fs warning (device dm-2): ext4_end_bio:317: I/O error -5 writing to inode 7865240 (offset 0 size 0 starting block 49226)
[2893469.707773] Buffer I/O error on device dm-2, logical block 49226
[2893469.707859] EXT4-fs warning (device dm-2): ext4_end_bio:317: I/O error -5 writing to inode 7865240 (offset 0 size 0 starting block 49229)
[2893469.707870] EXT4-fs warning (device dm-2): ext4_end_bio:317: I/O error -5 writing to inode 7864600 (offset 0 size 0 starting block 44853)
[2893469.707876] EXT4-fs warning (device dm-2): ext4_end_bio:317: I/O error -5 writing to inode 7864600 (offset 0 size 0 starting block 44858)
[2893469.726747] Aborting journal on device dm-2-8.
[2893469.768725] init: rsyslog main process (919) killed by BUS signal
[2893469.768766] init: rsyslog main process ended, respawning
[2893469.867576] Buffer I/O error on device dm-2, logical block 65044480
[2893469.867617] lost page write due to I/O error on dm-2
[2893469.867641] JBD2: Error -5 detected when updating journal superblock for dm-2-8.
[2893470.103486] Buffer I/O error on device dm-2, logical block 0
[2893470.103524] lost page write due to I/O error on dm-2
[2893470.103568] EXT4-fs error (device dm-2): ext4_journal_check_start:56: Detected aborted journal
[2893470.103684] EXT4-fs (dm-2): Remounting filesystem read-only
[2893470.103770] EXT4-fs (dm-2): previous I/O error to superblock detected
[2893470.197496] Buffer I/O error on device dm-2, logical block 0
[2893470.197584] lost page write due to I/O error on dm-2
[2893470.197720] Core dump to |/usr/share/apport/apport 439 7 0 439 pipe failed
[2893470.208395] init: rsyslog pre-start process (439) killed by BUS signal
[2893470.378616] Core dump to |/usr/share/apport/apport 442 7 0 442 pipe failed
[2893470.380183] init: startpar-bridge (rsyslog--stopped) main process (442) killed by BUS signal
[2893511.046711] Core dump to |/usr/share/apport/apport 1421 7 18446744073709551615 1421 pipe failed
[2903746.904285] EXT4-fs warning (device dm-1): __ext4_read_dirblock:681: error reading directory block (ino 174523398, block 0)
[2903773.782839] Aborting journal on device dm-1-8.
[2903773.801696] Buffer I/O error on device dm-1, logical block 1048084480
[2903773.801799] lost page write due to I/O error on dm-1
[2903773.801817] JBD2: Error -5 detected when updating journal superblock for dm-1-8.
[2903774.966747] Buffer I/O error on device dm-1, logical block 0
[2903774.968009] lost page write due to I/O error on dm-1
[2903774.968186] EXT4-fs error (device dm-1): ext4_journal_check_start:56: Detected aborted journal
[2903774.969608] EXT4-fs (dm-1): Remounting filesystem read-only
[2903774.970999] EXT4-fs (dm-1): previous I/O error to superblock detected
[2903775.026543] Buffer I/O error on device dm-1, logical block 0
[2903775.027901] lost page write due to I/O error on dm-1
[2903815.909779] Core dump to |/usr/share/apport/apport 3193 7 0 3193 pipe failed
[2929832.871589] init: Failed to write to log file /var/log/upstart/systemd-logind.log

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-3.13.0-34-generic 3.13.0-34.60
ProcVersionSignature: Ubuntu 3.13.0-34.60-generic 3.13.11.4
Uname: Linux 3.13.0-34-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 elo 21 13:53 seq
 crw-rw---- 1 root audio 116, 33 elo 21 13:53 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.14.1-0ubuntu3.3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory: 'iw'
CurrentDmesg: Error: command ['sh', '-c', 'dmesg | comm -13 --nocheck-order /var/log/dmesg -'] failed with exit code 1: comm: /var/log/dmesg: Permission denied
Date: Thu Aug 21 14:52:27 2014
InstallationDate: Installed on 2014-07-07 (45 days ago)
InstallationMedia: Ubuntu-Server 14.04 LTS "Trusty Tahr" - Release amd64 (20140416.2)
IwConfig:
 eth0 no wireless extensions.

 eth1 no wireless extensions.

 lo no wireless extensions.
Lsusb:
 Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: Red Hat Inc. OpenStack Nova
PciMultimedia:

ProcFB: 0 cirrusdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-34-generic root=/dev/mapper/vg0-lv0 ro
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
WifiSyslog:

dmi.bios.date: 01/01/2007
dmi.bios.vendor: Seabios
dmi.bios.version: 0.5.1
dmi.chassis.type: 1
dmi.chassis.vendor: Red Hat
dmi.modalias: dmi:bvnSeabios:bvr0.5.1:bd01/01/2007:svnRedHatInc.:pnOpenStackNova:pvr2013.1.3-2.el6:cvnRedHat:ct1:cvr:
dmi.product.name: OpenStack Nova
dmi.product.version: 2013.1.3-2.el6
dmi.sys.vendor: Red Hat Inc.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1359706

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.17 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.17-rc1-utopic/

[Sami Pietila (sampie)]

So far, I have not been able to reproduce the crash again even with exactly the same kernel.

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]