Latest Kernel Version 3.2.0-36 Causing EXT4-fs Errors

[Karlchen]

System: Ubuntu 12.04.1 x64 Unity desktop
---
lsb_release -rd; uname -a
Description: Ubuntu 12.04.1 LTS
Release: 12.04
Linux Voyager 3.2.0-36-generic #57-Ubuntu SMP Tue Jan 8 21:44:52 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
--
Have not experienced any severe problems with this system from Aug 2012 till Jan 20 2013, when kernel 3.2.0-35 was still the active kernel.
2013-01-20 21:01 kernel 3.2.0-36 was installed by the Ubuntu updater. No installation problems occurred.
Ever since K3.2.0-36 was the active kernel, the following problem occurred:

On every restart error messages like these could be read in /var/log/syslog:
syslog:Jan 22 00:44:37 Voyager kernel: [ 3.624421] EXT4-fs (sda5): orphan cleanup on readonly fs
syslog:Jan 22 00:44:37 Voyager kernel: [ 3.652470] EXT4-fs (sda5): ext4_orphan_cleanup: deleting unreferenced inode 2886456
syslog:Jan 22 00:44:37 Voyager kernel: [ 3.652671] EXT4-fs (sda5): ext4_orphan_cleanup: deleting unreferenced inode 2886232
syslog:Jan 22 00:44:37 Voyager kernel: [ 3.652688] EXT4-fs (sda5): ext4_orphan_cleanup: deleting unreferenced inode 2886450
syslog:Jan 22 00:44:37 Voyager kernel: [ 3.678434] EXT4-fs (sda5): ext4_orphan_cleanup: deleting unreferenced inode 2369342
syslog:Jan 22 00:44:37 Voyager kernel: [ 3.689458] EXT4-fs (sda5): 4 orphan inodes deleted
syslog:Jan 22 00:44:37 Voyager kernel: [ 3.689462] EXT4-fs (sda5): recovery complete

This happens no matter whether the system is rebooted (warm reboot) or shutdown and restarted later (cold reboot).

The relevant thing is that in the session which is shutdown the active kernel has to be K3.2.0-36.
The error will be detected in the next session started after reboot.

The workaround was going back to K3.2.0-35. During the first reboot the errors left behind by the previous K3.2.0-36 session were cleaned up. From then on sticking with K3.2.0-35 the errors would not return after any reboot.

Conclusion:
----------------
Telling from the symptoms Ubuntu 12.04.1 with K3.2.0-36 does not umount the root filesystem properly or does some other bad thing to leave behind orphaned inodes.
Returning to K3.2.0-35 will work around the problem

Note:
-------
(1) I have seen this problem on 3 different machines (different hardware) running Ubuntu 12.04.1 x64 and Ubuntu 12.04.1 x86.
      So it is not a problem caused by a flaw of one special machine.
(2) I have seen the same problem after updating Ubuntu 12.10 from K3.5.0-17 to K3.5.0-22. This suggests some change in the
      latest kernel editions which is common to the current Ubuntu kernels may be involved.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.2.0-36-generic 3.2.0-36.57
ProcVersionSignature: Ubuntu 3.2.0-36.57-generic 3.2.35
Uname: Linux 3.2.0-36-generic x86_64
NonfreeKernelModules: nvidia
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
ApportVersion: 2.0.1-0ubuntu17.1
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: karl 2234 F.... pulseaudio
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
Card0.Amixer.info:
 Card hw:0 'Intel'/'HDA Intel at 0xfdff4000 irq 44'
   Mixer name : 'Realtek ALC888'
   Components : 'HDA:10ec0888,1462735b,00100001'
   Controls : 45
   Simple ctrls : 21
Date: Sat Jan 26 10:50:56 2013
HibernationDevice: RESUME=UUID=20f6167d-0136-4d21-a1d9-fb36954a0815
InstallationMedia: Ubuntu 12.04 LTS "Precise Pangolin" - Release amd64 (20120425)
IwConfig:
 lo no wireless extensions.

 eth0 no wireless extensions.
MachineType: MEDIONPC MS-7358
MarkForUpload: True
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=de_DE.UTF-8
 SHELL=/bin/bash
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.2.0-36-generic root=UUID=7eb4fefd-d144-4bdb-9848-199970e85265 ro quiet splash vt.handoff=7
PulseList:
 Error: command ['pacmd', 'list'] failed with exit code 1: Home directory /home/karl not ours.
 No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-3.2.0-36-generic N/A
 linux-backports-modules-3.2.0-36-generic N/A
 linux-firmware 1.79.1
RfKill:

SourcePackage: linux
StagingDrivers: zram
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 07/24/2007
dmi.bios.vendor: Phoenix Technologies, LTD
dmi.bios.version: 6.00 PG
dmi.board.name: MS-7358
dmi.board.vendor: MICRO-STAR INTERNATIONAL CO., LTD
dmi.board.version: Fab D
dmi.chassis.type: 3
dmi.chassis.vendor: OEM
dmi.chassis.version: OEM
dmi.modalias: dmi:bvnPhoenixTechnologies,LTD:bvr6.00PG:bd07/24/2007:svnMEDIONPC:pnMS-7358:pvrOEM:rvnMICRO-STARINTERNATIONALCO.,LTD:rnMS-7358:rvrFabD:cvnOEM:ct3:cvrOEM:
dmi.product.name: MS-7358
dmi.product.version: OEM
dmi.sys.vendor: MEDIONPC

[Brad Figg (brad-figg)]

This change was made by a bot.

[Karlchen (karlchen)]

Correction:
---------------
Item (2) under "Note:" in my bug report should of course read:
 (2) I have seen the same problem after updating Ubuntu 12.10 from K3.5.0-17 to K3.5.0-22.

Karl

[Karlchen (karlchen)]

Here is a concise hard- and software configuration of the system "Voyager" which this report is based on.
Installed inxi 1.84.1 on Ubuntu 12.04.1 x64, which does not bring inxi along by default, and ran "inxi -Fx", because inxi can sum up the system configuration in a really concise way.
After all the system specifications might be relevant for the experienced problem.

Karl

[Karlchen (karlchen)]

Oops, the saved "inxi -Fx" output holds escape sequences making the text file unreadable. Here is the pure screen output:

System: Host: Voyager Kernel: 3.2.0-36-generic x86_64 (64 bit, gcc: 4.6.3) Desktop: Gnome Distro: Ubuntu 12.04 precise
Machine: System: MEDIONPC product: MS-7358
           Mobo: MICRO-STAR model: MS-7358 version: Fab D Bios: Phoenix version: 6.00 PG date: 07/24/2007
CPU: Quad core Intel Core2 Quad CPU (-MCP-) cache: 4096 KB flags: (lm nx sse sse2 sse3 ssse3 vmx) bmips: 19140.9
           Clock Speeds: 1: 1600.00 MHz 2: 1600.00 MHz 3: 1600.00 MHz 4: 1600.00 MHz
Graphics: Card: NVIDIA G86 [GeForce 8500 GT] bus-ID: 01:00.0
           X.Org: 1.11.3 drivers: nvidia (unloaded: nouveau,vesa,fbdev) Resolution: 1280x1024@51.0hz
           GLX Renderer: GeForce 8500 GT/PCIe/SSE2 GLX Version: 3.3.0 NVIDIA 295.40 Direct Rendering: Yes
Audio: Card: Intel 82801I (ICH9 Family) HD Audio Controller driver: snd_hda_intel bus-ID: 00:1b.0 Sound: ALSA ver: 1.0.24
Network: Card: Realtek RTL-8169 Gigabit Ethernet driver: r8169 ver: 2.3LK-NAPI port: ac00 bus-ID: 04:00.0
           IF: eth0 state: up speed: 100 Mbps duplex: full mac: 00:0a:cd:1d:12:51
Drives: HDD Total Size: 507.9GB (2.1% used) 1: id: /dev/sda model: ST3500830AS size: 500.1GB
           2: USB id: /dev/sde model: Premium size: 7.8GB
Partition: ID: / size: 46G used: 7.0G (17%) fs: ext4 ID: swap-1 size: 1.71GB used: 0.00GB (0%) fs: swap
           ID: swap-2 size: 8.22GB used: 0.00GB (0%) fs: swap
RAID: No RAID devices detected - /proc/mdstat and md_mod kernel raid module present
Sensors: None detected - is lm-sensors installed and configured?
Info: Processes: 194 Uptime: 3:17 Memory: 923.5/3261.8MB Runlevel: 2 Gcc sys: 4.6.3 Client: Shell inxi: 1.8.4

[Karlchen (karlchen)]

inix -Fx screen output, escape sequences removed, 3rd attempt finally worked, attachment is in pure text format and readable.

[viking777 (viking-f2s)]

I have an identical problem on Ubuntu 12.10 when using kernel 3.5.0-22 but not kernel 3.5.0-21

Jan 22 18:44:21 fujitsu kernel: [ 2.865532] EXT4-fs (sda2): orphan cleanup on readonly fs
Jan 22 18:44:21 fujitsu kernel: [ 2.866295] EXT4-fs (sda2): ext4_orphan_cleanup: deleting unreferenced inode 1310686
Jan 22 18:44:21 fujitsu kernel: [ 2.866328] EXT4-fs (sda2): ext4_orphan_cleanup: deleting unreferenced inode 1310684
Jan 22 18:44:21 fujitsu kernel: [ 2.866332] EXT4-fs (sda2): ext4_orphan_cleanup: deleting unreferenced inode 1310685
Jan 22 18:44:21 fujitsu kernel: [ 2.866339] EXT4-fs (sda2): ext4_orphan_cleanup: deleting unreferenced inode 57
Jan 22 18:44:21 fujitsu kernel: [ 2.866361] EXT4-fs (sda2): ext4_orphan_cleanup: deleting unreferenced inode 30
Jan 22 18:44:21 fujitsu kernel: [ 2.866368] EXT4-fs (sda2): ext4_orphan_cleanup: deleting unreferenced inode 262369
Jan 22 18:44:21 fujitsu kernel: [ 2.866388] EXT4-fs (sda2): 6 orphan inodes deleted

In my case the inodes 30 and 57 are repeatedly deleted after every boot which I find quite disturbing and I have now reverted to using only kernel 3.5.0-21.

inxi -Fx
System: Host: fujitsu Kernel: 3.5.0-21-generic x86_64 (64 bit, gcc: 4.7.2)
           Desktop: Gnome (Gtk 2.24.13) Distro: Ubuntu 12.10 quantal
Machine: System: FUJITSU product: LIFEBOOK AH532
           Mobo: FUJITSU model: FJNBB1C Bios: FUJITSU // Phoenix version: Version 1.09 date: 05/22/2012
CPU: Dual core Intel Core i5-3210M CPU (-HT-MCP-) cache: 3072 KB flags: (lm nx sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx) bmips: 9977.12
           Clock Speeds: 1: 1200.00 MHz 2: 1200.00 MHz 3: 1200.00 MHz 4: 1200.00 MHz
Graphics: Card: Intel 3rd Gen Core processor Graphics Controller bus-ID: 00:02.0
           X.Org: 1.13.0 drivers: intel (unloaded: fbdev,vesa) Resolution: 1366x768@60.0hz, 1280x1024@60.0hz
           GLX Renderer: Mesa DRI Intel Ivybridge Mobile GLX Version: 3.0 Mesa 9.0 Direct Rendering: Yes
Audio: Card: Intel 7 Series/C210 Series Chipset Family High Definition Audio Controller
           driver: snd_hda_intel bus-ID: 00:1b.0
           Sound: Advanced Linux Sound Architecture ver: 1.0.25
Network: Card-1: Realtek RTL8111/8168B PCI Express Gigabit Ethernet controller
           driver: r8169 ver: 2.3LK-NAPI port: 3000 bus-ID: 02:00.0
           IF: eth0 state: up speed: 100 Mbps duplex: full mac: 5c:9a:d8:69:59:e5
           Card-2: Intel Centrino Wireless-N 2230 driver: iwlwifi ver: in-tree: bus-ID: 08:00.0
           IF: wlan0 state: down mac: 68:5d:43:a6:60:0d
Drives: HDD Total Size: 1000.2GB (3.2% used) 1: id: /dev/sda model: TOSHIBA_MK5076GS size: 500.1GB
           2: USB id: /dev/sdb model: My_Passport_0740 size: 500.1GB
Partition: ID: / size: 26G used: 4.7G (20%) fs: ext4 ID: /home size: 116G used: 13G (12%) fs: ext4
           ID: swap-1 size: 6.30GB used: 0.00GB (0%) fs: swap
RAID: No RAID devices detected - /proc/mdstat and md_mod kernel raid module present
Sensors: System Temperatures: cpu: 51.0C mobo: N/A
           Fan Speeds (in rpm): cpu: N/A
Info: Processes: 180 Uptime: 10 min Memory: 761.8/5832.8MB Runlevel: 2 Gcc sys: 4.7.2 Client: Shell (fi...

[Karlchen (karlchen)]

Disk controller operating mode as reported by palimpsest (disk utility) might be relevant, too.
In fact, all affected systems that I have seen work in AHCI mode.

[Luis Henriques (henrix)]

Karlchen and viking777, we have identified a regression on the 3.5.0-22.34 (12.10) and 3.2.0-36.57 (12.04) kernels. This regression could in fact be causing the problems you're seing in your systems. It should have been solved by the kernels that are currently in -proposed (3.2.0-37.58 and 3.5.0-23.35).

Could you please give these kernels a try to see if the problem is solved?

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Karlchen (karlchen)]

Hello, Luis Henriques.

This seems to be good news indeed.

Started with my 32-bit Ubuntu 12.10 running on kernel 3.5.0-22, added quantal-proposed to the /etc/apt/sources.list file, updated to kernel 3.5.0-23 and rebooted in order to activate the new kernel.

---Synaptic history entry logging installation of kernel 3.5.0-23---
Commit Log for Sun Jan 27 15:18:30 2013

Die folgenden Pakete wurden aktualisiert:
linux-generic (3.5.0.22.28) to 3.5.0.23.29
linux-headers-generic (3.5.0.22.28) to 3.5.0.23.29
linux-headers-generic-pae (3.5.0.22.28) to 3.5.0.23.29
linux-image-generic (3.5.0.22.28) to 3.5.0.23.29
linux-libc-dev (3.5.0-22.34) to 3.5.0-23.35

Die folgenden Pakete wurden installiert:
linux-headers-3.5.0-23 (3.5.0-23.35)
linux-headers-3.5.0-23-generic (3.5.0-23.35)
linux-image-3.5.0-23-generic (3.5.0-23.35)
linux-image-extra-3.5.0-23-generic (3.5.0-23.35)
---Synaptic end

---uname -a output---
Linux unimatrix0 3.5.0-23-generic #35-Ubuntu SMP Thu Jan 24 13:05:29 UTC 2013 i686 i686 i686 GNU/Linux
---uname end

Used the system for a while an rebooted it, because the problem becomes visible only after rebooting.

---grep "EXT4-fs" /var/log/syslog output---
syslog:Jan 27 16:26:05 unimatrix0 kernel: [ 1.642452] EXT4-fs (sda6): INFO: recovery required on readonly filesystem
syslog:Jan 27 16:26:05 unimatrix0 kernel: [ 1.642461] EXT4-fs (sda6): write access will be enabled during recovery
syslog:Jan 27 16:26:05 unimatrix0 kernel: [ 3.993863] EXT4-fs (sda6): recovery complete
syslog:Jan 27 16:26:05 unimatrix0 kernel: [ 4.003148] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog:Jan 27 16:26:05 unimatrix0 kernel: [ 24.559045] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
---grep end---

This seemed to look better. No complaint about orphaned inodes. But the first line telling that "INFO: recovery required on readonly filesystem" made me suspicious. This "INFO" line does not appear in the syslog file under Ubuntu 12.04.1 K3.2.0-35.
But K3.5.0-23 might just be more verbose.

Rebooted again and this time did not select to launch Ubuntu 12.10 K3.5.0-23, but Ubuntu 12.04.1 K3.2.0-35 and run an fsck against the Ubuntu 12.10 partition which is /dev/sda6.

So the commandline run was: fsck -v -f /dev/fsck.
(Sorry, forgot to switch my terminal language to English. So the output is in German. Yet, it may be understandable nonetheless, I hope.)

---fsck -v -f /dev/sda6 output---
sudo fsck -v -f /dev/sda6
fsck von util-linux 2.20.1
e2fsck 1.42 (29-Nov-2011)
ubu1210: stelle das Journal wieder her <========== ubu1210 (partition name): recovering the journal
Durchgang 1: Prüfe Inodes, Blocks, und Größen
Durchgang 2: Prüfe Verzeichnis Struktur
Durchgang 3: Prüfe Verzeichnis Verknüpfungen
Durchgang 4: Überprüfe die Referenzzähler
Durchgang 5: Überprüfe Gruppe Zusammenfassung
Die Anzahl freier Blöcke ist falsch (10905020, gezählt=10904942). <=== free block count incorrect
Repariere<j>? ja

Die Anzahl freier Inodes ist falsch (2864274, gezählt=2864270). <== free inode count incorrect
Repariere<j>? ja

ubu1210: ***** DATEISYSTEM WURDE VERÄNDERT *****

281458 Inodes sind in Benutzung (8.95%)
     155 nicht zusammenhängende...

[Karlchen (karlchen)]

Good evening, Luis Henriques, folks.

Coming back to the Ubuntu 12.04.1 x64 system running kernel 3.2.0-36 which made me open this bug report. (cf. https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1105900/comments/0):

Tonight at 19:50 the kernel was updated to K3.2.0-37.

---Synaptic history---
Commit Log for Mon Jan 28 19:50:18 2013

Die folgenden Pakete wurden aktualisiert:
linux-generic (3.2.0.36.43) to 3.2.0.37.44
linux-headers-generic (3.2.0.36.43) to 3.2.0.37.44
linux-image-generic (3.2.0.36.43) to 3.2.0.37.44
linux-libc-dev (3.2.0-36.57) to 3.2.0-37.58

Die folgenden Pakete wurden installiert:
linux-headers-3.2.0-37 (3.2.0-37.58)
linux-headers-3.2.0-37-generic (3.2.0-37.58)
linux-image-3.2.0-37-generic (3.2.0-37.58)
---Synaptic history end---

The machine was rebooted to load the updated kernel.

---lsb_release -rd;uname -a output ---
$ lsb_release -rd;uname -a
Description: Ubuntu 12.04.1 LTS
Release: 12.04
Linux Voyager 3.2.0-37-generic #58-Ubuntu SMP Thu Jan 24 15:28:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
---lsb_release -rd;uname -a end---

---grep "EXT4-fs syslog" after first boot of K3.2.0-37---
syslog:Jan 28 20:00:57 Voyager kernel: [ 3.919628] EXT4-fs (sda5): orphan cleanup on readonly fs
syslog:Jan 28 20:00:57 Voyager kernel: [ 3.937261] EXT4-fs (sda5): ext4_orphan_cleanup: deleting unreferenced inode 2895492
syslog:Jan 28 20:00:57 Voyager kernel: [ 3.937489] EXT4-fs (sda5): ext4_orphan_cleanup: deleting unreferenced inode 2895491
syslog:Jan 28 20:00:57 Voyager kernel: [ 3.937500] EXT4-fs (sda5): ext4_orphan_cleanup: deleting unreferenced inode 2895490
syslog:Jan 28 20:00:57 Voyager kernel: [ 3.937517] EXT4-fs (sda5): ext4_orphan_cleanup: deleting unreferenced inode 2895489
syslog:Jan 28 20:00:57 Voyager kernel: [ 3.937525] EXT4-fs (sda5): ext4_orphan_cleanup: deleting unreferenced inode 2895488
syslog:Jan 28 20:00:57 Voyager kernel: [ 3.937539] EXT4-fs (sda5): 5 orphan inodes deleted
syslog:Jan 28 20:00:57 Voyager kernel: [ 3.937541] EXT4-fs (sda5): recovery complete
syslog:Jan 28 20:00:57 Voyager kernel: [ 3.969702] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
syslog:Jan 28 20:00:57 Voyager kernel: [ 21.609817] EXT4-fs (sda5): re-mounted. Opts: errors=remount-ro
---grep "EXT4-fs syslog" after first boot of K3.2.0-37 end---

This was to be expected because K3.2.0-36 had left behind open file handles on shutdown.

BUT:

---grep "EXT4-fs syslog" after second boot of K3.2.0-37---
syslog:Jan 28 20:12:07 Voyager kernel: [ 4.130957] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
syslog:Jan 28 20:12:07 Voyager kernel: [ 21.736968] EXT4-fs (sda5): re-mounted. Opts: errors=remount-ro
---grep "EXT4-fs syslog" after second boot of K3.2.0-37 end---

Great, no more cleanup needed on reboot. Yet, uptime had been only few minutes. So let us try again a little later.

---grep "EXT4-fs syslog" after third boot of K3.2.0-37---
syslog:Jan 28 22:35:27 Voyager kernel: [ 3.756446] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
syslog:Jan 28 22:35:27 Voyager kernel: [ 21.520713] EXT4-fs (sda5): re-mounted. Opts: errors=remount-...

[viking777 (viking-f2s)]

I can confirm that I am using AHCI mode also. I will start testing the 3.5.0-23 kernel today.

[viking777 (viking-f2s)]

These are my initial results. Using the 3.5.0-23 kernel 64 bit on Ubuntu 12.10.
Installed the kernel and did 5 reboots. No instance of orphaned inodes occurred:

grep -i orphan /var/log/syslog
ubuntu@fujitsu

The 'recovery' message occurred on every boot, but then again it also occurs on every boot with kernels -21 and -22, so there is no change there:

grep -i recovery /var/log/syslog
Jan 29 11:04:35 fujitsu kernel: [ 1.827272] EXT4-fs (sda2): INFO: recovery required on readonly filesystem
Jan 29 11:04:35 fujitsu kernel: [ 1.828256] EXT4-fs (sda2): write access will be enabled during recovery
Jan 29 11:04:35 fujitsu kernel: [ 2.452937] EXT4-fs (sda2): recovery complete

I will continue using the -23 kernel to see if uptime brings a recurrence of the orphaned inode problem.

[viking777 (viking-f2s)]

For my next experiment I rebooted from Ubuntu12.10 into Mint14 and forced an fsck on the Ubuntu partition.
Fsck found no errors.
However when I rebooted into Ubuntu the 'recovery' error was not present this time.
There continue to be no 'orphan' errors.

[viking777 (viking-f2s)]

Since forcing fsck from another distro, the warning on file system recovery on my Ubuntu partition (given in post 13), has not returned. Prior to forcing this check it occurred on every boot. This makes me wonder if the decision to set maximum mount count for all ext4 files systems to -1 was actually such a good idea?

sudo dumpe2fs -h /dev/sda2 | grep -i 'mount count'

dumpe2fs 1.42.5 (29-Jul-2012)
Mount count: 2
Maximum mount count: -1

Of course 2 mounts is not a lot to go on, but I will continue to monitor the situation.

Still no orphaned inodes on the -23 kernel so far.

[viking777 (viking-f2s)]

Update to comment 15. Ubuntu12.10, kernel 3.5.0-23 64bit.

After 2 days of use (mount count now 14) the 'recovery' error message recurred (see comment #13). The file system was "recovered" on every boot until I forced an fsck from another distro. As before, the fsck reported no file system errors, but once completed the 'recovery' message again stopped appearing.

I still have no 'orphaned inodes' messages on the -23 kernel.

[Karlchen (karlchen)]

Hello, Luis Henriques, folks.

As announced in comment #11 (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1105900/comments/11), I have checked /var/log/syslog on my Ubuntu 12.04.1 x64 system running kernel 3.2.0-37 for a few more days.

 ---(z)grep "EXT4-fs" syslog.?gz syslog.1 syslog---
syslog.3.gz:Jan 28 20:12:07 Voyager kernel: [ 4.130957] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
syslog.3.gz:Jan 28 20:12:07 Voyager kernel: [ 21.736968] EXT4-fs (sda5): re-mounted. Opts: errors=remount-ro
syslog.3.gz:Jan 28 22:35:27 Voyager kernel: [ 3.756446] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
syslog.3.gz:Jan 28 22:35:27 Voyager kernel: [ 21.520713] EXT4-fs (sda5): re-mounted. Opts: errors=remount-ro
syslog.3.gz:Jan 30 22:53:50 Voyager kernel: [ 4.067835] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
syslog.3.gz:Jan 30 22:53:50 Voyager kernel: [ 21.730656] EXT4-fs (sda5): re-mounted. Opts: errors=remount-ro
syslog.2.gz:Jan 31 20:02:10 Voyager kernel: [ 3.742789] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
syslog.2.gz:Jan 31 20:02:10 Voyager kernel: [ 6.382830] EXT4-fs (sda5): re-mounted. Opts: errors=remount-ro
syslog.1:Feb 1 19:41:15 Voyager kernel: [ 3.897188] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
syslog.1:Feb 1 19:41:15 Voyager kernel: [ 21.434926] EXT4-fs (sda5): re-mounted. Opts: errors=remount-ro
syslog:Feb 2 19:53:05 Voyager kernel: [ 4.071853] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
syslog:Feb 2 19:53:05 Voyager kernel: [ 21.618324] EXT4-fs (sda5): re-mounted. Opts: errors=remount-ro
 ---(z)grep "EXT4-fs" syslog.?gz syslog.1 syslog end---

The EXT4-fs entries recorded in the syslog file during the past few days seem to confirm that for Ubuntu 12.04.1 x64 the problem has been solved by installing kernel 3.2.0-37.
This is in accordance with another Ubuntu 12.04.1 x86 system where K3.2.0-37 has been installed and where syslog has not recorded any EXT4-fs errors or warnings so far.

So at this point in time I would say:

Ubuntu editions: Ubuntu 12.04.1 x64 / x86
Kernel version : Kernel 3.2.0-37
EXT4-fs status : reported problem(s) solved.

Kind regards,
Karl

[Karlchen (karlchen)]

Hello, Luis Henriques, viking777.

With respect to
+ Ubuntu edition: Ubuntu 12.10 x86
+ Kernel version : Kernel 3.5.0-23
I am afraid I have to confirm your reports, viking777.
Same behaviour here. "/" is busy on every single shutdown right before the sysem gets halted or rebooted. So "/" is never clean when the system comes up next time and has to replay its journal.

---(z)grep "EXT4-fs" syslog2.gz syslog.1 syslog ---
syslog.2.gz:Jan 31 23:19:27 unimatrix0 kernel: [ 1.688343] EXT4-fs (sda6): INFO: recovery required on readonly filesystem
syslog.2.gz:Jan 31 23:19:27 unimatrix0 kernel: [ 1.688352] EXT4-fs (sda6): write access will be enabled during recovery
syslog.2.gz:Jan 31 23:19:27 unimatrix0 kernel: [ 3.078213] EXT4-fs (sda6): recovery complete
syslog.2.gz:Jan 31 23:19:27 unimatrix0 kernel: [ 3.087535] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog.2.gz:Jan 31 23:19:27 unimatrix0 kernel: [ 23.731970] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
syslog.1:Feb 2 01:11:03 unimatrix0 kernel: [ 1.667425] EXT4-fs (sda6): INFO: recovery required on readonly filesystem
syslog.1:Feb 2 01:11:03 unimatrix0 kernel: [ 1.667435] EXT4-fs (sda6): write access will be enabled during recovery
syslog.1:Feb 2 01:11:03 unimatrix0 kernel: [ 4.640044] EXT4-fs (sda6): recovery complete
syslog.1:Feb 2 01:11:03 unimatrix0 kernel: [ 4.649186] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog.1:Feb 2 01:11:03 unimatrix0 kernel: [ 8.473379] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
syslog:Feb 3 14:18:10 unimatrix0 kernel: [ 1.718444] EXT4-fs (sda6): INFO: recovery required on readonly filesystem
syslog:Feb 3 14:18:10 unimatrix0 kernel: [ 1.718453] EXT4-fs (sda6): write access will be enabled during recovery
syslog:Feb 3 14:18:10 unimatrix0 kernel: [ 4.457884] EXT4-fs (sda6): recovery complete
syslog:Feb 3 14:18:10 unimatrix0 kernel: [ 4.467270] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
syslog:Feb 3 14:18:10 unimatrix0 kernel: [ 24.140801] EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
---(z)grep "EXT4-fs" syslog2.gz syslog.1 syslog end---

So at this point in time I would say:

Ubuntu editions: Ubuntu 12.10 x64 / x86
Kernel version : Kernel 3.5.0-23
EXT4-fs status : reported problem(s) not solved yet.

Kind regards,
Karl

[Matt Bush (mbbush)]

I get the same problem using 3.5.0-23 and an ext2 filesystem on my root partition.

[maxnax (maxnax)]

I've faced the same bug on Ubuntu 12.04 with both 3.2.0-36 and 3.5.0-22 . And besides numerous " inode xxx has zero dtime DELETED" at each reboot ,there was something even more sophisticated ...
Acronis linux based livecd fails to backup ext2 partition with those kernels !
Or rather not exactly so... Acronis creates backups and then even checks them OK , but fails to restore , leavin an unpartitioning space
But everything seems working fine since upgrading to 3.2.0-37 and 3.5.0-23 . Almost two weeks without any problem

[Salik Rafiq (salik-rafiq)]

I too am experiencing buddy errors with EXT4.
using 3.2.0-39-generic-pae
The volume giving the errors is a RAID 1 setup using dmraid. So I wonder if in this situation there is an issue there. This volume was upgraded from ext3 as well.

I don't see errors in the other ext4 volumes.

The errors I am seeing are

Apr 2 19:00:14 chamraid01 kernel: [883688.713934] EXT4-fs error (device md3): ext4_mb_generate_buddy:739: group 1627, 29388 clusters in bitmap, 29387 in gd
Apr 2 22:10:15 chamraid01 kernel: [895089.919766] EXT4-fs error (device md3): ext4_mb_generate_buddy:739: group 1639, 20958 clusters in bitmap, 20957 in gd
Apr 2 22:11:24 chamraid01 kernel: [895158.699530] EXT4-fs error (device md3): ext4_mb_generate_buddy:739: group 1854, 29521 clusters in bitmap, 29520 in gd
Apr 2 23:37:59 chamraid01 kernel: [900353.845445] EXT4-fs error (device md3): ext4_mb_generate_buddy:739: group 1592, 22283 clusters in bitmap, 22282 in gd
Apr 2 23:38:53 chamraid01 kernel: [900408.376493] EXT4-fs warning (device md3): empty_dir:1929: bad directory (dir #17039929) - no `.' or `..'
Apr 3 03:23:17 chamraid01 kernel: [913871.840137] EXT4-fs (md3): error count: 11
Apr 3 03:23:17 chamraid01 kernel: [913871.840157] EXT4-fs (md3): initial error at 1364782713: ext4_mb_generate_buddy:739
Apr 3 03:23:17 chamraid01 kernel: [913871.840170] EXT4-fs (md3): last error at 1364942279: ext4_mb_generate_buddy:739

I have another server which is using Ubuntu 10.04 and it reports 0 EXT4 errors, even though it has a much larger RAID 5 also using dmraid.

[Mathias Rodriguez (enanox)]

I am experiencing the same issue in Ubuntu 12.04 with kernel version 3.2.0-45 in a EXT4 volume. The error messages on init are similar to the listed before in this log.

[Salik Rafiq (salik-rafiq)]

This is interesting.
Just put on kernel 3.2.0-48 pae and over the past three days no buddy errors reported in the logs.

Will monitor over the next couple of weeks, so far, whatever was done to the kernel, appears to have cured my issue.

[Salik Rafiq (salik-rafiq)]

After one week my log shows a single buddy error. This is a distinct improvement. Hopefully the next kernel update will totally cure it.

[Salik Rafiq (salik-rafiq)]

Presently have updated to kernel 3.2.0-49-generic-pae and have monitored over two weeks and not a single ext4 buddy error has been created.

Can others move up to this kernel and validate. This seems to have cured my issue. Think I'll lock myself down at this one.

[Karlchen (karlchen)]

Hello, folks.

I know it has been a really long time since I have last posted in this bug report which after all has been opened by me.

Ubuntu 12.04 Kernel 3.2.0-37
----------------------------------------
In post #17 (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1105900/comments/17) I confirmed that starting with kernel 3.2.0-37 the reported problem/bug had been fixed.
As far as I can tell this is true for all later 3.2.0-x kernels, too.
On my Ubuntu 12.04, by now Ubuntu 12.04.3, the problem has never occurred again. And currently, I am on kernel 3.2.0-55.
Summary: Bug fixed in kernel 3.2.0-37 and all later kernels.

Ubuntu 12.10 Kernel 3.5.0-23
-----------------------------------------
In post #18 (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1105900/comments/18) I had stated that the problem still persisted. This was right with respect to the symptoms. This was wrong with respect to the cause.
Starting with 3.5.0.23 the "filesystem / busy" during the shutdown and the need to perform an fsck during startup has no longer been caused by the kernel.

As I know by now - shame on me for needing more than 6 months to find out - the remaining symptoms were caused by a shutdown problem which is discussed e.g. in these two Launchpad bug reports:
+ Upstartification of /etc/init.d/networking has lost deconfiguring-networking event causing bad side-effects
   (https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1061639)
+ In Quantal, the root filesystem is not cleanly unmounted at shutdown or reboot
   (https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1058987)

So whoever is still experiencing an unclean root filesystem on each startup should check those two bug reports.

From my point of view: Kernel bug fixed in kernel 3.5.0-23 and all later kernels.

Kind regards,
Karl

[Karlchen (karlchen)]

Have just changed the state of this report from "Confirmed" to "Fix released", because as a matter of fact the reported kernel bug has been fixed by
+ Kernel 3.2.0-37 (and later) ... applying to Ubuntu 12.04
+ Kernel 3.5.0-23 (and later) ... applying to Ubuntu 12.10 (and 12.04 with Kernel series 3.5.0-x)