I cannot find how it might be related, but we are seeing massive filesystem corruptions in virtual guests on kvm in Lucid.
Host was running several kernels, from stock Lucid up to 3.0.0-14-server. Guests were booted with several different kernels as well. We also changed storag backen form qcow, to raw and eventually to lvm base with no avail.
Usuall message just after going to RO:
[2012-03-01 04:39:06] EXT4-fs (vda): error count: 10
[2012-03-01 04:39:06] EXT4-fs (vda): initial error at 1323754623: htree_dirblock_to_tree:586: inode 371080: block 1229922
[2012-03-01 04:39:06] EXT4-fs (vda): last error at 1329327878: ext4_remount:3754: inode 170313: block 543763
[2012-03-02 04:40:50] EXT4-fs (vda): error count: 10
[2012-03-02 04:40:50] EXT4-fs (vda): initial error at 1323754623: htree_dirblock_to_tree:586: inode 371080: block 1229922
[2012-03-02 04:40:50] EXT4-fs (vda): last error at 1329327878: ext4_remount:3754: inode 170313: block 543763
[2012-03-03 04:42:38] EXT4-fs (vda): error count: 10
[2012-03-03 04:42:38] EXT4-fs (vda): initial error at 1323754623: htree_dirblock_to_tree:586: inode 371080: block 1229922
[2012-03-03 04:42:38] EXT4-fs (vda): last error at 1329327878: ext4_remount:3754: inode 170313: block 543763
[2012-03-04 04:44:25] EXT4-fs (vda): error count: 10
[2012-03-04 04:44:25] EXT4-fs (vda): initial error at 1323754623: htree_dirblock_to_tree:586: inode 371080: block 1229922
[2012-03-04 04:44:25] EXT4-fs (vda): last error at 1329327878: ext4_remount:3754: inode 170313: block 543763
[2012-03-04 20:34:20] EXT4-fs error (device vda): htree_dirblock_to_tree:587: inode #171186: block 546842: comm chown: bad entry in directory: rec_len is smaller than minimal - offset=0(0), inode=4210740, rec_len=0, name_len=0
[2012-03-04 20:34:20] Aborting journal on device vda-8.
[2012-03-04 20:34:20] EXT4-fs (vda): Remounting filesystem read-only
[2012-03-05 04:46:13] EXT4-fs (vda): error count: 11
[2012-03-05 04:46:13] EXT4-fs (vda): initial error at 1323754623: htree_dirblock_to_tree:586: inode 371080: block 1229922
[2012-03-05 04:46:13] EXT4-fs (vda): last error at 1330893259: htree_dirblock_to_tree:587: inode 171186: block 546842
I cannot find how it might be related, but we are seeing massive filesystem corruptions in virtual guests on kvm in Lucid. to_tree: 586: inode 371080: block 1229922 to_tree: 586: inode 371080: block 1229922 to_tree: 586: inode 371080: block 1229922 to_tree: 586: inode 371080: block 1229922 to_tree: 587: inode #171186: block 546842: comm chown: bad entry in directory: rec_len is smaller than minimal - offset=0(0), inode=4210740, rec_len=0, name_len=0 to_tree: 586: inode 371080: block 1229922 to_tree: 587: inode 171186: block 546842
Host was running several kernels, from stock Lucid up to 3.0.0-14-server. Guests were booted with several different kernels as well. We also changed storag backen form qcow, to raw and eventually to lvm base with no avail.
Usuall message just after going to RO:
[2012-03-01 04:39:06] EXT4-fs (vda): error count: 10
[2012-03-01 04:39:06] EXT4-fs (vda): initial error at 1323754623: htree_dirblock_
[2012-03-01 04:39:06] EXT4-fs (vda): last error at 1329327878: ext4_remount:3754: inode 170313: block 543763
[2012-03-02 04:40:50] EXT4-fs (vda): error count: 10
[2012-03-02 04:40:50] EXT4-fs (vda): initial error at 1323754623: htree_dirblock_
[2012-03-02 04:40:50] EXT4-fs (vda): last error at 1329327878: ext4_remount:3754: inode 170313: block 543763
[2012-03-03 04:42:38] EXT4-fs (vda): error count: 10
[2012-03-03 04:42:38] EXT4-fs (vda): initial error at 1323754623: htree_dirblock_
[2012-03-03 04:42:38] EXT4-fs (vda): last error at 1329327878: ext4_remount:3754: inode 170313: block 543763
[2012-03-04 04:44:25] EXT4-fs (vda): error count: 10
[2012-03-04 04:44:25] EXT4-fs (vda): initial error at 1323754623: htree_dirblock_
[2012-03-04 04:44:25] EXT4-fs (vda): last error at 1329327878: ext4_remount:3754: inode 170313: block 543763
[2012-03-04 20:34:20] EXT4-fs error (device vda): htree_dirblock_
[2012-03-04 20:34:20] Aborting journal on device vda-8.
[2012-03-04 20:34:20] EXT4-fs (vda): Remounting filesystem read-only
[2012-03-05 04:46:13] EXT4-fs (vda): error count: 11
[2012-03-05 04:46:13] EXT4-fs (vda): initial error at 1323754623: htree_dirblock_
[2012-03-05 04:46:13] EXT4-fs (vda): last error at 1330893259: htree_dirblock_
Or
[20768.343508] EXT3-fs error (device vda): htree_dirblock_ to_tree: bad entry in directory #837494: rec_len is smaller than minimal - offset=0, inode=4210740, rec_len=0, name_len=0 remove_ journal_ head: freeing b_committed_data remove_ journal_ head: freeing b_committed_data remove_ journal_ head: freeing b_committed_data buildd/ linux-lts- backport- oneiric- 3.0.0/fs/ ext3/inode. c:1571 ext3_ordered_ writepage+ 0x223/0x250( ) bcf>] warn_slowpath_ common+ 0x7f/0xc0 c2a>] warn_slowpath_ null+0x1a/ 0x20 f33>] ext3_ordered_ writepage+ 0x223/0x250 cf7>] __writepage+ 0x17/0x40 001>] write_cache_ pages+0x241/ 0x4d0 ce0>] ? set_page_ dirty+0x70/ 0x70 2e1>] generic_ writepages+ 0x51/0x80 345>] do_writepages+ 0x35/0x40 59e>] writeback_ single_ inode+0x10e/ 0x280 af3>] writeback_ sb_inodes+ 0xe3/0x1b0 d64>] writeback_ inodes_ wb+0xa4/ 0x170 943>] wb_writeback+ 0x2f3/0x430 f8f>] ? _raw_spin_ lock_irqsave+ 0x2f/0x40 c9f>] wb_do_writeback +0x21f/ 0x270 d9a>] bdi_writeback_ thread+ 0xaa/0x270 cf0>] ? wb_do_writeback +0x270/ 0x270 3c6>] kthread+0x96/0xa0 3a4>] kernel_ thread_ helper+ 0x4/0x10 330>] ? kthread_ worker_ fn+0x190/ 0x190 3a0>] ? gs_change+0x13/0x13
[20768.348149] Aborting journal on device vda.
[20768.352064] EXT3-fs (vda): error: remounting filesystem read-only
[20768.396397] __journal_
[20768.396405] __journal_
[20768.396407] __journal_
[20769.700102] ------------[ cut here ]------------
[20769.700125] WARNING: at /build/
[20769.700127] Hardware name: Bochs
[20769.700128] Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl sunrpc psmouse serio_raw virtio_balloon i2c_piix4 raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov floppy raid6_pq async_tx raid1 raid0 multipath linear
[20769.700146] Pid: 2496, comm: flush-253:0 Not tainted 3.0.0-14-server #23~lucid1-Ubuntu
[20769.700147] Call Trace:
[20769.700155] [<ffffffff81061
[20769.700158] [<ffffffff81061
[20769.700160] [<ffffffff811ea
[20769.700166] [<ffffffff81118
[20769.700169] [<ffffffff8111a
[20769.700171] [<ffffffff81118
[20769.700173] [<ffffffff8111a
[20769.700176] [<ffffffff8111a
[20769.700180] [<ffffffff81196
[20769.700183] [<ffffffff81196
[20769.700185] [<ffffffff81196
[20769.700187] [<ffffffff81197
[20769.700191] [<ffffffff815fb
[20769.700194] [<ffffffff81197
[20769.700196] [<ffffffff81197
[20769.700199] [<ffffffff81197
[20769.700203] [<ffffffff81084
[20769.700206] [<ffffffff81605
[20769.700208] [<ffffffff81084
[20769.700211] [<ffffffff81605
[20769.700212] ---[ end trace 787438b409e1b580 ]---
[2012-02-23 10:01:54] plackup[1101]: segfault at 404034 ip 0000000000404034 sp 00007fff61aad508 error 14 in perl[601000+1000] to_tree: 587: inode #135110: block 533749: comm grep: bad entry in directory: rec_len is smaller than minimal - offset=0(0), inode=4210740, rec_len=0, name_len=0 to_tree: 587: inode #135110: block 533749: comm grep: bad entry in directory: rec_len is smaller than minimal - offset=0(0), inode=4210740, rec_len=0, name_len=0 to_tree: 587: inode #135110: block 533749: comm grep: bad entry in directory: rec_len is smaller than minimal - offset=0(0), inode=4210740, rec_len=0, name_len=0 to_tree: 587: inode #135110: block 533749: comm grep: bad entry in directory: rec_len is smaller than minimal - offset=0(0), inode=4210740, rec_len=0, name_len=0
[2012-02-23 10:03:57] plackup[1092]: segfault at 404034 ip 0000000000404034 sp 00007fff61aad458 error 14 in perl[601000+1000]
[2012-02-23 10:09:53] EXT4-fs error (device vda): htree_dirblock_
[2012-02-23 10:09:53] Aborting journal on device vda-8.
[2012-02-23 10:09:53] EXT4-fs (vda): Remounting filesystem read-only
[2012-02-23 10:11:41] EXT4-fs error (device vda): htree_dirblock_
[2012-02-23 10:12:15] EXT4-fs error (device vda): htree_dirblock_
[2012-02-23 10:13:15] EXT4-fs error (device vda): htree_dirblock_
Interesting is inode 4210740 referred by all crashes (although I don't know why)