| 2011-01-24 06:29:34 |
Eugene San |
bug |
|
|
added bug |
| 2011-01-24 08:28:06 |
Eugene San |
description |
Terminated suspend causes ext4 corruptions which crashes kernel on mount attempt.
Affects Karmic, Lucid Maverick and probably Natty.
Valid for any combinations, for example : Suspend Lucid system -> Hard poweroff -> Connect Lucid's ext4 to Maverick system -> Observe Crash
Actually report describes two severe issues as one.
As immediate solution I propose:
1) Migration of filesystem recovery to initramfs-tools. Will help system to recover it self before root mounted.
2) Fix resume from suspend while bootdevice on USB.
Usually happens on systems with bootdevice on USB.
When system returns from suspend, there is a chance of ~25% for EXT4 on USB to become readonly with corrupted journal.
To fix affected EXT4, de-activation of auto-mount required or else you will observer as below:
[225221.546381] usb-storage: device found at 22
[225221.546386] usb-storage: waiting for device to settle before scanning
[225226.551130] usb-storage: device scan complete
[225226.555578] scsi 11:0:0:0: Direct-Access Generic Flash HS-CF 4.44 PQ: 0 ANSI: 0
[225226.558965] scsi 11:0:0:1: Direct-Access Generic Flash HS-COMBO 4.44 PQ: 0 ANSI: 0
[225226.560622] sd 11:0:0:0: Attached scsi generic sg2 type 0
[225226.562393] sd 11:0:0:1: Attached scsi generic sg3 type 0
[225226.622322] sd 11:0:0:0: [sdc] Attached SCSI removable disk
[225226.626721] sd 11:0:0:1: [sdd] Attached SCSI removable disk
[225233.188100] usb 1-2: new high speed USB device using ehci_hcd and address 5
[225233.323593] usb 1-2: configuration #1 chosen from 1 choice
[225233.324302] scsi12 : SCSI emulation for USB Mass Storage devices
[225233.324731] usb-storage: device found at 5
[225233.324741] usb-storage: waiting for device to settle before scanning
[225238.320747] usb-storage: device scan complete
[225238.322831] scsi 12:0:0:0: Direct-Access USB Mass Storage Device PQ: 0 ANSI: 0 CCS
[225238.324132] sd 12:0:0:0: Attached scsi generic sg4 type 0
[225238.325568] sd 12:0:0:0: [sde] 15954944 512-byte logical blocks: (8.16 GB/7.60 GiB)
[225238.326653] sd 12:0:0:0: [sde] Write Protect is off
[225238.326661] sd 12:0:0:0: [sde] Mode Sense: 03 00 00 00
[225238.326667] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.334023] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.334044] sde: sde1
[225238.338676] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.338693] sd 12:0:0:0: [sde] Attached SCSI removable disk
[225252.842616] EXT4-fs warning (device sde1): ext4_clear_journal_err: Filesystem error recorded from previous mount: IO failure
[225252.842629] EXT4-fs warning (device sde1): ext4_clear_journal_err: Marking fs in need of filesystem check.
[225252.842674] BUG: unable to handle kernel paging request at 02add000
[225252.842684] IP: [<c035c35a>] __percpu_counter_sum+0x2a/0x70
[225252.842701] *pde = 00000000
[225252.842707] Oops: 0000 [#1] SMP
[225252.842714] last sysfs file: /sys/devices/pci0000:00/0000:00:0b.0/host1/target1:0:0/1:0:0:0/block/sdb/sdb3/stat
[225252.842722] Modules linked in: btrfs zlib_deflate crc32c libcrc32c ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs reiserfs hidp aes_i586 aes_generic binfmt_misc ppdev rfcomm sco bridge stp bnep l2cap ip6table_filter ip6_tables iptable_filter ip_tables x_tables dm_crypt snd_hda_codec_nvhdmi snd_hda_codec_realtek nfsd exportfs snd_hda_intel nfs snd_hda_codec lockd snd_hwdep nfs_acl auth_rpcgss snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss lm90 snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq vga16fb sunrpc coretemp vgastate nvidia(P) snd_timer snd_seq_device snd agpgart soundcore btusb i2c_nforce2 gspca_zc3xx lp gspca_main snd_page_alloc videodev ftdi_sio shpchp parport v4l1_compat bluetooth usbserial usb_storage raid10 raid456 async_raid6_recov async_pq raid6_pq async_xor async_memcpy async_tx raid0 multipath linear dm_raid45 xor usbhid hid fbcon tileblit font bitblit softcursor vesafb raid1 ahci forcedeth ramzswap xvmalloc lzo_decompress lzo_compress
[225252.842886]
[225252.842894] Pid: 14666, comm: mount Tainted: P (2.6.32-28-generic #55-Ubuntu) To Be Filled By O.E.M.
[225252.842901] EIP: 0060:[<c035c35a>] EFLAGS: 00010297 CPU: 0
[225252.842909] EIP is at __percpu_counter_sum+0x2a/0x70
[225252.842914] EAX: 00000000 EBX: 00000000 ECX: 02add000 EDX: 00000000
[225252.842920] ESI: 00000000 EDI: ea39ec94 EBP: f0085d78 ESP: f0085d6c
[225252.842926] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[225252.842933] Process mount (pid: 14666, ti=f0084000 task=f02d5940 task.ti=f0084000)
[225252.842938] Stack:
[225252.842941] f5c4c000 03477fda 00000000 f0085dac c02a3856 e6984400 f0085d94 c058bf3e
[225252.842955] <0> 00000001 e6984400 03476c6a 00000000 f6e022c0 f5c4c000 dd309000 e6984400
[225252.842971] <0> f0085de4 c02a3a33 f5c4c000 c05a978d c06d59dc c06e4685 dd309000 f0085de4
[225252.842987] Call Trace:
[225252.842998] [<c02a3856>] ? ext4_commit_super+0xd6/0x1e0
[225252.843008] [<c058bf3e>] ? printk+0x1d/0x1f
[225252.843017] [<c02a3a33>] ? ext4_clear_journal_err+0xa3/0xd0
[225252.843026] [<c02cce5f>] ? jbd2_journal_load+0x4f/0xb0
[225252.843035] [<c02a5f0f>] ? ext4_load_journal+0x14f/0x2e0
[225252.843044] [<c02a71b6>] ? ext4_fill_super+0x1116/0x1930
[225252.843055] [<c020c172>] ? get_sb_bdev+0x162/0x1a0
[225252.843063] [<c02a60a0>] ? ext4_fill_super+0x0/0x1930
[225252.843072] [<c02a16f6>] ? ext4_get_sb+0x26/0x30
[225252.843080] [<c02a60a0>] ? ext4_fill_super+0x0/0x1930
[225252.843087] [<c020bce7>] ? vfs_kern_mount+0x67/0x170
[225252.843097] [<c0220623>] ? get_fs_type+0x33/0xb0
[225252.843105] [<c020be4e>] ? do_kern_mount+0x3e/0xe0
[225252.843113] [<c0223763>] ? do_mount+0x1c3/0x200
[225252.843120] [<c022380b>] ? sys_mount+0x6b/0xa0
[225252.843130] [<c01033ec>] ? syscall_call+0x7/0xb
[225252.843139] [<c0590000>] ? setjmp_pre_handler+0x30/0x90
[225252.843144] Code: 00 55 89 e5 57 89 c7 56 53 e8 23 24 23 00 8b 5f 04 b8 ff ff ff ff 8b 77 08 eb 1c 8d b6 00 00 00 00 8b 0c 85 00 3f 7a c0 8b 57 14 <8b> 14 0a 89 d1 c1 f9 1f 01 d3 11 ce 8d 48 01 a1 0c c0 59 c0 ba
[225252.843225] EIP: [<c035c35a>] __percpu_counter_sum+0x2a/0x70 SS:ESP 0068:f0085d6c
[225252.843236] CR2: 0000000002add000
[225252.843243] ---[ end trace 2bc710c0004faec8 ]--- |
Terminated suspend causes ext4 corruptions which crashes kernel on mount attempt.
Affects Karmic, Lucid Maverick and probably Natty.
Valid for any combinations, for example : Suspend Lucid system -> Hard poweroff -> Connect Lucid's ext4 to Maverick system -> Observe Crash
Actually report describes two severe issues as one.
As immediate solution I propose:
1) Migration of filesystem recovery to initramfs-tools. Will help system to recover it self before root mounted.
2) Fix resume from suspend while bootdevice on USB.
Usually happens on systems with bootdevice on USB.
When system returns from suspend, there is a chance of ~25% for EXT4 on USB to become readonly with corrupted journal.
To fix affected EXT4, de-activation of auto-mount required or else you will observer as below:
[225221.546381] usb-storage: device found at 22
[225221.546386] usb-storage: waiting for device to settle before scanning
[225226.551130] usb-storage: device scan complete
[225226.555578] scsi 11:0:0:0: Direct-Access Generic Flash HS-CF 4.44 PQ: 0 ANSI: 0
[225226.558965] scsi 11:0:0:1: Direct-Access Generic Flash HS-COMBO 4.44 PQ: 0 ANSI: 0
[225226.560622] sd 11:0:0:0: Attached scsi generic sg2 type 0
[225226.562393] sd 11:0:0:1: Attached scsi generic sg3 type 0
[225226.622322] sd 11:0:0:0: [sdc] Attached SCSI removable disk
[225226.626721] sd 11:0:0:1: [sdd] Attached SCSI removable disk
[225233.188100] usb 1-2: new high speed USB device using ehci_hcd and address 5
[225233.323593] usb 1-2: configuration #1 chosen from 1 choice
[225233.324302] scsi12 : SCSI emulation for USB Mass Storage devices
[225233.324731] usb-storage: device found at 5
[225233.324741] usb-storage: waiting for device to settle before scanning
[225238.320747] usb-storage: device scan complete
[225238.322831] scsi 12:0:0:0: Direct-Access USB Mass Storage Device PQ: 0 ANSI: 0 CCS
[225238.324132] sd 12:0:0:0: Attached scsi generic sg4 type 0
[225238.325568] sd 12:0:0:0: [sde] 15954944 512-byte logical blocks: (8.16 GB/7.60 GiB)
[225238.326653] sd 12:0:0:0: [sde] Write Protect is off
[225238.326661] sd 12:0:0:0: [sde] Mode Sense: 03 00 00 00
[225238.326667] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.334023] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.334044] sde: sde1
[225238.338676] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.338693] sd 12:0:0:0: [sde] Attached SCSI removable disk
[225252.842616] EXT4-fs warning (device sde1): ext4_clear_journal_err: Filesystem error recorded from previous mount: IO failure
[225252.842629] EXT4-fs warning (device sde1): ext4_clear_journal_err: Marking fs in need of filesystem check.
[225252.842674] BUG: unable to handle kernel paging request at 02add000
[225252.842684] IP: [<c035c35a>] __percpu_counter_sum+0x2a/0x70
[225252.842701] *pde = 00000000
[225252.842707] Oops: 0000 [#1] SMP
[225252.842714] last sysfs file: /sys/devices/pci0000:00/0000:00:0b.0/host1/target1:0:0/1:0:0:0/block/sdb/sdb3/stat
[225252.842722] Modules linked in: btrfs zlib_deflate crc32c libcrc32c ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs reiserfs hidp aes_i586 aes_generic binfmt_misc ppdev rfcomm sco bridge stp bnep l2cap ip6table_filter ip6_tables iptable_filter ip_tables x_tables dm_crypt snd_hda_codec_nvhdmi snd_hda_codec_realtek nfsd exportfs snd_hda_intel nfs snd_hda_codec lockd snd_hwdep nfs_acl auth_rpcgss snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss lm90 snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq vga16fb sunrpc coretemp vgastate nvidia(P) snd_timer snd_seq_device snd agpgart soundcore btusb i2c_nforce2 gspca_zc3xx lp gspca_main snd_page_alloc videodev ftdi_sio shpchp parport v4l1_compat bluetooth usbserial usb_storage raid10 raid456 async_raid6_recov async_pq raid6_pq async_xor async_memcpy async_tx raid0 multipath linear dm_raid45 xor usbhid hid fbcon tileblit font bitblit softcursor vesafb raid1 ahci forcedeth ramzswap xvmalloc lzo_decompress lzo_compress
[225252.842886]
[225252.842894] Pid: 14666, comm: mount Tainted: P (2.6.32-28-generic #55-Ubuntu) To Be Filled By O.E.M.
[225252.842901] EIP: 0060:[<c035c35a>] EFLAGS: 00010297 CPU: 0
[225252.842909] EIP is at __percpu_counter_sum+0x2a/0x70
[225252.842914] EAX: 00000000 EBX: 00000000 ECX: 02add000 EDX: 00000000
[225252.842920] ESI: 00000000 EDI: ea39ec94 EBP: f0085d78 ESP: f0085d6c
[225252.842926] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[225252.842933] Process mount (pid: 14666, ti=f0084000 task=f02d5940 task.ti=f0084000)
[225252.842938] Stack:
[225252.842941] f5c4c000 03477fda 00000000 f0085dac c02a3856 e6984400 f0085d94 c058bf3e
[225252.842955] <0> 00000001 e6984400 03476c6a 00000000 f6e022c0 f5c4c000 dd309000 e6984400
[225252.842971] <0> f0085de4 c02a3a33 f5c4c000 c05a978d c06d59dc c06e4685 dd309000 f0085de4
[225252.842987] Call Trace:
[225252.842998] [<c02a3856>] ? ext4_commit_super+0xd6/0x1e0
[225252.843008] [<c058bf3e>] ? printk+0x1d/0x1f
[225252.843017] [<c02a3a33>] ? ext4_clear_journal_err+0xa3/0xd0
[225252.843026] [<c02cce5f>] ? jbd2_journal_load+0x4f/0xb0
[225252.843035] [<c02a5f0f>] ? ext4_load_journal+0x14f/0x2e0
[225252.843044] [<c02a71b6>] ? ext4_fill_super+0x1116/0x1930
[225252.843055] [<c020c172>] ? get_sb_bdev+0x162/0x1a0
[225252.843063] [<c02a60a0>] ? ext4_fill_super+0x0/0x1930
[225252.843072] [<c02a16f6>] ? ext4_get_sb+0x26/0x30
[225252.843080] [<c02a60a0>] ? ext4_fill_super+0x0/0x1930
[225252.843087] [<c020bce7>] ? vfs_kern_mount+0x67/0x170
[225252.843097] [<c0220623>] ? get_fs_type+0x33/0xb0
[225252.843105] [<c020be4e>] ? do_kern_mount+0x3e/0xe0
[225252.843113] [<c0223763>] ? do_mount+0x1c3/0x200
[225252.843120] [<c022380b>] ? sys_mount+0x6b/0xa0
[225252.843130] [<c01033ec>] ? syscall_call+0x7/0xb
[225252.843139] [<c0590000>] ? setjmp_pre_handler+0x30/0x90
[225252.843144] Code: 00 55 89 e5 57 89 c7 56 53 e8 23 24 23 00 8b 5f 04 b8 ff ff ff ff 8b 77 08 eb 1c 8d b6 00 00 00 00 8b 0c 85 00 3f 7a c0 8b 57 14 <8b> 14 0a 89 d1 c1 f9 1f 01 d3 11 ce 8d 48 01 a1 0c c0 59 c0 ba
[225252.843225] EIP: [<c035c35a>] __percpu_counter_sum+0x2a/0x70 SS:ESP 0068:f0085d6c
[225252.843236] CR2: 0000000002add000
[225252.843243] ---[ end trace 2bc710c0004faec8 ]---
|
|
| 2011-01-24 08:37:10 |
Eugene San |
description |
Terminated suspend causes ext4 corruptions which crashes kernel on mount attempt.
Affects Karmic, Lucid Maverick and probably Natty.
Valid for any combinations, for example : Suspend Lucid system -> Hard poweroff -> Connect Lucid's ext4 to Maverick system -> Observe Crash
Actually report describes two severe issues as one.
As immediate solution I propose:
1) Migration of filesystem recovery to initramfs-tools. Will help system to recover it self before root mounted.
2) Fix resume from suspend while bootdevice on USB.
Usually happens on systems with bootdevice on USB.
When system returns from suspend, there is a chance of ~25% for EXT4 on USB to become readonly with corrupted journal.
To fix affected EXT4, de-activation of auto-mount required or else you will observer as below:
[225221.546381] usb-storage: device found at 22
[225221.546386] usb-storage: waiting for device to settle before scanning
[225226.551130] usb-storage: device scan complete
[225226.555578] scsi 11:0:0:0: Direct-Access Generic Flash HS-CF 4.44 PQ: 0 ANSI: 0
[225226.558965] scsi 11:0:0:1: Direct-Access Generic Flash HS-COMBO 4.44 PQ: 0 ANSI: 0
[225226.560622] sd 11:0:0:0: Attached scsi generic sg2 type 0
[225226.562393] sd 11:0:0:1: Attached scsi generic sg3 type 0
[225226.622322] sd 11:0:0:0: [sdc] Attached SCSI removable disk
[225226.626721] sd 11:0:0:1: [sdd] Attached SCSI removable disk
[225233.188100] usb 1-2: new high speed USB device using ehci_hcd and address 5
[225233.323593] usb 1-2: configuration #1 chosen from 1 choice
[225233.324302] scsi12 : SCSI emulation for USB Mass Storage devices
[225233.324731] usb-storage: device found at 5
[225233.324741] usb-storage: waiting for device to settle before scanning
[225238.320747] usb-storage: device scan complete
[225238.322831] scsi 12:0:0:0: Direct-Access USB Mass Storage Device PQ: 0 ANSI: 0 CCS
[225238.324132] sd 12:0:0:0: Attached scsi generic sg4 type 0
[225238.325568] sd 12:0:0:0: [sde] 15954944 512-byte logical blocks: (8.16 GB/7.60 GiB)
[225238.326653] sd 12:0:0:0: [sde] Write Protect is off
[225238.326661] sd 12:0:0:0: [sde] Mode Sense: 03 00 00 00
[225238.326667] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.334023] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.334044] sde: sde1
[225238.338676] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.338693] sd 12:0:0:0: [sde] Attached SCSI removable disk
[225252.842616] EXT4-fs warning (device sde1): ext4_clear_journal_err: Filesystem error recorded from previous mount: IO failure
[225252.842629] EXT4-fs warning (device sde1): ext4_clear_journal_err: Marking fs in need of filesystem check.
[225252.842674] BUG: unable to handle kernel paging request at 02add000
[225252.842684] IP: [<c035c35a>] __percpu_counter_sum+0x2a/0x70
[225252.842701] *pde = 00000000
[225252.842707] Oops: 0000 [#1] SMP
[225252.842714] last sysfs file: /sys/devices/pci0000:00/0000:00:0b.0/host1/target1:0:0/1:0:0:0/block/sdb/sdb3/stat
[225252.842722] Modules linked in: btrfs zlib_deflate crc32c libcrc32c ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs reiserfs hidp aes_i586 aes_generic binfmt_misc ppdev rfcomm sco bridge stp bnep l2cap ip6table_filter ip6_tables iptable_filter ip_tables x_tables dm_crypt snd_hda_codec_nvhdmi snd_hda_codec_realtek nfsd exportfs snd_hda_intel nfs snd_hda_codec lockd snd_hwdep nfs_acl auth_rpcgss snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss lm90 snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq vga16fb sunrpc coretemp vgastate nvidia(P) snd_timer snd_seq_device snd agpgart soundcore btusb i2c_nforce2 gspca_zc3xx lp gspca_main snd_page_alloc videodev ftdi_sio shpchp parport v4l1_compat bluetooth usbserial usb_storage raid10 raid456 async_raid6_recov async_pq raid6_pq async_xor async_memcpy async_tx raid0 multipath linear dm_raid45 xor usbhid hid fbcon tileblit font bitblit softcursor vesafb raid1 ahci forcedeth ramzswap xvmalloc lzo_decompress lzo_compress
[225252.842886]
[225252.842894] Pid: 14666, comm: mount Tainted: P (2.6.32-28-generic #55-Ubuntu) To Be Filled By O.E.M.
[225252.842901] EIP: 0060:[<c035c35a>] EFLAGS: 00010297 CPU: 0
[225252.842909] EIP is at __percpu_counter_sum+0x2a/0x70
[225252.842914] EAX: 00000000 EBX: 00000000 ECX: 02add000 EDX: 00000000
[225252.842920] ESI: 00000000 EDI: ea39ec94 EBP: f0085d78 ESP: f0085d6c
[225252.842926] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[225252.842933] Process mount (pid: 14666, ti=f0084000 task=f02d5940 task.ti=f0084000)
[225252.842938] Stack:
[225252.842941] f5c4c000 03477fda 00000000 f0085dac c02a3856 e6984400 f0085d94 c058bf3e
[225252.842955] <0> 00000001 e6984400 03476c6a 00000000 f6e022c0 f5c4c000 dd309000 e6984400
[225252.842971] <0> f0085de4 c02a3a33 f5c4c000 c05a978d c06d59dc c06e4685 dd309000 f0085de4
[225252.842987] Call Trace:
[225252.842998] [<c02a3856>] ? ext4_commit_super+0xd6/0x1e0
[225252.843008] [<c058bf3e>] ? printk+0x1d/0x1f
[225252.843017] [<c02a3a33>] ? ext4_clear_journal_err+0xa3/0xd0
[225252.843026] [<c02cce5f>] ? jbd2_journal_load+0x4f/0xb0
[225252.843035] [<c02a5f0f>] ? ext4_load_journal+0x14f/0x2e0
[225252.843044] [<c02a71b6>] ? ext4_fill_super+0x1116/0x1930
[225252.843055] [<c020c172>] ? get_sb_bdev+0x162/0x1a0
[225252.843063] [<c02a60a0>] ? ext4_fill_super+0x0/0x1930
[225252.843072] [<c02a16f6>] ? ext4_get_sb+0x26/0x30
[225252.843080] [<c02a60a0>] ? ext4_fill_super+0x0/0x1930
[225252.843087] [<c020bce7>] ? vfs_kern_mount+0x67/0x170
[225252.843097] [<c0220623>] ? get_fs_type+0x33/0xb0
[225252.843105] [<c020be4e>] ? do_kern_mount+0x3e/0xe0
[225252.843113] [<c0223763>] ? do_mount+0x1c3/0x200
[225252.843120] [<c022380b>] ? sys_mount+0x6b/0xa0
[225252.843130] [<c01033ec>] ? syscall_call+0x7/0xb
[225252.843139] [<c0590000>] ? setjmp_pre_handler+0x30/0x90
[225252.843144] Code: 00 55 89 e5 57 89 c7 56 53 e8 23 24 23 00 8b 5f 04 b8 ff ff ff ff 8b 77 08 eb 1c 8d b6 00 00 00 00 8b 0c 85 00 3f 7a c0 8b 57 14 <8b> 14 0a 89 d1 c1 f9 1f 01 d3 11 ce 8d 48 01 a1 0c c0 59 c0 ba
[225252.843225] EIP: [<c035c35a>] __percpu_counter_sum+0x2a/0x70 SS:ESP 0068:f0085d6c
[225252.843236] CR2: 0000000002add000
[225252.843243] ---[ end trace 2bc710c0004faec8 ]---
|
Terminated suspend causes ext4 corruptions which crashes kernel on mount attempt.
Affects Karmic, Lucid, Maverick and probably Natty.
Valid for any combinations, for example : Suspend Lucid system -> Poweroff (hard) Lucid system -> Connect Lucid's ext4 to Maverick system -> Observe Crash
Actually report describes two severe issues.
Usually happens on systems with bootdevice on USB.
When system returns from suspend, there is a chance of ~25% for EXT4 on USB to become readonly with corrupted journal.
Attempt to start affected EXT4 will result an useless initramfs prompt without ability to solve the issue locally.
To fix the issue additional Linux system required (with auto-mount de-activated!).
An attempt to mount affected EXT4 will results a crashed provided below.
As immediate solution I propose:
1) Migration of filesystem recovery to initramfs-tools. Will help system to recover it self before root mounted.
2) Fix resume from suspend while bootdevice on USB.
-------------------------------------------------------------------------------------
[225221.546381] usb-storage: device found at 22
[225221.546386] usb-storage: waiting for device to settle before scanning
[225226.551130] usb-storage: device scan complete
[225226.555578] scsi 11:0:0:0: Direct-Access Generic Flash HS-CF 4.44 PQ: 0 ANSI: 0
[225226.558965] scsi 11:0:0:1: Direct-Access Generic Flash HS-COMBO 4.44 PQ: 0 ANSI: 0
[225226.560622] sd 11:0:0:0: Attached scsi generic sg2 type 0
[225226.562393] sd 11:0:0:1: Attached scsi generic sg3 type 0
[225226.622322] sd 11:0:0:0: [sdc] Attached SCSI removable disk
[225226.626721] sd 11:0:0:1: [sdd] Attached SCSI removable disk
[225233.188100] usb 1-2: new high speed USB device using ehci_hcd and address 5
[225233.323593] usb 1-2: configuration #1 chosen from 1 choice
[225233.324302] scsi12 : SCSI emulation for USB Mass Storage devices
[225233.324731] usb-storage: device found at 5
[225233.324741] usb-storage: waiting for device to settle before scanning
[225238.320747] usb-storage: device scan complete
[225238.322831] scsi 12:0:0:0: Direct-Access USB Mass Storage Device PQ: 0 ANSI: 0 CCS
[225238.324132] sd 12:0:0:0: Attached scsi generic sg4 type 0
[225238.325568] sd 12:0:0:0: [sde] 15954944 512-byte logical blocks: (8.16 GB/7.60 GiB)
[225238.326653] sd 12:0:0:0: [sde] Write Protect is off
[225238.326661] sd 12:0:0:0: [sde] Mode Sense: 03 00 00 00
[225238.326667] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.334023] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.334044] sde: sde1
[225238.338676] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.338693] sd 12:0:0:0: [sde] Attached SCSI removable disk
[225252.842616] EXT4-fs warning (device sde1): ext4_clear_journal_err: Filesystem error recorded from previous mount: IO failure
[225252.842629] EXT4-fs warning (device sde1): ext4_clear_journal_err: Marking fs in need of filesystem check.
[225252.842674] BUG: unable to handle kernel paging request at 02add000
[225252.842684] IP: [<c035c35a>] __percpu_counter_sum+0x2a/0x70
[225252.842701] *pde = 00000000
[225252.842707] Oops: 0000 [#1] SMP
[225252.842714] last sysfs file: /sys/devices/pci0000:00/0000:00:0b.0/host1/target1:0:0/1:0:0:0/block/sdb/sdb3/stat
[225252.842722] Modules linked in: btrfs zlib_deflate crc32c libcrc32c ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs reiserfs hidp aes_i586 aes_generic binfmt_misc ppdev rfcomm sco bridge stp bnep l2cap ip6table_filter ip6_tables iptable_filter ip_tables x_tables dm_crypt snd_hda_codec_nvhdmi snd_hda_codec_realtek nfsd exportfs snd_hda_intel nfs snd_hda_codec lockd snd_hwdep nfs_acl auth_rpcgss snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss lm90 snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq vga16fb sunrpc coretemp vgastate nvidia(P) snd_timer snd_seq_device snd agpgart soundcore btusb i2c_nforce2 gspca_zc3xx lp gspca_main snd_page_alloc videodev ftdi_sio shpchp parport v4l1_compat bluetooth usbserial usb_storage raid10 raid456 async_raid6_recov async_pq raid6_pq async_xor async_memcpy async_tx raid0 multipath linear dm_raid45 xor usbhid hid fbcon tileblit font bitblit softcursor vesafb raid1 ahci forcedeth ramzswap xvmalloc lzo_decompress lzo_compress
[225252.842886]
[225252.842894] Pid: 14666, comm: mount Tainted: P (2.6.32-28-generic #55-Ubuntu) To Be Filled By O.E.M.
[225252.842901] EIP: 0060:[<c035c35a>] EFLAGS: 00010297 CPU: 0
[225252.842909] EIP is at __percpu_counter_sum+0x2a/0x70
[225252.842914] EAX: 00000000 EBX: 00000000 ECX: 02add000 EDX: 00000000
[225252.842920] ESI: 00000000 EDI: ea39ec94 EBP: f0085d78 ESP: f0085d6c
[225252.842926] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[225252.842933] Process mount (pid: 14666, ti=f0084000 task=f02d5940 task.ti=f0084000)
[225252.842938] Stack:
[225252.842941] f5c4c000 03477fda 00000000 f0085dac c02a3856 e6984400 f0085d94 c058bf3e
[225252.842955] <0> 00000001 e6984400 03476c6a 00000000 f6e022c0 f5c4c000 dd309000 e6984400
[225252.842971] <0> f0085de4 c02a3a33 f5c4c000 c05a978d c06d59dc c06e4685 dd309000 f0085de4
[225252.842987] Call Trace:
[225252.842998] [<c02a3856>] ? ext4_commit_super+0xd6/0x1e0
[225252.843008] [<c058bf3e>] ? printk+0x1d/0x1f
[225252.843017] [<c02a3a33>] ? ext4_clear_journal_err+0xa3/0xd0
[225252.843026] [<c02cce5f>] ? jbd2_journal_load+0x4f/0xb0
[225252.843035] [<c02a5f0f>] ? ext4_load_journal+0x14f/0x2e0
[225252.843044] [<c02a71b6>] ? ext4_fill_super+0x1116/0x1930
[225252.843055] [<c020c172>] ? get_sb_bdev+0x162/0x1a0
[225252.843063] [<c02a60a0>] ? ext4_fill_super+0x0/0x1930
[225252.843072] [<c02a16f6>] ? ext4_get_sb+0x26/0x30
[225252.843080] [<c02a60a0>] ? ext4_fill_super+0x0/0x1930
[225252.843087] [<c020bce7>] ? vfs_kern_mount+0x67/0x170
[225252.843097] [<c0220623>] ? get_fs_type+0x33/0xb0
[225252.843105] [<c020be4e>] ? do_kern_mount+0x3e/0xe0
[225252.843113] [<c0223763>] ? do_mount+0x1c3/0x200
[225252.843120] [<c022380b>] ? sys_mount+0x6b/0xa0
[225252.843130] [<c01033ec>] ? syscall_call+0x7/0xb
[225252.843139] [<c0590000>] ? setjmp_pre_handler+0x30/0x90
[225252.843144] Code: 00 55 89 e5 57 89 c7 56 53 e8 23 24 23 00 8b 5f 04 b8 ff ff ff ff 8b 77 08 eb 1c 8d b6 00 00 00 00 8b 0c 85 00 3f 7a c0 8b 57 14 <8b> 14 0a 89 d1 c1 f9 1f 01 d3 11 ce 8d 48 01 a1 0c c0 59 c0 ba
[225252.843225] EIP: [<c035c35a>] __percpu_counter_sum+0x2a/0x70 SS:ESP 0068:f0085d6c
[225252.843236] CR2: 0000000002add000
[225252.843243] ---[ end trace 2bc710c0004faec8 ]---
|
|
| 2011-01-25 15:28:53 |
Jeremy Foshee |
tags |
initramfs-tools |
initramfs-tools kernel-suspend |
|
| 2011-01-25 15:28:56 |
Jeremy Foshee |
tags |
initramfs-tools kernel-suspend |
initramfs-tools kernel-suspend needs-kernel-logs |
|
| 2011-01-25 15:29:00 |
Jeremy Foshee |
tags |
initramfs-tools kernel-suspend needs-kernel-logs |
initramfs-tools kernel-suspend needs-kernel-logs needs-upstream-testing |
|
| 2011-01-25 15:29:03 |
Jeremy Foshee |
tags |
initramfs-tools kernel-suspend needs-kernel-logs needs-upstream-testing |
initramfs-tools kernel-suspend kj-triage needs-kernel-logs needs-upstream-testing |
|
| 2011-01-25 15:29:08 |
Jeremy Foshee |
linux (Ubuntu): status |
New |
Incomplete |
|
| 2011-01-30 16:36:09 |
Eugene San |
summary |
Terminated suspend causes ext4 corruptions which crashes kernel on mount attempt |
Suspending with rootfs (mainly ext4) on USB, cause corruption which crashes kernel on mount attempt |
|
| 2011-01-30 16:37:13 |
Eugene San |
summary |
Suspending with rootfs (mainly ext4) on USB, cause corruption which crashes kernel on mount attempt |
[All releases] Suspending with rootfs (mainly ext4) on USB, cause silent corruptions which panics kernel on mount attempt. Making system unbootbale |
|
| 2011-01-30 16:37:24 |
Eugene San |
summary |
[All releases] Suspending with rootfs (mainly ext4) on USB, cause silent corruptions which panics kernel on mount attempt. Making system unbootbale |
[All releases] Suspending with rootfs (mainly ext4) on USB, cause silent corruptions which panics kernel on mount attempt. Making system unbootable |
|
| 2011-01-30 16:37:29 |
Eugene San |
summary |
[All releases] Suspending with rootfs (mainly ext4) on USB, cause silent corruptions which panics kernel on mount attempt. Making system unbootable |
[All releases] Suspending with rootfs (mainly ext4) on USB, cause silent corruptions which panics kernel on mount attempt. Making system unbootable. |
|
| 2011-01-30 16:37:51 |
Eugene San |
linux (Ubuntu): status |
Incomplete |
Confirmed |
|
| 2011-01-30 16:49:06 |
Eugene San |
summary |
[All releases] Suspending with rootfs (mainly ext4) on USB, cause silent corruptions which panics kernel on mount attempt. Making system unbootable. |
[All releases] Suspending with rootfs on USB, cause silent corruptions cause kernel panic on mount attempt, data loss and making system unbootable. |
|
| 2011-01-30 16:50:14 |
Eugene San |
summary |
[All releases] Suspending with rootfs on USB, cause silent corruptions cause kernel panic on mount attempt, data loss and making system unbootable. |
[All releases] Suspending with rootfs on USB, causes silent corruptions, kernel panic on mount attempt, data loss and leaving system unbootable. |
|
| 2011-01-30 16:55:15 |
Eugene San |
bug task added |
|
linux (Fedora) |
|
| 2011-01-30 18:39:51 |
Eugene San |
attachment added |
|
dmesg.log https://bugs.launchpad.net/ubuntu/+source/linux/+bug/706795/+attachment/1817996/+files/dmesg.log |
|
| 2011-01-30 18:59:39 |
Eugene San |
description |
Terminated suspend causes ext4 corruptions which crashes kernel on mount attempt.
Affects Karmic, Lucid, Maverick and probably Natty.
Valid for any combinations, for example : Suspend Lucid system -> Poweroff (hard) Lucid system -> Connect Lucid's ext4 to Maverick system -> Observe Crash
Actually report describes two severe issues.
Usually happens on systems with bootdevice on USB.
When system returns from suspend, there is a chance of ~25% for EXT4 on USB to become readonly with corrupted journal.
Attempt to start affected EXT4 will result an useless initramfs prompt without ability to solve the issue locally.
To fix the issue additional Linux system required (with auto-mount de-activated!).
An attempt to mount affected EXT4 will results a crashed provided below.
As immediate solution I propose:
1) Migration of filesystem recovery to initramfs-tools. Will help system to recover it self before root mounted.
2) Fix resume from suspend while bootdevice on USB.
-------------------------------------------------------------------------------------
[225221.546381] usb-storage: device found at 22
[225221.546386] usb-storage: waiting for device to settle before scanning
[225226.551130] usb-storage: device scan complete
[225226.555578] scsi 11:0:0:0: Direct-Access Generic Flash HS-CF 4.44 PQ: 0 ANSI: 0
[225226.558965] scsi 11:0:0:1: Direct-Access Generic Flash HS-COMBO 4.44 PQ: 0 ANSI: 0
[225226.560622] sd 11:0:0:0: Attached scsi generic sg2 type 0
[225226.562393] sd 11:0:0:1: Attached scsi generic sg3 type 0
[225226.622322] sd 11:0:0:0: [sdc] Attached SCSI removable disk
[225226.626721] sd 11:0:0:1: [sdd] Attached SCSI removable disk
[225233.188100] usb 1-2: new high speed USB device using ehci_hcd and address 5
[225233.323593] usb 1-2: configuration #1 chosen from 1 choice
[225233.324302] scsi12 : SCSI emulation for USB Mass Storage devices
[225233.324731] usb-storage: device found at 5
[225233.324741] usb-storage: waiting for device to settle before scanning
[225238.320747] usb-storage: device scan complete
[225238.322831] scsi 12:0:0:0: Direct-Access USB Mass Storage Device PQ: 0 ANSI: 0 CCS
[225238.324132] sd 12:0:0:0: Attached scsi generic sg4 type 0
[225238.325568] sd 12:0:0:0: [sde] 15954944 512-byte logical blocks: (8.16 GB/7.60 GiB)
[225238.326653] sd 12:0:0:0: [sde] Write Protect is off
[225238.326661] sd 12:0:0:0: [sde] Mode Sense: 03 00 00 00
[225238.326667] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.334023] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.334044] sde: sde1
[225238.338676] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.338693] sd 12:0:0:0: [sde] Attached SCSI removable disk
[225252.842616] EXT4-fs warning (device sde1): ext4_clear_journal_err: Filesystem error recorded from previous mount: IO failure
[225252.842629] EXT4-fs warning (device sde1): ext4_clear_journal_err: Marking fs in need of filesystem check.
[225252.842674] BUG: unable to handle kernel paging request at 02add000
[225252.842684] IP: [<c035c35a>] __percpu_counter_sum+0x2a/0x70
[225252.842701] *pde = 00000000
[225252.842707] Oops: 0000 [#1] SMP
[225252.842714] last sysfs file: /sys/devices/pci0000:00/0000:00:0b.0/host1/target1:0:0/1:0:0:0/block/sdb/sdb3/stat
[225252.842722] Modules linked in: btrfs zlib_deflate crc32c libcrc32c ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs reiserfs hidp aes_i586 aes_generic binfmt_misc ppdev rfcomm sco bridge stp bnep l2cap ip6table_filter ip6_tables iptable_filter ip_tables x_tables dm_crypt snd_hda_codec_nvhdmi snd_hda_codec_realtek nfsd exportfs snd_hda_intel nfs snd_hda_codec lockd snd_hwdep nfs_acl auth_rpcgss snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss lm90 snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq vga16fb sunrpc coretemp vgastate nvidia(P) snd_timer snd_seq_device snd agpgart soundcore btusb i2c_nforce2 gspca_zc3xx lp gspca_main snd_page_alloc videodev ftdi_sio shpchp parport v4l1_compat bluetooth usbserial usb_storage raid10 raid456 async_raid6_recov async_pq raid6_pq async_xor async_memcpy async_tx raid0 multipath linear dm_raid45 xor usbhid hid fbcon tileblit font bitblit softcursor vesafb raid1 ahci forcedeth ramzswap xvmalloc lzo_decompress lzo_compress
[225252.842886]
[225252.842894] Pid: 14666, comm: mount Tainted: P (2.6.32-28-generic #55-Ubuntu) To Be Filled By O.E.M.
[225252.842901] EIP: 0060:[<c035c35a>] EFLAGS: 00010297 CPU: 0
[225252.842909] EIP is at __percpu_counter_sum+0x2a/0x70
[225252.842914] EAX: 00000000 EBX: 00000000 ECX: 02add000 EDX: 00000000
[225252.842920] ESI: 00000000 EDI: ea39ec94 EBP: f0085d78 ESP: f0085d6c
[225252.842926] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[225252.842933] Process mount (pid: 14666, ti=f0084000 task=f02d5940 task.ti=f0084000)
[225252.842938] Stack:
[225252.842941] f5c4c000 03477fda 00000000 f0085dac c02a3856 e6984400 f0085d94 c058bf3e
[225252.842955] <0> 00000001 e6984400 03476c6a 00000000 f6e022c0 f5c4c000 dd309000 e6984400
[225252.842971] <0> f0085de4 c02a3a33 f5c4c000 c05a978d c06d59dc c06e4685 dd309000 f0085de4
[225252.842987] Call Trace:
[225252.842998] [<c02a3856>] ? ext4_commit_super+0xd6/0x1e0
[225252.843008] [<c058bf3e>] ? printk+0x1d/0x1f
[225252.843017] [<c02a3a33>] ? ext4_clear_journal_err+0xa3/0xd0
[225252.843026] [<c02cce5f>] ? jbd2_journal_load+0x4f/0xb0
[225252.843035] [<c02a5f0f>] ? ext4_load_journal+0x14f/0x2e0
[225252.843044] [<c02a71b6>] ? ext4_fill_super+0x1116/0x1930
[225252.843055] [<c020c172>] ? get_sb_bdev+0x162/0x1a0
[225252.843063] [<c02a60a0>] ? ext4_fill_super+0x0/0x1930
[225252.843072] [<c02a16f6>] ? ext4_get_sb+0x26/0x30
[225252.843080] [<c02a60a0>] ? ext4_fill_super+0x0/0x1930
[225252.843087] [<c020bce7>] ? vfs_kern_mount+0x67/0x170
[225252.843097] [<c0220623>] ? get_fs_type+0x33/0xb0
[225252.843105] [<c020be4e>] ? do_kern_mount+0x3e/0xe0
[225252.843113] [<c0223763>] ? do_mount+0x1c3/0x200
[225252.843120] [<c022380b>] ? sys_mount+0x6b/0xa0
[225252.843130] [<c01033ec>] ? syscall_call+0x7/0xb
[225252.843139] [<c0590000>] ? setjmp_pre_handler+0x30/0x90
[225252.843144] Code: 00 55 89 e5 57 89 c7 56 53 e8 23 24 23 00 8b 5f 04 b8 ff ff ff ff 8b 77 08 eb 1c 8d b6 00 00 00 00 8b 0c 85 00 3f 7a c0 8b 57 14 <8b> 14 0a 89 d1 c1 f9 1f 01 d3 11 ce 8d 48 01 a1 0c c0 59 c0 ba
[225252.843225] EIP: [<c035c35a>] __percpu_counter_sum+0x2a/0x70 SS:ESP 0068:f0085d6c
[225252.843236] CR2: 0000000002add000
[225252.843243] ---[ end trace 2bc710c0004faec8 ]---
|
Suspend of root filesystem on USB, causes FS corruptions which crashes kernel on mount attempt and making systen unbootable.
Affects Karmic, Lucid, Maverick and Natty.
Reconstruction:
---------------
a) Install any Ubuntu release on USB drive.
b) Boot into user session (preferably with ecryptfs enabled, which raises probably to almost always)
c) Suspend
d) Variant1: Resume -> Observer FS inconsistence and inability to continue operation with rootfs.
Afer hard reset (system unable to reboot/shutdown normally with readonly rootfs) the system is unable to boot due silently corrupted FS.
Variant2: Eject device -> Try to mount it on any Ubuntu release -> Observe Kernel panic.
Variant3: Hard reset the system (Before performing resume) -> Observe system inability to boot due to corrupted FS.
Actually report describes two severe issues:
--------------------------------------------
a) FS corruption during suspend
b) Kernel panic when trying to mount affected FS.
Facts:
------
*) Happens on systems with bootdevice on USB.
*) When system returns from suspend, there is a chance of ~25% for rootfs on USB to be remounted as readonly. Due to short inability to read from USB device (device settle delay required)
*) Attempt to start affected FS will result an useless initramfs prompt without ability to solve the issue locally.
*) To fix the issue additional Linux system required (with auto-mount de-activated!).
*) An attempt to mount affected EXT4 will results a crashed provided below.
Corruptions are silent, FS marked as clean, that's what causes Kernel panic when it tries to mount corrupted FS marked as clean.
*) Issue caused probably by partially synced disk cache.
*) All tested Kernels affected (Lucid, Maverick, Natty, Natty-Mainline-Daily, Fedora14, Redhat6).
*) Validated with: Vanilla Lucid, Maverick, Natty, Fedora14 and RedHat6.
*) Kernels above 2.6.38 manages to mark FS as dirty during failed mount attempt, so virtually corruption got fixed after 2 reboots but loss is there.
*) Bug affects all filesystems, but severe for EXT4(data loss while fixable with another system without automount) and BTRFS (unfixable due to failed mounttime fixes and there is no way to fix manually).
As immediate solution I propose:
--------------------------------
1) Fix suspend/resume whith bootdevice on USB.
2) Migration of filesystem recovery to initramfs-tools. Will help system to recover before corrupted rootfs mounted.
Below log of attempt to mount corrupted FS:
-------------------------------------------------------------------------------------
[225221.546381] usb-storage: device found at 22
[225221.546386] usb-storage: waiting for device to settle before scanning
[225226.551130] usb-storage: device scan complete
[225226.555578] scsi 11:0:0:0: Direct-Access Generic Flash HS-CF 4.44 PQ: 0 ANSI: 0
[225226.558965] scsi 11:0:0:1: Direct-Access Generic Flash HS-COMBO 4.44 PQ: 0 ANSI: 0
[225226.560622] sd 11:0:0:0: Attached scsi generic sg2 type 0
[225226.562393] sd 11:0:0:1: Attached scsi generic sg3 type 0
[225226.622322] sd 11:0:0:0: [sdc] Attached SCSI removable disk
[225226.626721] sd 11:0:0:1: [sdd] Attached SCSI removable disk
[225233.188100] usb 1-2: new high speed USB device using ehci_hcd and address 5
[225233.323593] usb 1-2: configuration #1 chosen from 1 choice
[225233.324302] scsi12 : SCSI emulation for USB Mass Storage devices
[225233.324731] usb-storage: device found at 5
[225233.324741] usb-storage: waiting for device to settle before scanning
[225238.320747] usb-storage: device scan complete
[225238.322831] scsi 12:0:0:0: Direct-Access USB Mass Storage Device PQ: 0 ANSI: 0 CCS
[225238.324132] sd 12:0:0:0: Attached scsi generic sg4 type 0
[225238.325568] sd 12:0:0:0: [sde] 15954944 512-byte logical blocks: (8.16 GB/7.60 GiB)
[225238.326653] sd 12:0:0:0: [sde] Write Protect is off
[225238.326661] sd 12:0:0:0: [sde] Mode Sense: 03 00 00 00
[225238.326667] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.334023] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.334044] sde: sde1
[225238.338676] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.338693] sd 12:0:0:0: [sde] Attached SCSI removable disk
[225252.842616] EXT4-fs warning (device sde1): ext4_clear_journal_err: Filesystem error recorded from previous mount: IO failure
[225252.842629] EXT4-fs warning (device sde1): ext4_clear_journal_err: Marking fs in need of filesystem check.
[225252.842674] BUG: unable to handle kernel paging request at 02add000
[225252.842684] IP: [<c035c35a>] __percpu_counter_sum+0x2a/0x70
[225252.842701] *pde = 00000000
[225252.842707] Oops: 0000 [#1] SMP
[225252.842714] last sysfs file: /sys/devices/pci0000:00/0000:00:0b.0/host1/target1:0:0/1:0:0:0/block/sdb/sdb3/stat
[225252.842722] Modules linked in: btrfs zlib_deflate crc32c libcrc32c ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs reiserfs hidp aes_i586 aes_generic binfmt_misc ppdev rfcomm sco bridge stp bnep l2cap ip6table_filter ip6_tables iptable_filter ip_tables x_tables dm_crypt snd_hda_codec_nvhdmi snd_hda_codec_realtek nfsd exportfs snd_hda_intel nfs snd_hda_codec lockd snd_hwdep nfs_acl auth_rpcgss snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss lm90 snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq vga16fb sunrpc coretemp vgastate nvidia(P) snd_timer snd_seq_device snd agpgart soundcore btusb i2c_nforce2 gspca_zc3xx lp gspca_main snd_page_alloc videodev ftdi_sio shpchp parport v4l1_compat bluetooth usbserial usb_storage raid10 raid456 async_raid6_recov async_pq raid6_pq async_xor async_memcpy async_tx raid0 multipath linear dm_raid45 xor usbhid hid fbcon tileblit font bitblit softcursor vesafb raid1 ahci forcedeth ramzswap xvmalloc lzo_decompress lzo_compress
[225252.842886]
[225252.842894] Pid: 14666, comm: mount Tainted: P (2.6.32-28-generic #55-Ubuntu) To Be Filled By O.E.M.
[225252.842901] EIP: 0060:[<c035c35a>] EFLAGS: 00010297 CPU: 0
[225252.842909] EIP is at __percpu_counter_sum+0x2a/0x70
[225252.842914] EAX: 00000000 EBX: 00000000 ECX: 02add000 EDX: 00000000
[225252.842920] ESI: 00000000 EDI: ea39ec94 EBP: f0085d78 ESP: f0085d6c
[225252.842926] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[225252.842933] Process mount (pid: 14666, ti=f0084000 task=f02d5940 task.ti=f0084000)
[225252.842938] Stack:
[225252.842941] f5c4c000 03477fda 00000000 f0085dac c02a3856 e6984400 f0085d94 c058bf3e
[225252.842955] <0> 00000001 e6984400 03476c6a 00000000 f6e022c0 f5c4c000 dd309000 e6984400
[225252.842971] <0> f0085de4 c02a3a33 f5c4c000 c05a978d c06d59dc c06e4685 dd309000 f0085de4
[225252.842987] Call Trace:
[225252.842998] [<c02a3856>] ? ext4_commit_super+0xd6/0x1e0
[225252.843008] [<c058bf3e>] ? printk+0x1d/0x1f
[225252.843017] [<c02a3a33>] ? ext4_clear_journal_err+0xa3/0xd0
[225252.843026] [<c02cce5f>] ? jbd2_journal_load+0x4f/0xb0
[225252.843035] [<c02a5f0f>] ? ext4_load_journal+0x14f/0x2e0
[225252.843044] [<c02a71b6>] ? ext4_fill_super+0x1116/0x1930
[225252.843055] [<c020c172>] ? get_sb_bdev+0x162/0x1a0
[225252.843063] [<c02a60a0>] ? ext4_fill_super+0x0/0x1930
[225252.843072] [<c02a16f6>] ? ext4_get_sb+0x26/0x30
[225252.843080] [<c02a60a0>] ? ext4_fill_super+0x0/0x1930
[225252.843087] [<c020bce7>] ? vfs_kern_mount+0x67/0x170
[225252.843097] [<c0220623>] ? get_fs_type+0x33/0xb0
[225252.843105] [<c020be4e>] ? do_kern_mount+0x3e/0xe0
[225252.843113] [<c0223763>] ? do_mount+0x1c3/0x200
[225252.843120] [<c022380b>] ? sys_mount+0x6b/0xa0
[225252.843130] [<c01033ec>] ? syscall_call+0x7/0xb
[225252.843139] [<c0590000>] ? setjmp_pre_handler+0x30/0x90
[225252.843144] Code: 00 55 89 e5 57 89 c7 56 53 e8 23 24 23 00 8b 5f 04 b8 ff ff ff ff 8b 77 08 eb 1c 8d b6 00 00 00 00 8b 0c 85 00 3f 7a c0 8b 57 14 <8b> 14 0a 89 d1 c1 f9 1f 01 d3 11 ce 8d 48 01 a1 0c c0 59 c0 ba
[225252.843225] EIP: [<c035c35a>] __percpu_counter_sum+0x2a/0x70 SS:ESP 0068:f0085d6c
[225252.843236] CR2: 0000000002add000
[225252.843243] ---[ end trace 2bc710c0004faec8 ]---
|
|
| 2011-01-30 21:11:51 |
Eugene San |
tags |
initramfs-tools kernel-suspend kj-triage needs-kernel-logs needs-upstream-testing |
initramfs-tools kernel-suspend kj-triage |
|
| 2011-02-25 17:47:13 |
Eugene San |
attachment added |
|
suspend resume cycle with debug https://bugs.launchpad.net/ubuntu/+source/linux/+bug/706795/+attachment/1871245/+files/mydebug-debug-bad.log |
|
| 2011-02-25 17:49:59 |
Eugene San |
attachment added |
|
reconstruction script https://bugs.launchpad.net/ubuntu/+source/linux/+bug/706795/+attachment/1871298/+files/test.sh |
|
| 2011-02-25 18:28:49 |
Eugene San |
summary |
[All releases] Suspending with rootfs on USB, causes silent corruptions, kernel panic on mount attempt, data loss and leaving system unbootable. |
[All releases] Suspend/Resume with rootfs on USB, causes filesystem corruptions and kernel panic on mount attempt, leaving system unbootable with data lost. |
|
| 2011-02-25 18:29:41 |
Eugene San |
description |
Suspend of root filesystem on USB, causes FS corruptions which crashes kernel on mount attempt and making systen unbootable.
Affects Karmic, Lucid, Maverick and Natty.
Reconstruction:
---------------
a) Install any Ubuntu release on USB drive.
b) Boot into user session (preferably with ecryptfs enabled, which raises probably to almost always)
c) Suspend
d) Variant1: Resume -> Observer FS inconsistence and inability to continue operation with rootfs.
Afer hard reset (system unable to reboot/shutdown normally with readonly rootfs) the system is unable to boot due silently corrupted FS.
Variant2: Eject device -> Try to mount it on any Ubuntu release -> Observe Kernel panic.
Variant3: Hard reset the system (Before performing resume) -> Observe system inability to boot due to corrupted FS.
Actually report describes two severe issues:
--------------------------------------------
a) FS corruption during suspend
b) Kernel panic when trying to mount affected FS.
Facts:
------
*) Happens on systems with bootdevice on USB.
*) When system returns from suspend, there is a chance of ~25% for rootfs on USB to be remounted as readonly. Due to short inability to read from USB device (device settle delay required)
*) Attempt to start affected FS will result an useless initramfs prompt without ability to solve the issue locally.
*) To fix the issue additional Linux system required (with auto-mount de-activated!).
*) An attempt to mount affected EXT4 will results a crashed provided below.
Corruptions are silent, FS marked as clean, that's what causes Kernel panic when it tries to mount corrupted FS marked as clean.
*) Issue caused probably by partially synced disk cache.
*) All tested Kernels affected (Lucid, Maverick, Natty, Natty-Mainline-Daily, Fedora14, Redhat6).
*) Validated with: Vanilla Lucid, Maverick, Natty, Fedora14 and RedHat6.
*) Kernels above 2.6.38 manages to mark FS as dirty during failed mount attempt, so virtually corruption got fixed after 2 reboots but loss is there.
*) Bug affects all filesystems, but severe for EXT4(data loss while fixable with another system without automount) and BTRFS (unfixable due to failed mounttime fixes and there is no way to fix manually).
As immediate solution I propose:
--------------------------------
1) Fix suspend/resume whith bootdevice on USB.
2) Migration of filesystem recovery to initramfs-tools. Will help system to recover before corrupted rootfs mounted.
Below log of attempt to mount corrupted FS:
-------------------------------------------------------------------------------------
[225221.546381] usb-storage: device found at 22
[225221.546386] usb-storage: waiting for device to settle before scanning
[225226.551130] usb-storage: device scan complete
[225226.555578] scsi 11:0:0:0: Direct-Access Generic Flash HS-CF 4.44 PQ: 0 ANSI: 0
[225226.558965] scsi 11:0:0:1: Direct-Access Generic Flash HS-COMBO 4.44 PQ: 0 ANSI: 0
[225226.560622] sd 11:0:0:0: Attached scsi generic sg2 type 0
[225226.562393] sd 11:0:0:1: Attached scsi generic sg3 type 0
[225226.622322] sd 11:0:0:0: [sdc] Attached SCSI removable disk
[225226.626721] sd 11:0:0:1: [sdd] Attached SCSI removable disk
[225233.188100] usb 1-2: new high speed USB device using ehci_hcd and address 5
[225233.323593] usb 1-2: configuration #1 chosen from 1 choice
[225233.324302] scsi12 : SCSI emulation for USB Mass Storage devices
[225233.324731] usb-storage: device found at 5
[225233.324741] usb-storage: waiting for device to settle before scanning
[225238.320747] usb-storage: device scan complete
[225238.322831] scsi 12:0:0:0: Direct-Access USB Mass Storage Device PQ: 0 ANSI: 0 CCS
[225238.324132] sd 12:0:0:0: Attached scsi generic sg4 type 0
[225238.325568] sd 12:0:0:0: [sde] 15954944 512-byte logical blocks: (8.16 GB/7.60 GiB)
[225238.326653] sd 12:0:0:0: [sde] Write Protect is off
[225238.326661] sd 12:0:0:0: [sde] Mode Sense: 03 00 00 00
[225238.326667] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.334023] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.334044] sde: sde1
[225238.338676] sd 12:0:0:0: [sde] Assuming drive cache: write through
[225238.338693] sd 12:0:0:0: [sde] Attached SCSI removable disk
[225252.842616] EXT4-fs warning (device sde1): ext4_clear_journal_err: Filesystem error recorded from previous mount: IO failure
[225252.842629] EXT4-fs warning (device sde1): ext4_clear_journal_err: Marking fs in need of filesystem check.
[225252.842674] BUG: unable to handle kernel paging request at 02add000
[225252.842684] IP: [<c035c35a>] __percpu_counter_sum+0x2a/0x70
[225252.842701] *pde = 00000000
[225252.842707] Oops: 0000 [#1] SMP
[225252.842714] last sysfs file: /sys/devices/pci0000:00/0000:00:0b.0/host1/target1:0:0/1:0:0:0/block/sdb/sdb3/stat
[225252.842722] Modules linked in: btrfs zlib_deflate crc32c libcrc32c ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs reiserfs hidp aes_i586 aes_generic binfmt_misc ppdev rfcomm sco bridge stp bnep l2cap ip6table_filter ip6_tables iptable_filter ip_tables x_tables dm_crypt snd_hda_codec_nvhdmi snd_hda_codec_realtek nfsd exportfs snd_hda_intel nfs snd_hda_codec lockd snd_hwdep nfs_acl auth_rpcgss snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss lm90 snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq vga16fb sunrpc coretemp vgastate nvidia(P) snd_timer snd_seq_device snd agpgart soundcore btusb i2c_nforce2 gspca_zc3xx lp gspca_main snd_page_alloc videodev ftdi_sio shpchp parport v4l1_compat bluetooth usbserial usb_storage raid10 raid456 async_raid6_recov async_pq raid6_pq async_xor async_memcpy async_tx raid0 multipath linear dm_raid45 xor usbhid hid fbcon tileblit font bitblit softcursor vesafb raid1 ahci forcedeth ramzswap xvmalloc lzo_decompress lzo_compress
[225252.842886]
[225252.842894] Pid: 14666, comm: mount Tainted: P (2.6.32-28-generic #55-Ubuntu) To Be Filled By O.E.M.
[225252.842901] EIP: 0060:[<c035c35a>] EFLAGS: 00010297 CPU: 0
[225252.842909] EIP is at __percpu_counter_sum+0x2a/0x70
[225252.842914] EAX: 00000000 EBX: 00000000 ECX: 02add000 EDX: 00000000
[225252.842920] ESI: 00000000 EDI: ea39ec94 EBP: f0085d78 ESP: f0085d6c
[225252.842926] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[225252.842933] Process mount (pid: 14666, ti=f0084000 task=f02d5940 task.ti=f0084000)
[225252.842938] Stack:
[225252.842941] f5c4c000 03477fda 00000000 f0085dac c02a3856 e6984400 f0085d94 c058bf3e
[225252.842955] <0> 00000001 e6984400 03476c6a 00000000 f6e022c0 f5c4c000 dd309000 e6984400
[225252.842971] <0> f0085de4 c02a3a33 f5c4c000 c05a978d c06d59dc c06e4685 dd309000 f0085de4
[225252.842987] Call Trace:
[225252.842998] [<c02a3856>] ? ext4_commit_super+0xd6/0x1e0
[225252.843008] [<c058bf3e>] ? printk+0x1d/0x1f
[225252.843017] [<c02a3a33>] ? ext4_clear_journal_err+0xa3/0xd0
[225252.843026] [<c02cce5f>] ? jbd2_journal_load+0x4f/0xb0
[225252.843035] [<c02a5f0f>] ? ext4_load_journal+0x14f/0x2e0
[225252.843044] [<c02a71b6>] ? ext4_fill_super+0x1116/0x1930
[225252.843055] [<c020c172>] ? get_sb_bdev+0x162/0x1a0
[225252.843063] [<c02a60a0>] ? ext4_fill_super+0x0/0x1930
[225252.843072] [<c02a16f6>] ? ext4_get_sb+0x26/0x30
[225252.843080] [<c02a60a0>] ? ext4_fill_super+0x0/0x1930
[225252.843087] [<c020bce7>] ? vfs_kern_mount+0x67/0x170
[225252.843097] [<c0220623>] ? get_fs_type+0x33/0xb0
[225252.843105] [<c020be4e>] ? do_kern_mount+0x3e/0xe0
[225252.843113] [<c0223763>] ? do_mount+0x1c3/0x200
[225252.843120] [<c022380b>] ? sys_mount+0x6b/0xa0
[225252.843130] [<c01033ec>] ? syscall_call+0x7/0xb
[225252.843139] [<c0590000>] ? setjmp_pre_handler+0x30/0x90
[225252.843144] Code: 00 55 89 e5 57 89 c7 56 53 e8 23 24 23 00 8b 5f 04 b8 ff ff ff ff 8b 77 08 eb 1c 8d b6 00 00 00 00 8b 0c 85 00 3f 7a c0 8b 57 14 <8b> 14 0a 89 d1 c1 f9 1f 01 d3 11 ce 8d 48 01 a1 0c c0 59 c0 ba
[225252.843225] EIP: [<c035c35a>] __percpu_counter_sum+0x2a/0x70 SS:ESP 0068:f0085d6c
[225252.843236] CR2: 0000000002add000
[225252.843243] ---[ end trace 2bc710c0004faec8 ]---
|
Suspend of root filesystem on USB, causes FS corruptions which crashes kernel
on mount attempt and making systen unbootable.
Affects (at least) RedHat6.0+, Fedora13+ and Ubuntu10+.
Reconstruction (Full):
---------------
a) Install any RedHat/Fedora/Ubuntu release on USB drive.
b) Boot into user session
c) Suspend
d) Resume
e) Repeat c,d (up to 4 times) until kernel informs about filesystem corruption.
f) Reset the system -> Observe system inability to boot and kernel panic due to
corrupted FS.
Reconstruction (Short):
---------------
a) Install any RedHat/Fedora/Ubuntu release on USB drive.
b) Boot into user session
c) Suspend
d) Hard reset the system (Before performing resume) -> Observe system inability
to boot and kernel panic due to corrupted FS.
Actually report describes two severe issues:
--------------------------------------------
a) FS corruption during suspend/resume
b) Kernel panic when trying to mount affected FS.
Facts:
------
*) Happens on systems with bootdevice on USB.
*) When system returns from suspend, there is a chance of ~25% for rootfs on
USB to be remounted as readonly. Due to short inability to read from USB device
(device settle delay required)
*) Attempt to start affected FS will result an useless initramfs prompt without
ability to solve the issue locally.
*) To fix the issue additional Linux system required (with auto-mount
de-activated!).
*) An attempt to mount affected EXT4 will results a crashed provided below.
Corruptions are silent, FS marked as clean, that's what causes Kernel panic
when it tries to mount corrupted FS marked as clean.
*) Issue caused probably by partially synced disk cache.
*) All tested Kernels affected (Lucid, Maverick, Natty, Natty-Mainline-Daily,
Fedora14, Redhat6).
*) Validated with: Mainline, Lucid, Maverick, Natty, Fedora14 and RedHat6.
*) Kernels above 2.6.38 manages to mark FS as dirty during failed mount
attempt, so virtually corruption got fixed after 2 reboots but loss is there.
*) Bug affects all filesystems, but severe for EXT4(data loss while fixable
with another system without automount) and BTRFS (unfixable due to failed
mounttime fixes and there is no way to fix manually). |
|
| 2011-02-25 20:43:59 |
filip |
bug |
|
|
added subscriber filip |
| 2011-03-26 07:16:23 |
Hendrik van Wyk |
bug |
|
|
added subscriber Hendrik van Wyk |
| 2011-09-12 21:49:31 |
Brad Figg |
tags |
initramfs-tools kernel-suspend kj-triage |
initramfs-tools kernel-suspend kj-triage maverick |
|
| 2012-01-01 08:16:16 |
Matthew Dutton |
bug |
|
|
added subscriber Matthew Dutton |
| 2012-07-26 14:18:04 |
penalvch |
tags |
initramfs-tools kernel-suspend kj-triage maverick |
initramfs-tools kernel-suspend kj-triage maverick needs-upstream-testing |
|
| 2012-07-26 14:18:16 |
penalvch |
tags |
initramfs-tools kernel-suspend kj-triage maverick needs-upstream-testing |
initramfs-tools kernel-suspend kj-triage lucid maverick natty needs-upstream-testing |
|
| 2012-07-26 14:19:06 |
penalvch |
linux (Ubuntu): importance |
Undecided |
Medium |
|
| 2012-07-26 14:19:06 |
penalvch |
linux (Ubuntu): status |
Confirmed |
Incomplete |
|
