HDD rename after 09/12 update (incl. GCC and glibc) caused boot failure

[gobble]

Hello
I would like to report a catastrophic failure after the 09/12 update. I have tried to recollect and list the exact sequence of events that occurred:
1.Update manager prompted me for updates to GCC and Glibc etc (as I recall from memory) on 09/12. Installed it and shutdown.
2.On 09/13, Boot failed and Ubuntu jumped to root prompt. After some exploring, I found that my two HDD had been renamed from sda and sdb to sdc and sdd. ( I made no bios changes).
3. I recalled seeing some fsck errors that scrolled by the screen so I ran fsck -y on sdc3but it could not auto repair
4. I mounted the sdc3 root partition (formerly sda3)
5. Tried to run vi /etc/fstab to change some entries but since PATH was not set ( it was only /bin:/sbin, I had to set it from cmdline). I located the path to vi and exported the PATH variable adding /usr/local/bin and sbin folders.
5. Tried to run vi but got command not found
6. ls command now showed NO files on the entire partition except lost+found!!
7. Booting from the install CD, df -h shows there is still data on the mounted partition with correct free/used stats: /dev/sda3 46G 36G 7.6G 83% /media

So it seems there were two problems: HDDs were reordered. fsck blew my inode tables.

Here is error fsck reports :
Group 130's block bitmap (4259840) is bad. Relocate? yes
Group 130's inode bitmap (4259841) is bad. Relocate? yes
Inodes that were part of corrupted orphan linked list found. Fix? Yes
/dev/sda3: ********** WARNING: Filesystem still has errors **********
/dev/sda3: 228946/3006464 files (2.0% non-contiguous), 9439856/12024652 blocks

I ran fsck a few times and now the messaged about orphan linked list does not appear.
------------
mke2fs 1.40.8 (13-Mar-2008)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
3006464 inodes, 12024652 blocks
601232 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
367 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
 4096000, 7962624, 11239424
------------

debugfs stats output:

Filesystem volume name: <none>
Last mounted on: <not available>
Filesystem UUID: cef9ef57-ebda-4be3-ad2a-995cc9ad6706
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype sparse_super large_file
Filesystem flags: signed_directory_hash
Default mount options: (none)
Filesystem state: not clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 3006464
Block count: 12024652
Reserved block count: 601232
Free blocks: 11883833
Free inodes: 3006453
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 1021
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 256
Filesystem created: Sun Jun 29 13:49:34 2008
Last mount time: n/a
Last write time: Sun Jun 29 13:49:49 2008
Mount count: 0
Maximum mount count: 39
Last checked: Sun Jun 29 13:49:34 2008
Check interval: 15552000 (6 months)
Next check after: Fri Dec 26 13:49:34 2008
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 128
Journal inode: 8
Default directory hash: tea
Directory Hash Seed: 40a8a3a2-1db1-4b58-8b3d-58d3add25fbf
Journal backup: inode blocks
Directories: 2

 Group 130: block bitmap at 4259840, inode bitmap at 4259841, inode table at 4259842
           32510 free blocks, 8192 free inodes, 0 used directories

------------------------

debugfs: icheck 130
Block Inode number
130 7

cd /
ls
2 (12) . 2 (12) .. 11 (4072) lost+found

I iterated thru most of the copies of the superblock, but no luck. There aren't any files present.

 open -s 163840 -b4096 /dev/sda3
 open -s 229376 -b4096 /dev/sda3

etc...

Since the directory structure is empty, I cannot debugfs:rdump either.

So now we have a situation where df reports filesystem 83% full, but the inode tables are empty.

I have no log files to give you since I cannot locate any on the partition, but you may find my other earlier CRs with dmesg, and other logs. system was Hardy Heron with Kernel 2.6.24.21.

So now I not only need a fix and also steps to recover the data on the partition. I do not want to re-install and lose GB of critical data from my home folder.

TIA
Let me know if you need any further information.

Regards

[gobble (gobbledegeek)]

PS: The /dev/sda3 you see here is from the Ubuntu 8.0.4 boot/install CD not the installed one that got blown up yesterday.
Just so there is no confusion.
Cheers

[gobble (gobbledegeek)]

OK I rdumped the lost+found, and it seems all the files are in there - some 36GB. Will work on recovery for each block/inode, lets not make this the subject of the CR. The main focus of this CR should be

1. did an Ubuntu update result in reordering of disks?
2. Was the fsck a result of a regression or just my bad luck that it broke?

TIA

[kernel-janitor (kernel-janitor)]

Hi gobble,

This bug was reported a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue? Can you try with the latest development release of Ubuntu? ISO CD images are available from http://cdimage.ubuntu.com/releases/ .

If it remains an issue, could you run the following command from a Terminal (Applications->Accessories->Terminal). It will automatically gather and attach updated debug information to this report.

apport-collect -p linux 270118

Also, if you could test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.

[This is an automated message. Apologies if it has reached you inappropriately; please just reply to this message indicating so.]

[gobble (gobbledegeek)]

It happened one dark and stormy night.... so far nothing has broken after that day.

Please close this.

Regards

[Leann Ogasawara (leannogasawara)]

Closing per the previous comment. Thanks.