2.6.35-devel is unable to mount root on mmc

[Till Harbaum]

The kernel built from the scripts at
https://code.launchpad.net/~beagleboard-kernel/+junk/2.6.35-devel

fails to mount the root directory from the mmc its booting off, although it's detecting the mmc just fine and the ext3 support is enabled in the kernel. This is happening on a revB7 board.

    1.972015] Waiting for root device /dev/mmcblk0p2...
[ 2.192993] usb 2-1: new low speed USB device using musb_hdrc and address 2
[ 2.333587] mmc0: new high speed SDHC card at address 0007
[ 2.339630] mmcblk0: mmc0:0007 SD04G 3.73 GiB (ro)
[ 2.344787] mmcblk0: p1 p2 p3
[ 2.355194] usb 2-1: device v046d pc00e is not supported
[ 2.412048] VFS: Cannot open root device "mmcblk0p2" or unknown-block(179,2)
[ 2.419128] Please append a correct "root=" boot option; here are the available partitions:
[ 2.427612] 1f00 512 mtdblock0 (driver?)
[ 2.432617] 1f01 1920 mtdblock1 (driver?)
[ 2.437683] 1f02 128 mtdblock2 (driver?)
[ 2.442687] 1f03 4096 mtdblock3 (driver?)
[ 2.447753] 1f04 255488 mtdblock4 (driver?)
[ 2.452819] b300 3911680 mmcblk0 driver: mmcblk
[ 2.458068] b301 40131 mmcblk0p1
[ 2.462432] b302 3365617 mmcblk0p2
[ 2.466796] b303 498015 mmcblk0p3
[ 2.471099] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,2)
[ 2.479675] [<c004a5d0>] (unwind_backtrace+0x0/0xf8) from [<c0550618>] (panic+0x6c/0xf0)
[ 2.487854] [<c0550618>] (panic+0x6c/0xf0) from [<c0009194>] (mount_block_root+0x188/0x230)
[ 2.496307] [<c0009194>] (mount_block_root+0x188/0x230) from [<c0009484>] (prepare_namespace+0x140/0x1d4)
[ 2.506011] [<c0009484>] (prepare_namespace+0x140/0x1d4) from [<c00086a0>] (kernel_init+0x13c/0x164)
[ 2.515258] [<c00086a0>] (kernel_init+0x13c/0x164) from [<c0045518>] (kernel_thread_exit+0x0/0x8)

The very same setup with a kernel built from
https://code.launchpad.net/~beagleboard-kernel/+junk/2.6-stable
works just fine.

[Till Harbaum (till-harbaum)]

A similar problem is reported in this thread:

http://e2e.ti.com/support/dsp/omap_applications_processors/f/42/t/28895.aspx?PageIndex=2

But no explanation/resolution has been found there.

[Robert Nelson (robertcnelson)]

Hi Tim,

Do you know what revision of the 2.6.35-devel you used?

It'll print it out at bootup, something like;

[ 0.000000] Linux version 2.6.35-rc5-d6 (root@debian-sheevaplug) (gcc version 4.4.4 (Debian 4.4.4-6) ) #1 PREEMPT Thu Jul 15 21:41:21 UTC 2010

Regards,

[Robert Nelson (robertcnelson)]

Hi Tim,

Just adding some more questions...

Can you provide a full boot log from the serial console. (from X-loader load to crash)

Thanks,

[Robert Nelson (robertcnelson)]

On Tue, Jul 20, 2010 at 2:24 PM, Till Harbaum <email address hidden> wrote:
> Hi Robert,
>
> i don't have access to the machine in question at this moment, but
> will add the missing information tomorrow.
>
> Regards,
>  Till
>
> BTW: I really enjoy your scripts and am i using them to run meego
> on the beagle.

Hi Till,

Not a problem, if you currently have access to the build machine, we
can tell with:

"bzr version-info" from the 2.6.35-devel directory

and from the version block of "cat version.sh"

voodoo@work-p4:~/bzr_repo/2.6.35-devel$ cat version.sh
#!/bin/bash

unset BUILD

KERNEL_REL=2.6.34
#STABLE_PATCH=.1
#PRE_RC=2.6.34-git16
RC_KERNEL=2.6.35
RC_PATCH=-rc5
ABI=8

Note, take a look at the ABI value i try to make sure the tree is
always buildable, but if it late and i'm heading home the ABI will be
a non constant value, if it's not tested..

Regards,

--
Robert Nelson
http://www.rcn-ee.com/

[Till Harbaum (till-harbaum)]

Full log from start to panic attached

[Robert Nelson (robertcnelson)]

Hi Till,

Thanks for the serial log, that's a strange one. What userspace lucid, maverick, other are you using? (i'm going to build a similar image)

In my own gcc testing, my beagle auto tests every newer rc kernel, here's that one in squeeze userspace for comparison:

http://rcn-ee.homeip.net:81/dl/gcc/sys/dmesg-162327-2.6.35-rc5-d7.log

The big thing with 2.6.35-rcX's series, I have not booted any with out a initramfs/uInitrd yet so we might have a regression there..

Looking at your log: "root=/dev/mmcblk0p2 rw rootwait" the 'rw' stands out, I've had problems in the past if that was not 'ro' in ubuntu/debian userspace.. It's been so long i don't remember what it did differently..

In-case we have a gcc regression, this one is pretty safe/compatible for all userspaces (except lucid/maverick)
http://rcn-ee.net/deb/squeeze/v2.6.35-rc5-d7/linux-image-2.6.35-rc5-d7_1.0squeeze_armel.deb

Regards,

[Till Harbaum (till-harbaum)]

Hi,

it's a meego rootfs i am intending to run (if that's your question), but how should the
rootfs have to do with this? It's not being loaded at all due to this issue.

But: Changing rw to ro allowed root to be mounted and to boot up. This has a strange
drawback: All other partitions (BOOT on mmcblk0p1 and swap on mmcblk0p3) are
read-only and i cannot e.g. enable swap (which is a show stopper in this case).
Strangely, the root partition itself is mounted rw and can be written.

Also the rw wasn't my idea, it's the default of u-boot as far as i understand. Erasing
nand and removing my boot.scr gives me a rw in the kernel options.

Regards,
  Till

[Till Harbaum (till-harbaum)]

That's really strange. You can see below that it claims to mount the rootfs read only. But as you can also see, i can just create a file and that file is still there after reboot (so it's not just some strange ram cache i am seeing). And you can also see swap not being mounted.

I am pretty sure you don't need a real rootfs for this. An empty second partition should do fine.

[ 1.967010] Waiting for root device /dev/mmcblk0p2...
[ 2.190277] usb 2-1: new low speed USB device using musb_hdrc and address 2
[ 2.307464] mmc0: new high speed SDHC card at address 0007
[ 2.313507] mmcblk0: mmc0:0007 SD04G 3.73 GiB (ro)
[ 2.318664] mmcblk0: p1 p2 p3
[ 2.351776] usb 2-1: device v046d pc00e is not supported
[ 2.415740] EXT3-fs: barriers not enabled
[ 2.422393] EXT3-fs (mmcblk0p2): mounted filesystem with writeback data mode
[ 2.429626] VFS: Mounted root (ext3 filesystem) readonly on device 179:2.
[ 2.436584] Freeing init memory: 240K
[ 2.440338] kjournald starting. Commit interval 5 seconds
                Welcome to Meego
[ OK ]

MeeGo release 1.0.80 (MeeGo)
Kernel 2.6.35-rc5-d7 on an armv7l

localhost.localdomain login: swapon: /dev/mmcblk0p3: swapon failed: Read-only file system
                                                                                         root
Password:
Last login: Wed Dec 31 16:00:18 on ttyS2
[root@localhost ~]# touch test
[root@localhost ~]# ls -la test
-rw-r--r-- 1 root root 0 Dec 31 16:00 test

[Robert Nelson (robertcnelson)]

Hi Till,

I'm going to start a full bisect from 2.6.34... While testing this tonight i got:

ubuntu@beagleboard:~$ mount | grep -v none | grep -v proc
/dev/mmcblk0p2 on / type ext4 (rw,errors=remount-ro)
/dev/mmcblk0p1 on /boot/uboot type vfat (ro)

the root file system comes up right, but my boot partition is ro...

ubuntu@beagleboard:~$ uname -a
Linux beagleboard 2.6.35-rc5-dl8 #1 PREEMPT Wed Jul 21 18:48:27 CDT 2010 armv7l GNU/Linux

Looking back at my logs I must have noticed it early in the 2.6.35's because I wrote a quick check (if ro remount as rw) for that issue in one of my boot scripts thinking it was an early rc bug..

Regards,

[Robert Nelson (robertcnelson)]

Okay, one step closer..

The regression occurs between snapshots in the merge 2.6.34-git6 (good) and 2.6.34-git7 (first bad).. (limitations of my current script..)

There's a big mmc merge in their for omap, so there's a lot that can be removed from the 11Mb difference between git6/7..

This script will generate it..

http://bazaar.launchpad.net/~beagleboard-kernel/+junk/2.6.35-devel/annotate/head:/tools/diff-snapshot.sh

I'll dig thru that diff and cut down the crude, but it looks like we just need to update the beagle board file's mmc properties..

Regards,

[Robert Nelson (robertcnelson)]

Hi Till,

Found the change that broke this:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=ed8303fc111e58530e22bd29b0d7e08dced75999

Reverted the change and it now correctly mounts the partitions..

ubuntu@beagleboard:~$ mount | grep -v none | grep -v proc
/dev/mmcblk0p2 on / type ext4 (rw,errors=remount-ro)
/dev/mmcblk0p1 on /boot/uboot type vfat (rw)

For now I just defined the code block out, but will investigate a better fix, since it should be correctly reading the gpio_wp line on the beagle and not ignoring it..

Latest bzr-version of 2.6.35-devel has the fix and i'm queye'ing up a new deb build... "2.6.35-rc5-d10" with this fix..

Regards,

[Robert Nelson (robertcnelson)]

Upstream change isn't correctly reading the gpio_wp line on the beagle.. (the kernel pin mux for the omap is still very young and we do alot in the bootloader...)

[Till Harbaum (till-harbaum)]

I can acknowledge that it's also working for my setup now. Thanks a lot!

[Till Harbaum (till-harbaum)]

I now have a beagleboard xm for testing and the same issue is back again. Mounting rw fails with the card not being mounted although it's being listed as being present. Mounting ro causes the entire card to be mounted ro.

The identical kernel boots on the b7 board just fine.

[Robert Nelson (robertcnelson)]

Yeah, my XM Rev A (256Mb) has the same issue. Although i haven't looked at it closely in awhile, is there any kind of "Write Protect" level on the mini-sd card holder? I don't think there is, well the kernel still thinks there is, we will have to add another check here for the XM, to ignore the gpio_wg...

Something like???

diff --git a/arch/arm/mach-omap2/board-omap3beagle.c b/arch/arm/mach-omap2/board-omap3beagle.c
index bec90ad..3f95e7e 100644
--- a/arch/arm/mach-omap2/board-omap3beagle.c
+++ b/arch/arm/mach-omap2/board-omap3beagle.c
@@ -379,7 +379,10 @@ static struct gpio_led gpio_leds[];
 static int beagle_twl_gpio_setup(struct device *dev,
                unsigned gpio, unsigned ngpio)
 {
- if (system_rev >= 0x20 && system_rev <= 0x34301000) {
+ if (cpu_is_omap3630()) {
+ mmc[0].gpio_wp = -EINVAL;
+ }
+ else if (system_rev >= 0x20 && system_rev <= 0x34301000) {
                omap_mux_init_gpio(23, OMAP_PIN_INPUT);
                mmc[0].gpio_wp = 23;
        } else {

Based from: http://git.kernel.org/?p=linux/kernel/git/tmlind/linux-omap-2.6.git;a=blob;f=arch/arm/mach-omap2/board-omap3beagle.c;h=6a6d2d7a04c6f0bd9f539b4c5f34510f46175bc1;hb=HEAD

Regards,

[Robert Nelson (robertcnelson)]

Actually are you sure? on my xm with 2.6.35-rc6-dl11

It's coming up as:
ubuntu@beagleboard:~$ mount | grep -v none | grep -v proc
/dev/mmcblk0p2 on / type ext4 (rw,errors=remount-ro)
/dev/mmcblk0p1 on /boot/uboot type vfat (rw)