Grub long sequential reads are extremely slow from USB2.0 HDD/FLASH drives

[Adam Niedling]

Grub long sequential reads are extremely slow from USB2.0 HDD/FLASH drives.
Reading vmlinuz(4.6MB) + initrd(18MB) takes several minutes (~0.5Mb/s)

Issues affects all grub versions until current in Natty (1.99~rc1-13ubuntu1).
But might be related to bad BIOS implementations.

Affected chipsets:
Intel 915 (todo)
Intel 965 (82801G)
Nvidia MCP79

Not-Affected chipsets:
Intel 965 (82801H)

Original description:
I'm using Hardy. Nothing happens for 50 seconds during bootup. There is only a blinking cursor on the screen after the grub menu. After the 50 seconds the splash screen appears. This problem only occurs on cold start, it doesn't happen when I just reboot. I have an HP nw8440 laptop. I'm attaching my dmesg and lspci -vv output.

Old Update: I've found out that initrd is read really slowly and this is causing the long delay during boot. Please read comment #27: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/247960/comments/27

[Chris Coulson (chrisccoulson)]

Thank you for taking the time to report this bug and make Ubuntu better. In order to get a better understanding of what might be happening here, I would like you to install 'bootchart' from the repositories. Bootchart is a boot sequence auditing tool, and produces a chart showing what processes were run during boot-up, how long they took and how much CPU and disk bandwidth they used. Once you have installed bootchart, a new chart will be saved in /var/log/bootchart each time you boot your machine. Could you please attach one of these charts to this bug report, as it may show what process is causing your machine to take so long to boot.

To install Bootchart, open a terminal (Applications -> Accessories -> Terminal) and type the following:

sudo apt-get install bootchart

Thanks!

[Adam Niedling (krychek)]

Thanks for the quick reply! I didn't know about this bootchart program but I like what it does now. I'm attaching the pic. I hope you can find out something, I can't seem to find what could be wrong.

[Adam Niedling (krychek)]

I just made a reboot now and the bootchart is the same.. so I think bootchart only starts working after the delay. I'm attaching both the new bootchart and the dmesg.

[Adam Niedling (krychek)]

So the first dmesg I've attached was made at a cold boot and this was at a reboot. The difference is huge. But it seems like the delay is 40 seconds long after all.

[Adam Niedling (krychek)]

I just though of something. Maybe this is delay is caused by the fact that this laptop has an encrypted hdd in it and Ubuntu runs from an external hdd. Poor Ubuntu tries to find out what that encrypted hdd is all about until it gives up?

[Chris Coulson (chrisccoulson)]

Your dmesg shows a delay very early on in the boot process:

[ 0.000000] Detected 1995.068 MHz processor.
[ 51.003457] Console: colour VGA+ 80x25

I'm assigning to the kernel for now. As per the kernel team bug policy, could you please provide the information requested at https://wiki.ubuntu.com/KernelTeamBugPolicies. You've already attached your dmesg, so we only need the following (attached to the report as individual uncompressed attachments):
1) Output of lspci -vvnn
2) Output of uname -a
3) Output of cat /proc/version_signature

Thanks

[Leann Ogasawara (leannogasawara)]

The Ubuntu Kernel Team is planning to move to the 2.6.27 kernel for the upcoming Intrepid Ibex 8.10 release. As a result, the kernel team would appreciate it if you could please test this newer 2.6.27 Ubuntu kernel. There are one of two ways you should be able to test:

1) If you are comfortable installing packages on your own, the linux-image-2.6.27-* package is currently available for you to install and test.

--or--

2) The upcoming Alpha5 for Intrepid Ibex 8.10 will contain this newer 2.6.27 Ubuntu kernel. Alpha5 is set to be released Thursday Sept 4. Please watch http://www.ubuntu.com/testing for Alpha5 to be announced. You should then be able to test via a LiveCD.

Please let us know immediately if this newer 2.6.27 kernel resolves the bug reported here or if the issue remains. More importantly, please open a new bug report for each new bug/regression introduced by the 2.6.27 kernel and tag the bug report with 'linux-2.6.27'. Also, please specifically note if the issue does or does not appear in the 2.6.26 kernel. Thanks again, we really appreicate your help and feedback.

[Rune K. Svendsen (runeks)]

I experienced this bug as well with a Hardy Heron Live CD. I just tried the Inpretid Ibex Alpha 6 Live CD and the problem isn't present anymore.

[Leann Ogasawara (leannogasawara)]

Based on the previous comment I'm marking this "Fix Released" for Intrepid. Adam, since you are the original bug reporter, if this isn't the case feel free to reopen by setting the status back to "New". Thanks.

[Launchpad Janitor (janitor)]

Per a decision made by the Ubuntu Kernel Team, bugs will longer be assigned to the ubuntu-kernel-team in Launchpad as part of the bug triage process. The ubuntu-kernel-team is being unassigned from this bug report. Refer to https://wiki.ubuntu.com/KernelTeamBugPolicies for more information. Thanks.

[Adam Niedling (krychek)]

Actually this is still an issue for me with Intrepid. However I don't have this issue when I'm booting Slax from usb pendrive. How could I provide more information?

[Chris Coulson (chrisccoulson)]

Adam, do you experience it using the Intrepid live CD? If so, could you try the latest Jaunty Alpha live CD to see if that exhibits the same issue?

Thanks

[Adam Niedling (krychek)]

I don't have this issue with live CDs. I tried both Intrepid and Jaunty alpha5. What could be wrong then?

[Adam Niedling (krychek)]

I still have this problem with Jaunty. After the grub menu I get a black screen with the blinking cursor for 40-50 seconds. After this the booting process goes fine. I think this is an issue with grub and has something to do with the fact that I'm booting from an external USB HDD. I also tried to boot with another laptop but the result was the same.

[Adam Niedling (krychek)]

No difference between the cold boot and reboot dmesg output with Jaunty.

[Leann Ogasawara (leannogasawara)]

Hi Adam,

Can you also test the latest upstream vanilla kernel for us? https://wiki.ubuntu.com/KernelMainlineBuilds . We'd really appreciate knowing if this exists upstream. Thanks.

[Adam Niedling (krychek)]

Hi Leann,

Thanks for taking care of this. I tested the latest kernel and the issue remains.

$ uname -a
Linux secrethostname 2.6.30-020630rc7-generic #020630rc7 SMP Sun May 24 01:38:23 UTC 2009 i686 GNU/Linux

[Adam Niedling (krychek)]

Just some update: 1. I've recently upgraded to grub2 but the issue remains.
2. Since I only have this issue when I do a cold boot, so I've discovered that if I enter into the BIOS and exit from it the system reboots, so I won't have the delay. I'm still wondering if this is a bug in Ubuntu..

[Adam Niedling (krychek)]

With grub2 instead of the blinking cursor I see this message for 50 seconds:

Booting 'Ubuntu, linux 2.6.28-13-generic'

[Linux-bzImage, setup=0x3000, size=0x3507d0]

_

[Adam Niedling (krychek)]

With the help of my local community I think I've found which process is taking a long time at boot. The initrd image is read really slowly. I created a new initrd after changing the MODULES parameter to dep in /etc/initramfs-tools/initramfs.conf . After this the size of initrd is reduced from 7 megs to just 3.2 and now I have only 30 seconds delay at boot. So initrd is read slowly during boot from a USB2.0 HDD.

[Nicolas Delvaux (malizor)]

Same bug here with an up-to-date Koala. (a 30 seconds delay on a USB2 HDD)

[ReimarBauer (reimarbauer)]

same problem with an uptodate 2009-11-12 jaunty on an internal drive.

currently I have the blinking cursor on 2 systems. Before the last online update they were both working fine.

I have no idea when it continues with the bootup sequence. waiting of 10mins nothing has happened

[ReimarBauer (reimarbauer)]

I solved it by reinstalling the MBR by grub.

[Adam Niedling (krychek)]

Could you tell me how exactly have you done that?

[Adam Niedling (krychek)]

Still an issue in Karmic.

[ReimarBauer (reimarbauer)]

after live cd boot
{{{
sudo su -
}}}
= prepare chroot =
{{{
mount /dev/sda3 /mnt
mount /dev/sda1 /mnt/boot
mount -t proc /proc /mnt/proc
mount -o bind /dev /mnt/dev
}}}
== chroot ==
{{{
chroot /mnt
}}}
== remount boot ==
for some unknown reason this was neccesary
{{{
umount /boot
mount /dev/sda1 /boot
}}}

= grub =
start grub
{{{
grub
}}}
the important part is to figure where stage1 is. then you can call setup
{{{
root (hd0,0)
grub> find /grub/stage1
find /grub/stage1
 (hd0,0)
grub> setup (hd0)
setup (hd0)
 Checking if "/boot/grub/stage1" exists... no
 Checking if "/grub/stage1" exists... yes
 Checking if "/grub/stage2" exists... yes
 Checking if "/grub/e2fs_stage1_5" exists... yes
 Running "embed /grub/e2fs_stage1_5 (hd0)"... failed (this is not fatal)
 Running "embed /grub/e2fs_stage1_5 (hd0,0)"... failed (this is not fatal)
 Running "install /grub/stage1 (hd0) /grub/stage2 p /grub/menu.lst "... succeeded
Done.
grub>
}}}

= reboot =
just reboot

[Adam Niedling (krychek)]

I had to reinstall grub2 after upgrading to Karmic because something was messed up and it's still slow.

[syntag (lotekware)]

Always had this same problem since Ubuntu 8 on Acer Travelmate 6292 and other laptops. ONLY happens when running Ubuntu from an external (USB) 2.5" HDD. I also tried 9.04 and 9.10 (Grub 2), and bought a different external HDD, but nothing changes, GRUB is still slow (about 20-30 seconds before the actual booting process starts).

[syntag (lotekware)]

BTW I use Reiser (v3) as a file system. Adam, Nicolas, what filesystem are you using?

[bwallum (rbw2)]

I have a similar delay problem using an external USB 2.0 hard drive. I boot up from it using karmic 32bit using Grub2. The external drive is IDE (PATA), 320GB, the file system is ext3, power comes from two USB connections. Does the same issue exist on SATAII external hard drives driven by USB power?

[syntag (lotekware)]

bwallum, one of the external HDDs I tried is a USB-powered SATAII drive and I get the same delay.

It looks like it's not due to the filesystem, kernel or distro version difference. Perhaps the problem has something to do with Grub's (and Grub2's) method of accessing external USB disks and/or mounting their filesystems.

Btw this looks related: http://ubuntuforums.org/showthread.php?t=961493
Here the external USB HDD is used as a backup only, yet the user describes the problem as: "Very slow boot at Grub stage2 (minutes, reports user). No other anomalies post boot. Normal boot time when an external backup hard drive is switched off."

[Adam Niedling (krychek)]

syntag: I'm using ext3.

[Nicolas Delvaux (malizor)]

I have this problem with both ext3 and ext4.

I agree with your diagnostic syntag.

[syntag (lotekware)]

Apparently the filesystem you choose only affects the mount time, which happens after Grub's "Starting up" message. With reiserfs I saw a delay of 1 or 2 seconds, with ext2 and ext3 no noticeable delay.

The 30-40 seconds delay problem appears (at least in my case) BEFORE that message even appears, right after the first message: "Boot from (hd0,0) reiserfs ID etc...", as if Grub is probing the usb device. But why does this happen only to usb hdds?

[syntag (lotekware)]

Ok, I saw many reports at ubuntuforums, launchpad, etc. where users complain about Grub loading slowly when the boot partition or MBR is not on sda. An usb disk is usually sdb, sdc... Moreover, when installing Ubuntu on sda my notebooks boot instantly. I'll check out Grub's source code, but I seem to recall that Canonical also added some code to its own version.

[valus (blaedhorn)]

I doubt the problem is in Grub. This happens when the BIOS doesn't support USB 2.0 at BOOT TIME, so it uses the slower 1.1 (you should see your hdd recognized as a 1.1 device when the system is starting). Once your OS loads, you have full 2.0 support, that's why lspci output is useless in this case, the problem is in the BIOS, not in linux. I know for sure that many Acer notebooks do this... don't know about HPs though.

[Adam Niedling (krychek)]

According to wikipedia USB1.0 has a speed of 183 KB/sec. My initrd image is 7.5 MBs. 7500/183=41. And I have around 40 seconds delay during boot. So my USB 2.0 HDD is treated as a USB 1.0 device? And not even USB 1.1. That's odd.

[valus (blaedhorn)]

Adam - I don't know if there is something else involved, but I saw a ~40 sec delay on a couple of Acer notebooks I borrowed (on the BIOS screen the external hdd was recognized as USB 1.1). On the same hdd connected to a Sony notebook (BIOS showed USB 2.0) ubuntu booted instantly.

[valus (blaedhorn)]

Anyway like I said before the 1.x problem is only at boot time, later when your os loads your hdd is treated as a 2.0 device. So this is not a bug in Grub, it's just poor quality firmware... (the notebooks i tried were just 1 or 2 years old).

[Adam Niedling (krychek)]

valus: as I mentioned above, if I enter the BIOS right after turning on the laptop and then exit it immediately the laptop restarts and I won't have the delay. Why do you think this is? I only have the delay at a cold boot, not when rebooting.

[Adam Niedling (krychek)]

I'm setting this to invalid as I've lost this Ubuntu installation. (Kernel panic on boot, couldn't fix it. I wasn't doing anything nasty, but something just went wrong.)

[ErikH (erik-heckers)]

Hello!

I have a similar problem after installing Ubuntu Server 10.10 to an Eee PC 4G.
It seems to be frozen for 30 seconds after select a boot entry in the grub menu and
then the kernel starts.
I opened an interactive grub shell and entered the grub commands manually.
After enter "linux ..." (4 MB kernel is loaded?) and "initrd ..." (10 MB initrd is loaded?) the
system hangs for a long time before I get a prompt again. That is definitely where the
system spends the 30 seconds during boot!
When I finally enter the "boot" command as last command in the grub shell the kernel boots
really fast and the login prompt is available after 3-4 seconds.

So the issue clearly is the time it takes for the "linux" and "initrd" command of grub
to finish whatever they do!

[Eugene San (eugenesan)]

I am re-opening the bug.

[Ralf (ralf-kaestner)]

I do not think that the issue is related generally to bad bioses not supporting USB2.0. Especially in my case I have an option in Bios to either set USB ports to 2.0 or 1.1 at boot. I can also clearly see that it becomes faster when select 2.0 but its still horrible slow to load the kernel+initrd from usb drive.

I have my boot partition on an USB stick which is quite fast when Ubuntu is booted. But when grub loads the files it takes far too long compared to the time it takes to read the same files in OS. I use ext2 and wanted to test if its better on vfat but was unable to get grub boot from the same stick after I just reformated it with vfat and grub-installed'ed it without errors.

[Bastian Schmidt (bastian-schmidt)]

I also noticed the slow reads when trying to create a multi-boot USB stick.
For a 38MB initrd it takes over 1 minute to load on all pcs tested. Using syslinux, the same initrd (from the same stick) is loaded in less then about 5seconds.
So this is definitely not a Bios issue.

[Rovano (rovano)]

I testing RR and Opensuse 12.3 on older PC.. Intel chipset G45, P 4 3GHz, ...
And found bug.
Loading kernel ~20minutes!
Same as:
https://bugzilla.novell.com/show_bug.cgi?id=808611
http://lists.opensuse.org/opensuse-factory/2013-03/msg00162.html

If create with dd(sudo dd if=/home/silvuska/raring-desktop-i386.iso of=/dev/sdc
 or volume bs=4k)... bug exist. If create with usb-creator, bug no reproduced.

With usb-creator kernel loading in few seconds.. on ubuntu or opensuse.

[Rovano (rovano)]

And with this script(syslinux,gpart) https://github.com/cyberorg/live-fat-stick same problem.

[Rovano (rovano)]

Repair result:
I testing RR and Opensuse 12.3 on older PC.. Intel chipset G45, P 4 3GHz, ...
And found bug.
Loading(no boot!) kernel Opensuse ~20minutes! Ubuntu 5minutes.
Is this an error similar to the one in the link?
https://bugzilla.novell.com/show_bug.cgi?id=808611
http://lists.opensuse.org/opensuse-factory/2013-03/msg00162.html

And with this script(syslinux,gpart) https://github.com/cyberorg/live-fat-stick same problem on Opensuse. Ubuntu not tested.

[jalonsom (jalonsom)]

Same here with ubuntu 13.04 installed in a USB stick
It is a fast stick. Boots in some computers in a few seconds, but in others it takes 6-7 minutes to boot.
The main delay is when loading initramfs, it takes 5+ minutes.
After that, boot sequence is normal.

[jimmy the saint (lowid95)]

I can report that this bug is still present in 14.04 Server Edition.

I am running an HP Proliant MicroServer G1610. The OS boots from a Sandisk Cruzer. It is a full installation, not a LiveUSB.

The delay between POST and GRUB is over 5 minutes. Reinstalling Grub does not resolve the issue. Once Grub shows up, the rest of the boot goes smoothly, and the USB drive works fine.

Replacing Grub2 with Extlinux (Syslinux) results in a normal boot time that can be measured in seconds between POST and the bootloader.

[Adam Niedling (krychek)]

jimmy: you might have a different problem because the delay that my original bug report is describing happens after grub, not before it.

[Goh Lip (gohlip)]

Adam, you're right in your original post that initrd is the cause of the delay.
Other non-Ubuntu based distros do not have the long 50 secs wait to boot OS at usb. (same kernel)
It's just 13 secs vs 50secs. (haven't tried Debian distros).

One would think after 5 years, something ought to be done about it.

[Adam Niedling (krychek)]

Someone who can still reproduce this issue should open a report upstream. The kernel devs will never take a look at this otherwise.