performs poorly on slow HDD

[Scott James Remnant (Canonical)]

Binary package hint: sreadahead

This is a tracking bug to improve performance on HDD disks, especially the really slow ones where sreadahead and the kernel have a deathmatch

[Martin Pitt (pitti)]

Speaking of which, is it actually deliberate that sreadahead runs for a very long time, without doing anything CPU/IO related at all? (see http://people.canonical.com/~pitti/tmp/tick-karmic-20090916-2.png for example).

[Scott James Remnant (Canonical) (canonical-scott)]

This may end up being a lynx project, it does not seem to be a simple "fix".

The recent change to the deadline scheduler has certainly improved things though

[Matej Kenda (matejken)]

Is this in any way related to bug 421116, which was fixed recently?

[Martin Pitt (pitti)]

Bug 421116 was about taking a lot of CPU for building the pack. As far as I understand, this bug is about sreadahead not doing any actual readahead (or doing it poorly) on startup when it already has a pack?

[Martin Pitt (pitti)]

E. g. here: http://people.canonical.com/~pitti/tmp/tick-karmic-20090924-2.png

Runs for a long time without doing any CPU/IO

[Robbie Williamson (robbiew)]

@Scott: Could you put in a summary of the current status for this bug please. Thanks

[Paul Sladen (sladen)]

Keybuk: how long *should* sreadahead be running for? Times of ~30 seconds runtime for sreadahead appear to be "normal" charts on (see bug #423924).

[Rami Al-Rfou' (rmyeid)]

I attached the bootchart with sreadahead enabled and disabled.
I disabled sreadahead by commenting the exec statement
exec /sbin/sreadahead -t 0
in /etc/init/sreadahead.conf

[Johan Kiviniemi (ion)]

Is this the correct bug report for this issue? On my quite new laptop (dualcore 2.2 GHz AMD CPU, 3 GiB of RAM, a 320 GB HDD), sreadahead consistently slows startup down by about 5.5 seconds.

I measured this with the following job:

start on login-session-start
task
script
    cat /proc/uptime >>/uptimes
    if [ "$(wc -l /uptimes | cut -d' ' -f1)" -lt 5 ]; then
        reboot
    fi
end script

The average measured time with sreadahead disabled is 33.07 s, and with sreadahead enabled, 38.55 s.

Could something as simple as making the rest of the startup wait for sreadahead to finish fix this? The problem might simply be caused by thrashing when sreadahead is doing its thing simultaneously with the rest of the system booting.

[Pavel Rojtberg (rojtberg)]

in case this bug cant be fix for karmic - are there any plans to revert to readahead and postpone sreadahead for lynx? It did a good job for HDDs and these are still used in the majority of the target systems.

[Scott James Remnant (Canonical) (canonical-scott)]

On Wed, 2009-10-14 at 21:40 +0000, Pavel Rojtberg wrote:

> in case this bug cant be fix for karmic - are there any plans to revert
> to readahead and postpone sreadahead for lynx? It did a good job for
> HDDs and these are still used in the majority of the target systems.
>
In the testing I've been able to do, readahead doesn't show any
improvement.

Scott
--
Scott James Remnant
<email address hidden>

[Martin Pitt (pitti)]

Scott James Remnant [2009-10-15 17:38 -0000]:
> In the testing I've been able to do, readahead doesn't show any
> improvement.

Hm, sreadahead does not do _any_ readahead for me right now. Is that
due to a kernel change which would also break readahead?

Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

[Scott James Remnant (Canonical) (canonical-scott)]

On Thu, 2009-10-15 at 18:53 +0000, Martin Pitt wrote:

> Scott James Remnant [2009-10-15 17:38 -0000]:
> > In the testing I've been able to do, readahead doesn't show any
> > improvement.
>
> Hm, sreadahead does not do _any_ readahead for me right now. Is that
> due to a kernel change which would also break readahead?
>
Not that I know of - which kernel are you using?

Scott
--
Scott James Remnant
<email address hidden>

[Hernando Torque (htorque)]

@Martin Pitt: Are you referring to the bar in the chart not being red? Maybe bootchart doesn't represent sreadahead very well, because when put in foreground you can see I/O going on while the bar is still gray: http://img.xrmb2.net/images/660462.png

[Martin Pitt (pitti)]

Scott James Remnant [2009-10-15 19:43 -0000]:
> Not that I know of - which kernel are you using?

Just what's in karmic (amd64 2.6.31-14.47-generic), but it's been that
slow for weeks, so it's nothing new.

--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

[Martin Pitt (pitti)]

Hernando Torque [2009-10-15 21:17 -0000]:
> @Martin Pitt: Are you referring to the bar in the chart not being red?

I did yes. That, and grub->gdm now taking 85 seconds, where it took
just 50 in jaunty.

> Maybe bootchart doesn't represent sreadahead very well, because when put
> in foreground you can see I/O going on while the bar is still gray:
> http://img.xrmb2.net/images/660462.png

It might also be a bootchart bug of course, assigning the I/O to wrong
processes. However, I don't quite believe that: As you see in my
bootchart [1], sreadahead starts and just has a tiny fraction of a
second before other processes kick in and use I/O. In the readahead
times it was the only active process for some 15 seconds, to avoid
trashing with other processes.

Martin

[1] http://people.canonical.com/~pitti/tmp/tick-karmic-20090924-2.png
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

[papukaija (papukaija)]

Is it possible to fix this bug before the final release of Karmic?

[Steve Langasek (vorlon)]

No, there's no way this will be fixed before final release. Redirecting to the SRU queue.

[Scott James Remnant (Canonical) (canonical-scott)]

Please try the following:

  sudo add-apt-repository ppa:ubuntu-boot/ppa
  sudo apt-get update
  sudo apt-get dist-upgrade

This should install an updated kernel package, and replace sreadahead with ureadahead.

The first reboot will reprofile your system, the second reboot should be substantially faster.

Please let me know how you get in (before/after bootcharts always appreciated)

[smooth (hsavio)]

The boot time did improve by ~ 5 Secs, which is faster but not substantially.
Also disk utilization has also dropped

I have attached the new bootchart.

[Martin Pitt (pitti)]

Scott,

it's working magnificiently here. Times from grub to gdm with no CPU/IO:

Default karmic (sreadahead): 57 seconds (http://people.canonical.com/~pitti/bootcharts/karmic-sreadahead.png)
Karmic without any *readahead: 45 seconds (http://people.canonical.com/~pitti/bootcharts/karmic-noreadahead.png)
Karmic with your PPA: 25 seconds (http://people.canonical.com/~pitti/bootcharts/karmic-ureadahead.png)

[Scott James Remnant (Canonical) (canonical-scott)]

This has been uploaded to karmic-proposed, since the kernel patch is there (though that might take some more cooking due to other patches in the kernel upload):

Changes:
 ureadahead (0.90.3-2) karmic-proposed; urgency=low
 .
   * über-readahead is a replacement for sreadahead that should
     significantly improve boot performance on rotational hard drives,
     especially those that had regressed in performance from jaunty to
     karmic.
 .
     It does this by pre-loading such things as ext2/3/4 inodes and opening
     files in as logical order as possible before loading all blocks in one
     pass across the disk.
 .
     On SSD, this behaves much as sreadahead used to, replacing that package
     with slightly improved tracing code.
 .
     This requires the kernel package also found in karmic-proposed.
 .
     LP: #432089.

[fossfreedom (fossfreedom)]

things improved here - if assuming I've read my boot chart correctly - from boot to GDM originally was 73 secs - with the PPA + ureadahead - 42 secs

[Rami Al-Rfou' (rmyeid)]

I attached below two boot charts one is before the update and the another is after. As I do not know how to interpret the bootcharts, I used my old wrest watch to calculate the boot times:

1-sreadahead
Grub-> GDM 35 seconds
GDM -> Ubuntu 45 seconds

2-ureadahead
Grub -> GDM 30 seconds
GDM -> Ubuntu 20 seconds (at most maybe less)

I still remember that it was working faster than this when I installed karmic alpha 3.

[Ryan (ubuntu-draziw)]

One more reboot since I saw fsck showing for a lot of the time on the last long boot. This is another ureadahead boot - 2 seconds faster than the prior sreadahead. (But still - 1:30 on a dual 2 ghz system)

[pbrufal (pbrufal)]

In my system, adding the PPA+ureadahead makes the system 5sec slower :? I'm using the last kernel (2.6.31-15.49)

80 sec before PPA
85 sec after PPA (tried 6 times)

[Martin Pitt (pitti)]

I accepted the source package into karmic-proposed, source NEWed. Will do "please verify" steps once it's through binary NEW.

[Martin Pitt (pitti)]

Accepted ureadahead into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Scott James Remnant (Canonical) (canonical-scott)]

On Tue, 2009-11-10 at 06:09 +0000, pbrufal wrote:

> In my system, adding the PPA+ureadahead makes the system 5sec slower :?
> I'm using the last kernel (2.6.31-15.49)
>
> 80 sec before PPA
> 85 sec after PPA (tried 6 times)
>
You didn't attach the "before" bootchart.

In the "after" bootchart, the boot time is actually only 55s; of which
it looks like 5s are waiting at the login screen.

Scott
--
Scott James Remnant
<email address hidden>

[FriedChicken (domlyons)]

After installing ureadahead from proposed booting takes about 10 s longer.

[ScottHW (publicw)]

I've been experiencing significantly slower boot times since upgrading to Karmic. Too bad, as one of the main features I thought we were going to look forward to was faster booting.

In Hardy, I was booting in ~0:39
In Jaunty, I was booting in ~-0.26
In Karmic, I was booting in ~1.22

Upgraded to ureadahead, I am booting in ~1.19

(sorry, I also apparently don't know how to attach multiple files...)

[ScottHW (publicw)]

Here is a Jaunty bootchart

[ScottHW (publicw)]

Here is a Karmic bootchart (with sreadahead)

[ScottHW (publicw)]

And finally, Karmic with ueadahead.

I've tried to pick random representative bootcharts for each of the described versions.

[Tormod Volden (tormodvolden)]

Boot times from grub to gnome-panel loaded (and CPU drop), on a dual-core 3GHz Intel, 2GB RAM, Samsung SP2504C SATA-II drive (using autologin):
- jaunty: ~43s (but not so stock install any longer)
- karmic-updates: 28s
- karmic-proposed: 22s

[Scott James Remnant (Canonical) (canonical-scott)]

On Thu, 2009-11-12 at 22:28 +0000, Bdopo wrote:

> Bootchart after installing the ureadhead from the boot ppa. 4 second
> difference.
>
This is still profiling, make sure you have a /var/lib/ureadahead/*pack
file and reboot again :-)

Scott
--
Scott James Remnant
<email address hidden>

[Scott James Remnant (Canonical) (canonical-scott)]

On Fri, 2009-11-13 at 08:35 +0000, ScottHW wrote:

> I've been experiencing significantly slower boot times since upgrading
> to Karmic. Too bad, as one of the main features I thought we were going
> to look forward to was faster booting.
>
> In Hardy, I was booting in ~0:39
> In Jaunty, I was booting in ~-0.26
> In Karmic, I was booting in ~1.22
>
> Upgraded to ureadahead, I am booting in ~1.19
>
Please open up your jaunty and karmic (ureadhead) bootcharts; that goes
for everyone else too ;-)

Now first, please ignore the "time:" bit at the top. That is *NOT* your
boot time, that is simply the number of seconds after boot that
bootchart exited (for whatever reason).

Reading jaunty, scroll down and look at the chart; look at what the last
thing happening before the chart is cut off: It's about 3/4 the way
down:

  [gdm ]
  [Xorg ]
        [sh]
    [xkbcomp]

So this chart is being cut off as soon as the X server is started, and
the other init scripts are finished.

Reading karmic, you'll notice that the chart is much wider and that the
CPU and I/O graphs of the last bit are empty. That's because bootchart
runs longer to capture _more_ of the boot.

Try and find the same gdm/Xorg pattern - it's not at the right hand side
of the graph is it? In fact it's almost exactly in the middle.

jaunty wasn't booting faster - the karmic bootchart just *also* includes
your desktop login as well (since we care about both)

Scott
--
Scott James Remnant
<email address hidden>

[Huy V. Le (huyvieto)]

Hi Scott,

When you say Jaunty wasn't booting faster, I'm not sure then what was faster in Jaunty.

See my previous post, How can we achieve the same performance, boot and load Firefox in less than 1 minutes.

I have attached some bootchart with/out elevator=noop.

Hope it can help to improve performance.

Regards,

Huy

[Tormod Volden (tormodvolden)]

On my moderately equipped laptop (1.6 GHz Pentium M, 1GB RAM, Samsung HM160HC P-ATA drive, ureadahead is not better then sreadahead when it comes to get my desktop ready to use. I think the time where disk and CPU has calmed down is the most telling figure for practical purposes. This time is pretty much the same. ureadahead seems to read a lot faster, but it blocks all other processes while it is running, afterwards these processes crave for CPU without using the HD much. With sreadahead both disk and CPU activity is evened out through the whole boot.

[Bdopo (safelydeliver-online)]

Hi Scott,
  Thanks for your reply. I checked and I do have a /var/lib/ureadahead/*pack
and I've rebooted several times with no great speed difference. Visibly, I can see the sequence of texts that flash on my screen go faster (almost like jaunty) with this new ppa installed. Attached is a bootchart of my most recent boot?field.comment=Hi Scott,
  Thanks for your reply. I checked and I do have a /var/lib/ureadahead/*pack
and I've rebooted several times with no great speed difference. Visibly, I can see the sequence of texts that flash on my screen go faster (almost like jaunty) with this new ppa installed. Attached is a bootchart of my most recent boot?field.comment=Hi Scott,
  Thanks for your reply. I checked and I do have a /var/lib/ureadahead/*pack
and I've rebooted several times with no great speed difference. Visibly, I can see the sequence of texts that flash on my screen go faster (almost like jaunty) with this new ppa installed. Attached is a bootchart of my most recent boot

[Alan Goldblatt (alan-b-goldblatt)]

@Tormod -- your setup seems to be quite different for those two bootcharts. The one for s- is loading a lot fewer services (no compiz, no panel, etc.). I think if you try s- vs. u- with otherwise identical setups you'll find u- to be much faster than s-. (Also, your boots are pretty nice and quick either way.... I wish mine would boot so fast.)

-Alan

[ErikBerglund (a-launchpad-snap-345-blog-no)]

In Jaunty SSHD and SMBD started in less than 20 seconds, now they both take over 45 seconds.

I've tried the following:
1. Upgrading to GRUB 2.
2. Disabling unused services.
3. Changing CONCURRENCY=none to CONCURRENCY=startpar in /etc/init.d/rc

Nothing has helped.
I'm booting from a 500 GB SATAII 7200rpm disk with ext4.

?field.comment=In Jaunty SSHD and SMBD started in less than 20 seconds, now they both take over 45 seconds.

I've tried the following:
1. Upgrading to GRUB 2.
2. Disabling unused services.
3. Changing CONCURRENCY=none to CONCURRENCY=startpar in /etc/init.d/rc

Nothing has helped.
I'm booting from a 500 GB SATAII 7200rpm disk with ext4.

[ErikBerglund (a-launchpad-snap-345-blog-no)]

Installed the PPA, boot is now noticably faster. SSHD and SMBD start in less than 25 seconds.

Still not as fast as Jaunty, though.

[Brian Pitts (bpitts)]

Compared to my previous attachment, this one is much better.

[Eric Appleman (erappleman)]

My system is similar to Tormo's

1.6 GHz Core Duo, 2 GB RAM, 54000 RPM 160 GB SATA HDD

[Eric Appleman (erappleman)]

btw, i have 4 partitions (in order). /root, /home, swap, windows.

[Shaji N V (nvshaji)]

Scot,

I have couple of issues -

1. sreadahead is required by metapackages ubuntu-desktop and ubuntu-netbook-remix, which get removed when I replace sreadahead with ureadahead. Minor problem - but it would be cleaner if package dependencies are sorted out.
2. First time when I installed ureadahead along with a new kernel and rebooted, it did recompile dkms modules ( have 3 modules) upon next boot. This process made next boot slower, and ureadahead profile to collect 5500 files
          strings /var/lib/ureadahead/pack | wc -l
Now i removed pack file, reinstalled ureadahead and this time pack has only 2400 files.

Probably profiling needs to be done on multiple boots and take the common files across to optimize this.

[FriedChicken (domlyons)]

@Scott:

> > In case you need it...
> > Everything in the list has 0 blocks.
> >
> What kind of filesystem are you using? Are you using wubi or anything?
>
>
>XFS on / and ext3 on /boot. No, it's a native installation.
(post #69)

I've trid this on two more systems, two of them with XFS (and ext3/4 for /boot) and one with JFS:

- XFS (1) and JFS: As above every entry has "0 blocks". ureadahead is doing something on booting but overall boot takes longer or is just as fast as on profling.

- XFS (2): the same and "cat /var/log/syslog | grep ureadahead" gives:
  ... init: ureadahead-other main process (915) terminated with status 4
(appears on every boot)
Also there's a delay of 5 s (sleep just after usplash and resume). I will attach the bootchart of this system.

I guess the last one is a seperate bug but what about the "0 blocks"-problem?

[FriedChicken (domlyons)]

I don't know why but profiling seems to be done several times... This chart is one of them.

[Robin S. Parriath (robinparriath)]

Doesn't seem to make much of a difference on my machine. I get the same 1 min 25-30 secs on stopwatch.

Attached are the bootcharts.

[Scott James Remnant (Canonical) (canonical-scott)]

ureadahead has now replaced sreadahead in both karmic (via -updates) and lucid

[Martin Pitt (pitti)]

It's only in -proposed. However, it matured enough now, and we got enough feedback to demonstrate that it is a huge improvement.

Copying over now.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ureadahead - 0.90.3-2

---------------
ureadahead (0.90.3-2) karmic-proposed; urgency=low

  * über-readahead is a replacement for sreadahead that should
    significantly improve boot performance on rotational hard drives,
    especially those that had regressed in performance from jaunty to
    karmic.

    It does this by pre-loading such things as ext2/3/4 inodes and opening
    files in as logical order as possible before loading all blocks in one
    pass across the disk.

    On SSD, this behaves much as sreadahead used to, replacing that package
    with slightly improved tracing code.

    This requires the kernel package also found in karmic-proposed.

    LP: #432089.
 -- Scott James Remnant <email address hidden> Mon, 09 Nov 2009 18:38:51 +0000

[Christian Loos (cloos)]

After upgrade from jaunty to karmic my boot time was rising from 1 minute to 10 minutes.
Even after I installed ureadahead and linux-image from proposed, boot time is also 10 minutes.
So from my point of view this bug isn't fixed. Attached is my dmesg.
If you need further information just contact me.

[Steve Langasek (vorlon)]

Even pessimal handling by ureadahead wouldn't account for a 10 minute boot time. You appear to have an unrelated issue; please file a separate bug report.

[ 5.120214] ieee1394: Host added: ID:BUS[0-00:1023] GUID[001106664555572b]
[ 556.653356] PM: Starting manual resume from disk

The gap in time in the logs happens right after initializing your firewire interface; you probably need to file a bug against the 'linux' package.

[Steve Langasek (vorlon)]

Also, this could be due to a buggy, non-standard script in your initramfs, so please include a full listing of all the scripts under /usr/share/initramfs-tools/scripts/ (ls -lR /usr/share/initramfs-tools/scripts/) in your bug report.

[Omegamormegil (omegamormegil)]

Christian,

Sorry you are still having trouble. Lots of people have reported that this
solution has made substantial improvements to their boot time, effectively
fixing this bug for all of them. If you are still experiencing a slow boot
time, it would be best to open a new bug report because you must have a
different issue.

Feel free to post a link to your new report here, so other people having the
same problem as you might have an easier time finding your report.

On Tue, Dec 1, 2009 at 3:10 PM, Christian Loos <email address hidden>wrote:

> After upgrade from jaunty to karmic my boot time was rising from 1 minute
> to 10 minutes.
> Even after I installed ureadahead and linux-image from proposed, boot time
> is also 10 minutes.
> So from my point of view this bug isn't fixed. Attached is my dmesg.
> If you need further information just contact me.
>
> ** Attachment added: "dmesg-2009-12-01.log"
> http://launchpadlibrarian.net/36308039/dmesg-2009-12-01.log
>
> ** Changed in: ureadahead (Ubuntu Karmic)
> Status: Fix Released => Fix Committed
>
> --
> performs poorly on slow HDD
> https://bugs.launchpad.net/bugs/432089
> You received this bug notification because you are a direct subscriber
> of a duplicate bug.
>
> Status in “sreadahead” package in Ubuntu: Won't Fix
> Status in “ureadahead” package in Ubuntu: Fix Released
> Status in “sreadahead” source package in Karmic: Fix Released
> Status in “ureadahead” source package in Karmic: Fix Committed
>
> Bug description:
> Binary package hint: sreadahead
>
> This is a tracking bug to improve performance on HDD disks, especially the
> really slow ones where sreadahead and the kernel have a deathmatch
>

[Jamie Lawler (jamie-lawler)]

You could also try disabling firewire in your BIOS and see if that solves the problem.

[Christian Loos (cloos)]

Steve, attached the initramfs scripts.

After the upgrade from jaunty to karmic and the 10 minutes boot time I tried different things:
- fresh karmic installation
- blacklisting the ieee1394 and ohci1394 modules
- disabling swap partition after reading bug 479611 that have the same dmesg entry after the delay like my one

all without success.

So I thought that this bug will tracking all related boot delays after upgrade from jaunty to karmic.

But ok I will open a new bug and hope someone can fix my problem.

[Christian Loos (cloos)]

Just created bug 491045.

[Tim (tzakharov)]

I saw this come through Karmic's updates. I almost started the install then noticed in the changelog:
"This requires the kernel package also found in karmic-proposed."

Since I don't have karmic-proposed enabled, does that mean installing this update could break my system?

[Martin Pitt (pitti)]

Tim [2009-12-02 0:52 -0000]:
> Since I don't have karmic-proposed enabled, does that mean installing
> this update could break my system?

No, the kernel is already in -updates now, and even if you don't run
that the worst thing that can happen that you get no readahead at all,
and thus no speedup.

[Dan Andreșan (danyer)]

Nice, ureadahead shave 50% of my boot time.

Now, sreadahead is left as a "transitional package" which can be removed.

When I want to remove it (I like things as clean as possible) it "helpfully" offer to remove ubuntu-desktop too.

Could you remove the dependency? Thanks.