ureadahead requires /var on root filesystem

[Roman Yepishev]

I have the following mountpoints defined that use different partitions:

/
/boot
/home
/opt
/usr
/var

These nodes are located on different partitions to enable greater control over the system and prevent unusable states.
ureadahead.conf uses the following event:
  start on starting mountall
Which fails for me since /var is not yet mounted when mountall starts leading to ureadahead errors about missing pack files.

ureadahead-other.conf in turn uses
  start on mounted DEVICE=[/UL]* MOUNTPOINT=/?
I have yet to find out what that means but I see that ureadahead-other is terminated with error status 4 (i.e. no pack file)

I think that ureadahead configuration files should be ready for such kind of layout.
I have fixed the first item by rewriting the rule as start on mounted MOUNTPOINT=/var but I don't know what to do with second item.

[Pavel Malyshev (afunix)]

I can confirm that.
I have 3 computers (2x karmic with latest updates, 1x lucid) with separate /var and all of them have empty /var/lib/ureadahead.

[Scott James Remnant (Canonical) (canonical-scott)]

Moved to Ubuntu bug tracker

[Scott James Remnant (Canonical) (canonical-scott)]

The FHS only really allows us /var to store things like this, sadly

[Pavel Malyshev (afunix)]

Can ureadahead collect data in memory until /var/lib/ureadahead becomes ready?

[Brian J. Murrell (brian-interlinx)]

~sigh~ Yet another bug with regard to boot-up and separate /var. Would it be at all possible for the users/developers/testers of all of this bootup (i.e. upstart, mountall, etc.) software to routinely build their (test) systems with a separate /var? This seems like an all-too-common use-case that is simply not getting tested before these releases are made for the general public to use.

I can understand bugs that get out due to corner cases, but a separate /var is far too far from being a "corner case".

[Piedro Kulman (piedro)]

So what's up now? Is there a workaround?
Should it be removed from installations with /var as separate partition?

"The FHS only really allows us /var to store things like this, sadly"

Does this mean there won't be any fix at all?

[Daniel Jour (danieloertwig)]

Hi there.
If have the same issue on my server system with new lucid installed.

Every 10. boot or so the system hangs up and must be killed.
This could possibly be linked to this issue here.

[Trey "WebStandardCSS" Brister (webstandardcss)]

[SOLVED] OK, so you thought you were smart and installed Ubuntu with /var on its own partition so that way runaway log files can't take down your whole system. (wasn't this theory valid back when hard drives we measured in Megabytes?)

Well I am that smart guy and I got the readahead error.

Here is what I did to fix it. (it worked for me YMMV)

Boot to Ubuntu live CD

Mount /var partition and root partition from the Hard drive. (Just click the drive in Places to see the disks.)

Edit /etc/fstab on the hard drive root filesystem and comment out the line containing /var

Enter the /var partition on the root filesystem and rename the three existing directories appending ".old" (without quotes) to the end of the filename.

Copy the contents of the var partition to the /var folder of the root partition

Copy the contents of the three ".old" directories to their respective directories inside /var on the root partition.

Reboot.

Now in places you should see an unmounted disk which used to be the /var/ partition

Also you should not see the unreadahead error any more.

[Brian J. Murrell (brian-interlinx)]

On Sat, 2010-05-15 at 13:26 +0000, Trey "WebStandardCSS" Brister wrote:
>
> Well I am that smart guy and I got the readahead error.

Not smart enough... :-)

> Boot to Ubuntu live CD

Not needed. If you want access to what's in the /var that's on the root
filesystem (i.e. what's under the /var that you mounted on /var) you can
simply bind mount the root filesystem somewhere else such as:

# mkdir /tmp/rootbind
# mount -o bind / /tmp/rootbind
# cd /tmp/rootbind/var

I wonder if this sort of thing could in fact be the solution to this
problem in an automated manner.

[Scott James Remnant (Canonical) (canonical-scott)]

The reason that the files are clumbed in /var/lib/ureadahead right now is that it needs to be easy to wipe them for reprofiling. Constant-reprofiling would mean we could just put the pack in $MOUNTPOINT/.ureadahead or something

The need for the temporary /var/lib/ureadahead/debug mount goes away too since /sys is now always mounted on boot by the kernel/init daemon

[Oliv' (omeunier2000)]

Thanks... but İ'm not sure İ understand what to do ;-)

Regars,

Olivier

________________________________
De : Scott James Remnant <email address hidden>
À : <email address hidden>
Envoyé le : Lun 17 mai 2010, 19h 33min 21s
Objet : [Bug 523484] Re: ureadahead requires /var on root filesystem

The reason that the files are clumbed in /var/lib/ureadahead right now
is that it needs to be easy to wipe them for reprofiling. Constant-
reprofiling would mean we could just put the pack in
$MOUNTPOINT/.ureadahead or something

The need for the temporary /var/lib/ureadahead/debug mount goes away too
since /sys is now always mounted on boot by the kernel/init daemon

--
ureadahead requires /var on root filesystem
https://bugs.launchpad.net/bugs/523484
You received this bug notification because you are a direct subscriber
of a duplicate bug.

Status in über-readahead: Invalid
Status in “ureadahead” package in Ubuntu: Triaged

Bug description:
I have the following mountpoints defined that use different partitions:

/
/boot
/home
/opt
/usr
/var

These nodes are located on different partitions to enable greater control over the system and prevent unusable states.
ureadahead.conf uses the following event:
  start on starting mountall
Which fails for me since /var is not yet mounted when mountall starts leading to ureadahead errors about missing pack files.

ureadahead-other.conf in turn uses
  start on mounted DEVICE=[/UL]* MOUNTPOINT=/?
I have yet to find out what that means but I see that ureadahead-other is terminated with error status 4 (i.e. no pack file)

I think that ureadahead configuration files should be ready for such kind of layout.
I have fixed the first item by rewriting the rule as start on mounted MOUNTPOINT=/var but I don't know what to do with second item.

To unsubscribe from this bug, go to:
https://bugs.launchpad.net/ureadahead/+bug/523484/+subscribe

[Scott James Remnant (Canonical) (canonical-scott)]

On Tue, 2010-05-18 at 08:11 +0000, Oliv' wrote:

> Thanks... but İ'm not sure İ understand what to do ;-)
>
Wait for it to be fixed in Maverick ;-)

Scott
--
Scott James Remnant
<email address hidden>

[Brian J. Murrell (brian-interlinx)]

On Tue, 2010-05-18 at 10:03 +0000, Scott James Remnant wrote:
>
> Wait for it to be fixed in Maverick ;-)

You are kidding right? Please tell me you are kidding.

I gotta say, it gets pretty tiring to continually be upgrading Ubuntu
releases only to find a whole bag of problems that will be "fixed in the
next release".

Between this issue the bag of issues that upstart/mountall have brought
to Lucid, what are people who only upgrade through LTS releases supposed
to do? Are we supposed to use our hacks (like always booting with
init=/bin/bash so that we can manually open a VT before we
exec /sbin/init and then use the VT to prod mountall/upstart along into
a full runtime boot) for the next two years with hope that the next LTS
release is better?

Sorry to bitch, but it seems I get more and more frustrated with the
quality of Ubuntu releases with every new one that comes along. Having
to manually (as in sit down at the console of) prod half a dozen
machines through booting is just the worst so far.

I can't imagine what people with a rack or data-centre full of them do.

b.

[Scott James Remnant (Canonical) (canonical-scott)]

On Tue, 2010-05-18 at 10:34 +0000, Brian J. Murrell wrote:

> You are kidding right? Please tell me you are kidding.
>
> I gotta say, it gets pretty tiring to continually be upgrading Ubuntu
> releases only to find a whole bag of problems that will be "fixed in the
> next release".
>
It's pretty tiring to be working flat out, doing twelve or sixteen hour
days every day, quite often working into weekends, not seeing your
friends or family and only seeing sunlight to shut it out of your office
because it's reflecting off the monitor and distracting you ...

<insert nervous break down here>

Seriously I'm not kidding, I work as hard as I can, and I just can't
perform superhuman feats.

If you have /var on a separate partition, you won't get working
ureadahead - this will be fixed in a future release, but due to the
fix's reliance on a kernel patch only *may* be backported into the LTS.

Scott
--
Scott James Remnant
<email address hidden>

[Jonathan (birge)]

'I gotta say, it gets pretty tiring to continually be upgrading Ubuntu
releases only to find a whole bag of problems that will be "fixed in the
next release"'

Welcome to linux. On the other hand, you didn't pay for it. On the third hand, half the problem here is that linux lets you see these error messages and also lets you do cool things like break your FS onto different partitions. Do you have any idea how many unresolved issues are flying by during Windows startup, but you just don't see them?

I have this problem, too, but put my two cents in for deferring it to Maverick. Slow boots aren't that big of a deal on a unix machine. How often are you people restarting your machines?

[Jonathan (birge)]

One thing to add, though: you guys REALLY should be testing with var as a separate partition. The whole reason for var is that it is files which grow and change, and the reason for splitting it off from the rest of the filesystem was so that you can put it on its own partition and not have it cause fragmentation on your other file systems. Thus, I agree with the other user that a separate /var partition is in no way a corner case. It's the smart way to do things, in fact, and the implicit message here is that Ubuntu isn't aimed at smart users.

[Michel (michel-crondor)]

A simple fix for this problem:
1. Create directory to store data in root filesystem: mkdir /.ureadahead
2. Mount your root fs somewhere, to be able to access it: mount / /mnt/tmp -o bind
3. Make two links to the new storage location:
rm -rf /var/lib/ureadahead
ln -s /.ureadahead /var/lib/ureadahead
ln -s /.ureadahead /mnt/tmp/var/lib/ureadahead # (create /mnt/tmp/var/lib if needed)
4. umount /mnt/tmp, reboot twice, and enjoy ureadhead!

[m4t (m4t)]

michel that is a great solution and worked for me (separate /var filesystem).

there is one thing missing though:

mkdir /.ureadahead/debugfs

-matt

[Paul Tansom (aptanet)]

To combine those last two comments, instead of using:

rm -rf /var/lib/ureadahead

I went for:

mv /var/lib/ureadahead /.ureadahead

which took an existing debugfs directory with it. That was the only content of the directory at the time. Other than that it all seems to be working fine, although I've not been aware of any performance improvement over what I was getting with the error and ending of ureadahead (but that's not the issue!).

[Brian J. Murrell (brian-interlinx)]

Will the bug be fixed for Maverick or is Maverick going to bet still
another release where running a system (sanely IMHO) with /var on a
separate partition is just not a supported configuration?

A separate /var is not a supported configuration. LOL.

[narcelio (narcelio)]

I´ve just upgraded my Ubuntu 10.04 to 10.10 and it can't boot anymore.

[PomCompot (pomme-compote-launchpad)]

Any proof of a link with ureadahead? Would be interesting for others to give some details on this (first, to know if we can make a migration with /var on another part).

[Brian J. Murrell (brian-interlinx)]

On Mon, 2010-10-25 at 01:53 +0000, narcelio wrote:
> I´ve just upgraded my Ubuntu 10.04 to 10.10 and it can't boot anymore.

Which you must be assuming is a problem with ureadahead, yes, or you
would not be posting to this bug.

More than likely your bug is not with ureadahead but one of the many
other bugs upstart/mountall and putting /var on a separate filesystem.

Indeed, it seems very likely that a responsible system deployment design
concept such as putting a filesystem that can grow without bounds on
it's own device (so as not to seize the whole system when it does fill
up) is outside the normal scope of what Ubuntu expects users to do and
therefore completely untested within Ubuntu's QA process.

Anyway, I'd suggest you start searching launchpad for the other bugs
with regard to Ubuntu not booting as I suspect ureadahead is not really
the problem.

[Ainsley Harewood (ainsley-rainstormgroup)]

Actually Brian I have a case like narcelio, I installed a fresh copy of Ubuntu 10.10 and created a separate partition for /var. After I tried to reboot and got unreadahead main process (336) terminated with status 5 and after that it just doesnt complete the booting process. The monitor comes off.

I am pretty sure that its /var because I installed it like half hour before and didnt put var on a separate partition and it worked perfectly.

[Brian J. Murrell (brian-interlinx)]

On Tue, 2010-10-26 at 20:31 +0000, Ainsley Harewood wrote:
> After I tried to
> reboot and got unreadahead main process (336) terminated with status 5
> and after that it just doesnt complete the booting process.

Right. This is a known and common problem. It's not caused by
ureadahead though. Yes, ureadahead does complain when /var is on a
separate filesystem but it doesn't prevent booting anyway. What is
preventing your booting is just not displaying any messages about it.
But again, it's not ureadahead. ureadahead's message just happens to be
the last thing printed before whatever else is preventing your boot from
completing breaks.

> I am pretty sure that its /var because I installed it like half hour
> before and didnt put var on a separate partition and it worked
> perfectly.

You are misunderstanding me. I am not saying your /var is NOT on a
separate filesystem. I'm just saying that ureadahead is not preventing
your system from booting.

[Alf Gaida (agaida)]

@Ainsley Harewood:
There is a simple proof that ureadahead don't harm your boot process. Chroot into the machine and purge ureadahead. Then reboot. If the machine dont boot, solve the problem. If the machine boot, your problem is solved.

To clearify some things: In a perfect world ureadahead should work on machines (servers) with a separate /var without problems. IMHO ureadahead is totaly useless on a server because it's another piece of software that could cause errors. Therefore there is no need to use ureadahed on a server. Ureadahead is dedicated to desktops to boost up the boot speed. How often you will boot a server? Once a month or year? The 20 or 40 secs are not worth any potential problem. Sorry for my bad english.

[Pavel Malyshev (afunix)]

So ureadahead will never support configurations with separate /var ?
Can this be fixed in upstart configuration with "start on mounted DEVICE=[/UL]* MOUNTPOINT=/var" as suggested in original bug description?

[hedgehog (hedgehogshiatus)]

This **very** simple fix worked for me:

[URL="http://ubuntuforums.org/showpost.php?p=9555086&postcount=9"][B]sentinella86[/B][/URL]

and if you want to track another bug report:

[URL="https://bugs.launchpad.net/ubuntu/+source/ureadahead/+bug/542334"][B]boot fails [...] ureadahead [...] terminated with status 5[/B][/URL]

HTH?

[Brian J. Murrell (brian-interlinx)]

On Sat, 2010-10-30 at 04:59 +0000, hedgehog wrote:
>
> and if you want to track another bug report:
>
> [URL="https://bugs.launchpad.net/ubuntu/+source/ureadahead/+bug/542334"][B]boot
> fails [...] ureadahead [...] terminated with status 5[/B][/URL]

Why would I want to track that bug report? It's just another one of
those (too many, IMHO) bug reports where it starts out with a developer
trying to resolve a bug and requests information from a user and after
the user supplies the information the developer just ignores the
response, and indeed the entire bug and the bug just turns into a
conglomeration of "me too"s with no further effort by the developer to
actually resolve the bug.

Following that bug would just be a waste to time, IMHO.

[Raubvogel (raubvogel)]

So, if ureadahead is not the guilty party, who is?

I have a few 10.04 LTS servers which I have configured with separated /var and they have been working fine so far <knock on wood>.

I have another 10.04 LTS machine in which I created a separate partition for /var but forgot to configure it during install. No problem, I thought, I just had to boot in single user mode (rescue mode in Linux?), edit fstab, move /var to /var.old, and create a new /var. So far I am wrong: if I try to reboot I too get the same error message. So, I deleted the pack files located in /var/lib/ureadahead/, added "console output" to /etc/init/ureadahead.conf and tried again. This is what I got:

udevd-work [298] open /dev/null failed: no such file or directory
udevd-work [300] open /dev/null failed: no such file or directory
udevd [108] : worker [199] did not accept message -1 (connection refused), kill

udevd-work [118]: inotify_add_watch (6, /dev/sda3, 10) failed: no such device or directory
init: ureadahead-other main process (309) terminated with status 5

Don't ask me why it can't find /dev/null. It could be completely unrelated to this. FYI, sda3 is the lVM device; inside it is /var. / is sda2 and /boot sda1. Also, after I restarted the machine using the /var that is on the root filesystem, I did get to see the

udevd-work [118]: inotify_add_watch (6, /dev/sda3, 10) failed: no such device or directory

message during boot time but it did not stop the boot.

[Tom Dickson (8-launchpad-bombcar-com)]

I had this problem, trying to boot Ubuntu 10.04 from a mirrored GPT system, failed until I moved var back to the / partition. Then it worked fine.

[PomCompot (pomme-compote-launchpad)]

@hedgehog#29
Brillant simple patch! Works like a charm now.

[Clint Byrum (clint-fewbar)]

I just have a couple of thoughts on this bug.

* The earlier comment that ureadahead could just buffer this in memory and write later wouldn't be all that difficult to implement. The potential for a lot of data is unknown to me (on my two systems I checked, the file is around 700kb), so that may be a limiting factor. However, in reading the code, it seams ureadahead pretty much already just builds all of this up in memory and then flushes it out at one time anyway.

The real issue is that there won't be a pack file to read from until the real /var from the last profiling attempt is mounted anyway.

So I was thinking we should just install a SIGUSR1 handler, then try to access /var/lib/ureadahead .. if that works.. ignore SIGUSR1, then plow on as we do now.

If not, go to sleep. Every time we get woken up, repeat the logic above. Then we can add this upstart job:

# ureadahead-filesystem
start on mounted

env MOUNTPOINT=""
instance $MOUNTPOINT

task
exec killall -SIGUSR1 ureadahead
# EOF

This will send ureadahead a SIGUSR1 to try and open /var/lib/ureadahead every time a filesystem is mounted.

I suppose we could use SIGSTOP and SIGCONT too, but this feels a little less prone to stepping on ptrace's toes.

[gaberlunzie (gerardda)]

I am running 11.04 64-bit and also have /var as a separate partition. Not only ureadahead(-other) but others such as dbus, gdm and dpkg cannot create files under /var/lib since the required directories are non-existent. The last thing I remember doing before boot failure was software upgrade this past week. Something seems to be blocking /var from use, creating a bottleneck giving errors that fsck cannot repair, namely:

init: ureadahead-other main process (...) terminated with status 4
init: ureadahead main process (...) terminated with status 5
init: dbus pre-start process (...) terminated with status 1

I can mount and chroot into my filesystem using live cd (which I'm doing now) to perform operations. The problem doesn't appear to be a broken grub2 one, as those repair operations as instructed run without complication, and fstab UUIDs checks out to be consistent. This situation looks like resort to yet another clean install situation for me, one too many, IMO.

[Pavel Malyshev (afunix)]

Is this another bug, which will never be fixed?

[Brian J. Murrell (brian-interlinx)]

On 11-06-12 02:10 AM, afunix wrote:
> Is this another bug, which will never be fixed?

Seems so. Throw it on the every growing pile. Like the NFS4/KRB5 bug
that has been ignored for going on (or is it over?) 6 months now.

TBH, I am starting to look at different distros. I am getting tired of
trading one bag of bugs for another bag of new bugs with every Ubuntu
distro upgrade and having to live with them for several more releases
while they are ignored on and on.

Ubuntu has lost it's shine AFAIC. It's getting dumbed down for the
masses and buggy as hell in the process.

[JasonPorter (jasonporter)]

This is still an issue in Ubuntu Server 11.04.

Isn't it a relatively common use case for /var to be on its own partition on a server?

Either fix the ureadahead behavior, or eliminate it from the server builds entirely. Most server admins don't care much about boot speed, but they do care about buggy exits during boot that spam the logs.

[maccus (maccus)]

I suppose this could easily be fixed within the package scripts. Just let the scripts check if the package is installed on a system with separate /var and modify the to be installed init file accordingly.

I currently use something like this in a local standalone postinstall script, i guess it could be modified to accomodate the ureadahead package:

if grep -q '/var' /etc/fstab; then

    grep -q '/var' /etc/init/ureadahead.conf || sed -i "s|^start on starting mountall|start on mounted MOUNTPOINT=/var|" /etc/init/ureadahead.conf

fi

[David F. (malteworld)]

@Maxim #39
The basic idea seems good, but there are more reliable ways to detect a separate /var partition:

! mountpoint -q /var || sed -i 's|^start on starting mountall|start on mounted MOUNTPOINT=/var|' /etc/init/ureadahead.conf

[Daniele Varrazzo (daniele-varrazzo)]

FYI: to mr "Wait for it to be fixed in Maverick ;-)": the bug is still there on Oneiric.

A fix on the line of #17 works nice.

[Tim Gehpunkt (rollercoaster)]

The fix linked in #29 is better and easier. I don't like to have variable files on / . Thats why we use seperate /var partitions ;)
This is the change to the upstart conf:
sudo sed -i 's+^start on starting mountall+start on mounted MOUNTPOINT=/var+' /etc/init/ureadahead.conf

@Alf Gaida
heretic question: is ureadahead installed in Ubuntu Server? ;)

But I'm also very disappointed of 10.04. It should not be called LTS, it should be called "Unsupported". I think we can wait until "zappy zebra" for this to be fixed.

[Alf Gaida (agaida)]

@rollercoaster
i finally fixed the problems with ubuntu ureadahead, mdadm, mountall and plymouth. i run my servers on debian stable or sid. ;)

[Nicolas Michel (nicolas-michel)]

Still there. On a production server (maverick). As you know, production servers are always setup with separated /var, /home, /tmp to make it stronger to disks full ...

Not really clean indeed seing all these errors at the boot process ... Even if it doesn't hurt.

[Davide Capodaglio (davidecapod)]

The bug is still there in 12.04...
it seems that nothing has been done about it....

[Bill Trost (trost-l)]

I'm currently running 11.10 and worked around the problem by copying /var/lib/ureadahead to / and the pointing a /var/lib/ureadahead symlink to /ureadahead both in the /var partition and from the root partition (accessed using the bind mount trick in comment 17). Now, when I next I'll get to see if it actually made any difference. I should know something by August!

[Pavel Malyshev (afunix)]

Why ureadahead does not have an option to change pack storage directory?
This bug can be workarounded easily if I'm able to create /ureadahead and set that option in ureadahead-other.conf

[Pavel Malyshev (afunix)]

So 2 years passed and this bug does not even have an assignment...

[Alf Gaida (agaida)]

There is a clear statement from JSR: There will be a fix eventually. Sooner or later. This was 2 years ago ...

[pqwoerituytrueiwoq (pqwoerituytrueiwoq)]

Just a reminder this is still a issue in raring (13.04) and this command fixed it
sudo sed -i 's+^start on starting mountall+start on mounted MOUNTPOINT=/var+' /etc/init/ureadahead.conf

ureadahead should check is /var is ready then continue, so the above would not be needed

i also get status 5 error when using mainline kernels, i am using saucy kernels
i have a separate /var partition

[Stéphane Graber (stgraber)]

Targeting this to 13.11, this bug got mentioned a few times on IRC over the past week in relation to Ubuntu Touch.

The current consensus is that Clint's idea (comment #34) is the best way to fix this in a generic way and that fix should definitely be implemented for 14.04 LTS.

[oldos2er (oldos2er)]

This bug exists for me in 14.04 alpha. All my top level root folders are mounted on a single ext4 partition.

[oldos2er (oldos2er)]

Addendum to post #52, running the command given in post #50 fixed the issue for me.

[Pavel Malyshev (afunix)]

Four years passed...
Seriously?
Trusty will be the third LTS release with the bug. What is the point to use LTS releases if such pure-Ubuntu bugs are not fixed fore years?

[Brian J. Murrell (brian-interlinx)]

On Sun, 2014-02-23 at 14:00 +0000, Pavel Malyshev wrote:
> Four years passed...
> What is the point to use LTS releases if such pure-Ubuntu bugs are not fixed fore years?

I asked myself that same question a while ago. This was my
conclusion/solution:

$ cat /etc/issue
Fedora release 20 (Heisenbug)
Kernel \r on an \m (\l)

Not that I think Canonical/Ubuntu cares that I no longer use Ubuntu.
From what I can see, it seems they are focused on TVs and mobile devices
now anyway and what's on the desktop is just a test ground for their
real markets and whatever they don't need in those markets just doesn't
get attention on the desktop any more.