Ubuntu

100% CPU usage in init if /dev/console is not available

Reported by Cedric Schieli on 2011-10-22
86
This bug affects 16 people
Affects Status Importance Assigned to Milestone
upstart (Ubuntu)
Undecided
Unassigned

Bug Description

When /dev/console is not available (for example when running as an OpenVZ guest), init consumes 100% time of one CPU core.
Here is a strace excerpt captured on the OpenVZ host :

...
waitid(P_ALL, 0, {}, WNOHANG|WEXITED|WSTOPPED|WCONTINUED, NULL) = 0
clock_gettime(CLOCK_MONOTONIC, {3903451, 826182267}) = 0
select(5, [0 3 4], [], [4], NULL) = 1 (in [0])
read(0, "", 1) = 0
waitid(P_ALL, 0, {}, WNOHANG|WEXITED|WSTOPPED|WCONTINUED, NULL) = 0
clock_gettime(CLOCK_MONOTONIC, {3903451, 826467596}) = 0
select(5, [0 3 4], [], [4], NULL) = 1 (in [0])
read(0, "", 1) = 0
waitid(P_ALL, 0, {}, WNOHANG|WEXITED|WSTOPPED|WCONTINUED, NULL) = 0
clock_gettime(CLOCK_MONOTONIC, {3903451, 827157279}) = 0
select(5, [0 3 4], [], [4], NULL) = 1 (in [0])
read(0, "", 1) = 0
...

Here is what happens :

- in init's main function, system_setup_console(CONSOLE_OUTPUT) fails, leaving file descriptors 0 and 1 uninitialized
- nih_main_loop_init eventually get called, initializing its interrupt_pipe on file descriptors 0 and 1
- we end up calling system_setup_console(CONSOLE_NONE), resetting file descriptors 0 and 1 to /dev/null
- in nih_main_loop, select is called on 0 (/dev/null instead of the pipe) which repeatedly appears to have changed

I'm attaching a small patch which fixes this.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in upstart (Ubuntu):
status: New → Confirmed
davidlyness (davidlyness) wrote :

I am affected by this and do not have the technical knowledge to modify and rebuild the upstart source code. Is there a fix that does not require altering the source, or is this likely to make it into an update some time soon?

NoahY (noahy) wrote :

I have the same basic problem, but the patch given doesn't do anything. Trying installing upstart 1.6.7, no better. init is always at 100% cpu.

James Hunt (jamesodhunt) wrote :

Thanks for reporting this issue. However, the console device is a device expected to exist on all Linux systems. The problem here is that Upstart *needs* access to the console: if it has no console access, jobs would behave differently in an OpenVZ container to running outside a container (bad). The patch above will not help jobs that have specified they want console access or ownership ("console output" or "console owner").

I am not an OpenVZ expert, but I do feel that the "fix" for this issue is for the OpenVZ environment to provide a /dev/console that Upstart is able to use such as would be provided by other virtualization technologies (LXC, qemu/kvm, Xen, VirtualBox) otherwise Upstart cannot guarantee job behaviour.

Cedric Schieli (cschieli) wrote :

I totally agree the proper fix would be that OpenVZ provides a /dev/console. My patch is rather a workaround.
Concerning jobs specifying console access, they already fallback to CONSOLE_NONE in job_process_spawn so they are working (maybe not exactly with the same behaviour though).
Concerning jobs specifying console ownership, they will fail indeed, but those jobs (rcS, mountall-shell, ...) really can't be used in a OpenVZ environment anyway.

NoahY (noahy) wrote :

Oh, I'm sorry for my stupidity... can confirm that the patch above works here - no more 100% cpu usage after rebooting.

Just wanted to confirm the patch is working for me, many thanks! Hope to see it in upstream soon :)

Ajunne (ajunne) wrote :

I have the same issue here, physical server runs CentOS 6.0, VE runs Ubuntu.

You can easily grant access to the physical version of /dev/console with the following command on your physical server:

vzctl set 105 --devices c:5:1:rw --save

Replace 105 with the number of you VE.

After this, reboot and init will no longer be hogging 100% cpu.

@Ajunne

Unfortunately that doesn't work for me.
After granting access to /dev/console to the VE (and rebooting it), I get
"No such device" instead of "Operation not permitted" when trying to access
the device and init still hogs the cpu.
Maybe it is a problem with the host (Debian 6.0,
linux-image-2.6.32-5-openvz-686 = 2.6.32-31) ? I'll investigate more deeply
in a few days.

James Hewitt (jammy) wrote :

I would consider this a regression - it was working fine in Natty, so has been broken for oneiric.

I would like to see a default ubuntu instance work without changes in an openvz environment - it would help encourage the use of ubuntu for virtualized environments. For me, ubuntu wouldn't start at all until I had made some changes to the init scripts.

This is definitely still broken :(

The specified vzctl command does not fix anything...

ahmadshamli (ahmadshamli87) wrote :

I'm using Parallel Virtuozzo. After upgrade from 11.04 to 11.10, init is using 100% of cpu all the time.

About the patch,how to use it.?

Cedric Schieli (cschieli) wrote :

I found another workaround which do not involve patching : simply delete
/dev/console in the VE and replace it with a plain file.

On Mon, Nov 7, 2011 at 4:45 AM, Cedric Schieli <email address hidden> wrote:
> I found another workaround which do not involve patching : simply delete
> /dev/console in the VE and replace it with a plain file.

Doesn't work. VE will not boot if the file is a regular file...
--
Kristian Erik Hermansen
https://profiles.google.com/kristian.hermansen

2GooD (david+launchpad) wrote :

It seems like there is a patch for this in the upstart trunk already?

http://bazaar.launchpad.net/~upstart-devel/upstart/trunk/revision/1326

Scott James Remnant (scott) wrote :

Please don't subscribe people to bugs without their permission

2GooD (david+launchpad) wrote :

I'm sorry Scott, I will not do that again. Thanks for your patience.

David Schoen (neerolyte) wrote :

> On Mon, Nov 7, 2011 at 4:45 AM, Cedric Schieli <email address hidden> wrote:
> > I found another workaround which do not involve patching : simply delete
> > /dev/console in the VE and replace it with a plain file.
>
> Doesn't work. VE will not boot if the file is a regular file...

I was able to work around this by replacing /dev/console with a symlink to /dev/null and rebooting, i.e.:
# rm /dev/console
# cd /dev
# ln -s null console
# reboot

This was on something the vendor is calling a "Virtuozzo" which I think might just be OpenVZ under the hood.

James Hewitt (jammy) wrote :

This has actually stopped happening on my server, after an upgrade by my hosting provider.

For those still affected by the issue, I have created a PPA that contains the latest oneiric package, with just this one upstream bugfix added:
https://code.launchpad.net/~jammy/+archive/upstart.fix-880049

Its waiting to build on launchpad now, packages should be available in a few hours.

Torsten Knodt (torsknod) wrote :

I would like to add something to the opinion that openvz should fix this.
I agree that it should be fixed by openvz.
But many people are not in control of the openvz/ Virtuozzo installation, because they just have rent a virtual server. And as it worked before it should still work because else those affected people don't trust into the dependability of Ubuntu any more. And by the way, there are many fixes for bad hardware in the kernel, why shouldn't there be fixes in Ubuntu for bad virtual hardware.

On Wed, Nov 23, 2011 at 6:08 PM, David Schoen <email address hidden> wrote:
> I was able to work around this by replacing /dev/console with a symlink to /dev/null and rebooting, i.e.:
> # rm /dev/console
> # cd /dev
> # ln -s null console
> # reboot

Container will not boot (only mount) if the above is attempted. Not a
solution...
--
Kristian Erik Hermansen
https://profiles.google.com/kristian.hermansen

pva (pva) wrote :

And openvz thinks that this is bug in upstart: http://bugzilla.openvz.org/show_bug.cgi?id=2165#c4

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Related questions

Remote bug watches

Bug watches keep track of this bug in other bug trackers.