Failed to create pty - disabling logging for job [SRU]

[Tim Heckman]

[PROBLEM]

This bug causes Upstart to display 1 error message / job that is started that should have its output logged (by default ''all'' jobs are thus affected) if it is unable to create a pseudoterminal device, which is required for Upstart job logging.

[IMPACT]

This bug affects all users that:

- boot their systems without an initramfs
- have non-standard /dev setups (for example a static /dev).

Backporting this fix to precise makes sense since:

- precise is an LTS.
- this bug is variously annoying/intimidating to users.
- this bug stops Upstart from logging job output at boot time.
- the comments on this bug show that users are already running in the environments outlined above.

[FIX DETAILS]

Upstart will now mount /dev as a devtmpfs filesystem if it fails to find expected device entries (/dev/ptmx and /dev/pts/) on the disk at boot time.
If any of the following devices do not exist (after any /dev mount attempt):

- /dev/ptmx (for pty support, required for Upstart job logging).
- /dev/null
- /dev/tty
- /dev/console
- /dev/kmsg (for early message logging)

[TEST CASE]

To reproduce this issue:

1) hold down control key at power-on to enter grub menu.
2) press 'e' to edit the top (default) kernel command-line.
3) modify the line beginning with "linux" such that:
  3.1) "quiet" is removed.
  3.2) "splash" is removed.
  3.3) "root=" is set to the *device name*, rather than a UUID
       (generally, this will be something like "root=/dev/sda1").
  3.4) add "rootfstype=<filesystem_type>"
       (this will be either "rootfstype=ext4" or "rootfstype=ext3").
  3.5) You'll now have a linux line something like this:
       "linux /boot/vmlinuz-3.2.0-27-generic root=/dev/sda1 rootfstype=ext4 ro"
4) *delete* the line beginning "initrd" entirely.
5) Press Control+x or F10 to boot.
6) Watch the boot messages and you will see various error messages including the following repeated multiple times:
   - "init: Failed to create pty - disabling logging for job".

[REGRESSION POTENTIAL]

None identified.

This fix is in upstream Upstart and quantal as of upstart version 1.5-0ubuntu8 (added to archive on 8 August 2012).

[PRE-SRU DESCRIPTION]

I believe this bug is in Upstart due to similar bug that was recently fixed:

- https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/936667

I am booting Ubuntu 12.04 LTS (x86) as a domU under Xen. After the previous bug was fixed, I still received the errors but the system now boots which is an improvement. During the boot process I see a few combinations of this error:

====
init: Failed to create pty - disabling logging for job
init: Temporary process spawn error: No such file or directory
====

After those two stop, I see a few more of these errors:

====
init: Failed to create pty - disabling logging for job
====

The above error also prints if I type "poweroff" or "shutdown"

This system's ultimate destiny is to operate using a custom-compiled Linux kernel that does not exist on the filesystem. Both the Ubuntu-supplied Linux kernel and the custom-compiled Linux kernel are afflicted by this bug.

Any suggestions would be greatly appreciated.

-Tim

//edit: added more details, edited for clarity.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[James Hunt (jamesodhunt)]

Hi Tim - the messages you are receiving are expected behaviour since Upstart is signalling an error condition - the environment it expects is not correct. Upstart currently expects /dev/pts to be mounted and the /dev/ptmx device to exist by the time it starts. On Ubuntu, this is currently handled by the initramfs. It is also possible from what you say that your custom kernel does not have full pty support enabled? A workaround for the messages is to boot with '--no-log' (see http://upstart.ubuntu.com/cookbook/#command-line-options) with the implication that all job output is discarded (as it used to be before job logging was added to Upstart).

[Tim Heckman (theckman)]

Hello James,

Thank you for taking the time to look in to this bug as well as get back to me. The explanation of the issue definitely makes the problem more clear. However, I am intrigued by the fact that something so important is assumed, and that Upstart errors when it encounters something different (such as there being no initramfs thus /dev/pts not being mounted). However, I'm not intimately familiar with the inner workings, but I would hope there's a way to fix it without having to implement the '--no-log' workaround.

The custom-compiled kernel that's being used to boot the system should have sufficient PTY support enabled:

----
~# zgrep -i "pty" /proc/config.gz
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
----

Thank you for the suggestion for the workaround. Sadly, I don't think that's something I can implement due to some restrictions with the environment that Ubuntu 12.04 LTS will be running in (at least in my case). But if this isn't something you guys are able to resolve in Upstart before release, alternative options may need to be looked at. I would also love to be able to have job logging enabled, as I feel it is an important addition to Upstart.

Thank you again for your time, James. I look forward to seeing how this issue irons out.

-Tim

[Adam Honse (calcprogrammer1)]

I'm having this same bug on the ARM port of Ubuntu. I have LEGACY_PTYS=y, LEGACY_PTY_COUNT=256, and UNIX98_PTYS=y on my custom 2.6.35 kernel (this is for the HP TouchPad tablet computer btw) and after the system is booted there are /dev/ptmx, /dev/pts, and /dev/ptyXX entries in the system. I get the same errors on my other ARM device (Archos 43 tablet with 2.6.37 kernel). I think the main issue I have isn't that job logging isn't enabled, it's that the dmesg is spammed with error messages. In the event that this error occurs it should not spew 100's of lines of errors, it should print a single, detailed, informative message and permanently disable logging for the session so the error doesn't print again.

[James Hunt (jamesodhunt)]

@Adam: the point here is that Upstart is signalling an error, albeit a soft one. However, in all cases it will be a transitory error since at some point in the boot /dev/pts _will_ be mounted.

For i386/amd64 Ubuntu images, Ubuntu currently uses an initramfs which is responsible for setting up /dev/pts. In the case of ARM where you have no initramfs, /dev/pts will not be mounted when Upstart starts. Currently the problem will 'correct itself' once mountall has finished since it will mount /dev/pts if not already mounted. So, currently if you have no initramfs you will get one message / job that Upstart started before /dev/pts is available.

The full solution to this problem involves a bit of redesign since the mountall model doesn't work well if you have no initramfs -- unless you disable job logging. A pragmatic fix would be to have Upstart mount /dev and /dev/pts itself and if that fails continue to display the message with the addition of the job name so it is clear which jobs will not have their output logged.

[Tim Heckman (theckman)]

James: It seems that this is a chicken and egg problem when someone doesn't have an initramfs. I'm quite alright with logging being disabled if "/dev/pts" isn't mounted. The problem I have is the large amount of unhelpful errors that are printed on the console. If this could be a concise error, that would be great:

"init: Failed to create pty - temporarily disabling job logging"

And that should be the only error, in my opinion. On the opposing side, I can see where having the specific job that wasn't logged noted. However, that doesn't happen now. And if it flooded the console, I don't think I'd be for it.

Again, without knowing how Upstart processes jobs (maybe I should fix that), I would probably be okay with something like:

"init: Failed to create pty - disabling job logging for the following:
Job1, Job2, Job3, etc."

Simply add a job to the end of the comma-separated list if it isn't logged.

-Tim

[Benjamin Kerensa (bkerensa)]

Thanks for the feedback on having a more concise error message perhaps this is something that could be improved.

[Henri Hulski (henri-hulski)]

I have exactly the same error on a brandnew precise-upgrade on an i386 system (Toshiba SM-30 Laptop).

[Kevin McCaughey (ohforfs)]

The behaviour seems wrong to me - I believe it needs to be changed so that it follows the previous expected (and acceptable) model. As a bit of a metaphor, if Toyota released a new car but all the new cars had a red warning light on them and Toyota's fix was "You all have to press a button in the boot and cover the warning light with tape" there would be an uproar.

Really, I think this needs to be fixed at the system end rather than hundreds of thousands of "fixes" being applied at the user end.

[Christopher M. Balz (christopherbalz)]

In my case, a seemingly plain-vanilla, straightforward upgrade from Ubuntu 10.04 LTS on Ubuntu-supported hardware (leo1 from System76) to Ubuntu 12.04 resulted in an inability to boot into anything more than GRUB, and also (once the system was booted) no DNS.

I detail my successful fix, and the process getting there, here: http://chrismbalz.typepad.com/computerburn/2012/05/rough-upgrade-from-ubuntu-1004-lts-to-1204-lts.html

[Tim Heckman (theckman)]

Christopher: Would you be able to clarify how your upgrade going sideways is related to a bug in Upstart when /dev/pts isn't mounted by an initrd?

-Tim

[Goswin von Brederlow (goswin-v-b)]

James Hunt (jamesodhunt) wrote on 2012-04-25: #5

@Adam: the point here is that Upstart is signalling an error, albeit a soft one. However, in all cases it will be a transitory error since at some point in the boot /dev/pts _will_ be mounted.

Aparently that isn't the case. I'm booting with NFS-root so no initrd and mountall also can't be used. Instead I have an upstart job that mounts /dev and /dev/pts and creates /dev/ptmx (see https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/1006517) and the error only goes away after restarting upstart, not once /dev/pts + ptmx is available.

[fresh-nes (fresh-nes)]

Hi,

I am working on a Pandaboard ES.
I try to boot my Pandaboard over NFS, using Ubuntu 12.04
My NFS connection is OK, but it stops during the start of modules.
When I boot the same fs, using an SD card, it works fine.
(I use the kernel option --no-log in order to avoid "init: Failed to create pty - disabling logging for job")

====
…
[ 9.431549] Sending DHCP requests .
[ 11.005828] smsc95xx 1-1.1:1.0: eth0: link up, 100Mbps, full-duplex, lpa 0xCDE1
[ 11.871856] ., OK
[ 11.909912] IP-Config: Got DHCP answer from 10.1.2.21, my address is 10.1.9.22
[ 11.918365] IP-Config: Complete:
[ 11.921752] device=eth0, addr=10.1.9.22, mask=255.255.0.0, gw=10.1.2.251,
[ 11.928161] host=10.1.9.22, domain=xxx.xxx, nis-domain=(none),
[ 11.936035] bootserver=10.1.2.21, rootserver=10.1.9.40, rootpath=
[ 11.943511] Waiting 2sec before mounting root device…
[ 13.963836] VFS: Mounted root (nfs filesystem) on device 0:14.
[ 13.970977] devtmpfs: mounted
[ 13.974334] Freeing init memory: 420K
* Starting network connection manager[ OK ]
* Starting mDNS/DNS-SD daemon[ OK ]
* Starting User Mode Init Manager for TI shared transport[ OK ]
* Starting bluetooth daemon[ OK ]
* Starting load fallback graphics devices[ OK ]
* Starting Userspace bootsplash[ OK ]
* Stopping load fallback graphics devices[ OK ]
====

And nothing else.

Could you explain me how to modify my upstart in order to boot my Pandaboard over NFS, using Ubuntu 12.04

Best regards,
PixMe

[Tim Heckman (theckman)]

James,

Is it conceivable that we will see a fix released in time for the Ubuntu 12.04.1 release in August? I think it's taken quite a bit longer than need to triage this bug and determine the priority of the fix. It's affecting quite a large number of people on different platforms. I'm simply trying to understand what the priority is, if any, to get a fix in place.

Are paying customers of Canonical still having this issue? Is this the type of support Ubuntu users can expect on an LTS release?

----

PixMe: Not sure the comments of this bug are the proper place to get help for such an issue. I'm not familiar with Pandaboard, but I'm fairly certain this bug is completely unrelated. I'd suggest reaching out to the community forum.

-Tim

[Colin Watson (cjwatson)]

So, running without an initramfs isn't currently a supported configuration due mainly to bugs such as this, although we would like to make it so in future for various reasons. As such I can't currently justify a priority higher than Medium. However, if the fix turns out not to be too intrusive (on which I'll have to defer to James) then it would be a reasonable target for 12.04.1 as the impact is clearly serious for those who encounter it.

[Alexey Subbotin (avs234)]

The most important system processes run without logging, and you can't justify a higher priority?
This issue looks _critical_ to me, all the more so as the users are already having obscure problems they're unable to trace.
Hope you solve it soon and wish you good luck!

[sagar shiwarkar (shiwarkarsagar)]

I got this message in vnc when I ran virtual machine of ubuntu 12.04 with the default kernel 2.6.32-41-server then with the updated kernel 3.2.0-24-generic but error still persist.
I can login to the vm may be because --no-logs option in the /boot/grub.cfg.
output of the kernel ring buffer

[ 124.901316] init: Failed to create pty - disabling logging for job
[ 124.901937] init: Temporary process spawn error: No such file or directory
[ 124.906374] init: Failed to create pty - disabling logging for job
[ 124.906962] init: Temporary process spawn error: No such file or directory

regards
-sagar

[Steve Langasek (vorlon)]

James has taken a crack at this and found it to appear not to be unintrusive, or at least not tractable in time for .1. Postponing to .2.

[Alexey Subbotin (avs234)]

good! anyway better than a crowd decided to change init etc with _no_
knowledge at all

On Thu, Jul 26, 2012 at 8:24 PM, Steve Langasek <
<email address hidden>> wrote:

> James has taken a crack at this and found it to appear not to be
> unintrusive, or at least not tractable in time for .1. Postponing to
> .2.
>
> ** Changed in: upstart (Ubuntu Precise)
> Milestone: ubuntu-12.04.1 => ubuntu-12.04.2
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/980917
>
> Title:
> Failed to create pty - disabling logging for job
>
> Status in “upstart” package in Ubuntu:
> Triaged
> Status in “upstart” source package in Precise:
> Triaged
>
> Bug description:
> I believe this bug is in Upstart due to similar bug that was recently
> fixed:
>
> - https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/936667
>
> I am booting Ubuntu 12.04 LTS (x86) as a domU under Xen. After the
> previous bug was fixed, I still received the errors but the system now
> boots which is an improvement. During the boot process I see a few
> combinations of this error:
>
> ====
> init: Failed to create pty - disabling logging for job
> init: Temporary process spawn error: No such file or directory
> ====
>
> After those two stop, I see a few more of these errors:
>
> ====
> init: Failed to create pty - disabling logging for job
> ====
>
> The above error also prints if I type "poweroff" or "shutdown"
>
> This system's ultimate destiny is to operate using a custom-compiled
> Linux kernel that does not exist on the filesystem. Both the Ubuntu-
> supplied Linux kernel and the custom-compiled Linux kernel are
> afflicted by this bug.
>
> Any suggestions would be greatly appreciated.
>
> -Tim
>
> //edit: added more details, edited for clarity.
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/980917/+subscriptions
>

[Launchpad Janitor (janitor)]

This bug was fixed in the package upstart - 1.5-0ubuntu8

---------------
upstart (1.5-0ubuntu8) quantal; urgency=low

  * Merge of latest upstream including doc fixes, fixes for booting
    on initramfs-less systems and improved error handling in spawn
    failure scenarios (LP: #980917, #1032101).
 -- James Hunt <email address hidden> Wed, 08 Aug 2012 10:05:34 +0100

[Alexey Subbotin (avs234)]

>> This bug was fixed in the package upstart - 1.5-0ubuntu8
Thanks a lot! Works like a charm now.

[Pavel Malyshev (afunix)]

Will this fix be ported in 12.04.1 or it has already escaped test plan for .1?

[Bobby Powers (bobbypowers)]

what needs to happen to have this backported into precise?

[Bobby Powers (bobbypowers)]

I should note that I've manually installed the quantal 1.5-0ubuntu8 upstart package in my precise VM and the 'failed to create pty' errors are gone, and everything else of mine is working fine

[James Hunt (jamesodhunt)]

There is currently no formal SRU branch for Precise, hence the debdiffs above. These changes are also in branch:

lp:~jamesodhunt/ubuntu/precise/upstart/sru-bug-980917

[James Hunt (jamesodhunt)]

Upstart version 1.5-0ubuntu7.1 has been uploaded to precise-proposed.

[Chris Halse Rogers (raof)]

Hello Tim, or anyone else affected,

Accepted upstart into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/upstart/1.5-0ubuntu7.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Hongwu Li (lihongwu1987)]

The fix released here fixed my issue, upstart version: 1.5-0ubuntu7.1
$ dpkg -s upstart
Package: upstart
Status: install ok installed
Multi-Arch: foreign
Priority: required
Section: admin
Installed-Size: 1172
Maintainer: James Hunt <email address hidden>
Architecture: i386
Version: 1.5-0ubuntu7.1
tag changed to 'verification-done'

[James Hunt (jamesodhunt)]

Branch lp:~jamesodhunt/ubuntu/precise/upstart/sru-bug-980917 updated and new package created and uploaded (upstart_1.5-0ubuntu7.2) to correct build failure seen on ARM.

Note that the changes are purely test code changes: the code for Upstart itself has not changed in this new version.

[Colin Watson (cjwatson)]

Hello Tim, or anyone else affected,

Accepted upstart into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/upstart/1.5-0ubuntu7.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Alessio Igor Bogani (abogani)]

The 1.5-0ubuntu7.2 fix the bug here (Ubuntu Core 12.04 on armhf).

[Adam Conrad (adconrad)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package upstart - 1.5-0ubuntu7.2

---------------
upstart (1.5-0ubuntu7.2) precise-proposed; urgency=low

  * Cherry-pick newer test code to fix test failure seen
    on ARM (LP: #980917).
 -- James Hunt <email address hidden> Thu, 17 Jan 2013 11:39:27 +0000

[Ronald (ronald645)]

Posting this as an informative notification:

It seems this fix probably causes regression bug #1105807.