Machine fails to commission when console=ttyS0 is present on kernel opts

[Julian Edwards]

Maas is pxe booting systems. The first time a system is booted, maas knows nothing about it, so we need some default kernel parameters. The primary target for these systems is datacenter, where there is at least some likelyhood that /dev/ttyS0 is logged (or could be logged). On boot of the system, we want to see log messages. So, we were passing to the kernel cmdline:
   console=tty1 console=ttyS0
The hope was that kernel messages would go both places, and /dev/console (which is opened by upstart) would go to ttyS0 if it was present and /dev/tty1 otherwise. It seems that the kernel is not at all smart about how it does this, though. On a system without a serial device, /dev/console gets hooked up, but writes to it fail with "Input/output error".
  Cloud-init uses 'console output' in upstart as it wants data to be viewable on the console. This causes an error, though, as writes to /dev/console fail in this case.

Additional information:
 * upstart closes all file descriptors (stdin/stdout/stderr) and re-opens them, explicitly setting /dev/console as its stdin, stdout, stderr. So, even an attempt to redirect /sbin/init output elsewhere (such as to /dev/tty0 if /dev/ttyS0 did not exist) from the initramfs would be unsuccessful.
 * kexec'ing with different parameters from inside the initramfs (removing the bad console=) really isn't an option, as the kernel/initrmafs is pxebooted, and we dont even have simple access to those.

Relative links:
 * kernel serial console doc: http://www.kernel.org/doc/Documentation/serial-console.txt
 * thread about re-assigning /dev/console: http://<email address hidden>/msg246433.html

Other info from system:
 http://paste.ubuntu.com/1261109/ shows the /var/lib/cloud/instance/userdata.txt
 http://pastebin.ubuntu.com/1261098/ shows /var/log/cloud-init.log
 http://paste.ubuntu.com/1261129/ shows the output when running "cloud-init --debug modules --mode=final" manually. In this case it finishes and successfully commissions.

Running:
sudo start cloud-final
shows:
start: Job failed to start

Also:
$ runlevel
N 2
$ sudo status rc
rc stop/waiting

[Julian Edwards (julian-edwards)]

dmesg: http://paste.ubuntu.com/1261143/

[Julian Edwards (julian-edwards)]

Tarball of all of /var/log

[Julian Edwards (julian-edwards)]

backdoor@10-0-0-100:~$ echo HI MOM | sudo tee /dev/console
sudo: unable to resolve host 10-0-0-100
HI MOM
tee: /dev/console: Input/output error

[Scott Moser (smoser)]

OK. So, heres a summary of the problem.

 * system boots with 'console=tty1 console=ttyS0'
 * system may not actually have a serial device (see mesg for more info)
 * kernel output does get to video console (and ttyS0 locations on systems where that is available)
 * all cloud-init jobs have 'console output', and some run successfully, but then one fails.
 * echo HI MOM | sudo tee /dev/console
    results in:
   tee: /dev/console: Input/output error
 * starting job afterwards as 'start cloud-init-final' will die immediately
   running the same thing it *would* run by hand (but with output not to /dev/console) will run to completion.

So it certainly seems to me that kernel assigned /dev/ttyS0 to /dev/console and upstarts' stdin and stdout there. and then later that device 'went bad'.

I was under the impression that kernel went through these and assigned /dev/console to the last valid entry, but it appears i may be wrong [1] on that.

--
[1] http://www.kernel.org/doc/Documentation/serial-console.txt

[Scott Moser (smoser)]

It seems that what we need to do is make this configurable (bug 1044503) and then possibly default to no 'console=' parameters.