add console=tty1 to cloud-image kernel boot parameters

[Jeremiah Snapp]

I use VMware vSphere Client's to manage my Ubuntu 12.04 cloud-image based vm's. vSphere is able to display the console of the vm as it boots but because the default kernel boot parameter is set to use console=ttyS0 I can't see anything until booting is done unless I manually add console=tty0 first. Can you add console=tty0 after console=ttyS0 so boot process will display automatically? I would really appreciate it.

Related bugs:
  bug 1122245: booting from a cloud image hangs until virsh console is used

[Scott Moser (smoser)]

The kernel actually does a good job of handling multiple 'console=' items. It then writes the same data to each of them (if they're valid devices).

However, when init (upstart) takes over, it only writes to /dev/console. So whatever the /dev/console means to the kernel is the only place that output from upstart or it subprocesses will go.

I think the kernel picks the last of the 'console=' items on the command line to tie to /dev/console. It may take the last *valid* entry, i'm not sure.

So, when upstart starts, and then starts running jobs (such as cloud-init) their output goes to /dev/console, and not to the other 'console=' places specified on the command line.

What all that means, is that I think we can fairly harmlessly add additional 'console=' lines to the command line, and have kernel messages go there. However, upstart messages will *not* go to those places.

[Jeremiah Snapp (jeremiah.snapp)]

Scott, I think the following info is relevant to your comment.

http://www.kernel.org/doc/Documentation/serial-console.txt

"You can specify multiple console= options on the kernel command line.
Output will appear on all of them. The last device will be used when
you open /dev/console. So, for example:

 console=ttyS1,9600 console=tty0

defines that opening /dev/console will get you the current foreground
virtual console, and kernel messages will appear on both the VGA
console and the 2nd serial port (ttyS1 or COM2) at 9600 baud."

[Clint Byrum (clint-fewbar)]

Jeremiah, thanks for the bug report. It would be pretty awful to lose the upstart messages going to the serial console on EC2, so I don't think this is a valid solution.

Perhaps we could add a vsphere-guest package which adds this to the kernel parameters in grub somehow. That would mean you still wouldn't see messages on the first boot, but debugging at that point can be handled by manually adding the option.

[Jeremiah Snapp (jeremiah.snapp)]

Thanks Clint. I understand that we can't lose any messages going to the serial console. So I experimented and I have a different solution that I think would satisfy.

My main goal is to *by default* be able to see in the VMware console the kernel messages as well as the first few messages from cloud-init that show cloud-init is searching for the EC2 datasource by default and is timing out. Sending this info only to the serial port means anyone unfamiliar with running a cloud-image in VMware (or other hypervisor) may not understand that cloud-init is behind the scenes searching for the EC2 datasource and timing out. They would sit there wondering why it's taking so long to boot. *Maybe* they'd figure out they can add console=tty1 to their kernel boot parameters so they can see what is going on. I personally think Ubuntu server is great but this could be frustrating to others and potentially lead them to believe something just ain't right with Ubuntu servers.

My new solution is to add console=tty1 *before* console=ttyS0. Note that it's tty1 and not tty0.

This produces the same results as expected for the serial port. All messages are output to serial as usual.

This also outputs all kernel messages and just enough of the cloud-init messages to tty1 which makes it visible in VMware's vSphere console where you can watch it search for the EC2 datasource and timeout. The remainder of the cloud-init messages (including the cloud-init final message) are not visible in the console so you'd have to log in and check the cloud-init log in /var/log to see it's status but that's much better than nothing at all. When the server is done booting the VMware console displays the tty1 as expected which is why I used tty1 instead of tty0.

I tested all this by using VMware's ability to output serial port to a file.

I have attached a file for the serial port output for each of the following grub kernel parameter entries:

linux /boot/vmliuz-3.2.0-24-virtual root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0

linux /boot/vmliuz-3.2.0-24-virtual root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0 ds=nocloud

The first shows cloud-init searching for EC2 datasource and timing out.

The second shows the results when cloud-init is configured with ds=nocloud.

Do you think this sounds reasonable?

[Ben Howard (darkmuggle-deactivatedaccount)]

Assigning to myself for consideration in the cloud images. A change like this seems innocent enough.

[Jeremiah Snapp (jeremiah.snapp)]

Thanks Ben. I'm not familiar with how these changes actually make it into new releases. If this change actually gets made would it make it into 12.04.1 in August?

[Scott Moser (smoser)]

Ben,
  Please do not change the images just yet.
  I'd really rather not make a change without knowing explicitly what happens in each case of the following, given the command line 'console=tty1 console=ttyS0':

a.) kvm -nographic
b.) no serial device present

For each of these, where does console output end up?

I'm also concerned about arm arch, where ttyX might not be meaningful. It seems likely that the kernel has sane behavior, but I'd like to know for sure.

case 'b' is (I believe) the case that is affecting Jeremiah. Because there is no serial device, the kernel then is selecting tty0 as the /dev/console device, and upstart is putting 'console output' cloud-init job output there. However, in the case where there *is* a serial device, I honestly cannot understand how the tty1 param would change anything.

The change does seem reasonable, but I just want to avoid making arbitrary changes that we do not completely understand.

[Ben Howard (darkmuggle-deactivatedaccount)]

Re comment #9

Scott,

I have no intention of changing anything until a through test is done for the _exact_ concerns that your raising.

> The change does seem reasonable, but I just want to avoid making arbitrary changes that we do not completely understand.
My comment about the change being "innocent enough" is misleading, since it implies that that I think there would be no reprocussions.

> I'm also concerned about arm arch, where ttyX might not be meaningful. It seems likely that the kernel has sane behavior, but I'd like to know for sure.

For ARM, tty1 would mean no console output. Right now we configure the ARM images to use the ARM serial console seperately anyway.

~Ben

[Scott Moser (smoser)]

Ben,
  I'd like to test this and see what falls out. For quantal, can we add 'console=tty1' to the command line in the dailies?
  Then, I'd like to get reports on a many scenarios as possible:
 * kvm -nographic
 * kvm -curses
 * openstack
 * ec2

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Scott Moser (smoser)]

This should be in the next quantal builds.

[Jeremiah Snapp (jeremiah.snapp)]

That's great! Thanks for working this in.

Will it make it into 12.04 at all or will it only be in 12.10 and beyond?

[Scott Moser (smoser)]

Jeremiah,
  The process for making changes to stable releases is the "Stable Release Updates" [1]. It does not 100% apply here since it is only a change to the image building process. But similar care should be taken. I'd like to see it bake a while in the cloud-image in quantal to avoid regression.

--
https://wiki.ubuntu.com/StableReleaseUpdates