ssh host key fingerprint no longer available in the console log

Bug #431103 reported by Mathias Gug on 2009-09-17
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
ec2-init (Ubuntu)
High
Scott Moser
Karmic
High
Scott Moser
linux-ec2 (Ubuntu)
High
Tim Gardner
Karmic
High
Tim Gardner

Bug Description

Binary package hint: ec2-init

When booting an instance the generated public key for sshd used to be printed
in the console so that one could verify the public key upon the first
connection.

After more than 20 minutes of uptime the console log ends with:

[ 1.847386] kjournald starting. Commit interval 5 seconds

[ 1.847406] EXT3-fs: mounted filesystem with writeback data mode.

[ 3.043120]

[ 3.043132] ***************************************************************

[ 3.043138] ***************************************************************

[ 3.043143] ** WARNING: Currently emulating unsupported memory accesses **

[ 3.043147] ** in /lib/tls glibc libraries. The emulation is **

[ 3.043152] ** slow. To ensure full performance you should **

[ 3.043156] ** install a 'xen-friendly' (nosegneg) version of **

[ 3.043161] ** the library, or disable tls support by executing **

[ 3.043165] ** the following as root: **

[ 3.043170] ** mv /lib/tls /lib/tls.disabled **

[ 3.043174] ** Offending process: dbus-uuidgen (pid=1119) **

[ 3.043179] ***************************************************************

[ 3.043184] ***************************************************************

[ 3.043188]

ProblemType: Bug
Architecture: i386
Date: Thu Sep 17 00:52:01 2009
DistroRelease: Ubuntu 9.10
NonfreeKernelModules: fbcon tileblit font bitblit softcursor i915 drm i2c_algo_bit i2c_core intel_agp agpgart
Package: ec2-init 0.4.999-0ubuntu1
PackageArchitecture: all
ProcEnviron:
 LANG=en_GB.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: User Name 2.6.31-300.2-ec2
SourcePackage: ec2-init
Uname: Linux 2.6.31-300-ec2 i686

Mathias Gug (mathiaz) wrote :
Mathias Gug (mathiaz) wrote :

Going through the /var/log/syslog file, the fingerprint is printed there. However there is a line related to rsyslog just after the line printed in the console:

rsyslogd-2039: Could no open output file '/dev/xconsole' [try http://www.rsyslog.com/e/2039 ]

Mathias Gug (mathiaz) wrote :

Upon more investigation, it seems that the default rsyslog configuration is incorrect for ec2:

/etc/rsyslog.d/50-default.conf:

daemon.*;mail.*;\
        news.err;\
        *.=debug;*.=info;\
 *.=notice;*.=warn |/dev/xconsole

Mathias Gug (mathiaz) wrote :

Replacing /dev/xconsole with /dev/xvc0 will fix the problem. All syslog messages will be printed to the console and thus be available via the console-output command.

We may wanna restrict what will be send to /dev/xvc0 (or add an ec2 specific configuration file in /etc/rsyslog.d/) and adjust the logger call in the ec2-init script to match the filters.

Changed in ec2-init (Ubuntu):
status: New → Triaged
importance: Undecided → High
Scott Moser (smoser) on 2009-09-17
Changed in ec2-init (Ubuntu):
assignee: nobody → Scott Moser (smoser)
milestone: none → ubuntu-9.10-beta
Scott Moser (smoser) wrote :

I've verified that putting the following stanza into /etc/rsyslog.d/60-ec2.conf fixes the problem:

daemon.*;mail.*;\
 news.err;\
 *.=debug;*.=info;\
 *.=notice;*.=warn |/dev/xvc0

we do still need to trim it back a bit, but that can be easily enough written by vm-builder if we dont see a more appropriate way.

Scott Moser (smoser) wrote :

Ok, prepare for long windedness.

I've tried to determine if the regression shown here was due to a kernel
change (including ramdisk) or a user space change. To do that, I ran
several different combinations of root and initrd and checked console
output.

aliases:
- ec2-krd = aki-a71cf9ce / aki-9c04e4f5
  ec2-vmlinuz-2.6.21.7-2.fc8xen.i386
  ec2-initrd-2.6.21.7-2.fc8xen.i386
- karmic-krd = aki-841efeed / ari-9a1efef3
  ubuntu-kernel-2.6.31-300-ec2-xen-i386-20090910-test-02
  ubuntu-ramdisk-2.6.31-300-ec2-xen-i386-20090910-test-02.manifest.xml
- intrepid-krd = aki-714daa18 / ari-7e4daa17
  vmlinuz-2.6.27-23-xen-i386
  initrd.img-2.6.27-23-xen-i386
- alpha5 = ami-3520c05c
  karmic-i386-alpha5.manifest.xml
- alpha5.1 = ami-a40fefcd
  karmic-i386-alpha5.1.manifest.xml
- alpha6 = ami-fa658593
  karmic-i386-alpha6.manifest.xml

- Note, aki-9c04e4f5 / ari-9e04e4f7 == aki-841efeed / ari-9a1efef3
  I simply re-published the same manifest in a different bucket generating
  the different numbers

root_________| kern/rd | instance | result
alpha5_______| amazon-ec2 | i-c374a0ab | Yes : init messages get to console
alpha5______ | karmic-krd | i-ab73a7c3 | No : no console output past kernel
alpha5.1_____| karmic-krd | i-9573a7fd | No : no console output past kernel
alpha6_______| amazon-ec2 | i-0778ac6f | Yes : init messages ("Deactivating swap", "init:")
alpha6_______| karmic-krd | i-c726f2af | No : no console output past kernel

Note, the amazon-ec2 kernels dont completly boot with our user space, so
the output isn't as clear as you'd like, but you do see output from the
init process and sysvinit.

Its not completely straightforward, but I believe that the above shows a regression in the kernel/initrd rather than in userspace init or general makeup of image.

John Johansen (jjohansen) wrote :

While it appears there is some kernel interaction it isn't just a kernel issue. I did further testing and came up with some interesting results.

     ami | aki | ari | full boot | init message go to console
fedora ami-2547a34c | fedora aki-b51cf9dc | fedora ari-b31cf9da | Yes | Yes
ubuntu ami-1a658573 | fedora aki-b51cf9dc | fedora ari-b31cf9da | No | Yes
ubuntu ami-1a658573 | aki-9c1efef5 | ari-901efef9 | Yes | No
fedora ami-2547a34c | aki-9c1efef5 | ari-901efef9 | Yes | Yes

A fedora userspace with the karmic kernel/initrd will boot and display init messages to the console.

> A fedora userspace with the karmic kernel/initrd will boot and display
> init messages to the console.

Interesting. Thank you for your effort. You have any clues?

Chuck Short (zulcss) wrote :

I have a theroy that I am testing right now

Chuck Short (zulcss) wrote :

Hi,

So my testing so far has been the following:

I think there might be a bug with the interaction of new xen framebuffer drivers and the older versions of Xen that Amazon runs.

I have added a kernel patch to my local ec2 tree which changes the console from xvc0 to tty1. This will allow console messages to be displayed on ec2 so that users can see the ssh keys, boot up messages, etc. I will send the patch today to the kernel team mailing list for approval to be included in the karmic kernel.

Regards
chuck

Scott Moser (smoser) on 2009-09-25
Changed in linux-ec2 (Ubuntu Karmic):
importance: Undecided → High
Chuck Short (zulcss) wrote :

Hi,

Can you try with with aki-305dbd59 and ari-365dbd5f.

I was able to get a proper boot console with the above kernel and ramdisk.

Thanks
chuck

Eric Hammond (esh) wrote :

This combination worked for me as far as seeing the host ssh key in the console output:

  ami-fa658593 + aki-305dbd59 + ari-365dbd5f

I did note that it spent almost 5 minutes in "pending" state which would be a concern if it continues to happen.

There were also a few more warning type messages than I am used to seeing which somebody might want to investigate.

Console output attached.

Scott Moser (smoser) wrote :

> I did note that it spent almost 5 minutes in "pending" state which would
> be a concern if it continues to happen.

I had noticed longer 'pending' times today then normal myself, and not
with this new kernel, but other things that i was testing. Its something
to watch for, but I think probably unrelated to any of our changes.

Tim Gardner (timg-tpi) on 2009-09-26
Changed in linux-ec2 (Ubuntu Karmic):
assignee: nobody → Tim Gardner (timg-tpi)
status: New → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux-ec2 - 2.6.31-300.3

---------------
linux-ec2 (2.6.31-300.3) karmic; urgency=low

  [ Chuck Short ]

  * SAUCE: ec2: Default domU console to tty.
    - LP: #431103

  [ Tim Gardner ]

  * [Config] Remove dependency on wireless-crda
    - LP: #434755
  * [Config] Drop bootloaders as a recommendation
    - LP: #434755
  * [Config] Drop virtual sub flavours

 -- Tim Gardner <email address hidden> Fri, 25 Sep 2009 15:18:34 -0600

Changed in linux-ec2 (Ubuntu Karmic):
status: Fix Committed → Fix Released
Scott Moser (smoser) wrote :

I just uploaded to canonical-testing-us:
  aki-80a84be9 ubuntu-kernel-2.6.31-300.3-ec2-i386.manifest.xml
  aki-8aa84be3 ubuntu-kernel-2.6.31-300.3-ec2-x86_64.manifest.xml
  ari-90a84bf9 ubuntu-ramdisk-2.6.31-300.3-ec2-i386.manifest.xml
  ari-8ea84be7 ubuntu-ramdisk-2.6.31-300.3-ec2-x86_64.manifest.xml

I then booted the i386 build karmic-i386-20090926.1 (ami-28a34041) with:
  run-instances ami-28a34041 --kernel aki-80a84be9 --ramdisk ari-90a84bf9

The happy result is we have console output again, including the ssh keys.

So, with linux-image-2.6.31-300-ec2 at version 2.6.31-300.3,
a.) this should be fixed in beta
b.) There is no ec2-init task required

Changed in ec2-init (Ubuntu Karmic):
status: Triaged → Invalid
Changed in linux-ec2 (Ubuntu Karmic):
milestone: none → ubuntu-9.10-beta
Changed in ec2-init (Ubuntu Karmic):
milestone: ubuntu-9.10-beta → none
Scott Moser (smoser) wrote :

Just marking here, the beta release has proper console output. Thanks zul and kernel team.

tags: added: iso-testing
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers