Desktop smoke installation jobs do not complete due to possible comms loss between utah server and client

[Para Siva]

The desktop smoke installation jobs do not complete with desktop images of 20131129 due to, possibly, utah server losing comms with the client.

Though the installation on the VM continues fine, the server side console logging stops at:
...
...
WARNING: jobtimeout was already set by /etc/utah/config
WARNING: xml was already set by /etc/utah/config
WARNING: logpath was already set by /etc/utah/config

This means that the server does not know that the installation on the client is complete, Hence the following bits of server debug messages:
...
...
2013-11-29 09:58:13,809 rsyslog INFO: Waiting 8999s for: system started
2013-11-29 12:28:14,083 rsyslog ERROR: Timeout (9000) occurred for system started message.
Remaining messages: install started, install complete

The impacted jobs are:
http://ci.ubuntu.com/smokeng/trusty/desktop/amd64/20131129/5180/
and
http://ci.ubuntu.com/smokeng/trusty/desktop/i386/20131129/5181/

rsyslog version on trusty *appear to have changed on this image from 5.8.11-2ubuntu4 to 7.4.4-1ubuntu1 and I am wondering if that's any reason for this issue.

[Para Siva (psivaa)]

The issue appears to be due to installer syslog being not copied to /var/log/installer/ after the latest rsyslog upgrade. I've reported bug 1256695 for this.

[Paul Larson (pwlars)]

7.4.4-1ubuntu2 should fix this in the next image after 20121204. Closing this since the real issue was with rsyslog, not utah itself.

[Andy Doan (doanac)]

I just tried a desktop image from today that does have rsyslog 7.4.4-1ubuntu2 and things are still broke. Let me fill in the details.

We configure our VM via libvirt to have a serial device(/dev/ttyS0) that writes to a file on the host. During the desktop install we have some early preseed logic that adds an /etc/rsyslog.d config file that directs syslog messages to /dev/ttyS0. Under recent images, nothing is showing up in the file on the host end. For a quick sanity check I ran the following command in the VM:

 echo ANDY > /dev/ttyS0

This resulted in content showing up in the file on the host. So I think this means rsyslog is still not working.

[Andy Doan (doanac)]

More debugging courtesy of jibel's suggestion:

rsyslogd runs as syslog (which is in the syslog and adm groups). ttyS0 is writeable to root and dialout

[Andy Doan (doanac)]

here's the rsyslog bug I've opened:

 https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/1258245