libvirt: virtlogd: qemu 2.6.0 doesn't log boot message

[Markus Zoeller (markus_z)]

Description
===========
With libvirt 1.3.3 and Qemu 2.6.0 char devices are able to log their
stdout/stderr into a log file via a "<log>" XML element in the domain XML.
This feature solves the long standing bug 832507 (which can flood the log)
by a built-in log rotation. It also removes the mutually exclusivity of
"serial console" and "get console output".

Unfortunately, an (assumed) issue in Qemu prevents the logging of the
boot messages of the guest *unless* the connection to the char device is
already established.

Steps to reproduce
==================
A chronological list of steps which will bring off the
issue you noticed:
* Ensure to have the code of https://review.openstack.org/#/c/323765/14 applied
* Ensure to have libvirt 1.3.3 and qemu 2.6.0
* Launch an instance
* execute: nova console-log <server>

Expected result
===============
The CLI returns the boot messages.

Actual result
=============
The result is an empty string.

If I connect to the used console (via Horizon for example), execute "echo 'foo'" and things like that, the next call of "nova console-log <server>" *does* return those executed commands (but still not the boot messages). If I reboot withing the console, *then* the boot messages will be logged too.

Environment
===========
1. OpenStack version: master (Newton)
   $ git log --oneline -4
   201e231 libvirt: virtlogd: use virtlogd for char devices
   73e931a libvirt: simplify "get_console_output" interface
   ec94d7b libvirt: fix live-migration with serial console check
   fcb3dbf Merge "Fix error message for VirtualInterfaceUnplugException"

2. Which hypervisor did you use?
   libvirt 1.3.3 and kvm-qemu 2.6.0

2. Which storage type did you use?
   (For example: Ceph, LVM, GPFS, ...)
   What's the version of that?

3. Which networking type did you use?
   neutron + ovs

Logs & Configs
==============
N/A

[Matt Riedemann (mriedem)]

Is this a blocker for https://blueprints.launchpad.net/nova/+spec/libvirt-virtlogd ?

[Markus Zoeller (markus_z) (mzoeller)]

@Matt, no, I don't think so. After discussing this with danpb we agreed to use a new "workarounds" config option [1]. When this gets fixed fast in Qemu then we can still exploit it in Newton with a small follow up patch which deprecates that config option and activates the feature by default.

[1] https://review.openstack.org/#/c/323765/16/nova/virt/libvirt/driver.py@4325

[Markus Zoeller (markus_z) (mzoeller)]

The bug report for the Qemu project: https://bugs.launchpad.net/qemu/+bug/1599214

[Markus Zoeller (markus_z) (mzoeller)]

Marked as duplicate. The issue was in qemu and got fixed with v2.7.0. Also, Nova doesn't yet have a dependency to virtlogd. This is still under development with https://review.openstack.org/#/c/323765/

[Markus Zoeller (markus_z) (mzoeller)]

FWIW, I tested this successfully with a Devstack setup:

# =================================
# OpenStack version infos
# =================================
[12:18:24 stack@qemu-src ~/devstack ] $ ./tools/info.sh
os|distro=trusty
os|vendor=Ubuntu
os|release=14.04
git|cinder|master[c9900ad]
git|devstack|master[148d0e6]
git|glance|master[9d407fe]
git|horizon|master[cf0a206]
git|keystone|master[29b7102]
git|neutron|master[7311e56]
git|nova|master[5820172]
git|requirements|master[789dd5c]
[...]

# ====================================
# Hypervisor version infos
# ====================================
[12:19:33 stack@qemu-src ~/devstack ] $ sudo virsh -c qemu:///system version
Compiled against library: libvirt 1.3.3
Using library: libvirt 1.3.3
Using API: QEMU 1.3.3
Running hypervisor: QEMU 2.7.0

# =================================
# Devstack patches
# =================================
http://paste.openstack.org/show/584815/

# ==================================
# Build environment
# ==================================
https://github.com/markuszoeller/openstack/tree/master/scripts/vagrant/libvirt-qemu-source-U1404-VB