nova instance console log empty

[Ryan Beisner]

Nova instance console log is empty on Xenial-Ocata with libvirt 2.5.0-3ubuntu2~cloud0

[James Page (james-page)]

Serial console snippet from a trusty/mitaka install

    <serial type='file'>
      <source path='/srv/nova/instances/0a33a19d-a310-4861-bf16-1bd5df8448b7/console.log'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <serial type='pty'>
      <source path='/dev/pts/21'/>
      <target port='1'/>
      <alias name='serial1'/>
    </serial>
    <console type='file'>
      <source path='/srv/nova/instances/0a33a19d-a310-4861-bf16-1bd5df8448b7/console.log'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>

[James Page (james-page)]

Serial console snippet from xenial/ocata

    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <log file='/var/lib/nova/instances/5d80faa1-9bda-451e-a1c8-8fc6481d6ba6/console.log' append='off'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <log file='/var/lib/nova/instances/5d80faa1-9bda-451e-a1c8-8fc6481d6ba6/console.log' append='off'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>

[James Page (james-page)]

Note that this is with OpenStack Ocata, with the libvirt from zesty which is version 2.5.0.

This should be using the virtlogd capabilities of libvirt/qemu, but for some reason that's not actually working.

[James Page (james-page)]

related commit:

  https://github.com/openstack/nova/commit/1f659251c7509cab045024044a6b8d642ad85aef

[James Page (james-page)]

Might be related to bug 1599214 however qemu in use is 2.8.0 (and the empty log file issue should have been fixed in 2.7.0)

[James Page (james-page)]

OK so a bit more information

I was able to virsh console to a running domain, login to the instance, reboot it and then disconnect from the console.

For the duration of the access, the console.log file was populated with the same data; after disconnect, the log file gets no more data.

[Ryan Harper (raharper)]

I think this is the relevant qemu patch that we might need:

Discussion:
https://www.spinics.net/linux/fedora/libvir/msg142550.html

Patch:
http://patchwork.ozlabs.org/patch/721974/

[ Christian Ehrhardt  (paelzer)]

Thanks Ryan for doing all the search work!

@James - I have prepared a preliminary zesty version in a ppa at https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2512/+packages (currently building)

Since you very likely need that for your Xenial backport instead of the zesty version I pushed the commits to my LP repo, and you might want to experiment with those two on top of yours then:
https://git.launchpad.net/~paelzer/ubuntu/+source/qemu/commit/?id=273876e81575127d55cb6f1de700f5d96f144662
https://git.launchpad.net/~paelzer/ubuntu/+source/qemu/commit/?id=0db1cac03f93890e13b3a3152367509ff3a66457

There are a lot of CVEs for qemu 2.8 and I wanted to wait for a new Debian release to pick those up. That will still get into zesty release, but it could take a while, and I assume your schedule might be more aggressive.
I you happen to need this content here in zesty earlier please let me know and I'll plan for an extra upload/testing cycle.

[ Christian Ehrhardt  (paelzer)]

If anybody wants to test, can be done against zesty ppa https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2522/+packages

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:2.8+dfsg-3ubuntu1

---------------
qemu (1:2.8+dfsg-3ubuntu1) zesty; urgency=medium

  * Merge with Debian;
    This fixes several CVEs that were reported against qemu 2.8 and also
    includes a few important functional backports (LP: #1667033); remaining
    changes:
    - add qemu-kvm init script and defaults file
      (d/qemu-system-common.qemu-kvm.*)
    - d/rules, d/qemu-kvm-init: add and install script loading kvm
      modules and handling /etc/default/qemu-kvm
    - qemu-system-common.preinst: add kvm group if needed
    - Enable nesting by default on intel.
      - set default module option
      - re-load kvm_intel.ko if it was loaded without nested=1
      - d/p/ubuntu/expose-vmx_qemu64cpu.patch: enable nested kvm by
        default in qemu64 cpu type.
    - Enable svm by default for qemu64 on amd
    - d/p/ubuntu/define-ubuntu-machine-types.patch, d/qemu-system-x86.NEWS:
      define distro machine types to ease future live vm migration (includes
      all former follow up fixes).
    - Make qemu-system-common depend on qemu-block-extra
    - Make qemu-utils depend on qemu-block-extra
    - s390x support
      - Create qemu-system-s390x package
      - Include s390-ccw.img firmware
    - qemu-system-common.postinst:
      - change acl placed by udev, and add udevadm trigger.
    - d/qemu-kvm-init, d/kvm.powerpc, d/control-in: check SMT on ppc64el
    - Several changes were applied but missing in the changelog so far
      - d/qemu-system-ppc.links provide usr/bin/qemu-system-ppc64le symlink
      - arch aware kvm wrapper
      - update VCS links
      - let qemu-utils recommend sharutils
      - disable x32 architecture
    - Enable seccomp for ppc64el
    - Enable numa support for s390x
    - d/qemu-system-common.qemu-kvm.init: fix lintian error type
      init.d-script-missing-dependency-on-remote_fs
    - d/qemu-system-common.postinst: fix lintian error type
      command-with-path-in-maintainer-script
    - Transition qemu-kvm to a systemd unit
    - d/qemu-kvm-init, d/kvm.powerpc ppc64el SMT check avoid unwanted output
    - d/qemu-kvm-init, d/kvm.powerpc ppc64el SMT check keep output local so
      that it shows up where the user expects (sytemctl status, kvm stdout)
    - d/qemu-kvm-init ppc64el warn on expected second level kvm-hv load failure
    - add arch aware kvm wrapper for s390x
  * Dropped Changes (in Debian now):
    - d/p/ubuntu/ctrl-a-b-fix-fb5e19d2.patch: char: fix ctrl-a b not working
    - d/control-in: change dependencies for fix of wrong acl for newly
      created device node on ubuntu
    - have qemu-system-arm suggest: qemu-efi; this should be a stronger
      relationship, but qemu-efi is still in universe right now.
    - Disable glusterfs (Universe dependency)
    - no more skip disable libiscsi on Ubuntu
    - d/rules, d/control-in: avoid people editing d/control
  * Added Changes:
    - d/control: bump libseccomp-dev dependency as enabling libseccomp for
      power makes 2.3 the minimum level.

 -- Christian Ehrhardt <email address hidden> Wed, 01 Mar 2017 14:23:16 +0100

[Tytus Kurek (tkurek)]

@Christian:

I've just tried with "2.8+dfsg-2ubuntu2~ppa1" version from "https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2512" PPA and the log keeps being empty.

[ Christian Ehrhardt  (paelzer)]

Hi Tytus,
this ppa was only meant to pretest things for Xenial + Cloud Archive Ocata.
In Xenial there is a newer Qemu version SRUed in the meantime than the ppa had, so even enabling it won't install anything that fixes it for you - I'll kill the ppa to avoid this misunderstanding next time.

The situation as of now should be the following:
- In Zesty it should be fixed since 1:2.8+dfsg-3ubuntu1
- In Xenial+UCA-Ocata should be on 1:2.8+dfsg-3ubuntu2~cloud0 atm and fixed as well

Could you check the latter or newer and report back here?