nova instance console log empty

Bug #1667033 reported by Ryan Beisner on 2017-02-22
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Undecided
Unassigned
libvirt (Ubuntu)
Undecided
Unassigned
nova (Ubuntu)
Critical
Unassigned
qemu (Ubuntu)
High
ChristianEhrhardt

Bug Description

Nova instance console log is empty on Xenial-Ocata with libvirt 2.5.0-3ubuntu2~cloud0

James Page (james-page) wrote :
James Page (james-page) wrote :

Serial console snippet from a trusty/mitaka install

    <serial type='file'>
      <source path='/srv/nova/instances/0a33a19d-a310-4861-bf16-1bd5df8448b7/console.log'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <serial type='pty'>
      <source path='/dev/pts/21'/>
      <target port='1'/>
      <alias name='serial1'/>
    </serial>
    <console type='file'>
      <source path='/srv/nova/instances/0a33a19d-a310-4861-bf16-1bd5df8448b7/console.log'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>

James Page (james-page) wrote :

Serial console snippet from xenial/ocata

    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <log file='/var/lib/nova/instances/5d80faa1-9bda-451e-a1c8-8fc6481d6ba6/console.log' append='off'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <log file='/var/lib/nova/instances/5d80faa1-9bda-451e-a1c8-8fc6481d6ba6/console.log' append='off'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>

James Page (james-page) wrote :

Note that this is with OpenStack Ocata, with the libvirt from zesty which is version 2.5.0.

This should be using the virtlogd capabilities of libvirt/qemu, but for some reason that's not actually working.

Changed in nova (Ubuntu):
status: New → Triaged
importance: Undecided → Critical
James Page (james-page) wrote :

Might be related to bug 1599214 however qemu in use is 2.8.0 (and the empty log file issue should have been fixed in 2.7.0)

James Page (james-page) wrote :

OK so a bit more information

I was able to virsh console to a running domain, login to the instance, reboot it and then disconnect from the console.

For the duration of the access, the console.log file was populated with the same data; after disconnect, the log file gets no more data.

Ryan Harper (raharper) wrote :

I think this is the relevant qemu patch that we might need:

Discussion:
https://www.spinics.net/linux/fedora/libvir/msg142550.html

Patch:
http://patchwork.ozlabs.org/patch/721974/

Changed in qemu (Ubuntu):
status: New → Triaged
importance: Undecided → High
assignee: nobody → ChristianEhrhardt (paelzer)
ChristianEhrhardt (paelzer) wrote :

Thanks Ryan for doing all the search work!

@James - I have prepared a preliminary zesty version in a ppa at https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2512/+packages (currently building)

Since you very likely need that for your Xenial backport instead of the zesty version I pushed the commits to my LP repo, and you might want to experiment with those two on top of yours then:
https://git.launchpad.net/~paelzer/ubuntu/+source/qemu/commit/?id=273876e81575127d55cb6f1de700f5d96f144662
https://git.launchpad.net/~paelzer/ubuntu/+source/qemu/commit/?id=0db1cac03f93890e13b3a3152367509ff3a66457

There are a lot of CVEs for qemu 2.8 and I wanted to wait for a new Debian release to pick those up. That will still get into zesty release, but it could take a while, and I assume your schedule might be more aggressive.
I you happen to need this content here in zesty earlier please let me know and I'll plan for an extra upload/testing cycle.

James Page (james-page) on 2017-02-23
Changed in nova:
status: New → Invalid
Changed in nova (Ubuntu):
status: Triaged → Invalid
Changed in libvirt (Ubuntu):
status: New → Invalid
ChristianEhrhardt (paelzer) wrote :

If anybody wants to test, can be done against zesty ppa https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2522/+packages

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu - 1:2.8+dfsg-3ubuntu1

---------------
qemu (1:2.8+dfsg-3ubuntu1) zesty; urgency=medium

  * Merge with Debian;
    This fixes several CVEs that were reported against qemu 2.8 and also
    includes a few important functional backports (LP: #1667033); remaining
    changes:
    - add qemu-kvm init script and defaults file
      (d/qemu-system-common.qemu-kvm.*)
    - d/rules, d/qemu-kvm-init: add and install script loading kvm
      modules and handling /etc/default/qemu-kvm
    - qemu-system-common.preinst: add kvm group if needed
    - Enable nesting by default on intel.
      - set default module option
      - re-load kvm_intel.ko if it was loaded without nested=1
      - d/p/ubuntu/expose-vmx_qemu64cpu.patch: enable nested kvm by
        default in qemu64 cpu type.
    - Enable svm by default for qemu64 on amd
    - d/p/ubuntu/define-ubuntu-machine-types.patch, d/qemu-system-x86.NEWS:
      define distro machine types to ease future live vm migration (includes
      all former follow up fixes).
    - Make qemu-system-common depend on qemu-block-extra
    - Make qemu-utils depend on qemu-block-extra
    - s390x support
      - Create qemu-system-s390x package
      - Include s390-ccw.img firmware
    - qemu-system-common.postinst:
      - change acl placed by udev, and add udevadm trigger.
    - d/qemu-kvm-init, d/kvm.powerpc, d/control-in: check SMT on ppc64el
    - Several changes were applied but missing in the changelog so far
      - d/qemu-system-ppc.links provide usr/bin/qemu-system-ppc64le symlink
      - arch aware kvm wrapper
      - update VCS links
      - let qemu-utils recommend sharutils
      - disable x32 architecture
    - Enable seccomp for ppc64el
    - Enable numa support for s390x
    - d/qemu-system-common.qemu-kvm.init: fix lintian error type
      init.d-script-missing-dependency-on-remote_fs
    - d/qemu-system-common.postinst: fix lintian error type
      command-with-path-in-maintainer-script
    - Transition qemu-kvm to a systemd unit
    - d/qemu-kvm-init, d/kvm.powerpc ppc64el SMT check avoid unwanted output
    - d/qemu-kvm-init, d/kvm.powerpc ppc64el SMT check keep output local so
      that it shows up where the user expects (sytemctl status, kvm stdout)
    - d/qemu-kvm-init ppc64el warn on expected second level kvm-hv load failure
    - add arch aware kvm wrapper for s390x
  * Dropped Changes (in Debian now):
    - d/p/ubuntu/ctrl-a-b-fix-fb5e19d2.patch: char: fix ctrl-a b not working
    - d/control-in: change dependencies for fix of wrong acl for newly
      created device node on ubuntu
    - have qemu-system-arm suggest: qemu-efi; this should be a stronger
      relationship, but qemu-efi is still in universe right now.
    - Disable glusterfs (Universe dependency)
    - no more skip disable libiscsi on Ubuntu
    - d/rules, d/control-in: avoid people editing d/control
  * Added Changes:
    - d/control: bump libseccomp-dev dependency as enabling libseccomp for
      power makes 2.3 the minimum level.

 -- Christian Ehrhardt <email address hidden> Wed, 01 Mar 2017 14:23:16 +0100

Changed in qemu (Ubuntu):
status: Triaged → Fix Released
Tytus Kurek (tkurek) wrote :

@Christian:

I've just tried with "2.8+dfsg-2ubuntu2~ppa1" version from "https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2512" PPA and the log keeps being empty.

ChristianEhrhardt (paelzer) wrote :

Hi Tytus,
this ppa was only meant to pretest things for Xenial + Cloud Archive Ocata.
In Xenial there is a newer Qemu version SRUed in the meantime than the ppa had, so even enabling it won't install anything that fixes it for you - I'll kill the ppa to avoid this misunderstanding next time.

The situation as of now should be the following:
- In Zesty it should be fixed since 1:2.8+dfsg-3ubuntu1
- In Xenial+UCA-Ocata should be on 1:2.8+dfsg-3ubuntu2~cloud0 atm and fixed as well

Could you check the latter or newer and report back here?

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments