No support for double output to serial console on qemu-kvm s390x

[Frank Heimes]

A case came up where qemu-kvm should use double logging on (multiple) sclpconsole which is currently not supported on s390x.
Any idea or alternatively virtio console or something else possible?

Example (obviously non-working):

qemu-system-s390x: -drive file=/var/lib/libvirt/images/62d6ec4a-355d-4bbe-8ac8-52778b8a2107,format=raw,if=none,id=drive-virtio-disk0,serial=62d6ec4a-355d-4bbe-8ac8-52778b8a2107: 'serial' is deprecated, please use the corresponding option of '-device' instead 2019-01-15T13:32:53.547903Z qemu-system-s390x: -device sclpconsole,chardev=charconsole1,id=console1: Multiple VT220 operator consoles are not supported 2019-01-15T13:32:53.547913Z qemu-system-s390x: -device sclpconsole,chardev=charconsole1,id=console1: SCLP event initialization failed.

/usr/bin/kvm -name guest=vm1,debug-threads=on -S -object secret,id=masterKey0,format=ra
w,file=/var/lib/libvirt/qemu/domain-1-vm1/master-key.aes -machine s390-ccw-virtio-bionic,ac
cel=kvm,usb=off,dump-guest-core=off -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,th
reads=1 -uuid 211cd0ab-7a81-4a46-8c86-014f612976ea -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-vm1/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -drive file=/var/lib/libvirt/images/62d6ec4a-355d-4bbe-8ac8-52778b8a2107,format=raw,if=none,id=drive-virtio-disk0,serial=62d6ec4a-355d-4bbe-8ac8-52778b8a2107 -device virtio-blk-ccw,scsi=off,devno=fe.0.0002,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=26 -device virtio-net-ccw,netdev=hostnet0,id=net0,mac=52:54:00:4b:d0:26,devno=fe.0.0001 -add-fd set=2,fd=29 -chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on -device sclpconsole,chardev=charserial0,id=serial0 -chardev pty,id=charconsole1 -device sclpconsole,chardev=charconsole1,id=console1 -device virtio-balloon-ccw,id=balloon0,devno=fe.0.0000 -msg timestamp=on

[Christian Ehrhardt  (paelzer)]

To avoid confusion, on x86 this is generated from a libvirt xml that has two contole entries on the same port - one

@Frank - As discussed this comes from MAAS, you might want to ask them for their XML template.

And on s390x 2 x sclp was picked due to that which obviously can't work.
I would wonder if an extra virto-serial could do the trick.

Furthermore I just realized that Openstack does the same like in https://pastebin.canonical.com/190844/ ; so maybe they already have the right s390x combination to get this running somehow - you might ask them as well.

[Christian Ehrhardt  (paelzer)]

Found it - this is the template they render into for x86 (working)
...
        <controller type='virtio-serial' index='0'>
          <address type='pci' domain='0x0000'
            bus='0x00' slot='0x05' function='0x0'/>
        </controller>
        <serial type='pty'>
          <log file="/var/log/libvirt/qemu/{name}-serial0.log" append="off" />
          <target port='0'/>
        </serial>
        <console type='pty'>
          <target type='serial' port='0'/>
        </console>
...

And this is the failing one for s390x:
...
      <devices>
        <serial type='pty'>
          <log file="/var/log/libvirt/qemu/{name}-serial0.log" append="off" />
          <target port='0'/>
        </serial>
        <console type='pty'>
          <target type='sclp' port='0'/>
          <alias name='console0'/>
        </console>
      </devices>
...

I guess they'd appreciate a suggested to work libvirt XML over a better qemu commandline

[bugproxy (bugproxy)]

A 2nd console could be the sclplmconsole.

Something like

<console type='file'>
<source path='/tmp/test$1.log'/>
<target type='sclplm' port='1'/>

seems to work.

The guest kernel then needs to have 2 console statements

..... console=ttyS0 console=ttyS1 ....

[bugproxy (bugproxy)]

So what is the use case? Really having 2 consoles so that one can log like
<console type='pty'>
<target type='sclp' port='0'/>
</console>
<console type='pty'>
<log file='/var/log/libvirt/qemu/{name}-serial0.log' append='off'/>
<target type='sclplm' port='1'/>
</console>

Or just logging the primary console like

<console type='pty'>
<log file='/var/log/libvirt/qemu/{name}-serial0.log' append='off'/>
<target type='sclp' port='0'/>
</console>

[Christian Ehrhardt  (paelzer)]

I'd think the use case was mostly for logging to a file.
x86 serial in the past needed some tweaks to be able to log but at the same time be interactive - maybe that was the reason for the double entries.

I tried the latter suggestion with just sclp + log file and it was interactive and logging for me.

@Frank - would you hand that over to the MAAS team so that they can evaluate which would work for them? From the "ask the z experts POV" this is solved unless we find a deficiency in the suggested configuration.

[Frank Heimes (fheimes)]

Yes, since this looks like a valid alternate way I've already shared this with newell (subscribed him) who is working on LP 1811986 / LP 1811992.
Just shared it with andreserl, too.

[Newell Jensen (newell-jensen)]

Finally got some time to test out the first xml snippet in #4 above and although I was able to get console output in the VM the log file stopped at:

[ 13.527057] systemd[1]: media-root\x2dro.mount: Found ordering cycle on -.mou
nt/start
[ 13.527063] systemd[1]: media-root\x2dro.mount: Found dependency on media-roo
t\x2dro.mount/start
[ 13.527066] systemd[1]: media-root\x2dro.mount: Unable to break cycle startin
g with media-root\x2dro.mount/start
[ 13.628426] Loading iSCSI transport class v2.0-870.
[ 13.688597] iscsi: registered transport (tcp)
[ 13.742825] systemd-journald[475]: Received request to flush runtime journal
from PID 1

Might be something with systemd-journal obviously?

As of what I am seeing, this is a still a broken solution but maybe you guys can find out what is going on that is crippling the log file output.

Thanks

[bugproxy (bugproxy)]

FWIW

the first xml snippet will only long console output as sclplm is an additional console and any login will still happen at the first console (sclp).

the 2nd snipped will keep everything on one console (console output and tty interaction)

Please pick what you need.

[bugproxy (bugproxy)]

@Canonical, any update available ?

[Christian Ehrhardt  (paelzer)]

Thanks for that - I think we are done and have to thank @IBM.
We have pinged MAAS to consider the insight in their next updates.

[Newell Jensen (newell-jensen)]

Sorry for the delay in responding to this. As Chrisitian correctly assumed in his first paragraph, yes to get this to work for x86 we needed two serial consoles and I assume this was was the case without testing out the second xml snippet. Just tried it and it worked so will be pushing this to MAAS. Cheers.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-01-23 06:32 EDT-------

------- Comment From <email address hidden> 2019-01-24 04:58 EDT-------

------- Comment From <email address hidden> 2019-02-14 08:35 EDT-------

------- Comment From <email address hidden> 2019-03-15 08:45 EDT-------

[Frank Heimes (fheimes)]

Since MAAS 2.6 supersedes 2.5 - I'm changing to project entry to Fix Released (like 2.6 is).

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-08-07 09:10 EDT-------
IBM bugzilla status-> closed, Fix Release by Canonical