[Power 9] ISST-LTE:KVM:Ubuntu1804:BostonLC:boslcp3: Unable to start the guest with more than 1 thread on ubuntu1804 KVM host machine.

[bugproxy]

== Comment: #0 - INDIRA P. JOGA <email address hidden> - 2018-01-07 13:18:32 ==
Problem Description:
===================
Not able to start the ubutnu1804 guest with more than 1 thread on ubuntu1804 KVM host machine

Steps to re-create:
==================
> Installed Ubuntu1804 on boslcp3 host.

root@boslcp3:/home# uname -a
Linux boslcp3 4.13.0-17-generic #20-Ubuntu SMP Mon Nov 6 10:03:08 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux
root@boslcp3:/home# uname -r
4.13.0-17-generic

> Installed qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils packages

root@boslcp3:/home# virsh version
Compiled against library: libvirt 3.6.0
Using library: libvirt 3.6.0
Using API: QEMU 3.6.0
Running hypervisor: QEMU 2.10.1

> root@boslcp3:~# ppc64_cpu --smt
SMT is off

> Defined the guest boslcp3g1 from host machine

root@boslcp3:/home# virsh list --all
 Id Name State
----------------------------------------------------
 - boslcp3g1 shut off

Please cherry pick

commit 45c940ba490df28cb87b993981a5f63df6bbb8db
Author: Paul Mackerras <email address hidden>
Date: Fri Nov 18 17:43:30 2016 +1100

KVM: PPC: Book3S HV: Treat POWER9 CPU threads as independent subcores

> Started the guest and it fails with the error

root@boslcp3:~# virsh start --console boslcp3g1
error: Failed to start domain boslcp3g1
error: internal error: process exited while connecting to monitor: 2018-01-05T02:54:37.762120Z qemu-system-ppc64: -chardev pty,id=charserial0: char device redirected to /dev/pts/3 (label charserial0)
2018-01-05T02:54:37.829236Z qemu-system-ppc64: Cannot support more than 1 threads on PPC with KVM

> Unable to start the guest with more than 1 Thread from Ubuntu1804 KVM host machine.

XML:
****
root@boslcp3:/home# virsh dumpxml boslcp3g1
<domain type='kvm'>
  <name>boslcp3g1</name>
  <uuid>95374879-0ed3-4562-a00f-e47d9aaf285c</uuid>
  <maxMemory slots='16' unit='KiB'>10485760</maxMemory>
  <memory unit='KiB'>6291456</memory>
  <currentMemory unit='KiB'>6291456</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB' nodeset='0'/>
    </hugepages>
  </memoryBacking>
  <vcpu placement='static' current='16'>32</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='ppc64le' machine='pseries-2.10'>hvm</type>
    <boot dev='hd'/>
    <boot dev='network'/>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='forbid'>power9</model>
    <topology sockets='2' cores='8' threads='2'/>
    <numa>
      <cell id='0' cpus='0-7' memory='3145728' unit='KiB'/>
      <cell id='1' cpus='8-15' memory='3145728' unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>coredump-restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-ppc64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/disk/by-id/wwn-0x600507680183050d28000000000002a4-part1'/>
      <target dev='sda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='wwn-0x600507680183050d28000000000002a4-part2'/>
      <target dev='sdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='wwn-0x600507680183050d28000000000002a4-part3'/>
      <target dev='sdc' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='qemu-xhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <model name='spapr-pci-host-bridge'/>
      <target index='0'/>
    </controller>
    <controller type='pci' index='1' model='pci-root'>
      <model name='spapr-pci-host-bridge'/>
      <target index='1'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:e5:81:1f'/>
      <source dev='enP2p1s0f0' mode='bridge'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
      <address type='spapr-vio' reg='0x30000000'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
      <address type='spapr-vio' reg='0x30000000'/>
    </console>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>
    <panic model='pseries'/>
  </devices>
</domain>

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-01-15 02:20 EDT-------
Found this patch not available in Ubuntu 18.04. which is related to this issue.

commit 45c940ba490df28cb87b993981a5f63df6bbb8db
Author: Paul Mackerras <email address hidden>
Date: Fri Nov 18 17:43:30 2016 +1100

KVM: PPC: Book3S HV: Treat POWER9 CPU threads as independent subcores

[Joseph Salisbury (jsalisbury)]

That patch has been in The 18.04 kernel does have that patch applied:

git describe --contains 45c940b
v4.13~1729^2~18^2~10

[Joseph Salisbury (jsalisbury)]

I was starting to type in the last comment:

The patch has been in mainline since 4.10-rc1, so any Ubuntu kernel with a 4.10 or newer kernel has commit 45c940ba490df28cb87b993981a5f63df6bbb8db

[Joseph Salisbury (jsalisbury)]

Could there be another missing commit?

[Christian Ehrhardt  (paelzer)]

We are working on qemu 2.11 which IMHO could be the reason for your issues as well.
Do you happen to know if there are qemu patches post 2.10 needed as well for the P9 machines?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-01-19 00:03 EDT-------
Adding Leonardo to check if he is aware of any qemu patches post 2.10.

[Christian Ehrhardt  (paelzer)]

If you want to experiment with early qemu 2.11 on this take a look at the ppa at:
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3108

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-02-05 01:07 EDT-------
Test machine is just stabilised with latest PNOR & BMC levels. Will try to experiment with early qemu 2.11 whenever system is free from other tasks.

Regards,
Indira

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-02-11 23:57 EDT-------
Test machine is blocked with bug#164521, once system is released from dev will verify issue the early qemu 2.11 verison.

[Christian Ehrhardt  (paelzer)]

2.11 is now in Bionic - you don't need the ppa anymore to test this.
Looking forward to hear from that test.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-02-15 06:50 EDT-------
> Tested with qemu 2.11 verison and able to start the guests with more than 1 thread.

root@boslcp3:~# uname -r
4.13.0-32-generic

root@boslcp3:~# virsh version
Compiled against library: libvirt 4.0.0
Using library: libvirt 4.0.0
Using API: QEMU 4.0.0
Running hypervisor: QEMU 2.11.0

> Started below guests with threads 2 & 4 without any issues.

root@boslcp3:~# virsh dumpxml boslcp3g1 | grep threads
<topology sockets='2' cores='8' threads='2'/>
root@boslcp3:~#

root@boslcp3g1:~# lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
NUMA node(s): 2
Model: 2.1 (pvr 004e 1201)
Model name: POWER9 (architected), altivec supported
Hypervisor vendor: KVM
Virtualization type: para
L1d cache: 32K
L1i cache: 32K
NUMA node0 CPU(s): 0-7
NUMA node1 CPU(s): 8-15
root@boslcp3g1:~#

root@boslcp3g1:~# ppc64_cpu --smt
SMT=2

> root@boslcp3:~# virsh dumpxml boslcp3g5 | grep threads
<topology sockets='2' cores='4' threads='4'/>

root@boslcp3g5:~# lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 4
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Model: 2.1 (pvr 004e 1201)
Model name: POWER9 (architected), altivec supported
Hypervisor vendor: KVM
Virtualization type: para
L1d cache: 32K
L1i cache: 32K
NUMA node0 CPU(s): 0-15

root@boslcp3g5:~# ppc64_cpu --smt
SMT=4
root@boslcp3g5:~#

Regards,
Indira

[Christian Ehrhardt  (paelzer)]

Thanks Indira for the retest with the new versions.
And if I am reading it correct that means the libvirt/qemu uploads fixed this issue in 18.04.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-02-20 06:42 EDT-------
(In reply to comment #17)
> Thanks Indira for the retest with the new versions.
> And if I am reading it correct that means the libvirt/qemu uploads fixed
> this issue in 18.04.

Yes issued fixed with 18.04

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-10-05 02:20 EDT-------
As per comment#16, this issue is fixed & tested in the bionic kernel.
root@boslcp3:~# uname -r
4.13.0-32-generic

Regards,
Indira