ubuntu_kvm_smoke_test failed with "qemu-system-ppc64le: Requested safe cache capability level not supported by kvm" on Eoan P8

[Po-Hsu Lin]

Issue found on 5.3.0-56.50 with Power8 node "modoc"
qemu-kvm package version: 1:4.0+dfsg-0ubuntu9.6

Reproduce rate 2 out of 2

The test will fail with:
 + uvt-simplestreams-libvirt sync --source http://cloud-images.ubuntu.com/daily release=eoan arch=ppc64el
 + uvt-kvm create bjf-test release=eoan arch=ppc64el
 uvt-kvm: error: libvirt: internal error: qemu unexpectedly closed the monitor: 2020-05-28T10:22:52.552105Z qemu-system-ppc64le: Requested safe cache capability level not supported by kvm, try cap-cfpc=broken

There was a similar bug back in 2019, power8 machines need FW update or qemu/libvirt code - cap-cfpc=broken (bug 1839065)

[Po-Hsu Lin (cypressyew)]

This test works with P9 node "baltar" on Eoan.

Trying to reproduce this on another P8 node "wichita"

[Po-Hsu Lin (cypressyew)]

This issue is not a regression for this cycle, as it can be reproduce on 5.3.0-55-generic with qemu-kvm 4.0+dfsg-0ubuntu9.6

And this issue can be traced back to 2019-12-11 (bug 1798358)

[Frank Heimes (fheimes)]

Hi, please ignore my ignorance, but does this now mean that the test needs to be black listed for P8 on Eoan ?
Or is here a fix needed in Eoan's qemu-kvm?
(I think it's just a black listing, right?)

[Po-Hsu Lin (cypressyew)]

This issue can be found on Focal with this P8 node "modoc" as well.

[Po-Hsu Lin (cypressyew)]

Hello Frank,

I am not sure if a FW upgrade like the one mentioned in bug 1839065 (it was for node "Wichita") will be helpful?

Or maybe we will need a fix for qemu-kvm in Eoan / Focal, as this ubuntu_kvm_smoke test has passed with B-5.3 on this node "modoc"

A quick glance for this test on different releases / kernels on this P8 node "modoc":
  * F-5.4: failed with this issue
  * E-5.3: failed with this issue
  * B-5.3: passed
  * B-4.15-ibm-gt: failed with bug 1882518
  * B-4.15: passed
  * X-4.15: passed
  * X-4.4: passed

(If we can confirm that this is not supported on E/F for modoc, we can blacklist it)

[Frank Heimes (fheimes)]

So 'Wichita' as well as 'Modoc' are both Tuleta/P8 systems (exact same type and model),
and also both are marked as FW updated, but they are obviously on different FW levels.
(I was only able to get the high level FW levels so far.)
wichita: FW860.70 (SV860_205)
modoc: FW860.20 (SV860_082)

Since cpaelzer successfully used wichita in LP 1839065 (with the newer FW version):
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1839065/comments/10
FW860.70 (SV860_205) (t) FW860.70 (SV860_205) (p) FW860.70 (SV860_205) (b)
I think it's indeed worth a try to update modoc, too - the same FW should run on modoc.

But 'wichita' is available right now in MAAS, I think it might make sense to test on wichita first, to get (hopefully) a confirmation that it works there, and then updating modoc in a follow on step!? (iirc a note was sent out some time ago on how to update the Power systems FW)

[Po-Hsu Lin (cypressyew)]

A quick test for the kvm smoke test on witchita:
 * Focal - Test passed
 * Eoan - cpu soft lockup when running "sudo ppc64_cpu --smt=off", syslog could be found here https://paste.ubuntu.com/p/pwtD8VB5kk/

This soft lockup did not occur on Eoan modoc, sine it will try to disable smt before trying to fetch the image with uvt-simplestreams-libvirt

[Andrew Cloke (andrew-cloke)]

Next step would be to update modoc firmware to the same level as witchita.

[Andrew Cloke (andrew-cloke)]

Po-Hsu Lin mentioned that he is happy to attempt a firmware update at a suitable moment.

[Po-Hsu Lin (cypressyew)]

With the firmware updated to SV860_205 on node "modoc". The KVM can now be created and tested on Eoan without any issue.

(Not even the cpu soft lockup issue on witchita with "sudo ppc64_cpu --smt=off", that might worth another bug to get investigated, I can open one if you need.)

Thanks for your help, I will close this bug now.