[power9][Ubuntu 18.04][4.15.0-10-generic][libvirt-bin-4.0.0-1ubuntu2] KVM guest unable start from libvirt "could not find capabilities for arch=ppc64le " (kvm)

[bugproxy]

== Comment: #0 - Satheesh Rajendran <email address hidden> - 2018-02-17 11:44:57 ==
---Problem Description---
virsh domcapabilities fails with "unable to find any emulator to serve 'ppc64le' architecture", and libvirt unable to define/start a guest with error "could not find capabilities for arch=ppc64le ",
and virt-install complains as host does not have virtualisation support looks like the root cause of all these errors are same.

Host is able to boot the kvm guest from qemu fine.

#/usr/bin/kvm -M pseries -nographic -serial /dev/pts/0 -monitor stdio /home/sath/avocado-fvt-wrapper/data/avocado-vt/images/ubuntu-18.04-ppc64le.qcow2 -device virtio-scsi-pci -m 5120 -smp 1 -vga none
(qemu) info status
VM status: running
(qemu) info kvm
kvm support: enabled
(qemu) info cpus
* CPU #0: nip=0x000000000dbe4acc thread_id=12917
(qemu)

Env:
4.15.0-10-generic
ii qemu-kvm 1:2.11+dfsg-1ubuntu1 ppc64el QEMU Full virtualization on x86 hardware
ii libvirt-bin 4.0.0-1ubuntu2 ppc64el programs for the libvirt library

# lsmod|grep kvm
kvm_hv 132653 1
kvm 223734 4 kvm_hv

Contact Information = <email address hidden>

---uname output---
Linux ltc-boston8 4.15.0-10-generic #11-Ubuntu SMP Tue Feb 13 18:21:52 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

Machine Type = power9

---Steps to Reproduce---
 virsh domcapabilities
error: failed to get emulator capabilities
error: invalid argument: unable to find any emulator to serve 'ppc64le' architecture ----------------NOK

2. # virsh domcapabilities --emulatorbin /usr/bin/qemu-system-ppc64
error: failed to get emulator capabilities
error: internal error: QEMU / QMP failed: qemu-system-ppc64: could not acquire pid file: Permission denied --------------------NOK

3.# virsh define vm1
error: Failed to define domain from vm1
error: invalid argument: could not find capabilities for arch=ppc64le ---------------NOK

# cat vm1
<domain type='kvm'>
  <name>vm1</name>
  <memory unit='G'>8</memory>
  <vcpu placement='static'>1</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='ppc64le' machine='pseries-bionic'>hvm</type>
    <boot dev='hd'/>
  </os>
  <cpu>
    <topology sockets='1' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/sath/avocado-fvt-wrapper/data/avocado-vt/images/ubuntu-18.04-ppc64le.qcow2'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='scsi' index='0' model='virtio-scsi'>

<alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='usb' index='0'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:a7:a8:a9'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
      <alias name='serial0'/>
      <address type='spapr-vio' reg='0x30000000'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
      <address type='spapr-vio' reg='0x30000000'/>
    </console>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>
    <panic model='pseries'/>
  </devices>
  <seclabel type='dynamic' model='apparmor' relabel='yes'>
  </seclabel>
</domain>

4) #/usr/bin/virt-install --connect=qemu:///system --hvm
ERROR Host does not support virtualization type 'hvm' -----------NOK

Userspace tool common name: ii libvirt-bin 4.0.0-1ubuntu2 ppc64el programs for the libvirt library

The userspace tool has the following bit modes: both

Userspace rpm: ii libvirt-bin 4.0.0-1ubuntu2 ppc64el programs for the libvirt library

Userspace tool obtained from project website: na

*Additional Instructions for <email address hidden>:
-Post a private note with access information to the machine that the bug is occuring on.
-Attach ltrace and strace of userspace application.

== Comment: #3 - Shivaprasad G. Bhat <email address hidden> - 2018-02-20 00:44:49 ==
This is some apparmor issue.

The qemu is not able to access /var/lib/libvirt directory when run as qemu:qemu.

I changed the user to root:root, the things are working fine now.

Want this to be mirrored to Canonical to figure out more on why the issue with non-root user.

Thanks,
Shiva

== Comment: #6 - SRIKANTH AITHAL <email address hidden> - 2018-02-20 02:19:49 ==
(In reply to comment #3)
> This is some apparmor issue.
>
> The qemu is not able to access /var/lib/libvirt directory when run as
> qemu:qemu.
>
> I changed the user to root:root, the things are working fine now.
>
> Want this to be mirrored to Canonical to figure out more on why the issue
> with non-root user.
>
> Thanks,
> Shiva

Can we have this mirrored to Canonical please?
Since we have workaround I am reducing severity to ship, still this issue should be resolved noting that customers would prefer using non-root users.

[bugproxy (bugproxy)]

Default Comment by Bridge

[Christian Ehrhardt  (paelzer)]

Quote the essential part of the bug:
"This is some apparmor issue.
The qemu is not able to access /var/lib/libvirt directory when run as qemu:qemu.
I changed the user to root:root, the things are working fine now."

Note: The default is libvirt-qemu:kvm which works just fine.

Not sure where/why set qemu:qemu user/group for qemu.
But if I do set so in /etc/libvirt/qemu.conf and create such a user (qemu:qemu is no default as I said).
With that set still the referred domcapabilities commands work.
If with qemu/qemu I try to start a guest it fails:
  qemu-system-ppc64: failed to initialize KVM: Permission denied
As we have:
crw-rw---- 1 root kvm 10, 232 Feb 20 11:42 /dev/kvm

The attached ./sos_commands/kernel/dmesg has no apparmor Denies.

I have to set this to incomplete for now and need to ask:
- why user qemu:qemu?
  - if you set non default users you might need to change a lot to get it working.
- I read apparmor issues, but found none in the logs, could you provide the messages of the apparmor denies
- the sosreport holds a lot, but just not e.g. the qemu.conf to set the user
TL;DR - I need to see the actual failing config and the error/Denial by apparmor

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-02-20 09:51 EDT-------
Yes(In reply to comment #10)
> If with qemu/qemu I try to start a guest it fails:
> qemu-system-ppc64: failed to initialize KVM: Permission denied
> As we have:
> crw-rw---- 1 root kvm 10, 232 Feb 20 11:42 /dev/kvm
>
> The attached ./sos_commands/kernel/dmesg has no apparmor Denies.
>

Correct. There are no denies.

> I have to set this to incomplete for now and need to ask:
> - why user qemu:qemu?

My bad, typos.. The user is "qemu" on Redhat, "libvirt-qemu" on Debian variants.

> - if you set non default users you might need to change a lot to get it
> working.

Now that you confirmed there might not be an apparmor issue here, I see libvirt-qemu doesnt have permissions to open directory '/var/lib/libvirt/'.

I see libvirt-daemon-system.postinst file has
adduser --quiet \
--system \
--ingroup kvm \
--quiet \
--disabled-login \
--disabled-password \
--home /var/lib/libvirt \
--no-create-home \
--gecos "Libvirt Qemu" \
$PARAMETER_UID \
libvirt-qemu

So, Now I see the libvirt-qemu is created during the install of libvirt-daemon-system and has --no-create-home. And I see the directory
ls -lrt /var/lib/libvirt/ -d
drwx------ 7 root root 4096 Feb 17 08:23 /var/lib/libvirt/
has no permission for non-root user.

Now I am not sure if it is the package which created this directory with such permissions Or it is the test automation scripts.

Satheesh,
Can you confirm if your test suite creates /var/lib/libvirt directory as part of test setup ? Or does it alter the permissions as we see above ? On hosts where things are working I see the directory to have "drwxr-xr-x"..

Thanks,
Shivaprasad

[Christian Ehrhardt  (paelzer)]

Hi,
help sorting out dirs and required access here:
$ ll /var/lib/libvirt/
drwxr-xr-x 7 root root 4096 Feb 8 07:55 ./
drwx--x--x 2 root root 4096 Dec 14 13:15 boot/
drwxr-xr-x 2 root root 4096 Feb 19 14:08 dnsmasq/
drwx--x--x 2 root root 4096 Dec 14 13:15 images/
drwx------ 2 root root 4096 Dec 14 13:15 sanlock/

On the above only libvirt needs to access (and libvirt runs as root)

drwxr-x--- 8 libvirt-qemu kvm 4096 Feb 14 08:02 qemu/

this is the one dir qemu (read potentially untrusted guest code) needs to access.
So only this dir is on libvirt-qemu:kvm ownership.

So "libvirt-qemu doesnt have permissions to open directory '/var/lib/libvirt/'.":
Yes, and it usually doesn't need.

I hope this helps when you double check your setup.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-02-20 11:53 EDT-------
Looks like before installing libvirt the `/var/lib/libvirt/images/` folder got created by testbed(jenkins) as the base folder for the slave is configured as /var/lib/libvirt/images/,

Just to make sure, I have deleted the complete /var/lib/libvirt folder and removed libvirt package and reinstalled it back, I could see now the /var/lib/libvirt images folder permissions are intact

VMs are starting fine now.

#virsh domcapabilities
<domainCapabilities>
<path>/usr/bin/qemu-system-ppc64</path>
<domain>kvm</domain>
<machine>pseries-bionic</machine>
<arch>ppc64le</arch>
<vcpu max='1024'/>
...

# ls -l libvirt
ls: cannot access 'libvirt': No such file or directory

------During libvirt package install
# ls -l libvirt
total 16
drwxr-xr-x 2 root root 4096 Feb 2 01:31 boot
drwxr-xr-x 2 root root 4096 Feb 2 01:31 images
drwxr-xr-x 3 root root 4096 Feb 20 11:34 qemu
drwxr-xr-x 2 root root 4096 Feb 2 01:31 sanlock

# ls -l libvirt
total 16
drwx--x--x 2 root root 4096 Feb 2 01:31 boot
drwx--x--x 2 root root 4096 Feb 2 01:31 images
drwxr-x--- 3 libvirt-qemu libvirt-qemu 4096 Feb 20 11:34 qemu
drwx------ 2 root root 4096 Feb 2 01:31 sanlock

# ls -l libvirt
total 20
drwx--x--x 2 root root 4096 Feb 2 01:31 boot
drwxr-xr-x 2 root root 4096 Feb 20 11:39 dnsmasq
drwx--x--x 2 root root 4096 Feb 2 01:31 images
drwxr-x--- 9 root root 4096 Feb 20 11:38 qemu --------I could see qemu folder permissions are getting altered during package install
drwx------ 2 root root 4096 Feb 2 01:31 sanlock

# ls -l
total 20
drwx--x--x 2 root root 4096 Feb 2 01:31 boot
drwxr-xr-x 2 root root 4096 Feb 20 11:39 dnsmasq
drwx--x--x 2 root root 4096 Feb 2 01:31 images
drwxr-x--- 9 root root 4096 Feb 20 11:38 qemu
drwx------ 2 root root 4096 Feb 2 01:31 sanlock

I hope this should be fine?

Regards,
-Satheesh

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-02-20 12:16 EDT-------
Little confusing, I just tried creating a folder in /var/lib as root user it did create with below permission, which is same as current libvirt folder permission, even if the jenkins created folder should had the same. something missing still?

drwxr-xr-x 2 root root 4096 Feb 20 12:12 temp

Regards,
-Satheesh

[Christian Ehrhardt  (paelzer)]

Glad that it works now per comment #5

Yes folder permissions are changed on install - they are created first and then chown'ed.

I didn't get the last issue of manually creating /var/lib/temp as root:root - that should be unrelated right?

TL;DR - it seems to work as set up by the package and if you can convince the jenkins setup of the testbed to not mess with it things are good.
Setting invalid per former comments - feel free to continue to discuss here whatever that temp dir in the last comment was about.

[Satheesh Rajendran (sathnaga)]

I meant by, even if the jenkins created a folder `/var/lib/libvirt/images` in first place, it should have had the right permissions as how `temp` folder I created above has the same permissions as the install'ed libvirt folder now, I do not find a convincing reason why it did not work in the first place.

I hope the env does not have any local workarounds to make it work and new install of host
again gets this issue :-)

Regards,
-Satheesh.