[Ubuntu 18.04] Memory hotplug fails with error 'virSecurityManagerSetMemoryLabel'

[bugproxy]

== Comment: #0 - NAGESWARA R. SASTRY <email address hidden> - 2018-02-13 23:52:48 ==
---Problem Description---
While doing memory hotplug seeing the following error from libvirt

error: Failed to attach device from mem_hp_512m.xml
error: this function is not supported by the connection driver: virSecurityManagerSetMemoryLabel

Contact Information = <email address hidden>

---uname output---
Linux hostname 4.13.0-32-generic #35-Ubuntu SMP Thu Jan 25 09:05:20 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

Machine Type = 5104-22C, TN71-BP012

---Debugger---
A debugger is not configured

---Steps to Reproduce---
 1. Install host with Ubuntu18.04
2. Start the Ubuntu18.04 guest with memory hotplug enabled guest xml tags
3. Do the memory hotplug
# virsh attach-device virt-tests-vm1-nrs mem_hp_512m.xml --live
error: Failed to attach device from mem_hp_512m.xml
error: this function is not supported by the connection driver: virSecurityManagerSetMemoryLabel

Will attach guest xml, memory hotplug xml, sosreport

== Comment: #1 - NAGESWARA R. SASTRY <email address hidden> - 2018-02-13 23:54:07 ==

== Comment: #2 - NAGESWARA R. SASTRY <email address hidden> - 2018-02-13 23:54:39 ==

== Comment: #3 - NAGESWARA R. SASTRY <email address hidden> - 2018-02-14 00:08:07 ==

== Comment: #4 - NAGESWARA R. SASTRY <email address hidden> - 2018-02-14 00:09:06 ==
# dpkg -l | grep "libvirt\|qemu"
ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu
ii libvirt-bin 4.0.0-1ubuntu2 ppc64el programs for the libvirt library
ii libvirt-clients 4.0.0-1ubuntu2 ppc64el Programs for the libvirt library
ii libvirt-daemon 4.0.0-1ubuntu2 ppc64el Virtualization daemon
ii libvirt-daemon-driver-storage-rbd 4.0.0-1ubuntu2 ppc64el Virtualization daemon RBD storage driver
ii libvirt-daemon-system 4.0.0-1ubuntu2 ppc64el Libvirt daemon configuration files
ii libvirt-dev:ppc64el 4.0.0-1ubuntu2 ppc64el development files for the libvirt library
ii libvirt-glib-1.0-0:ppc64el 1.0.0-1 ppc64el libvirt GLib and GObject mapping library
ii libvirt0:ppc64el 4.0.0-1ubuntu2 ppc64el library for interfacing with different virtualization systems
ii python-libvirt 4.0.0-1 ppc64el libvirt Python bindings
ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu1 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.11+dfsg-1ubuntu1 ppc64el QEMU Full virtualization on x86 hardware
ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.11+dfsg-1ubuntu1 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.11+dfsg-1ubuntu1 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.11+dfsg-1ubuntu1 ppc64el QEMU utilities

== Comment: #5 - NAGESWARA R. SASTRY <email address hidden> - 2018-02-14 00:09:33 ==
kernel version:
# uname -a
Linux ltc-boston128 4.13.0-32-generic #35-Ubuntu SMP Thu Jan 25 09:05:20 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

== Comment: #6 - NAGESWARA R. SASTRY <email address hidden> - 2018-02-14 01:12:28 ==

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[Frank Heimes (fheimes)]

The steps to reproduce in the bug description say 'install 18.04' and the uname command prints a 4.13.0-32 kernel.

Since I assume that your setup is an 18.04 host with an 18.04 guest, the kernel (or better the Ubuntu 18.04 daily build) in use is _very_ outdated.
Current 18.04 kernels are 4.15:
 linux-generic | 4.15.0.10.11 | bionic | ppc64el
 linux-generic | 4.15.0.12.13 | bionic-proposed | ppc64el
Even libvirt (libvirt-bin 4.0.0-1ubuntu2) seems to be quite old - current is:
 libvirt-bin | 4.0.0-1ubuntu4 | bionic | ppc64el

Please always update your system to the very latest level before testing, especially on releases that are currently in development - like 18.04 today - that potentially get updated packages virtually every day. So please re-try on latest packages.

[Christian Ehrhardt  (paelzer)]

Just freed up my test system from the verification of the next upload.
Trying to reproduce the issue ...

Starting a trivial uvt guest:
$ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=ppc64el label=daily release=bionic
$ uvt-kvm create --password=ubuntu cpaelzer-bionic release=bionic arch=ppc64el label=daily

Then adding the potential hotplug memory (without further changes)
That is 1.5G current and 10G max in 16 slots
  <maxMemory slots='16' unit='KiB'>10485760</maxMemory>
  <memory unit='KiB'>1572864</memory>
  <currentMemory unit='KiB'>1572864</currentMemory>

Attaching the hotplug mem xml (also simplified):
<memory model='dimm'>
        <target>
                <size unit='KiB'>524288</size>
        </target>
</memory>

I can confirm the reported issue with that:
ubuntu@wichita:~$ virsh attach-device cpaelzer-bionic hp512m.xml --live
error: Failed to attach device from hp512m.xml
error: this function is not supported by the connection driver: virSecurityManagerSetMemoryLabel

[Christian Ehrhardt  (paelzer)]

Sometimes there are new methods not directly implemented for Apparmor, if they are made required this can be an issue as in this case.

E.g. in the case of
+ .domainSetSecurityInputLabel = virSecurityApparmorSetInputLabel,
+ .domainRestoreSecurityInputLabel = virSecurityApparmoRestoreInputLabel,
this was internal so far, but now opened up.

I implemented some of them recently, but the issue reported indicates a few more are missing.
I checked which as of today apparmor could grow to be complete again and found:

    virSecurityDomainSetMemoryLabel domainSetSecurityMemoryLabel;
    virSecurityDomainRestoreMemoryLabel domainRestoreSecurityMemoryLabel;

    virSecurityDomainSetInputLabel domainSetSecurityInputLabel;
    virSecurityDomainRestoreInputLabel domainRestoreSecurityInputLabel;

Both are existing quite a while and never (so far) were crucial.
But some other change made them be a hard requirement.

I'll need to look at the arguments passed to implement and also check a few more details.
As it seems atm
mem: nvdimms have a path, provide a rw rule for them
inp: passthrough devs input->source.evdev seems to be a path, provide a rw rule for them

[Christian Ehrhardt  (paelzer)]

As assumed this can be triggered on x86 as well:
  <maxMemory slots='4' unit='KiB'>2097152</maxMemory>
  <memory unit='KiB'>524288</memory>
  <currentMemory unit='KiB'>524288</currentMemory>
...
  <cpu>
    <numa>
      <cell id='0' cpus='0' memory='524288' unit='KiB'/>
    </numa>
  </cpu>

hp512m.xml
<memory model='dimm'>
        <target>
                <size unit='KiB'>524288</size>
                <node>0</node>
        </target>
</memory>

root@b:~# virsh attach-device b-test hp512.xml --live
error: Failed to attach device from hp512.xml
error: this function is not supported by the connection driver: virSecurityManagerSetMemoryLabel

[Christian Ehrhardt  (paelzer)]

(very) preliminary changes building in [1] at the moment.
I'll need to iterate on them and submit them upstream once working for me.

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3200/+packages

[Christian Ehrhardt  (paelzer)]

In addition while testing I found that one of our last round of stable updates also makes this a segfault. This stalls resolving this a bit, at least it won't be submitted upstream the next hour as I planned.

[Christian Ehrhardt  (paelzer)]

While waiting here see bug 1756915 for the segfault until resolved.

[Christian Ehrhardt  (paelzer)]

Tested successfully from ppa - submitted upstream as part of an AppArmor related series.
=> https://www.redhat.com/archives/libvir-list/2018-March/msg01171.html

[Christian Ehrhardt  (paelzer)]

The fixes will get a respin, so I took this change out of the currently ongoing upload (to unblock it). I'll work on this one to be ready right after as much as possible.

[Christian Ehrhardt  (paelzer)]

V2 submitted at https://www.redhat.com/archives/libvir-list/2018-March/msg01240.html

[Christian Ehrhardt  (paelzer)]

V3: https://www.redhat.com/archives/libvir-list/2018-March/msg01278.html

[Christian Ehrhardt  (paelzer)]

Changes accepted upstream, preparing an upload and pushing it through regression tests before doing so.

[Christian Ehrhardt  (paelzer)]

No regression triggered, uploading ...

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 4.0.0-1ubuntu7

---------------
libvirt (4.0.0-1ubuntu7) bionic; urgency=medium

  * Fix nvdimm memory and passthrough input devices for hotplug via
    domain security callbacks backporting upstream commits (LP: #1755153).
    - d/p/ubuntu-aa/lp1755153-apparmor-add-Set-Restore-InputLabel.patch
    - d/p/ubuntu-aa/lp1755153-apparmor-add-Set-Restore-MemoryLabel.patch
  * Fix nvdimm memory and passthrough input devices in initial guest
    description via virt-aa-helper (LP: #1757085).
    - d/p/ubuntu-aa/lp1757085-virt-aa-helper-nvdimm-memory.patch
    - d/p/ubuntu-aa/lp1757085-virt-aa-helper-passthrough-input.patch

 -- Christian Ehrhardt <email address hidden> Wed, 21 Mar 2018 08:30:47 +0100

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-03-26 06:26 EDT-------
Seeing permission denied error

# virsh attach-device nrs mem_hp_512mb.xml --live
error: Failed to attach device from mem_hp_512mb.xml
error: cannot limit locked memory of process 25026 to 33554432: Permission denied

Will upload guest xml and libvirt log.

# dpkg -l | grep "qemu\|libvirt"
ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu
ii libvirt-bin 4.0.0-1ubuntu7 ppc64el programs for the libvirt library
ii libvirt-clients 4.0.0-1ubuntu7 ppc64el Programs for the libvirt library
ii libvirt-daemon 4.0.0-1ubuntu7 ppc64el Virtualization daemon
ii libvirt-daemon-driver-storage-rbd 4.0.0-1ubuntu7 ppc64el Virtualization daemon RBD storage driver
ii libvirt-daemon-system 4.0.0-1ubuntu7 ppc64el Libvirt daemon configuration files
ii libvirt-dev:ppc64el 4.0.0-1ubuntu7 ppc64el development files for the libvirt library
ii libvirt-glib-1.0-0:ppc64el 1.0.0-1 ppc64el libvirt GLib and GObject mapping library
ii libvirt0:ppc64el 4.0.0-1ubuntu7 ppc64el library for interfacing with different virtualization systems
ii python-libvirt 4.0.0-1 ppc64el libvirt Python bindings
ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu5 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.11+dfsg-1ubuntu5 ppc64el QEMU Full virtualization on x86 hardware
ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.11+dfsg-1ubuntu5 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.11+dfsg-1ubuntu5 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.11+dfsg-1ubuntu5 ppc64el QEMU utilities

# uname -a
Linux ltc-boston17 4.15.0-12-generic #13 SMP Thu Mar 22 14:16:58 CDT 2018 ppc64le ppc64le ppc64le GNU/Linux

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2018-03-26 06:29 EDT-------

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-03-26 06:32 EDT-------
from dmesg can see some audit related messages

[168891.101273] audit: type=1400 audit(1522059829.586:46305): apparmor="DENIED" operation="setrlimit" info="cap_sys_resource" error=-13 profile="/usr/sbin/libvirtd" pid=18008 comm="libvirtd" rlimit=memlock value=33554432 peer="libvirt-7dceb080-8d9b-4911-884b-4783a4545e5c"
[168891.101316] audit: type=1400 audit(1522059829.586:46306): apparmor="DENIED" operation="setrlimit" info="cap_sys_resource" error=-13 profile="/usr/sbin/libvirtd" pid=18008 comm="libvirtd" rlimit=memlock value=33554432 peer="libvirt-7dceb080-8d9b-4911-884b-4783a4545e5c"
[168891.348015] audit: type=1400 audit(1522059829.834:46307): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-7dceb080-8d9b-4911-884b-4783a4545e5c" pid=25133 comm="apparmor_parser"

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2018-03-26 06:30 EDT-------

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-03-26 07:36 EDT-------
Hi ,

I have treid hotplug attach memory today on kvm system and i still see its failing as below

libvirt version info:
*******************
root@boslcp3:~# virsh version
Compiled against library: libvirt 4.0.0
Using library: libvirt 4.0.0
Using API: QEMU 4.0.0
Running hypervisor: QEMU 2.11.1

root@boslcp3:/home# virsh dumpxml boslcp3g3 | grep max
<maxMemory slots='16' unit='KiB'>20971520</maxMemory>
root@boslcp3:/home# virsh dumpxml boslcp3g3 | grep mem
<memory unit='KiB'>14680064</memory>
<cell id='0' cpus='0-31' memory='7340032' unit='KiB'/>
<cell id='1' cpus='43-63' memory='7340032' unit='KiB'/>
<memballoon model='virtio'>
</memballoon>

root@boslcp3:/home# virsh attach-device boslcp3g3 mem-hp.xml --live
error: Failed to attach device from mem-hp.xml
error: this function is not supported by the connection driver: virSecurityManagerSetMemoryLabel

root@boslcp3:/home# cat mem-hp.xml
<memory model='dimm'>
<target>
<size unit='GiB'>2</size>
<node>0</node>
</target>
</memory>

Regards,
Indira

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-04-02 01:53 EDT-------
Do we have any updates on this issue?

[Christian Ehrhardt  (paelzer)]

I have used this successfully with non locked memory.
I wonder about comment #22 - would you mind restarting libvirtd and report "dpkg -l libvirt-daemon-system" instead of only "virsh version"?

The remaining issue (comment #20) you see with prlimit is an issue in apparmor.
TL;DR it is allowed in the apparmor profile but failing due to apparmor.

This already affected passthrough hotplug in the past and is known to IBM.
See bug 1679704

I'd not mark this one here as a dup, as I fixed the initially reported problem.
Instead I'd ask you to chime in on the referred bug to raise its priority now that we know more things being affected.

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2018-03-26 06:29 EDT-------

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2018-03-26 06:30 EDT-------

[Christian Ehrhardt  (paelzer)]

Once more - I tried on x86 and ppc64 for this this morning.
The same case as outlined in comment #7.

TL;DR:
- x86 works now, the reported issue of virSecurityManagerSetMemoryLabel missing is fixed
- ppc64 also passes the reported issue
- ppc64 hangs further down on the attach at bug 1679704

ppc:
ubuntu@wichita:~$ virsh attach-device cpaelzer-bionic hp512.xml
error: Failed to attach device from hp512.xml
error: cannot limit locked memory of process 10121 to 96468992: Permission denied

x86:
ubuntu@node-horsea:~$ virsh attach-device cpaelzer-bionic hp512.xml
Device attached successfully

For the apparmor issue:
[ 774.341567] audit: type=1400 audit(1522915593.238:41): apparmor="DENIED" operation="setrlimit" info="cap_sys_resource" error=-13 profile="/usr/sbin/libvirtd" pid=8376 comm="libvirtd" rlimit=memlock value=96468992 peer="libvirt-70a586a2-ef34-4954-91ea-9a6ecab52da3"
[ 774.341606] audit: type=1400 audit(1522915593.238:42): apparmor="DENIED" operation="setrlimit" info="cap_sys_resource" error=-13 profile="/usr/sbin/libvirtd" pid=8376 comm="libvirtd" rlimit=memlock value=96468992 peer="libvirt-70a586a2-ef34-4954-91ea-9a6ecab52da3"

Both are on a 4.15 kernel of Bionic.
Memory allocation works differently on these arches, which likely is the reason it follow-on fails at bug 1679704 while x86 is good.

But that still means the bug here is solved, I'm happy that you chime in on bug 1679704 and help me to raise the prio.
But lets stop updating this bug here, unless you really (=REALLY) think the problem reported here is reoccurring.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-06-04 05:11 EDT-------
Since this bug is resolved and has been tested already, closing...

------- Comment From <email address hidden> 2018-06-04 05:12 EDT-------
Thanks Everyone!