Realtime feature mlockall: Cannot allocate memory

Bug #1708305 reported by Jorge Niedbalski on 2017-08-02
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Medium
Unassigned
Declined for Newton by Corey Bryant
Mitaka
Medium
Jorge Niedbalski
Ocata
Medium
Jorge Niedbalski
libvirt (Ubuntu)
Medium
Unassigned
Xenial
Medium
Jorge Niedbalski
Zesty
Medium
Jorge Niedbalski
Artful
Medium
Unassigned

Bug Description

[Impact]

 * Guest definitions that uses locked memory + hugepages fail to spawn

 * Backport upstream fix to solve that issue

[Environment]

root@buneary:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial

root@buneary:~# uname -r
4.10.0-29-generic

Reproducible also with the 4.4 kernel.

[Detailed Description]

When a guest memory backing stanza is defined using the <locked/> stanza + hugepages,
as follows:

  <memoryBacking>
    <hugepages>
      <page size='1' unit='GiB' nodeset='0'/>
      <page size='1' unit='GiB' nodeset='1'/>
    </hugepages>
    <nosharedpages/>
    <locked/>
  </memoryBacking>

(Full guest definition: http://paste.ubuntu.com/25229162/)

The guest fails to start due to the following error:

2017-08-02 20:25:03.714+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10.12 (Christian Ehrhardt <email address hidden> Wed, 19 Jul 2017 08:28:14 +0200), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.14), hostname: buneary.seg
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name reproducer2 -S -machine pc-i440fx-2.5,accel=kvm,usb=off -cpu host -m 124928 -realtime mlock=on -smp 32,sockets=16,cores=1,threads=2 -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=64424509440,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0-15,memdev=ram-node0 -object memory-backend-file,id=ram-node1,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=66571993088,host-nodes=1,policy=bind -numa node,nodeid=1,cpus=16-31,memdev=ram-node1 -uuid 2460778d-979b-4024-9a13-0c3ca04b18ec -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-reproducer2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/uvtool/libvirt/images/test-ds.qcow,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on
Domain id=14 is tainted: host-cpu
char device redirected to /dev/pts/1 (label charserial0)

mlockall: Cannot allocate memory

2017-08-02T20:25:37.732772Z qemu-system-x86_64: locking memory failed
2017-08-02 20:25:37.811+0000: shutting down

This seems to be due to the setrlimit for RLIMIT_MEMLOCK is too low for mlockall
to work given the large amount of memory.

There is a libvirt upstream patch that enforces the existence of the
hard_limit stanza when using with <locked/> in the memory backing settings.

https://github.com/libvirt/libvirt/commit/c2e60ad0e5124482942164e5fec088157f5e716a

Memory locking can only work properly if the memory locking limit
for the QEMU process has been raised appropriately: the default one
is extremely low, so there's no way the guest will fit in there.

The commit https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b is also required when using hugepages and the locked stanza.

[Test Case]
 * Define a guest that uses the following stanzas (See for a full guest reference: http://paste.ubuntu.com/25288141/)

  <memory unit='GiB'>120</memory>
  <currentMemory unit='GiB'>120</currentMemory>

  <memoryBacking>
    <hugepages>
      <page size='1' unit='GiB' nodeset='0'/>
      <page size='1' unit='GiB' nodeset='1'/>
    </hugepages>
    <nosharedpages/>
    <locked/>
  </memoryBacking>

* virsh define guest.xml
* virsh start guest.xml
* Without the fix, the following error will be raised and the guest
will not start.

root@buneary:/home/ubuntu# virsh start reproducer2
error: Failed to start domain reproducer2
error: internal error: process exited while connecting to monitor: mlockall: Cannot allocate memory
2017-08-11T03:59:54.936275Z qemu-system-x86_64: locking memory failed

* With the fix, the error shouldn't be displayed and the guest started

[Suggested Fix]

* https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b (Proposed)

[Regression Potential]

 * There is one (theoretical) thing to think of - the change increases
   the lock resource limits for the spawned qemu. Maybe that could be
   used as an attack. Fortunately the defnition does have neither
   locked nor hugepages by default so opt-in, and while the guest can
   do all kind of things when exploited it can only hardly change it's
   (virtual) physical memory to increase what the host might be locking.
   Yet worth to mention IMHO

 * The general regression is rather low as I said "locked" is not
   default and the code path is only affecting domains configured that
   way. That makes the change rather safe for the overall user - and the
   one using locked likely need it.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in libvirt (Ubuntu):
status: New → Confirmed
tags: added: canonical-bootstack
Peter Sabaini (peter-sabaini) wrote :

Just to confirm that I can see this issue as well, and also that configuring memtune/hard_limit to $LARGE_VALUE makes it go away

With the workaround available to globally set the memtune/hard_limit this is prio low, but certainly sounds like a valid issue.

Depending if we merge libvirt 3.6 this will be part of that merge or a backport for artful and then consider SRUs from there.

FYI: I put this onto my tracking list, but will need a while to get back to that.

tags: added: libvirt-3.6
Changed in libvirt (Ubuntu Artful):
importance: Undecided → Low
Changed in libvirt (Ubuntu Artful):
status: Confirmed → In Progress
importance: Low → High
importance: High → Medium
assignee: nobody → Jorge Niedbalski (niedbalski)
description: updated
Changed in libvirt (Ubuntu Artful):
assignee: Jorge Niedbalski (niedbalski) → nobody
Changed in libvirt (Ubuntu Artful):
status: In Progress → Confirmed
Changed in libvirt (Ubuntu Xenial):
status: New → In Progress
Changed in libvirt (Ubuntu Zesty):
status: New → In Progress
Changed in libvirt (Ubuntu Xenial):
importance: Undecided → High
importance: High → Medium
Changed in libvirt (Ubuntu Zesty):
importance: Undecided → Medium
Changed in libvirt (Ubuntu Xenial):
assignee: nobody → Jorge Niedbalski (niedbalski)
Changed in libvirt (Ubuntu Zesty):
assignee: nobody → Jorge Niedbalski (niedbalski)
Jorge Niedbalski (niedbalski) wrote :

Hello Christian,

Attached you can find a xenial SRU patch based on commit (https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b).

Here is the guest definition used on my testing (http://paste.ubuntu.com/25288141/)

Without the patch, the error is reproduced.

root@buneary:/home/ubuntu# virsh start reproducer2
error: Failed to start domain reproducer2
error: internal error: process exited while connecting to monitor: mlockall: Cannot allocate memory
2017-08-11T03:59:54.936275Z qemu-system-x86_64: locking memory failed

With the patch the guest is initialized correctly.

root@buneary:/var/lib/uvtool/libvirt/images# virsh start reproducer2
Domain reproducer2 started

I ran the following systemtap probe (probe kernel.function("sys_setrlimit") { printf("locked: %d -> %s\n", pid(), sprintf("%s, %s", _rlimit_resource_str($resource), _struct_rlimit_u($rlim))) } )

And the output shows that the RLIMIT_MEMLOCK process limit is set to 9007199254740991 (VIR_DOMAIN_MEMORY_PARAM_UNLIMITED)

root@buneary:/home/ubuntu# stap -g qemu_probe.stap -kv -g --suppress-time-limits
Pass 1: parsed user script and 465 library scripts using 112796virt/48440res/6664shr/42136data kb, in 130usr/40sys/164real ms.
Pass 2: analyzed script: 1 probe, 5 functions, 100 embeds, 0 globals using 164772virt/101848res/7988shr/94112data kb, in 660usr/120sys/776real ms.
Pass 3: translated to C into "/tmp/stapVXuoMi/stap_30601_src.c" using 164772virt/102040res/8180shr/94112data kb, in 10usr/0sys/5real ms.
Pass 4: compiled C into "stap_30601.ko" in 2160usr/540sys/3231real ms.
Pass 5: starting run.
locked: 31028 -> RLIMIT_MEMLOCK, [9007199254740991,9007199254740991]

Changed in libvirt (Ubuntu Artful):
status: Confirmed → Fix Released
status: Fix Released → Fix Committed

The attachment "fix-1708305-xenial.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Jorge Niedbalski (niedbalski) wrote :

Attaching the patch for Zesty, please look the previous comment for
all the context.

Package for testing is available on this PPA https://launchpad.net/~niedbalski/+archive/ubuntu/lp1708305/+packages

description: updated
tags: added: sts-sru-needed

Hi Jorge,
thanks for your prep work on this.
Checking prereqs.

#1 fixed in Dev?
I read the "there is a patch" before as "there is a -new- patch", but checked now.
This is in since version 3.2 which makes it fix released in Artful already.
While I work on 3.6 we don't need it to go on with this as a fix in the dev version is a prereq.

#2 Patch?
I reviewed your changes and I'm ok with the patch, headers and backport.
Also in general while I'm not so sure on unlimited in general I see why they can't set a better value int his case - and since this is as-upstream this is good.

#3 SRU Queue?
Last SRU just complete, I think we are good adding a new one to the queue.

#4 Regressions
Thanks for pre-testing on your side already, gives some extra confidence.
I also checked via a (very basic) sniff test, full tests will have to be done on proposed but that we always do.
Therefore ok for now to go on.

#5 SRU Template
You already provided half of it, it misses a few things like regression considerations.
I can add those, but then nothing seems to block us from sponsoring this for SRU review IMO.

Changed in libvirt (Ubuntu Artful):
status: Fix Committed → Fix Released

Umm one thing - I checked qemu on the SRU queue, but while killing qemu due to limits it is obviously a libvirt change.

That said there is an ongoing libvirt SRU already.
It is all green on tests and verifications, but needs some time to complete following the SRU policy. That time this upload will have to wait until being considered and accepted by the SRU Team.
As I outlined below when I set low prio (I understand it is more for some people) I don't think it is overall urgent enough to override the other SRU that is already almost complete.

description: updated

Changes are in git, and pushed for SRU acceptance.

@SRU Team:
1. currently there is one libvirt SRU ongoing (on T/X - bug 1707400)
2. it is known and clear that acceptance of this SRU here prereqs that the referenced one has completed
=> So please accept this once the other completed (which looks good other than waiting the few days the process defines).

Changed in cloud-archive:
status: New → Fix Committed
importance: Undecided → Medium
Jorge Niedbalski (niedbalski) wrote :

Attached is the patch for T/M UCA.

Jorge Niedbalski (niedbalski) wrote :

Attached is the patch for Z/O UCA.

Hello Jorge, or anyone else affected,

Accepted libvirt into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/2.5.0-3ubuntu5.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in libvirt (Ubuntu Zesty):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-zesty
Edward Hope-Morley (hopem) wrote :

just for clarity i'm nominating for newton and it will be marked as invalid since there is no libvirt in newton uca

The former SRU now completed, could one on the SRU Team move the Xenial from unapproved to proposed as well?

Chris J Arges (arges) wrote :

Hello Jorge, or anyone else affected,

Accepted libvirt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/1.3.1-1ubuntu10.14 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in libvirt (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed-xenial
Jorge Niedbalski (niedbalski) wrote :

OK, I commented out the memtune section of the guest

  <!--<memtune>
    <hard_limit unit='GiB'>128</hard_limit>
  </memtune>-->

without a hard_limit set and started it, checking with the following stap script:

root@buneary:/home/ubuntu# stap -e 'probe kernel.function("sys_setrlimit") { printf("locked: %d -> %s\n", pid(), sprintf("%s, %s", _rlimit_resource_str($resource), _struct_rlimit_u($rlim))) }' -kv -g --suppress-time-limits
Pass 1: parsed user script and 465 library scripts using 112804virt/48432res/6652shr/42144data kb, in 160usr/20sys/176real ms.
Pass 2: analyzed script: 1 probe, 5 functions, 100 embeds, 0 globals using 164780virt/101832res/7968shr/94120data kb, in 740usr/140sys/2122real ms.
Pass 3: translated to C into "/tmp/stapXqZhh3/stap_54310_src.c" using 164780virt/102024res/8160shr/94120data kb, in 10usr/0sys/13real ms.
Pass 4: compiled C into "stap_54310.ko" in 2040usr/540sys/4985real ms.
Pass 5: starting run.

It looks like the process limit has been applied correctly and the guest starts:

locked: 54734 -> RLIMIT_MEMLOCK, [9007199254740991,9007199254740991]

Marking the xenial verification as done.

tags: added: verification-done-xenial
removed: verification-needed-xenial
Jorge Niedbalski (niedbalski) wrote :

Repeated the previous attempt with a zesty machine, same results.

root@zesty:/home/ubuntu# stap -e 'probe kernel.function("sys_setrlimit") { printf("locked: %d -> %s\n", pid(), sprintf("%s, %s", _rlimit_resource_str($resource), _struct_rlimit_u($rlim))) }' -kv -g --suppress-time-limits
Pass 1: parsed user script and 465 library scripts using 112796virt/48420res/6644shr/42136data kb, in 140usr/50sys/318real ms.
Pass 2: analyzed script: 1 probe, 5 functions, 100 embeds, 0 globals using 164772virt/101816res/7956shr/94112data kb, in 750usr/290sys/3419real ms.
Pass 3: translated to C into "/tmp/stap2amlcj/stap_54795_src.c" using 164772virt/102008res/8148shr/94112data kb, in 0usr/0sys/5real ms.
Pass 4: compiled C into "stap_54795.ko" in 2180usr/530sys/3318real ms.
Pass 5: starting run.
locked: 55222 -> RLIMIT_MEMLOCK, [9007199254740991,9007199254740991]

Marking the zesty verification as done.

tags: added: verification-done verification-done-zesty
removed: verification-needed verification-needed-zesty
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libvirt - 2.5.0-3ubuntu5.5

---------------
libvirt (2.5.0-3ubuntu5.5) zesty; urgency=medium

  * d/p/bug-1708305-qemu-Fix-memory-locking-limit-calculation.patch:
    Remove memlock limit when using <memoryBacking><locked/>.
    (LP: #1708305).

 -- Jorge Niedbalski <email address hidden> Fri, 11 Aug 2017 00:34:01 -0400

Changed in libvirt (Ubuntu Zesty):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for libvirt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libvirt - 1.3.1-1ubuntu10.14

---------------
libvirt (1.3.1-1ubuntu10.14) xenial; urgency=medium

  * d/p/bug-1708305-qemu-Fix-memory-locking-limit-calculation.patch:
    Remove memlock limit when using <memoryBacking><locked/>.
    (LP: #1708305).

 -- Jorge Niedbalski <email address hidden> Thu, 10 Aug 2017 22:50:46 -0400

Changed in libvirt (Ubuntu Xenial):
status: Fix Committed → Fix Released

Hello Jorge, or anyone else affected,

Accepted libvirt into mitaka-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:mitaka-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-mitaka-needed to verification-mitaka-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-mitaka-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-mitaka-needed
James Page (james-page) wrote :

Hello Jorge, or anyone else affected,

Accepted libvirt into ocata-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ocata-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ocata-needed to verification-ocata-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ocata-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-ocata-needed
Jorge Niedbalski (niedbalski) wrote :

Verified on mitaka cloud http://pastebin.ubuntu.com/25621886/

Jorge Niedbalski (niedbalski) wrote :

Verified on ocata cloud http://pastebin.ubuntu.com/25621949/

tags: added: verification-mitaka-done verification-ocata-done
removed: verification-mitaka-needed verification-ocata-needed

The verification of the Stable Release Update for libvirt has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

James Page (james-page) wrote :

This bug was fixed in the package libvirt - 1.3.1-1ubuntu10.14~cloud0
---------------

 libvirt (1.3.1-1ubuntu10.14~cloud0) trusty-mitaka; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 libvirt (1.3.1-1ubuntu10.14) xenial; urgency=medium
 .
   * d/p/bug-1708305-qemu-Fix-memory-locking-limit-calculation.patch:
     Remove memlock limit when using <memoryBacking><locked/>.
     (LP: #1708305).

James Page (james-page) wrote :

The verification of the Stable Release Update for libvirt has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

James Page (james-page) wrote :

This bug was fixed in the package libvirt - 2.5.0-3ubuntu5.5~cloud0
---------------

 libvirt (2.5.0-3ubuntu5.5~cloud0) xenial-ocata; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 libvirt (2.5.0-3ubuntu5.5) zesty; urgency=medium
 .
   * d/p/bug-1708305-qemu-Fix-memory-locking-limit-calculation.patch:
     Remove memlock limit when using <memoryBacking><locked/>.
     (LP: #1708305).

Changed in cloud-archive:
status: Fix Committed → Fix Released
tags: added: sts-sru-done
removed: sts-sru-needed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers