Activity log for bug #1708305

Date Who What changed Old value New value Message
2017-08-02 22:20:10 Jorge Niedbalski bug added bug
2017-08-02 22:20:24 Jorge Niedbalski nominated for series Ubuntu Artful
2017-08-02 22:20:24 Jorge Niedbalski nominated for series Ubuntu Xenial
2017-08-02 22:20:24 Jorge Niedbalski nominated for series Ubuntu Zesty
2017-08-03 09:02:05 Launchpad Janitor libvirt (Ubuntu): status New Confirmed
2017-08-03 09:03:03 Peter Sabaini tags canonical-bootstack
2017-08-03 09:05:02 Dominique Poulain bug added subscriber Dominique Poulain
2017-08-03 17:04:15 Eric Desrochers bug task added libvirt (Ubuntu Artful)
2017-08-03 17:04:20 Eric Desrochers bug task added libvirt (Ubuntu Xenial)
2017-08-03 17:04:27 Eric Desrochers bug task added libvirt (Ubuntu Zesty)
2017-08-07 10:09:58 Christian Ehrhardt  bug added subscriber ChristianEhrhardt
2017-08-07 10:11:07 Christian Ehrhardt  tags canonical-bootstack canonical-bootstack libvirt-3.6
2017-08-07 10:24:41 Christian Ehrhardt  libvirt (Ubuntu Artful): importance Undecided Low
2017-08-10 21:22:52 Jorge Niedbalski libvirt (Ubuntu Artful): status Confirmed In Progress
2017-08-10 21:22:57 Jorge Niedbalski libvirt (Ubuntu Artful): importance Low High
2017-08-10 21:23:04 Jorge Niedbalski libvirt (Ubuntu Artful): importance High Medium
2017-08-10 21:23:09 Jorge Niedbalski libvirt (Ubuntu Artful): assignee Jorge Niedbalski (niedbalski)
2017-08-10 21:24:02 Jorge Niedbalski description [Environment] root@buneary:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.2 LTS Release: 16.04 Codename: xenial root@buneary:~# uname -r 4.10.0-29-generic Reproducible also with the 4.4 kernel. [Description] When a guest memory backing stanza is defined using the <locked/> stanza + hugepages, as follows: <memoryBacking> <hugepages> <page size='1' unit='GiB' nodeset='0'/> <page size='1' unit='GiB' nodeset='1'/> </hugepages> <nosharedpages/> <locked/> </memoryBacking> (Full guest definition: http://paste.ubuntu.com/25229162/) The guest fails to start due to the following error: 2017-08-02 20:25:03.714+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10.12 (Christian Ehrhardt <christian.ehrhardt@canonical.com> Wed, 19 Jul 2017 08:28:14 +0200), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.14), hostname: buneary.seg LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name reproducer2 -S -machine pc-i440fx-2.5,accel=kvm,usb=off -cpu host -m 124928 -realtime mlock=on -smp 32,sockets=16,cores=1,threads=2 -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=64424509440,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0-15,memdev=ram-node0 -object memory-backend-file,id=ram-node1,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=66571993088,host-nodes=1,policy=bind -numa node,nodeid=1,cpus=16-31,memdev=ram-node1 -uuid 2460778d-979b-4024-9a13-0c3ca04b18ec -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-reproducer2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/uvtool/libvirt/images/test-ds.qcow,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on Domain id=14 is tainted: host-cpu char device redirected to /dev/pts/1 (label charserial0) mlockall: Cannot allocate memory 2017-08-02T20:25:37.732772Z qemu-system-x86_64: locking memory failed 2017-08-02 20:25:37.811+0000: shutting down This seems to be due to the setrlimit for RLIMIT_MEMLOCK is too low for mlockall to work given the large amount of memory. There is a libvirt upstream patch that enforces the existence of the hard_limit stanza when using with <locked/> in the memory backing settings. https://github.com/libvirt/libvirt/commit/c2e60ad0e5124482942164e5fec088157f5e716a Memory locking can only work properly if the memory locking limit for the QEMU process has been raised appropriately: the default one is extremely low, so there's no way the guest will fit in there. [Suggested Fix] * https://github.com/libvirt/libvirt/commit/c2e60ad0e5124482942164e5fec088157f5e716a [Environment] root@buneary:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.2 LTS Release: 16.04 Codename: xenial root@buneary:~# uname -r 4.10.0-29-generic Reproducible also with the 4.4 kernel. [Description] When a guest memory backing stanza is defined using the <locked/> stanza + hugepages, as follows:   <memoryBacking>     <hugepages>       <page size='1' unit='GiB' nodeset='0'/>       <page size='1' unit='GiB' nodeset='1'/>     </hugepages>     <nosharedpages/>     <locked/>   </memoryBacking> (Full guest definition: http://paste.ubuntu.com/25229162/) The guest fails to start due to the following error: 2017-08-02 20:25:03.714+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10.12 (Christian Ehrhardt <christian.ehrhardt@canonical.com> Wed, 19 Jul 2017 08:28:14 +0200), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.14), hostname: buneary.seg LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name reproducer2 -S -machine pc-i440fx-2.5,accel=kvm,usb=off -cpu host -m 124928 -realtime mlock=on -smp 32,sockets=16,cores=1,threads=2 -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=64424509440,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0-15,memdev=ram-node0 -object memory-backend-file,id=ram-node1,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=66571993088,host-nodes=1,policy=bind -numa node,nodeid=1,cpus=16-31,memdev=ram-node1 -uuid 2460778d-979b-4024-9a13-0c3ca04b18ec -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-reproducer2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/uvtool/libvirt/images/test-ds.qcow,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on Domain id=14 is tainted: host-cpu char device redirected to /dev/pts/1 (label charserial0) mlockall: Cannot allocate memory 2017-08-02T20:25:37.732772Z qemu-system-x86_64: locking memory failed 2017-08-02 20:25:37.811+0000: shutting down This seems to be due to the setrlimit for RLIMIT_MEMLOCK is too low for mlockall to work given the large amount of memory. There is a libvirt upstream patch that enforces the existence of the hard_limit stanza when using with <locked/> in the memory backing settings. https://github.com/libvirt/libvirt/commit/c2e60ad0e5124482942164e5fec088157f5e716a Memory locking can only work properly if the memory locking limit for the QEMU process has been raised appropriately: the default one is extremely low, so there's no way the guest will fit in there. The commit https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b is also required when using hugepages and the locked stanza. [Suggested Fix] * https://github.com/libvirt/libvirt/commit/c2e60ad0e5124482942164e5fec088157f5e716a * https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b
2017-08-10 21:40:55 Jorge Niedbalski libvirt (Ubuntu Artful): assignee Jorge Niedbalski (niedbalski)
2017-08-10 21:50:11 Jorge Niedbalski libvirt (Ubuntu Artful): status In Progress Confirmed
2017-08-11 02:37:17 Jorge Niedbalski libvirt (Ubuntu Xenial): status New In Progress
2017-08-11 02:37:21 Jorge Niedbalski libvirt (Ubuntu Zesty): status New In Progress
2017-08-11 02:37:32 Jorge Niedbalski libvirt (Ubuntu Xenial): importance Undecided High
2017-08-11 02:37:35 Jorge Niedbalski libvirt (Ubuntu Xenial): importance High Medium
2017-08-11 02:37:38 Jorge Niedbalski libvirt (Ubuntu Zesty): importance Undecided Medium
2017-08-11 02:37:51 Jorge Niedbalski libvirt (Ubuntu Xenial): assignee Jorge Niedbalski (niedbalski)
2017-08-11 02:37:54 Jorge Niedbalski libvirt (Ubuntu Zesty): assignee Jorge Niedbalski (niedbalski)
2017-08-11 04:18:16 Jorge Niedbalski libvirt (Ubuntu Artful): status Confirmed Fix Released
2017-08-11 04:18:22 Jorge Niedbalski libvirt (Ubuntu Artful): status Fix Released Fix Committed
2017-08-11 04:19:03 Jorge Niedbalski attachment added fix-1708305-xenial.debdiff https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1708305/+attachment/4930463/+files/fix-1708305-xenial.debdiff
2017-08-11 04:21:23 Ubuntu Foundations Team Bug Bot tags canonical-bootstack libvirt-3.6 canonical-bootstack libvirt-3.6 patch
2017-08-11 04:21:30 Ubuntu Foundations Team Bug Bot bug added subscriber Ubuntu Sponsors Team
2017-08-11 04:32:24 Jorge Niedbalski description [Environment] root@buneary:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.2 LTS Release: 16.04 Codename: xenial root@buneary:~# uname -r 4.10.0-29-generic Reproducible also with the 4.4 kernel. [Description] When a guest memory backing stanza is defined using the <locked/> stanza + hugepages, as follows:   <memoryBacking>     <hugepages>       <page size='1' unit='GiB' nodeset='0'/>       <page size='1' unit='GiB' nodeset='1'/>     </hugepages>     <nosharedpages/>     <locked/>   </memoryBacking> (Full guest definition: http://paste.ubuntu.com/25229162/) The guest fails to start due to the following error: 2017-08-02 20:25:03.714+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10.12 (Christian Ehrhardt <christian.ehrhardt@canonical.com> Wed, 19 Jul 2017 08:28:14 +0200), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.14), hostname: buneary.seg LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name reproducer2 -S -machine pc-i440fx-2.5,accel=kvm,usb=off -cpu host -m 124928 -realtime mlock=on -smp 32,sockets=16,cores=1,threads=2 -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=64424509440,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0-15,memdev=ram-node0 -object memory-backend-file,id=ram-node1,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=66571993088,host-nodes=1,policy=bind -numa node,nodeid=1,cpus=16-31,memdev=ram-node1 -uuid 2460778d-979b-4024-9a13-0c3ca04b18ec -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-reproducer2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/uvtool/libvirt/images/test-ds.qcow,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on Domain id=14 is tainted: host-cpu char device redirected to /dev/pts/1 (label charserial0) mlockall: Cannot allocate memory 2017-08-02T20:25:37.732772Z qemu-system-x86_64: locking memory failed 2017-08-02 20:25:37.811+0000: shutting down This seems to be due to the setrlimit for RLIMIT_MEMLOCK is too low for mlockall to work given the large amount of memory. There is a libvirt upstream patch that enforces the existence of the hard_limit stanza when using with <locked/> in the memory backing settings. https://github.com/libvirt/libvirt/commit/c2e60ad0e5124482942164e5fec088157f5e716a Memory locking can only work properly if the memory locking limit for the QEMU process has been raised appropriately: the default one is extremely low, so there's no way the guest will fit in there. The commit https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b is also required when using hugepages and the locked stanza. [Suggested Fix] * https://github.com/libvirt/libvirt/commit/c2e60ad0e5124482942164e5fec088157f5e716a * https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b [Environment] root@buneary:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.2 LTS Release: 16.04 Codename: xenial root@buneary:~# uname -r 4.10.0-29-generic Reproducible also with the 4.4 kernel. [Description] When a guest memory backing stanza is defined using the <locked/> stanza + hugepages, as follows:   <memoryBacking>     <hugepages>       <page size='1' unit='GiB' nodeset='0'/>       <page size='1' unit='GiB' nodeset='1'/>     </hugepages>     <nosharedpages/>     <locked/>   </memoryBacking> (Full guest definition: http://paste.ubuntu.com/25229162/) The guest fails to start due to the following error: 2017-08-02 20:25:03.714+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10.12 (Christian Ehrhardt <christian.ehrhardt@canonical.com> Wed, 19 Jul 2017 08:28:14 +0200), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.14), hostname: buneary.seg LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name reproducer2 -S -machine pc-i440fx-2.5,accel=kvm,usb=off -cpu host -m 124928 -realtime mlock=on -smp 32,sockets=16,cores=1,threads=2 -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=64424509440,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0-15,memdev=ram-node0 -object memory-backend-file,id=ram-node1,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=66571993088,host-nodes=1,policy=bind -numa node,nodeid=1,cpus=16-31,memdev=ram-node1 -uuid 2460778d-979b-4024-9a13-0c3ca04b18ec -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-reproducer2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/uvtool/libvirt/images/test-ds.qcow,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on Domain id=14 is tainted: host-cpu char device redirected to /dev/pts/1 (label charserial0) mlockall: Cannot allocate memory 2017-08-02T20:25:37.732772Z qemu-system-x86_64: locking memory failed 2017-08-02 20:25:37.811+0000: shutting down This seems to be due to the setrlimit for RLIMIT_MEMLOCK is too low for mlockall to work given the large amount of memory. There is a libvirt upstream patch that enforces the existence of the hard_limit stanza when using with <locked/> in the memory backing settings. https://github.com/libvirt/libvirt/commit/c2e60ad0e5124482942164e5fec088157f5e716a Memory locking can only work properly if the memory locking limit for the QEMU process has been raised appropriately: the default one is extremely low, so there's no way the guest will fit in there. The commit https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b is also required when using hugepages and the locked stanza. [Test Case] * Define a guest that uses the following stanzas (See for a full guest reference: http://paste.ubuntu.com/25288141/) <memory unit='GiB'>120</memory> <currentMemory unit='GiB'>120</currentMemory> <memoryBacking> <hugepages> <page size='1' unit='GiB' nodeset='0'/> <page size='1' unit='GiB' nodeset='1'/> </hugepages> <nosharedpages/> <locked/> </memoryBacking> * virsh define guest.xml * virsh start guest.xml * Without the fix, the following error will be raised and the guest will not start. root@buneary:/home/ubuntu# virsh start reproducer2 error: Failed to start domain reproducer2 error: internal error: process exited while connecting to monitor: mlockall: Cannot allocate memory 2017-08-11T03:59:54.936275Z qemu-system-x86_64: locking memory failed * With the fix, the error shouldn't be displayed and the guest started [Suggested Fix] * https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b (Proposed)
2017-08-11 04:32:34 Jorge Niedbalski tags canonical-bootstack libvirt-3.6 patch canonical-bootstack libvirt-3.6 patch sts-sru-needed
2017-08-11 04:36:35 Jorge Niedbalski attachment added fix-1708305-zesty.debdiff https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1708305/+attachment/4930472/+files/fix-1708305-zesty.debdiff
2017-08-11 05:50:02 Christian Ehrhardt  libvirt (Ubuntu Artful): status Fix Committed Fix Released
2017-08-11 06:03:23 Christian Ehrhardt  description [Environment] root@buneary:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.2 LTS Release: 16.04 Codename: xenial root@buneary:~# uname -r 4.10.0-29-generic Reproducible also with the 4.4 kernel. [Description] When a guest memory backing stanza is defined using the <locked/> stanza + hugepages, as follows:   <memoryBacking>     <hugepages>       <page size='1' unit='GiB' nodeset='0'/>       <page size='1' unit='GiB' nodeset='1'/>     </hugepages>     <nosharedpages/>     <locked/>   </memoryBacking> (Full guest definition: http://paste.ubuntu.com/25229162/) The guest fails to start due to the following error: 2017-08-02 20:25:03.714+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10.12 (Christian Ehrhardt <christian.ehrhardt@canonical.com> Wed, 19 Jul 2017 08:28:14 +0200), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.14), hostname: buneary.seg LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name reproducer2 -S -machine pc-i440fx-2.5,accel=kvm,usb=off -cpu host -m 124928 -realtime mlock=on -smp 32,sockets=16,cores=1,threads=2 -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=64424509440,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0-15,memdev=ram-node0 -object memory-backend-file,id=ram-node1,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=66571993088,host-nodes=1,policy=bind -numa node,nodeid=1,cpus=16-31,memdev=ram-node1 -uuid 2460778d-979b-4024-9a13-0c3ca04b18ec -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-reproducer2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/uvtool/libvirt/images/test-ds.qcow,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on Domain id=14 is tainted: host-cpu char device redirected to /dev/pts/1 (label charserial0) mlockall: Cannot allocate memory 2017-08-02T20:25:37.732772Z qemu-system-x86_64: locking memory failed 2017-08-02 20:25:37.811+0000: shutting down This seems to be due to the setrlimit for RLIMIT_MEMLOCK is too low for mlockall to work given the large amount of memory. There is a libvirt upstream patch that enforces the existence of the hard_limit stanza when using with <locked/> in the memory backing settings. https://github.com/libvirt/libvirt/commit/c2e60ad0e5124482942164e5fec088157f5e716a Memory locking can only work properly if the memory locking limit for the QEMU process has been raised appropriately: the default one is extremely low, so there's no way the guest will fit in there. The commit https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b is also required when using hugepages and the locked stanza. [Test Case] * Define a guest that uses the following stanzas (See for a full guest reference: http://paste.ubuntu.com/25288141/) <memory unit='GiB'>120</memory> <currentMemory unit='GiB'>120</currentMemory> <memoryBacking> <hugepages> <page size='1' unit='GiB' nodeset='0'/> <page size='1' unit='GiB' nodeset='1'/> </hugepages> <nosharedpages/> <locked/> </memoryBacking> * virsh define guest.xml * virsh start guest.xml * Without the fix, the following error will be raised and the guest will not start. root@buneary:/home/ubuntu# virsh start reproducer2 error: Failed to start domain reproducer2 error: internal error: process exited while connecting to monitor: mlockall: Cannot allocate memory 2017-08-11T03:59:54.936275Z qemu-system-x86_64: locking memory failed * With the fix, the error shouldn't be displayed and the guest started [Suggested Fix] * https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b (Proposed) [Impact] * Guest definitions that uses locked memory + hugepages fail to spawn * Backport upstream fix to solve that issue [Environment] root@buneary:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.2 LTS Release: 16.04 Codename: xenial root@buneary:~# uname -r 4.10.0-29-generic Reproducible also with the 4.4 kernel. [Detailed Description] When a guest memory backing stanza is defined using the <locked/> stanza + hugepages, as follows:   <memoryBacking>     <hugepages>       <page size='1' unit='GiB' nodeset='0'/>       <page size='1' unit='GiB' nodeset='1'/>     </hugepages>     <nosharedpages/>     <locked/>   </memoryBacking> (Full guest definition: http://paste.ubuntu.com/25229162/) The guest fails to start due to the following error: 2017-08-02 20:25:03.714+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10.12 (Christian Ehrhardt <christian.ehrhardt@canonical.com> Wed, 19 Jul 2017 08:28:14 +0200), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.14), hostname: buneary.seg LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name reproducer2 -S -machine pc-i440fx-2.5,accel=kvm,usb=off -cpu host -m 124928 -realtime mlock=on -smp 32,sockets=16,cores=1,threads=2 -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=64424509440,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0-15,memdev=ram-node0 -object memory-backend-file,id=ram-node1,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=66571993088,host-nodes=1,policy=bind -numa node,nodeid=1,cpus=16-31,memdev=ram-node1 -uuid 2460778d-979b-4024-9a13-0c3ca04b18ec -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-reproducer2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/uvtool/libvirt/images/test-ds.qcow,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on Domain id=14 is tainted: host-cpu char device redirected to /dev/pts/1 (label charserial0) mlockall: Cannot allocate memory 2017-08-02T20:25:37.732772Z qemu-system-x86_64: locking memory failed 2017-08-02 20:25:37.811+0000: shutting down This seems to be due to the setrlimit for RLIMIT_MEMLOCK is too low for mlockall to work given the large amount of memory. There is a libvirt upstream patch that enforces the existence of the hard_limit stanza when using with <locked/> in the memory backing settings. https://github.com/libvirt/libvirt/commit/c2e60ad0e5124482942164e5fec088157f5e716a Memory locking can only work properly if the memory locking limit for the QEMU process has been raised appropriately: the default one is extremely low, so there's no way the guest will fit in there. The commit https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b is also required when using hugepages and the locked stanza. [Test Case]  * Define a guest that uses the following stanzas (See for a full guest reference: http://paste.ubuntu.com/25288141/)   <memory unit='GiB'>120</memory>   <currentMemory unit='GiB'>120</currentMemory>   <memoryBacking>     <hugepages>       <page size='1' unit='GiB' nodeset='0'/>       <page size='1' unit='GiB' nodeset='1'/>     </hugepages>     <nosharedpages/>     <locked/>   </memoryBacking> * virsh define guest.xml * virsh start guest.xml * Without the fix, the following error will be raised and the guest will not start. root@buneary:/home/ubuntu# virsh start reproducer2 error: Failed to start domain reproducer2 error: internal error: process exited while connecting to monitor: mlockall: Cannot allocate memory 2017-08-11T03:59:54.936275Z qemu-system-x86_64: locking memory failed * With the fix, the error shouldn't be displayed and the guest started [Suggested Fix] * https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b (Proposed) [Regression Potential] * There is one (theoretical) thing to think of - the change increases the lock resource limits for the spawned qemu. Maybe that could be used as an attack. Fortunately the defnition does have neither locked nor hugepages by default so opt-in, and while the guest can do all kind of things when exploited it can only hardly change it's (virtual) physical memory to increase what the host might be locking. Yet worth to mention IMHO * The general regression is rather low as I said "locked" is not default and the code path is only affecting domains configured that way. That makes the change rather safe for the overall user - and the one using locked likely need it.
2017-08-11 13:08:32 Corey Bryant bug task added cloud-archive
2017-08-11 13:09:10 Corey Bryant nominated for series cloud-archive/mitaka
2017-08-11 13:09:10 Corey Bryant bug task added cloud-archive/mitaka
2017-08-11 13:09:10 Corey Bryant nominated for series cloud-archive/ocata
2017-08-11 13:09:10 Corey Bryant bug task added cloud-archive/ocata
2017-08-11 13:09:59 Corey Bryant cloud-archive: status New Fix Committed
2017-08-11 13:10:03 Corey Bryant cloud-archive: importance Undecided Medium
2017-08-11 13:10:05 Corey Bryant cloud-archive/mitaka: importance Undecided Medium
2017-08-11 13:10:06 Corey Bryant cloud-archive/ocata: importance Undecided Medium
2017-08-11 13:10:17 Corey Bryant cloud-archive/ocata: assignee Jorge Niedbalski (niedbalski)
2017-08-11 13:10:25 Corey Bryant cloud-archive/mitaka: assignee Jorge Niedbalski (niedbalski)
2017-08-11 13:10:33 Corey Bryant cloud-archive/mitaka: status New Triaged
2017-08-11 13:10:36 Corey Bryant cloud-archive/ocata: status New Triaged
2017-08-11 13:37:07 Jorge Niedbalski attachment added fix-1708305-trusty-mitaka.debdiff https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1708305/+attachment/4930778/+files/fix-1708305-trusty-mitaka.debdiff
2017-08-11 13:37:24 Jorge Niedbalski cloud-archive/mitaka: status Triaged In Progress
2017-08-11 13:37:27 Jorge Niedbalski cloud-archive/ocata: status Triaged In Progress
2017-08-11 13:37:51 Jorge Niedbalski attachment added fix-1708305-ocata.debdiff https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1708305/+attachment/4930779/+files/fix-1708305-ocata.debdiff
2017-08-17 22:19:28 Brian Murray libvirt (Ubuntu Zesty): status In Progress Fix Committed
2017-08-17 22:19:31 Brian Murray bug added subscriber Ubuntu Stable Release Updates Team
2017-08-17 22:19:33 Brian Murray bug added subscriber SRU Verification
2017-08-17 22:19:38 Brian Murray tags canonical-bootstack libvirt-3.6 patch sts-sru-needed canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-needed verification-needed-zesty
2017-08-21 14:45:26 Edward Hope-Morley nominated for series cloud-archive/newton
2017-08-23 12:36:39 Chris J Arges libvirt (Ubuntu Xenial): status In Progress Fix Committed
2017-08-23 12:36:46 Chris J Arges tags canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-needed verification-needed-zesty canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-needed verification-needed-xenial verification-needed-zesty
2017-08-23 12:36:59 Chris J Arges removed subscriber Ubuntu Sponsors Team
2017-08-24 13:01:30 Jorge Niedbalski tags canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-needed verification-needed-xenial verification-needed-zesty canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-done-xenial verification-needed verification-needed-zesty
2017-08-24 13:17:54 Jorge Niedbalski tags canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-done-xenial verification-needed verification-needed-zesty canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-done verification-done-xenial verification-done-zesty
2017-08-24 13:22:48 Khawar Munir Abbasi bug added subscriber Khawar Munir Abbasi
2017-08-24 13:22:54 Khawar Munir Abbasi removed subscriber Khawar Munir Abbasi
2017-08-24 13:23:31 Khawar Munir Abbasi bug added subscriber Khawar Munir Abbasi
2017-08-24 13:23:32 Khawar Munir Abbasi removed subscriber Khawar Munir Abbasi
2017-08-24 13:52:22 Khawar Munir Abbasi bug added subscriber Khawar Munir Abbasi
2017-08-24 13:52:26 Khawar Munir Abbasi removed subscriber Khawar Munir Abbasi
2017-08-24 13:52:28 Khawar Munir Abbasi bug added subscriber Khawar Munir Abbasi
2017-08-24 13:52:45 Khawar Munir Abbasi removed subscriber Khawar Munir Abbasi
2017-08-24 13:53:23 Khawar Munir Abbasi bug added subscriber Khawar Munir Abbasi
2017-08-28 14:08:09 Łukasz Zemczak removed subscriber Ubuntu Stable Release Updates Team
2017-08-28 14:08:08 Launchpad Janitor libvirt (Ubuntu Zesty): status Fix Committed Fix Released
2017-08-31 13:04:15 Launchpad Janitor libvirt (Ubuntu Xenial): status Fix Committed Fix Released
2017-09-04 14:34:50 James Page cloud-archive/mitaka: status In Progress Fix Committed
2017-09-04 14:34:51 James Page tags canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-done verification-done-xenial verification-done-zesty canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-done verification-done-xenial verification-done-zesty verification-mitaka-needed
2017-09-04 14:37:18 James Page cloud-archive/ocata: status In Progress Fix Committed
2017-09-04 14:37:19 James Page tags canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-done verification-done-xenial verification-done-zesty verification-mitaka-needed canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-done verification-done-xenial verification-done-zesty verification-mitaka-needed verification-ocata-needed
2017-09-26 17:56:04 Jorge Niedbalski tags canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-done verification-done-xenial verification-done-zesty verification-mitaka-needed verification-ocata-needed canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-done verification-done-xenial verification-done-zesty verification-mitaka-done verification-ocata-done
2017-10-02 14:44:58 James Page cloud-archive/mitaka: status Fix Committed Fix Released
2017-10-02 14:46:04 James Page cloud-archive/ocata: status Fix Committed Fix Released
2017-12-04 14:59:17 Edward Hope-Morley cloud-archive: status Fix Committed Fix Released
2017-12-04 14:59:41 Edward Hope-Morley tags canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-done verification-done-xenial verification-done-zesty verification-mitaka-done verification-ocata-done canonical-bootstack libvirt-3.6 patch sts-sru-done verification-done verification-done-xenial verification-done-zesty verification-mitaka-done verification-ocata-done