2017-08-02 22:20:10 |
Jorge Niedbalski |
bug |
|
|
added bug |
2017-08-02 22:20:24 |
Jorge Niedbalski |
nominated for series |
|
Ubuntu Artful |
|
2017-08-02 22:20:24 |
Jorge Niedbalski |
nominated for series |
|
Ubuntu Xenial |
|
2017-08-02 22:20:24 |
Jorge Niedbalski |
nominated for series |
|
Ubuntu Zesty |
|
2017-08-03 09:02:05 |
Launchpad Janitor |
libvirt (Ubuntu): status |
New |
Confirmed |
|
2017-08-03 09:03:03 |
Peter Sabaini |
tags |
|
canonical-bootstack |
|
2017-08-03 09:05:02 |
Dominique Poulain |
bug |
|
|
added subscriber Dominique Poulain |
2017-08-03 17:04:15 |
Eric Desrochers |
bug task added |
|
libvirt (Ubuntu Artful) |
|
2017-08-03 17:04:20 |
Eric Desrochers |
bug task added |
|
libvirt (Ubuntu Xenial) |
|
2017-08-03 17:04:27 |
Eric Desrochers |
bug task added |
|
libvirt (Ubuntu Zesty) |
|
2017-08-07 10:09:58 |
Christian Ehrhardt |
bug |
|
|
added subscriber ChristianEhrhardt |
2017-08-07 10:11:07 |
Christian Ehrhardt |
tags |
canonical-bootstack |
canonical-bootstack libvirt-3.6 |
|
2017-08-07 10:24:41 |
Christian Ehrhardt |
libvirt (Ubuntu Artful): importance |
Undecided |
Low |
|
2017-08-10 21:22:52 |
Jorge Niedbalski |
libvirt (Ubuntu Artful): status |
Confirmed |
In Progress |
|
2017-08-10 21:22:57 |
Jorge Niedbalski |
libvirt (Ubuntu Artful): importance |
Low |
High |
|
2017-08-10 21:23:04 |
Jorge Niedbalski |
libvirt (Ubuntu Artful): importance |
High |
Medium |
|
2017-08-10 21:23:09 |
Jorge Niedbalski |
libvirt (Ubuntu Artful): assignee |
|
Jorge Niedbalski (niedbalski) |
|
2017-08-10 21:24:02 |
Jorge Niedbalski |
description |
[Environment]
root@buneary:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial
root@buneary:~# uname -r
4.10.0-29-generic
Reproducible also with the 4.4 kernel.
[Description]
When a guest memory backing stanza is defined using the <locked/> stanza + hugepages,
as follows:
<memoryBacking>
<hugepages>
<page size='1' unit='GiB' nodeset='0'/>
<page size='1' unit='GiB' nodeset='1'/>
</hugepages>
<nosharedpages/>
<locked/>
</memoryBacking>
(Full guest definition: http://paste.ubuntu.com/25229162/)
The guest fails to start due to the following error:
2017-08-02 20:25:03.714+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10.12 (Christian Ehrhardt <christian.ehrhardt@canonical.com> Wed, 19 Jul 2017 08:28:14 +0200), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.14), hostname: buneary.seg
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name reproducer2 -S -machine pc-i440fx-2.5,accel=kvm,usb=off -cpu host -m 124928 -realtime mlock=on -smp 32,sockets=16,cores=1,threads=2 -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=64424509440,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0-15,memdev=ram-node0 -object memory-backend-file,id=ram-node1,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=66571993088,host-nodes=1,policy=bind -numa node,nodeid=1,cpus=16-31,memdev=ram-node1 -uuid 2460778d-979b-4024-9a13-0c3ca04b18ec -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-reproducer2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/uvtool/libvirt/images/test-ds.qcow,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on
Domain id=14 is tainted: host-cpu
char device redirected to /dev/pts/1 (label charserial0)
mlockall: Cannot allocate memory
2017-08-02T20:25:37.732772Z qemu-system-x86_64: locking memory failed
2017-08-02 20:25:37.811+0000: shutting down
This seems to be due to the setrlimit for RLIMIT_MEMLOCK is too low for mlockall
to work given the large amount of memory.
There is a libvirt upstream patch that enforces the existence of the
hard_limit stanza when using with <locked/> in the memory backing settings.
https://github.com/libvirt/libvirt/commit/c2e60ad0e5124482942164e5fec088157f5e716a
Memory locking can only work properly if the memory locking limit
for the QEMU process has been raised appropriately: the default one
is extremely low, so there's no way the guest will fit in there.
[Suggested Fix]
* https://github.com/libvirt/libvirt/commit/c2e60ad0e5124482942164e5fec088157f5e716a |
[Environment]
root@buneary:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial
root@buneary:~# uname -r
4.10.0-29-generic
Reproducible also with the 4.4 kernel.
[Description]
When a guest memory backing stanza is defined using the <locked/> stanza + hugepages,
as follows:
<memoryBacking>
<hugepages>
<page size='1' unit='GiB' nodeset='0'/>
<page size='1' unit='GiB' nodeset='1'/>
</hugepages>
<nosharedpages/>
<locked/>
</memoryBacking>
(Full guest definition: http://paste.ubuntu.com/25229162/)
The guest fails to start due to the following error:
2017-08-02 20:25:03.714+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10.12 (Christian Ehrhardt <christian.ehrhardt@canonical.com> Wed, 19 Jul 2017 08:28:14 +0200), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.14), hostname: buneary.seg
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name reproducer2 -S -machine pc-i440fx-2.5,accel=kvm,usb=off -cpu host -m 124928 -realtime mlock=on -smp 32,sockets=16,cores=1,threads=2 -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=64424509440,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0-15,memdev=ram-node0 -object memory-backend-file,id=ram-node1,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=66571993088,host-nodes=1,policy=bind -numa node,nodeid=1,cpus=16-31,memdev=ram-node1 -uuid 2460778d-979b-4024-9a13-0c3ca04b18ec -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-reproducer2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/uvtool/libvirt/images/test-ds.qcow,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on
Domain id=14 is tainted: host-cpu
char device redirected to /dev/pts/1 (label charserial0)
mlockall: Cannot allocate memory
2017-08-02T20:25:37.732772Z qemu-system-x86_64: locking memory failed
2017-08-02 20:25:37.811+0000: shutting down
This seems to be due to the setrlimit for RLIMIT_MEMLOCK is too low for mlockall
to work given the large amount of memory.
There is a libvirt upstream patch that enforces the existence of the
hard_limit stanza when using with <locked/> in the memory backing settings.
https://github.com/libvirt/libvirt/commit/c2e60ad0e5124482942164e5fec088157f5e716a
Memory locking can only work properly if the memory locking limit
for the QEMU process has been raised appropriately: the default one
is extremely low, so there's no way the guest will fit in there.
The commit https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b is also required when using hugepages and the locked stanza.
[Suggested Fix]
* https://github.com/libvirt/libvirt/commit/c2e60ad0e5124482942164e5fec088157f5e716a
* https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b |
|
2017-08-10 21:40:55 |
Jorge Niedbalski |
libvirt (Ubuntu Artful): assignee |
Jorge Niedbalski (niedbalski) |
|
|
2017-08-10 21:50:11 |
Jorge Niedbalski |
libvirt (Ubuntu Artful): status |
In Progress |
Confirmed |
|
2017-08-11 02:37:17 |
Jorge Niedbalski |
libvirt (Ubuntu Xenial): status |
New |
In Progress |
|
2017-08-11 02:37:21 |
Jorge Niedbalski |
libvirt (Ubuntu Zesty): status |
New |
In Progress |
|
2017-08-11 02:37:32 |
Jorge Niedbalski |
libvirt (Ubuntu Xenial): importance |
Undecided |
High |
|
2017-08-11 02:37:35 |
Jorge Niedbalski |
libvirt (Ubuntu Xenial): importance |
High |
Medium |
|
2017-08-11 02:37:38 |
Jorge Niedbalski |
libvirt (Ubuntu Zesty): importance |
Undecided |
Medium |
|
2017-08-11 02:37:51 |
Jorge Niedbalski |
libvirt (Ubuntu Xenial): assignee |
|
Jorge Niedbalski (niedbalski) |
|
2017-08-11 02:37:54 |
Jorge Niedbalski |
libvirt (Ubuntu Zesty): assignee |
|
Jorge Niedbalski (niedbalski) |
|
2017-08-11 04:18:16 |
Jorge Niedbalski |
libvirt (Ubuntu Artful): status |
Confirmed |
Fix Released |
|
2017-08-11 04:18:22 |
Jorge Niedbalski |
libvirt (Ubuntu Artful): status |
Fix Released |
Fix Committed |
|
2017-08-11 04:19:03 |
Jorge Niedbalski |
attachment added |
|
fix-1708305-xenial.debdiff https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1708305/+attachment/4930463/+files/fix-1708305-xenial.debdiff |
|
2017-08-11 04:21:23 |
Ubuntu Foundations Team Bug Bot |
tags |
canonical-bootstack libvirt-3.6 |
canonical-bootstack libvirt-3.6 patch |
|
2017-08-11 04:21:30 |
Ubuntu Foundations Team Bug Bot |
bug |
|
|
added subscriber Ubuntu Sponsors Team |
2017-08-11 04:32:24 |
Jorge Niedbalski |
description |
[Environment]
root@buneary:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial
root@buneary:~# uname -r
4.10.0-29-generic
Reproducible also with the 4.4 kernel.
[Description]
When a guest memory backing stanza is defined using the <locked/> stanza + hugepages,
as follows:
<memoryBacking>
<hugepages>
<page size='1' unit='GiB' nodeset='0'/>
<page size='1' unit='GiB' nodeset='1'/>
</hugepages>
<nosharedpages/>
<locked/>
</memoryBacking>
(Full guest definition: http://paste.ubuntu.com/25229162/)
The guest fails to start due to the following error:
2017-08-02 20:25:03.714+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10.12 (Christian Ehrhardt <christian.ehrhardt@canonical.com> Wed, 19 Jul 2017 08:28:14 +0200), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.14), hostname: buneary.seg
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name reproducer2 -S -machine pc-i440fx-2.5,accel=kvm,usb=off -cpu host -m 124928 -realtime mlock=on -smp 32,sockets=16,cores=1,threads=2 -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=64424509440,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0-15,memdev=ram-node0 -object memory-backend-file,id=ram-node1,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=66571993088,host-nodes=1,policy=bind -numa node,nodeid=1,cpus=16-31,memdev=ram-node1 -uuid 2460778d-979b-4024-9a13-0c3ca04b18ec -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-reproducer2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/uvtool/libvirt/images/test-ds.qcow,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on
Domain id=14 is tainted: host-cpu
char device redirected to /dev/pts/1 (label charserial0)
mlockall: Cannot allocate memory
2017-08-02T20:25:37.732772Z qemu-system-x86_64: locking memory failed
2017-08-02 20:25:37.811+0000: shutting down
This seems to be due to the setrlimit for RLIMIT_MEMLOCK is too low for mlockall
to work given the large amount of memory.
There is a libvirt upstream patch that enforces the existence of the
hard_limit stanza when using with <locked/> in the memory backing settings.
https://github.com/libvirt/libvirt/commit/c2e60ad0e5124482942164e5fec088157f5e716a
Memory locking can only work properly if the memory locking limit
for the QEMU process has been raised appropriately: the default one
is extremely low, so there's no way the guest will fit in there.
The commit https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b is also required when using hugepages and the locked stanza.
[Suggested Fix]
* https://github.com/libvirt/libvirt/commit/c2e60ad0e5124482942164e5fec088157f5e716a
* https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b |
[Environment]
root@buneary:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial
root@buneary:~# uname -r
4.10.0-29-generic
Reproducible also with the 4.4 kernel.
[Description]
When a guest memory backing stanza is defined using the <locked/> stanza + hugepages,
as follows:
<memoryBacking>
<hugepages>
<page size='1' unit='GiB' nodeset='0'/>
<page size='1' unit='GiB' nodeset='1'/>
</hugepages>
<nosharedpages/>
<locked/>
</memoryBacking>
(Full guest definition: http://paste.ubuntu.com/25229162/)
The guest fails to start due to the following error:
2017-08-02 20:25:03.714+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10.12 (Christian Ehrhardt <christian.ehrhardt@canonical.com> Wed, 19 Jul 2017 08:28:14 +0200), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.14), hostname: buneary.seg
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name reproducer2 -S -machine pc-i440fx-2.5,accel=kvm,usb=off -cpu host -m 124928 -realtime mlock=on -smp 32,sockets=16,cores=1,threads=2 -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=64424509440,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0-15,memdev=ram-node0 -object memory-backend-file,id=ram-node1,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=66571993088,host-nodes=1,policy=bind -numa node,nodeid=1,cpus=16-31,memdev=ram-node1 -uuid 2460778d-979b-4024-9a13-0c3ca04b18ec -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-reproducer2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/uvtool/libvirt/images/test-ds.qcow,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on
Domain id=14 is tainted: host-cpu
char device redirected to /dev/pts/1 (label charserial0)
mlockall: Cannot allocate memory
2017-08-02T20:25:37.732772Z qemu-system-x86_64: locking memory failed
2017-08-02 20:25:37.811+0000: shutting down
This seems to be due to the setrlimit for RLIMIT_MEMLOCK is too low for mlockall
to work given the large amount of memory.
There is a libvirt upstream patch that enforces the existence of the
hard_limit stanza when using with <locked/> in the memory backing settings.
https://github.com/libvirt/libvirt/commit/c2e60ad0e5124482942164e5fec088157f5e716a
Memory locking can only work properly if the memory locking limit
for the QEMU process has been raised appropriately: the default one
is extremely low, so there's no way the guest will fit in there.
The commit https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b is also required when using hugepages and the locked stanza.
[Test Case]
* Define a guest that uses the following stanzas (See for a full guest reference: http://paste.ubuntu.com/25288141/)
<memory unit='GiB'>120</memory>
<currentMemory unit='GiB'>120</currentMemory>
<memoryBacking>
<hugepages>
<page size='1' unit='GiB' nodeset='0'/>
<page size='1' unit='GiB' nodeset='1'/>
</hugepages>
<nosharedpages/>
<locked/>
</memoryBacking>
* virsh define guest.xml
* virsh start guest.xml
* Without the fix, the following error will be raised and the guest
will not start.
root@buneary:/home/ubuntu# virsh start reproducer2
error: Failed to start domain reproducer2
error: internal error: process exited while connecting to monitor: mlockall: Cannot allocate memory
2017-08-11T03:59:54.936275Z qemu-system-x86_64: locking memory failed
* With the fix, the error shouldn't be displayed and the guest started
[Suggested Fix]
* https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b (Proposed) |
|
2017-08-11 04:32:34 |
Jorge Niedbalski |
tags |
canonical-bootstack libvirt-3.6 patch |
canonical-bootstack libvirt-3.6 patch sts-sru-needed |
|
2017-08-11 04:36:35 |
Jorge Niedbalski |
attachment added |
|
fix-1708305-zesty.debdiff https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1708305/+attachment/4930472/+files/fix-1708305-zesty.debdiff |
|
2017-08-11 05:50:02 |
Christian Ehrhardt |
libvirt (Ubuntu Artful): status |
Fix Committed |
Fix Released |
|
2017-08-11 06:03:23 |
Christian Ehrhardt |
description |
[Environment]
root@buneary:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial
root@buneary:~# uname -r
4.10.0-29-generic
Reproducible also with the 4.4 kernel.
[Description]
When a guest memory backing stanza is defined using the <locked/> stanza + hugepages,
as follows:
<memoryBacking>
<hugepages>
<page size='1' unit='GiB' nodeset='0'/>
<page size='1' unit='GiB' nodeset='1'/>
</hugepages>
<nosharedpages/>
<locked/>
</memoryBacking>
(Full guest definition: http://paste.ubuntu.com/25229162/)
The guest fails to start due to the following error:
2017-08-02 20:25:03.714+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10.12 (Christian Ehrhardt <christian.ehrhardt@canonical.com> Wed, 19 Jul 2017 08:28:14 +0200), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.14), hostname: buneary.seg
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name reproducer2 -S -machine pc-i440fx-2.5,accel=kvm,usb=off -cpu host -m 124928 -realtime mlock=on -smp 32,sockets=16,cores=1,threads=2 -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=64424509440,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0-15,memdev=ram-node0 -object memory-backend-file,id=ram-node1,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=66571993088,host-nodes=1,policy=bind -numa node,nodeid=1,cpus=16-31,memdev=ram-node1 -uuid 2460778d-979b-4024-9a13-0c3ca04b18ec -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-reproducer2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/uvtool/libvirt/images/test-ds.qcow,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on
Domain id=14 is tainted: host-cpu
char device redirected to /dev/pts/1 (label charserial0)
mlockall: Cannot allocate memory
2017-08-02T20:25:37.732772Z qemu-system-x86_64: locking memory failed
2017-08-02 20:25:37.811+0000: shutting down
This seems to be due to the setrlimit for RLIMIT_MEMLOCK is too low for mlockall
to work given the large amount of memory.
There is a libvirt upstream patch that enforces the existence of the
hard_limit stanza when using with <locked/> in the memory backing settings.
https://github.com/libvirt/libvirt/commit/c2e60ad0e5124482942164e5fec088157f5e716a
Memory locking can only work properly if the memory locking limit
for the QEMU process has been raised appropriately: the default one
is extremely low, so there's no way the guest will fit in there.
The commit https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b is also required when using hugepages and the locked stanza.
[Test Case]
* Define a guest that uses the following stanzas (See for a full guest reference: http://paste.ubuntu.com/25288141/)
<memory unit='GiB'>120</memory>
<currentMemory unit='GiB'>120</currentMemory>
<memoryBacking>
<hugepages>
<page size='1' unit='GiB' nodeset='0'/>
<page size='1' unit='GiB' nodeset='1'/>
</hugepages>
<nosharedpages/>
<locked/>
</memoryBacking>
* virsh define guest.xml
* virsh start guest.xml
* Without the fix, the following error will be raised and the guest
will not start.
root@buneary:/home/ubuntu# virsh start reproducer2
error: Failed to start domain reproducer2
error: internal error: process exited while connecting to monitor: mlockall: Cannot allocate memory
2017-08-11T03:59:54.936275Z qemu-system-x86_64: locking memory failed
* With the fix, the error shouldn't be displayed and the guest started
[Suggested Fix]
* https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b (Proposed) |
[Impact]
* Guest definitions that uses locked memory + hugepages fail to spawn
* Backport upstream fix to solve that issue
[Environment]
root@buneary:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial
root@buneary:~# uname -r
4.10.0-29-generic
Reproducible also with the 4.4 kernel.
[Detailed Description]
When a guest memory backing stanza is defined using the <locked/> stanza + hugepages,
as follows:
<memoryBacking>
<hugepages>
<page size='1' unit='GiB' nodeset='0'/>
<page size='1' unit='GiB' nodeset='1'/>
</hugepages>
<nosharedpages/>
<locked/>
</memoryBacking>
(Full guest definition: http://paste.ubuntu.com/25229162/)
The guest fails to start due to the following error:
2017-08-02 20:25:03.714+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10.12 (Christian Ehrhardt <christian.ehrhardt@canonical.com> Wed, 19 Jul 2017 08:28:14 +0200), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.14), hostname: buneary.seg
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name reproducer2 -S -machine pc-i440fx-2.5,accel=kvm,usb=off -cpu host -m 124928 -realtime mlock=on -smp 32,sockets=16,cores=1,threads=2 -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=64424509440,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0-15,memdev=ram-node0 -object memory-backend-file,id=ram-node1,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=yes,size=66571993088,host-nodes=1,policy=bind -numa node,nodeid=1,cpus=16-31,memdev=ram-node1 -uuid 2460778d-979b-4024-9a13-0c3ca04b18ec -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-reproducer2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/uvtool/libvirt/images/test-ds.qcow,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on
Domain id=14 is tainted: host-cpu
char device redirected to /dev/pts/1 (label charserial0)
mlockall: Cannot allocate memory
2017-08-02T20:25:37.732772Z qemu-system-x86_64: locking memory failed
2017-08-02 20:25:37.811+0000: shutting down
This seems to be due to the setrlimit for RLIMIT_MEMLOCK is too low for mlockall
to work given the large amount of memory.
There is a libvirt upstream patch that enforces the existence of the
hard_limit stanza when using with <locked/> in the memory backing settings.
https://github.com/libvirt/libvirt/commit/c2e60ad0e5124482942164e5fec088157f5e716a
Memory locking can only work properly if the memory locking limit
for the QEMU process has been raised appropriately: the default one
is extremely low, so there's no way the guest will fit in there.
The commit https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b is also required when using hugepages and the locked stanza.
[Test Case]
* Define a guest that uses the following stanzas (See for a full guest reference: http://paste.ubuntu.com/25288141/)
<memory unit='GiB'>120</memory>
<currentMemory unit='GiB'>120</currentMemory>
<memoryBacking>
<hugepages>
<page size='1' unit='GiB' nodeset='0'/>
<page size='1' unit='GiB' nodeset='1'/>
</hugepages>
<nosharedpages/>
<locked/>
</memoryBacking>
* virsh define guest.xml
* virsh start guest.xml
* Without the fix, the following error will be raised and the guest
will not start.
root@buneary:/home/ubuntu# virsh start reproducer2
error: Failed to start domain reproducer2
error: internal error: process exited while connecting to monitor: mlockall: Cannot allocate memory
2017-08-11T03:59:54.936275Z qemu-system-x86_64: locking memory failed
* With the fix, the error shouldn't be displayed and the guest started
[Suggested Fix]
* https://github.com/libvirt/libvirt/commit/7e667664d28f90bf6916604a55ebad7e2d85305b (Proposed)
[Regression Potential]
* There is one (theoretical) thing to think of - the change increases
the lock resource limits for the spawned qemu. Maybe that could be
used as an attack. Fortunately the defnition does have neither
locked nor hugepages by default so opt-in, and while the guest can
do all kind of things when exploited it can only hardly change it's
(virtual) physical memory to increase what the host might be locking.
Yet worth to mention IMHO
* The general regression is rather low as I said "locked" is not
default and the code path is only affecting domains configured that
way. That makes the change rather safe for the overall user - and the
one using locked likely need it. |
|
2017-08-11 13:08:32 |
Corey Bryant |
bug task added |
|
cloud-archive |
|
2017-08-11 13:09:10 |
Corey Bryant |
nominated for series |
|
cloud-archive/mitaka |
|
2017-08-11 13:09:10 |
Corey Bryant |
bug task added |
|
cloud-archive/mitaka |
|
2017-08-11 13:09:10 |
Corey Bryant |
nominated for series |
|
cloud-archive/ocata |
|
2017-08-11 13:09:10 |
Corey Bryant |
bug task added |
|
cloud-archive/ocata |
|
2017-08-11 13:09:59 |
Corey Bryant |
cloud-archive: status |
New |
Fix Committed |
|
2017-08-11 13:10:03 |
Corey Bryant |
cloud-archive: importance |
Undecided |
Medium |
|
2017-08-11 13:10:05 |
Corey Bryant |
cloud-archive/mitaka: importance |
Undecided |
Medium |
|
2017-08-11 13:10:06 |
Corey Bryant |
cloud-archive/ocata: importance |
Undecided |
Medium |
|
2017-08-11 13:10:17 |
Corey Bryant |
cloud-archive/ocata: assignee |
|
Jorge Niedbalski (niedbalski) |
|
2017-08-11 13:10:25 |
Corey Bryant |
cloud-archive/mitaka: assignee |
|
Jorge Niedbalski (niedbalski) |
|
2017-08-11 13:10:33 |
Corey Bryant |
cloud-archive/mitaka: status |
New |
Triaged |
|
2017-08-11 13:10:36 |
Corey Bryant |
cloud-archive/ocata: status |
New |
Triaged |
|
2017-08-11 13:37:07 |
Jorge Niedbalski |
attachment added |
|
fix-1708305-trusty-mitaka.debdiff https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1708305/+attachment/4930778/+files/fix-1708305-trusty-mitaka.debdiff |
|
2017-08-11 13:37:24 |
Jorge Niedbalski |
cloud-archive/mitaka: status |
Triaged |
In Progress |
|
2017-08-11 13:37:27 |
Jorge Niedbalski |
cloud-archive/ocata: status |
Triaged |
In Progress |
|
2017-08-11 13:37:51 |
Jorge Niedbalski |
attachment added |
|
fix-1708305-ocata.debdiff https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1708305/+attachment/4930779/+files/fix-1708305-ocata.debdiff |
|
2017-08-17 22:19:28 |
Brian Murray |
libvirt (Ubuntu Zesty): status |
In Progress |
Fix Committed |
|
2017-08-17 22:19:31 |
Brian Murray |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
2017-08-17 22:19:33 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
2017-08-17 22:19:38 |
Brian Murray |
tags |
canonical-bootstack libvirt-3.6 patch sts-sru-needed |
canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-needed verification-needed-zesty |
|
2017-08-21 14:45:26 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/newton |
|
2017-08-23 12:36:39 |
Chris J Arges |
libvirt (Ubuntu Xenial): status |
In Progress |
Fix Committed |
|
2017-08-23 12:36:46 |
Chris J Arges |
tags |
canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-needed verification-needed-zesty |
canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-needed verification-needed-xenial verification-needed-zesty |
|
2017-08-23 12:36:59 |
Chris J Arges |
removed subscriber Ubuntu Sponsors Team |
|
|
|
2017-08-24 13:01:30 |
Jorge Niedbalski |
tags |
canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-needed verification-needed-xenial verification-needed-zesty |
canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-done-xenial verification-needed verification-needed-zesty |
|
2017-08-24 13:17:54 |
Jorge Niedbalski |
tags |
canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-done-xenial verification-needed verification-needed-zesty |
canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-done verification-done-xenial verification-done-zesty |
|
2017-08-24 13:22:48 |
Khawar Munir Abbasi |
bug |
|
|
added subscriber Khawar Munir Abbasi |
2017-08-24 13:22:54 |
Khawar Munir Abbasi |
removed subscriber Khawar Munir Abbasi |
|
|
|
2017-08-24 13:23:31 |
Khawar Munir Abbasi |
bug |
|
|
added subscriber Khawar Munir Abbasi |
2017-08-24 13:23:32 |
Khawar Munir Abbasi |
removed subscriber Khawar Munir Abbasi |
|
|
|
2017-08-24 13:52:22 |
Khawar Munir Abbasi |
bug |
|
|
added subscriber Khawar Munir Abbasi |
2017-08-24 13:52:26 |
Khawar Munir Abbasi |
removed subscriber Khawar Munir Abbasi |
|
|
|
2017-08-24 13:52:28 |
Khawar Munir Abbasi |
bug |
|
|
added subscriber Khawar Munir Abbasi |
2017-08-24 13:52:45 |
Khawar Munir Abbasi |
removed subscriber Khawar Munir Abbasi |
|
|
|
2017-08-24 13:53:23 |
Khawar Munir Abbasi |
bug |
|
|
added subscriber Khawar Munir Abbasi |
2017-08-28 14:08:09 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
2017-08-28 14:08:08 |
Launchpad Janitor |
libvirt (Ubuntu Zesty): status |
Fix Committed |
Fix Released |
|
2017-08-31 13:04:15 |
Launchpad Janitor |
libvirt (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
2017-09-04 14:34:50 |
James Page |
cloud-archive/mitaka: status |
In Progress |
Fix Committed |
|
2017-09-04 14:34:51 |
James Page |
tags |
canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-done verification-done-xenial verification-done-zesty |
canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-done verification-done-xenial verification-done-zesty verification-mitaka-needed |
|
2017-09-04 14:37:18 |
James Page |
cloud-archive/ocata: status |
In Progress |
Fix Committed |
|
2017-09-04 14:37:19 |
James Page |
tags |
canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-done verification-done-xenial verification-done-zesty verification-mitaka-needed |
canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-done verification-done-xenial verification-done-zesty verification-mitaka-needed verification-ocata-needed |
|
2017-09-26 17:56:04 |
Jorge Niedbalski |
tags |
canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-done verification-done-xenial verification-done-zesty verification-mitaka-needed verification-ocata-needed |
canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-done verification-done-xenial verification-done-zesty verification-mitaka-done verification-ocata-done |
|
2017-10-02 14:44:58 |
James Page |
cloud-archive/mitaka: status |
Fix Committed |
Fix Released |
|
2017-10-02 14:46:04 |
James Page |
cloud-archive/ocata: status |
Fix Committed |
Fix Released |
|
2017-12-04 14:59:17 |
Edward Hope-Morley |
cloud-archive: status |
Fix Committed |
Fix Released |
|
2017-12-04 14:59:41 |
Edward Hope-Morley |
tags |
canonical-bootstack libvirt-3.6 patch sts-sru-needed verification-done verification-done-xenial verification-done-zesty verification-mitaka-done verification-ocata-done |
canonical-bootstack libvirt-3.6 patch sts-sru-done verification-done verification-done-xenial verification-done-zesty verification-mitaka-done verification-ocata-done |
|