Large memory guests, "error: monitor socket did not show up: No such file or directory"
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ubuntu Cloud Archive |
Fix Released
|
Undecided
|
Unassigned | ||
Mitaka |
Fix Released
|
Medium
|
Unassigned | ||
Ocata |
Fix Released
|
Medium
|
Unassigned | ||
libvirt (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Xenial |
Fix Released
|
Undecided
|
Unassigned | ||
Yakkety |
Won't Fix
|
Undecided
|
Unassigned | ||
Zesty |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
[Description]
- Configured a machine with 32 static VCPUs, 160GB of RAM using 1G
hugepages on a NUMA capable machine.
Domain definition (http://
- Once started (virsh start).
Libvirt log.
LC_ALL=C PATH=/usr/
Then the following error is raised.
virsh start reproducer2
error: Failed to start domain reproducer2
error: monitor socket did not show up: No such file or directory
- The fix is done via backports, as a TL;DR the change does:
1. instead of sleeping too short (1ms) in a loop for very long start
small but exponentially increase for the few cases that need long.
That way fast actions are done fast, but long actions are no cpu-hogs
2. huge guests get ~1s per 1Gb extra timeout to come up, that allows
huge guests to initialize properly.
[Impact]
* Cannot start virtual machines with large pools of memory allocated
on NUMA nodes.
[Test Case]
* this is a tradeoff of memory clearing speed vs guest size.
Once the clearing of guest memory exceeds ~30 seconds the issue will
trigger.
* Guest must be backed by huge pages as otherwise the kernel will fault
in on demand instead of needing the initial clear.
* One way to "slow down" is to Configure a Machine with multiple NUMA
nodes.
root@
1048576KiB: 60
root@
1048576KiB: 62
* Another one to slow down the init is to just use a really heg guest. In
the example 122G guest was enough. (full guest definition:
http://
<memory unit='GiB'
<currentMemory unit='GiB'
<memoryBacking>
<hugepages>
<page size='1' unit='GiB' nodeset='0'/>
<page size='1' unit='GiB' nodeset='1'/>
</hugepages>
</memoryBacking>
<cpu mode='host-
<topology sockets='16' cores='1' threads='2'/>
<numa>
<cell id='0' cpus='0-15' memory='60' unit='GiB' memAccess=
<cell id='1' cpus='16-31' memory='62' unit='GiB' memAccess=
</numa>
</cpu>
* Define the guest, and try to start it.
$ virsh define reproducer.xml
$ virsh start reproducer
* Verify that the following error is raised:
root@buneary:
error: Failed to start domain reproducer2
error: monitor socket did not show up: No such file or directory
[Expected Behavior]
* Machine is started without issues as displayed https:/
[Regression Potential]
* The behavior on timeouts around starting a guest changed. We backported
the fix along with a fix to that new behavior (where guests seemed to
wait forever due to the exponential wait).
Still the "allowed" wait time is increased, but users might expect it
instantly as they are used from their laptop.
Now if one starts a 1TB guest the allowed time is base+1000s.
A user might think a while it is broken or hanging, but there is no way
to avoid that.
OTOH before the fix it would have failed to start after 30 seconds so
not really a regression IMHO.
[Other Info]
description: | updated |
tags: | added: sts-sru-needed |
description: | updated |
tags: | added: sts |
Referred change is upstream in Libvirt 3.2, so assuming that is fixed.
But check on X-Z SRUs