lstopo command in nailgun-agent hangs when there are VMs booted with realtime scheduling.
nailgun-agent is being executed on every deployed Compute host every 30 seconds. Among the commands which are executed is the
'lstopo' command which is part of get_numa_topology function.
lstopo command will go one by one through all the existing CPUs of the System, including the ones allocated to tenant VMs by the vcpu_pin_set parameter in nova.conf, set it's affinity and bind itself to each Core.
The problem occurs when there are VMs with following extra_specs defined in the Nova flavor:
hw:cpu_realtime=yes
hw:cpu_realtime_mask=<vCPU number to set emulator pin>
As a result of those extra specs, following is defined for the VM in libvirt-xml
Example:
<vcpu placement='static'>8</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='10'/>
<vcpupin vcpu='1' cpuset='11'/>
<vcpupin vcpu='2' cpuset='12'/>
<vcpupin vcpu='3' cpuset='13'/>
<vcpupin vcpu='4' cpuset='14'/>
<vcpupin vcpu='5' cpuset='15'/>
<vcpupin vcpu='6' cpuset='16'/>
<vcpupin vcpu='7' cpuset='17'/>
<emulatorpin cpuset='10'/>
<vcpusched vcpus='1-7' scheduler='fifo' priority='1'/>
</cputune>
Due to vcpusched_vcpus parameter in libvirt, the tasks which the VM's GuestOS is executing will have a higher priority than the lstopo command which is trying to bind the pCPU belonging to that VM. Due to this, the command execution waits indefinitely. The result is that nailgun-agent is hanging on the Compute host as well, and the Compute is shown as Offline in 'fuel node'
printout.
Following is requested:
'lstopo' is being executed towards an already provisioned/deployed node. This is not needed at this point of time, as CPU topology will not change once the Compute is deployed. It should be enough to run it once, only during server discovery. Once the node is provisioned, lstopo is not needed any more. Same applies for other commands inside nailgun-agent, like 'lshw'.
MOS 9.2 build 596, November 2017
Fix proposed to branch: 9.0/mitaka
Change author: Oleksiy Molchanov <email address hidden>
Review: https:/