nova allows suboptimal emulator tread pinning for realtime guests

[sean mooney]

Today when ever you use a realtime guest you are required to enable cpu pinning and other feature such as specifying a real time core mask via hw:cpu_realtime_mask or hw_cpu_realtime_mask.

In the Victoria release this requirement was relaxed somewhat with the introduction of mixed cpu policy guest that are assigned pinned and floating cores.

https://github.com/openstack/nova/commit/9fc63c764429c10f9041e6b53659e0cbd595bf6b

It is now possible to allocate all cores in an instance to realtime and
omit the ``hw:cpu_realtime_mask`` extra spec. This requires specifying the
``hw:emulator_threads_policy`` extra spec.

https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/releasenotes/notes/bug-1884231-16acf297d88b122e.yaml

However while that works well it also possible to hw:cpu_realtime_mask but not specify hw:emulator_threads_policy which leads to sub optimal xml generation for the libvirt driver.

This is reported downstream as https://bugzilla.redhat.com/show_bug.cgi?id=1700390 for older releases that predate the changes referenced above.

Though in revaluation of this a possible improvement can be made as detailed in https://bugzilla.redhat.com/show_bug.cgi?id=1700390#c11

Today if we have a 2 core V where guest cpu 0 is non realtime and guest cpu 1 is realtime we
.e.g. hw:cpu_policy=dedicated hw:cpu_realtime=True hw:cpu_realtime_mask=^0
would generate the xml as follows
  <vcpupin vcpu="0" cpuset="0"/>
  <vcpupin vcpu="1" cpuset="1"/>
  <vcpusched vcpus='1' scheduler='fifo' priority='1'/>
  <emulatorpin cpuset="0,1"/>

This is because the default behavior when no emulator_threads_policy is specifed is for the emulator thread to float over all the vm cores.

But a slight modification to the XML could be made to have a more optimal default in this case
useing the cpu_realtime_mask we can instead restrict the emulator thread to float over the non realtime cores with realtime priortiy.

  <vcpupin vcpu="0" cpuset="0"/>
  <vcpupin vcpu="1" cpuset="1"/>
  <vcpusched vcpus='1' scheduler='fifo' priority='1'/>
  <emulatorpin cpuset="0"/>
  <emulatorsched scheduler='fifo' priority='1'>

This will ensure that if QEMU need to process a request for a device attach for example
that the emulator thread has higher priority then the guest vCPUs that deal with guest house keeping task but will not interrupt the realtime cores.

This would give many of the benefits of emulator_threads_policy=share or emulator_threads_policy=isolate without increase resource usage or requiring any config,flavor or image changes. this should also be a backportable solution to this problem.

This is especially important given realtime hosts often are deployed with the kernel 'isolcpus' parameter which mean that the kernel will not load balance the emulator thread across the range and will instead leave it on the core it initially spawned on. today you could get lucky and it could be spawn on core 0 in which case the new behavior would be the same or it could get spawned on core 1. When the emulator thread is spawned on core 1 since it has less priority then the vcpu thread it will only run if the guest vCPU idles resulting in the inability for QEMU to process device attach and other QEMU monitor commands form libvirt or the user.

Workaround
----------

If you set the flavor property `hw:cpu_realtime`, ensure to also set:

(1) `hw:emulator_thread_policy=isolate` or `hw:emulator_thread_policy=share`; and

(2) set `cpu_shared_set` (this defines which physical CPUs will be used for best-effort guest vCPU resources) in `nova.conf`

[sean mooney (sean-k-mooney)]

note this is a wishlist bug since you can and should just use emulator_threads_policy today but we are not doing a simple and obvios optimaiation that would greatly imporve the UX of realtime guests.