Recent vCPU features are disabled

[Alexey Stupnikov]

The fix for bug #1618473 introduced a number of performance issues for MOS end users, since recent x86 extensions are no longer available for MOS compute's vCPUs. Here is the most important ones:
  - aes
  - pclmuldq
  - x2apic
  - avx (NOT avx2)
  - ssse3, sse4_1, sse4_2

This problem can be critical for micro and nano instances and will affect customer experience (there are a number of complaints in the web about cloud providers failing to provide some CPU instruction set extensions).

It is also worth mentioning that there is Ubuntu bug #1524069 with the same complaints.

[Alexei Sheplyakov (asheplyakov)]

> - aes
> - pclmuldq
> - x2apic
> - avx (NOT avx2)
> - ssse3, sse4_1, sse4_2

Attempt to use these instructions in the guest will fail if the hypervisor is not aware how to enable/handle them properly. Therefore the hypervisor (the host kernel kvm module) is the best source of knowledge about which CPU flags should be advertised to the guest.

For example, (Intel) CPU executes AVX instructions only if XCR0.AVX bit is set,
otherwise it raises #GP exception, which eventually translates either to SIGSEGV delivered
to the guest process which was trying to execute such an instruction, or the *guest* kernel panic (if the guest kernel itself was trying to execute the instruction in question).

Setting XCR0 is a privileged instruction, so the guest kernel can't set it, the hypervisor
(the host kernel kvm module) should set it. The hypervisor should be aware of AVX instructions for this to work (which is not the case if the *host* kernel is old enough).

Thus the guest can't use AVX instructions without proper hypervisor support no matter if these instructions are advertised to the guest or not.

[Alexei Sheplyakov (asheplyakov)]

> The fix for bug #1618473 introduced a number of performance issues for MOS end users

This statement is not backed by any actual measurements, adjusting the bug importance accordingly

[Ivan Suzdal (isuzdal)]

I guess what best way is disable avx2 (or any other) feature [0] instead of changing cpu_mode.

[0] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1524069/comments/19

[Alexey Stupnikov (astupnikov)]

Alexei, I have tried to say, that QEMU will use its default profile, which is qemu-64 (at lest it did so in my lab), so a lot of features will not be available to guest. I don't try to say that computes should advertise avx2 instructions to guest, since this will lead to segfaults with recent kernels.

About testing. I don't think that AES encryption speed with and without AES-NI, especially for micro and nano instances, can be a theme for discussion. I can say the same about other encryption and calculation extensions.

[Alexey Stupnikov (astupnikov)]

Please check again this issue's importance based on the arguments from my previous comment.

[Alexey Stupnikov (astupnikov)]

Nova team, can you please check if we can do something about comment https://bugs.launchpad.net/mos/+bug/1621067/comments/3

[Alexey Stupnikov (astupnikov)]

@asheplyakov I can't agree with the point that QEMU will select the optimal vCPU features to advertise to the guest. In my experience, it always selects qemu-64 profile with slight modifications for KVM hypervisor (tested for two different envs).

[Alexey Stupnikov (astupnikov)]

Users are interested in those features: [1], [2], [3]

Vendors are also thinking they are important: [4]

[1] http://security.stackexchange.com/questions/11864/is-intel-aesni-supported-by-xen-in-vps-cloud-hosting-environments
[2] https://forums.aws.amazon.com/thread.jspa?threadID=122148
[3] https://www.phoronix.com/forums/forum/hardware/processors-memory/39178-benchmarking-amazon-s-new-ec2-c3-instance-types

[4] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Security_Guide/sect-Security_Guide-Encryption-OpenSSL_Intel_AES-NI_Engine.html

Moving bug to confirmed.

[Roman Podoliaka (rpodolyaka)]

Alexey St.

>>> Nova team, can you please check if we can do something about comment https://bugs.launchpad.net/mos/+bug/1621067/comments/3

I'm not sure, what you want to see here: a statement be added to libvirt nova driver code, that would add this section to a domain XML for all guests? Or make it a conditional statement based on what, qemu version / image? Nova just *does not* know what is there inside an image, and I don't think we can rely on the metadata to be set properly for all uploaded images.

In my opinion the clean way to fix this would be to set the cpu_model in nova.conf based on your host configuration after performing tests with most popular cloud images. Which is kind of what we did, it's just that we set a *safe* default and what you want too see is one, that would allow for performance optimizations.

I'd rather we keep the one that works for popular cloud images, but at the same time send a clear message to operators, that they can use the cpu_model option in nova.conf to configure a CPU model that better corresponds to the hardware they have, if they have tested it properly.

[Alexey Stupnikov (astupnikov)]

Roman, thanks for you answer. I would like to ask you to check is it possible to add two configuration parameters: cpu_features_enable and cpu_features_disable to configure feature require and feature disable to libvirt config template [1]?
[1]: https://libvirt.org/formatdomain.html#elementsCPU

[Alexey Stupnikov (astupnikov)]

Roman, thanks for you answer. I would like to ask you to check is it possible to add two configuration parameters: cpu_features_enable and cpu_features_disable to configure MULTIPLE feature require and feature disable to libvirt config template [1]?
[1]: https://libvirt.org/formatdomain.html#elementsCPU

[Roman Podoliaka (rpodolyaka)]

Alexey,

The current way to do that is to specify the desired CPU model in nova.conf. I'm not sure we want to provide more flexibility than that. But even if we reconsider that in the future, config options will not be back ported to stable releases.

[Roman Podoliaka (rpodolyaka)]

By the way, I checked if AVX2 instructions actually work inside an instance and they *do*:

http://paste.openstack.org/show/568987/

So far it's only the ubuntu kernel which fails on start. I'm not sure we should disable AVX2 or any other features.

[Alexey Stupnikov (astupnikov)]

Roman, I have other ideas how to fix it, but it will take much more time from customers. The problem is that real CPU's feature sets are not well aligned with ones defined in libvirt, the other issue is some CPUs having customised feature sets (BIOS/UEFI settings changed or being suboptimal). As a result, it is not a solid option to use pre-defined CPU profiles without exclusion/adding some parameters.

I don't have any other valuable propositions for nova, but IMO described issue is certainly a bug (though I am not sure about its importance). I am not 100% sure about nova, but it is certainly in the product and it is not configuration issue we can fix using Fuel. So your word is needed here.

[Alexey Stupnikov (astupnikov)]

Wow, it works for Fedora cloud image:
[fedora@testvm1 ~]$ cat /proc/cpuinfo | grep avx2
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer xsave avx f16c rdrand hypervisor lahf_lm abm vnmi ept fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt

[Roman Podoliaka (rpodolyaka)]

Alexey,

>> I don't have any other valuable propositions for nova, but IMO described issue is certainly a bug (though I am not sure about its importance). I am not 100% sure about nova, but it is certainly in the product and it is not configuration issue we can fix using Fuel. So your word is needed here.

I gave you my take on this: operators can choose any cpu model they want in nova config (or enable features pass-through) after deployment, when they tested this configuration with the cloud images, that are going to be used in this cloud. For now we could probably live with a "safe" default.

I'm all for investigating this further, because, as we see, other cloud images with newer kernels or just applications that use AVX2 work properly, so it's only Ubuntu kernel which is affected. We could ask MOS Linux or Alexey Sheplyakov to continue the investigation and try to understand what is the root cause of the failure.

[Alexey Stupnikov (astupnikov)]

It is possible to define custom CPU profiles in /usr/share/libvirt/cpu_map.xml, so this bug has a clear WA in nova. Roman's approach looks pretty reasonable. At least when customers are not complaining about this change.

[Dmitry Sutyagin (dsutyagin)]

folks, it's not possible. You can define anything there but QEMU uses it's own internal database, libvirt uses cpu_map.xml only to match current host cpu agaisnt this file, so adding definitions there will not work, QEMU will not take these new custom definitions. I tried adding a custom def in there, and set nova.conf to use cpu_mode=custom and cpu_model=mycustomcpu - the result is QEMU fails with "qemu-system-x86_64: Unable to find CPU definition: mycustomcpu". We need a new feature to solve this issue, which is to add smth like "cpu_feature=..." which takes smth like "disable=avx2,disable=...", parses it and sets "<feature policy='disable' name='avx2'/>" in VM xml

[Dmitry Sutyagin (dsutyagin)]

my bad, we can do "SandyBridge,+blahblah" and list all modules except the problematic one(s)