Forcing CPU flags makes the guest kernel Oops

[Dmitry Tyzhnenko]

dos.py forces CPU flags so qemu claims avx2 instructions are usable
from the guest, and in fact they are not, which causes an Oops in raid6_pq code:

https://drive.google.com/file/d/0B8U7EvTbuAOldHVjU0dpd0VXSTQ/view?usp=sharing
https://drive.google.com/file/d/0B8U7EvTbuAOlbEJaRHBJcGwyR2c/view?usp=sharing
https://drive.google.com/file/d/0B8U7EvTbuAOlMFhYY3dSVlFjbFU/view?usp=sharing

Thus the (virtualized) nodes are unable to boot after the provisioning.

Fuel ISO 7.0-164

Scenario:
  1. Create new environment
  2. Choose Neutron, VLAN
  3. Choose Ceilometer
  4. Add 5 controller
  5. Add 2 compute
  6. Add 1 cinder
  7. Add 2 mongo
  8. Verify networks
  9. Deploy the environment
 10. Verify networks
 11. Run OSTF tests

Expected result:
  All steps pass

Actual result:
  Step 9 has failed after provisioning nodes

Fuel version
{
    'build_id': '2015-08-12_15-16-47',
    'build_number': '164',
    'release_versions': {'2015.1.0-7.0': {'VERSION': {
        'build_id': '2015-08-12_15-16-47',
        'build_number': '164',
        'api': '1.0',
        'fuel-library_sha': '22f848670e49d89fc04aaed4d8efd1b07360cbe7',
        'nailgun_sha': 'fff6bda090fac15c48b27cca7832a70f8e381101',
        'feature_groups': ['mirantis'],
        'fuel-nailgun-agent_sha': 'e01693992d7a0304d926b922b43f3b747c35964c',
        'openstack_version': '2015.1.0-7.0',
        'fuel-agent_sha': '57145b1d8804389304cd04322ba0fb3dc9d30327',
        'production': 'docker',
        'python-fuelclient_sha': '26fc025e0fc5791b62e5ed8561a6016bf8a406bc',
        'astute_sha': 'e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc',
        'fuel-ostf_sha': '58220583f10fa47f12291488ef77854809c68310',
        'release': '7.0',
        'fuelmain_sha': '67e5214c0dc5d4ba6da4ae651cef9934800459a9',
        }}},
    'auth_required': true,
    'api': '1.0',
    'fuel-library_sha': '22f848670e49d89fc04aaed4d8efd1b07360cbe7',
    'nailgun_sha': 'fff6bda090fac15c48b27cca7832a70f8e381101',
    'feature_groups': ['mirantis'],
    'fuel-nailgun-agent_sha': 'e01693992d7a0304d926b922b43f3b747c35964c',
    'openstack_version': '2015.1.0-7.0',
    'fuel-agent_sha': '57145b1d8804389304cd04322ba0fb3dc9d30327',
    'production': 'docker',
    'python-fuelclient_sha': '26fc025e0fc5791b62e5ed8561a6016bf8a406bc',
    'astute_sha': 'e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc',
    'fuel-ostf_sha': '58220583f10fa47f12291488ef77854809c68310',
    'release': '7.0',
    'fuelmain_sha': '67e5214c0dc5d4ba6da4ae651cef9934800459a9',
    }

[Alexei Sheplyakov (asheplyakov)]

Please specify the host kernel version and (host) qemu version.

[Alexei Sheplyakov (asheplyakov)]

Also please post the exact qemu command line

[Dmitry Tyzhnenko (dtyzhnenko)]

$ uname -a
Linux cz7329 3.13.0-61-generic #100-Ubuntu SMP Wed Jul 29 11:21:34 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ dpkg-query -l '*qemu*' | grep 'ii'
ii ipxe-qemu 1.0.0+git-20131111.c3d1e78-2ubuntu1.1 all PXE boot firmware - ROM images for qemu
ii qemu-keymaps 2.0.0+dfsg-2ubuntu1.16 all QEMU keyboard maps
ii qemu-kvm 2.0.0+dfsg-2ubuntu1.16 amd64 QEMU Full virtualization
ii qemu-system-common 2.0.0+dfsg-2ubuntu1.16 amd64 QEMU full system emulation binaries (common files)
ii qemu-system-x86 2.0.0+dfsg-2ubuntu1.16 amd64 QEMU full system emulation binaries (x86)
ii qemu-utils 2.0.0+dfsg-2ubuntu1.16 amd64 QEMU utilities

$ qemu-system-x86_64 --version
QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.16), Copyright (c) 2003-2008 Fabrice Bellard

[Alexei Sheplyakov (asheplyakov)]

@Dmitry,

please post the exact qemu command line.

[Alexei Sheplyakov (asheplyakov)]

Sounds like a configuration issue (i.e. wrong -cpu).
The very same combination of kernel and qemu works for me just fine.
qemu command line arguments can be found here: http://paste.openstack.org/show/419002

[Dmitry Tyzhnenko (dtyzhnenko)]

@asheplyakov

http://paste.openstack.org/show/419019/

[Alexei Sheplyakov (asheplyakov)]

> -cpu Nehalem,+invpcid,+erms,+bmi2,+smep,+avx2

Removing +avx2 makes the guest kernel boot successfully.

In general using certain advanced CPU instructions in the guest might require additional support from
the hypervisor (kvm kernel module or qemu process). Perhaps -cpu flags should be removeed so qemu
will guess the correct flags (i.e. an intersection between the available CPU instructions and ones which
qemu is able to handle).

[Dmitry Tyzhnenko (dtyzhnenko)]

Workaround :
 We can run test with false in DRIVER_USE_HOST_CPU variable for use autodetection in qemu

[Dmitry Tyzhnenko (dtyzhnenko)]

Reduce importance to medium because we have workaround

[Timur Nurlygayanov (tnurlygayanov)]

The issue reproduced on many environments, here is another description of this issue:
https://bugs.launchpad.net/fuel/+bug/1520088

[Timur Nurlygayanov (tnurlygayanov)]

ERROR FROM FUEL:
    Deployment Failed
    Error
    Too many nodes failed to provision
    Some nodes have an error status after deployment. Redeployment is needed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/250326

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/250326
Committed: https://git.openstack.org/cgit/openstack/fuel-devops/commit/?id=22d64bd96a841dfc2330b5d2069fb9d0862c357c
Submitter: Jenkins
Branch: master

commit 22d64bd96a841dfc2330b5d2069fb9d0862c357c
Author: Georgy <email address hidden>
Date: Thu Nov 26 14:05:06 2015 +0300

    Update libvirt cpu config builder

    Closes-Bug: 1485047

    Change-Id: I6dd7e2b99cae68bc32e71d82bd23c33e6bf7495e

[Georgy Dyuldin (g-dyuldin)]

Fuel ISO 8.0-230