[regression] OSTF tests for Nova failed on configuration with Cinder LVM, Sahara, Ceilometer and Ironic

[Timur Nurlygayanov]

Note:
The issue reproduced on Tempest CI, so, this is a regression issue.

Steps To Reproduce:
1) Deploy the following configuration: 3 controllers with mongo, 2 compute node with cinder backend, 2 Ironic nodes
2) Run OSTF tests

Observed Results:
All VM-related OSTF tests failed (boot VM from volume and etc.)

Expected Results:
All OSTF tests passed.

Please see attached diagnostic snapshot for more detailed information.

[Ivan Kolodyazhny (e0ne)]

Exception in the bug description is not related to the issue. It's common error during OpenStack installation.

According to OSTF results, all tests with 'boot instance' step faild. I've found the following error in nova-compute logs:

failed to initialize KVM: Inappropriate ioctl for device

[Timur Nurlygayanov (tnurlygayanov)]

Cinder team reviewed the env and it looks like Nova can't start the VM

[Timur Nurlygayanov (tnurlygayanov)]

We added tag blocker-for-qa because this is a blocker issue for Tempest

[Roman Podoliaka (rpodolyaka)]

The VM boot fails with:

2016-09-15 10:55:39.975+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64 -name instance-00000001 -S -machine pc-i440fx-vivid,accel=kvm,usb=off -m 64 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 2dddaaee-ff93-4520-8fe0-6adedf713f02 -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=13.1.1,serial=a7d1fe7b-b18b-4259-bfc2-b054a5fcd761,uuid=2dddaaee-ff93-4520-8fe0-6adedf713f02,family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000001.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/dev/disk/by-path/ip-10.109.3.3:3260-iscsi-iqn.2010-10.org.openstack:volume-c8c53351-8d4f-4ccf-9fe7-9bd51fa46076-lun-1,if=none,id=drive-virtio-disk0,format=raw,serial=c8c53351-8d4f-4ccf-9fe7-9bd51fa46076,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/nova/instances/2dddaaee-ff93-4520-8fe0-6adedf713f02/disk.config,if=none,id=drive-virtio-disk25,format=raw,cache=directsync -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk25,id=virtio-disk25 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:5b:be:ba,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/2dddaaee-ff93-4520-8fe0-6adedf713f02/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on

Domain id=2 is tainted: high-privileges
char device redirected to /dev/pts/11 (label charserial1)
kvm version too old
failed to initialize KVM: Inappropriate ioctl for device
2016-09-15 10:55:40.181+0000: shutting down

Looks like it's just a poor error reporting from qemu: it just can not start a KVM VM.

This happens because kvm module is not loaded, but nova-compute is configured to use it:

root@node-7:~# grep virt_type /etc/nova/nova-compute.conf
virt_type=kvm
root@node-7:~# lsmod | grep kvm
root@node-7:~# grep vmx /proc/cpuinfo
root@node-7:~# grep svm /proc/cpuinfo
root@node-7:~#

^ nor can it be loaded, because the host does not support virtualization extensions.

If you try to start a KVM VM manually, you'll see the very same error:

root@node-7:~# qemu-system-x86_64 -enable-kvm -m 512 -hda trusty-server-cloudimg-amd64-disk1.img -machine pc-1.0,accel=kvm,usb=off -rtc base=utc -serial mon:stdio -vga none -nographic
kvm version too old
failed to initialize KVM: Inappropriate ioctl for device

While TCG (software emulation) acceleration works as expected:

root@node-7:~# qemu-system-x86_64 -m 512 -hda trusty-server-cloudimg-amd64-disk1.img -machin...

[Timur Nurlygayanov (tnurlygayanov)]

We found the errors in the configuration of the virtualization on the bare-metal hosts, the issue was fixed on the 3 QA environments and now it works fine.

Status changed to Invalid.