fuel_migration failed on node srv52-bud with KVM issue

[Maksym Strukov]

https://mirantis.testrail.com/index.php?/tests/view/1906357

Scenario:
1. Create cluster
2. Run OSTF tests
3. Run Network check
4. Migrate fuel-master to VM

Test failed with message (https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.fuel_master_migrate/18/testReport/(root)/fuel_migration/fuel_migration/)

TimeoutError: The phrase Rebooting to begin the data sync process not found in /var/log/fuel-migrate.log file on remote node

Log /var/log/fuel-migrate.log contain:

+ virsh -c qemu+ssh://root@10.109.5.4/system define /tmp/vm.xml
Domain fuel_master defined from /tmp/vm.xml

+ virsh -c qemu+ssh://root@10.109.5.4/system autostart fuel_master
Domain fuel_master marked as autostarted

+ virsh -c qemu+ssh://root@10.109.5.4/system start fuel_master
error: Failed to start domain fuel_master
error: unsupported configuration: Domain requires KVM, but it is not available. Check that virtualization is enabled in the host BIOS, and host configuration is setup to load the kvm modules.

Previously test passed ok on nodes 76, 77, 48, 72, 73, 75, 74 etc. But failed on srv52-bud.

[Maksym Strukov (unbelll)]

There is a chance, that server configured in wrong way. I can't login with my key. Assigned to devops.

[Andrey Nikitin (heos)]

Fixed, added 'search' domain into /etc/resolv.conf files.

[Alexandr Kostrikov (akostrikov-mirantis)]

I am reassigning because we have not configured kvm as in https://docs.fuel-infra.org/fuel-dev/devops.html#optional-enabling-nested-paging

There is on 51
ssh <email address hidden> "cat /etc/modprobe.d/qemu-system-x86.conf"

/* *\
  Welcome to the Fuel server srv51-bud.infra.mirantis.net
  All granted users could be authenticated via LDAP

  If you'd like to authenticate with public keys,
    please use auth.mirantis.com to upload your key

  On any questions please do not hesitate to contact
  --
    <email address hidden>
\* */
options kvm_intel nested=1

But there is no on 52
ssh <email address hidden> "cat /etc/modprobe.d/qemu-system-x86.conf"
Warning: the ECDSA host key for 'srv52-bud.infra.mirantis.net' differs from the key for the IP address '172.18.162.35'
Offending key for IP in /home/akostrikov/.ssh/known_hosts:31
Matching host key in /home/akostrikov/.ssh/known_hosts:83
Are you sure you want to continue connecting (yes/no)? yes

/* *\
  Welcome to the Fuel server srv52-bud.infra.mirantis.net
  All granted users could be authenticated via LDAP

  If you'd like to authenticate with public keys,
    please use auth.mirantis.com to upload your key

  On any questions please do not hesitate to contact
  --
    <email address hidden>
\* */
cat: /etc/modprobe.d/qemu-system-x86.conf: No such file or directory

[Andrey Nikitin (heos)]

srv52 has newest version of qemu - 2.4. As I can see this version of qemu has no file /etc/modprobe.d/qemu-system-x86.conf by default.

So, we should deeply investigate the problem to find a root cause. On this period srv52-bud is disabled in Jenkins.

[Dennis Dmitriev (ddmitriev)]

There is config from my local server:

(devops-venv-2.9.9)root@srv09-kha:/etc/modprobe.d# ls -la /etc/modprobe.d/
total 44
drwxr-xr-x 2 root root 4096 Sep 9 01:04 .
drwxr-xr-x 108 root root 4096 Sep 9 13:16 ..
-rw-r--r-- 1 root root 325 Apr 10 2014 blacklist-ath_pci.conf
-rw-r--r-- 1 root root 1603 Apr 10 2014 blacklist.conf
-rw-r--r-- 1 root root 210 Apr 10 2014 blacklist-firewire.conf
-rw-r--r-- 1 root root 677 Apr 10 2014 blacklist-framebuffer.conf
-rw-r--r-- 1 root root 583 Apr 10 2014 blacklist-rare-network.conf
-rw-r--r-- 1 root root 1077 Apr 10 2014 blacklist-watchdog.conf
-rw-r--r-- 1 root root 456 Apr 14 2014 fbdev-blacklist.conf
-rw-r--r-- 1 root root 347 Apr 10 2014 iwlwifi.conf
-rw-r--r-- 1 root root 104 Apr 10 2014 mlx4.conf

(devops-venv-2.9.9)root@srv09-kha:/etc/modprobe.d# cat /etc/issue
Ubuntu 14.04.3 LTS \n \l

(devops-venv-2.9.9)root@srv09-kha:/etc/modprobe.d# qemu-system-x86_64 --version
QEMU emulator version 2.4.0 (Debian 1:2.4+dfsg-ubuntu2), Copyright (c) 2003-2008 Fabrice Bellard

(devops-venv-2.9.9)root@srv09-kha:/etc/modprobe.d# cat /sys/module/kvm_intel/parameters/nested
Y

There is no file /etc/modprobe.d/qemu-system-x86.conf , but 'nested' is enabled for module 'kvm_intel'.

There were installed, in the following order:
- qemu 2.0.0 from upstream Ubuntu repo;
- qemu 2.0.0 from devops repo;
- qemu 2.4 from devops repo

[Andrey Nikitin (heos)]

I've found two commits into debian's repository about qemu module (http://anonscm.debian.org/cgit/pkg-qemu/qemu.git/):

1. About removing of qemu's initscript: http://anonscm.debian.org/cgit/pkg-qemu/qemu.git/diff/?id=ed9e65a393cd86525127813aee8afc0251f04199
2. About removing of "qemu-system-x86.install" file. http://anonscm.debian.org/cgit/pkg-qemu/qemu.git/commit/debian?id=e537fce5f9875430c9e9302f66535f1a6b5af142

So, I think, we should create that file by puppet manifests or study how to start that module.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Igor Shishkin <email address hidden>
Review: https://review.fuel-infra.org/13683

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13683
Submitter: Igor Shishkin <email address hidden>
Branch: master

Commit: 1e2a8f7b1bc7bb5efd3afa61a350ad2da06db8ab
Author: Igor Shishkin <email address hidden>
Date: Thu Nov 5 15:34:07 2015

Adding stub for qemu=2.4 missed modprobe file

Change-Id: I05d4c3cc79b98c7d72f51cae3dd1d47c5b8acb03
Closes-Bug: #1506582
Closes-Bug: #1513394
Related-Bug: #1484080
Signed-off-by: Igor Shishkin <email address hidden>