KVM module not loaded during deployment of compute nodes in MOS 6.0

[Aleksandr Shaposhnikov]

Steps to reproduce:

1. Deploy MOS 6.0
2. Deploy cluster with Ubuntu+HA+KVM.
3. kvm and kvm_intel/kvm_amd modules on compute nodes not loaded.

version:
api: '1.0'
astute_sha: c3e7c7a18528cf9acca48021488a93dff74f5c97
auth_required: true
build_id: 2014-10-16_19-59-04
build_number: '72'
feature_groups:
- mirantis
fuellib_sha: 677c2809bd602ed6f793b03df49ef8b0f8dcb7e7
fuelmain_sha: 5cf06aac43ccb4a6031fbfa87ff9f9a729314daa
nailgun_sha: b83eaf18cbcc36393f8ac1e7732a6395546a7ca8
ostf_sha: de177931b53fbe9655502b73d03910b8118e25f1
production: docker
release: '6.0'
release_versions:
  2014.2-6.0:
    VERSION:
      api: '1.0'
      astute_sha: c3e7c7a18528cf9acca48021488a93dff74f5c97
      build_id: 2014-10-16_19-59-04
      build_number: '72'
      feature_groups:
      - mirantis
      fuellib_sha: 677c2809bd602ed6f793b03df49ef8b0f8dcb7e7
      fuelmain_sha: 5cf06aac43ccb4a6031fbfa87ff9f9a729314daa
      nailgun_sha: b83eaf18cbcc36393f8ac1e7732a6395546a7ca8
      ostf_sha: de177931b53fbe9655502b73d03910b8118e25f1
      production: docker
      release: '6.0'

[Aleksandr Shaposhnikov (alashai8)]

Just deployed new #53 ISO and cluster based on Ubuntu HA.

Here is grep from compute node:
root@node-22:~# lsmod
Module Size Used by
dm_multipath 27401 0
scsi_dh 14873 1 dm_multipath
nbd 17744 0
ib_iser 38796 0
rdma_cm 48728 1 ib_iser
ib_cm 48393 1 rdma_cm
iw_cm 18793 1 rdma_cm
ib_sa 34134 2 rdma_cm,ib_cm
ib_mad 48127 2 ib_cm,ib_sa
ib_core 82802 6 ib_iser,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad
ib_addr 14337 1 rdma_cm
iscsi_tcp 18587 0
libiscsi_tcp 25377 1 iscsi_tcp
libiscsi 57717 3 ib_iser,iscsi_tcp,libiscsi_tcp
scsi_transport_iscsi 82166 4 ib_iser,iscsi_tcp,libiscsi
ipt_MASQUERADE 12880 3
iptable_nat 13151 1
nf_nat_ipv4 13316 1 iptable_nat
nf_nat 26902 3 ipt_MASQUERADE,iptable_nat,nf_nat_ipv4
xt_conntrack 12760 1
ipt_REJECT 12576 2
xt_CHECKSUM 12549 1
iptable_mangle 12734 1
xt_tcpudp 12924 6
bridge 112036 0
stp 12976 1 bridge
llc 14597 2 bridge,stp
ebtable_nat 12807 0
ebtables 31209 1 ebtable_nat
iscsi_target_mod 300369 1
target_core_mod 322353 2 iscsi_target_mod
configfs 35771 3 iscsi_target_mod,target_core_mod
xt_state 12578 1
xt_CT 13046 1
iptable_raw 12678 1
xt_comment 12504 32
xt_multiport 12798 26
ip6table_filter 12815 0
ip6_tables 27502 1 ip6table_filter
iptable_filter 12810 1
ip_tables 27716 4 iptable_nat,iptable_mangle,iptable_raw,iptable_filter
x_tables 34194 16 ipt_MASQUERADE,xt_conntrack,ipt_REJECT,xt_CHECKSUM,iptable_mangle,xt_tcpudp,ebtables,xt_state,xt_CT,iptable_raw,xt_comment,xt_multiport,ip6table_filter,ip6_tables,iptable_filter,ip_tables
openvswitch 62733 0
psmouse 104093 0
gpio_ich 13526 0
serio_raw 13413 0
mei_me 18418 0
mei 78537 1 mei_me
lpc_ich 21163 0
joydev 17613 0
nf_conntrack_ipv6 18946 0
nf_defrag_ipv6 34817 1 nf_conntrack_ipv6
xfs 910633 1
ioatdma 68040 0
nf_conntrack_ipv4 19159 3
nf_conntrack 92358 9 ipt_MASQUERADE,iptable_nat,nf_nat_ipv4,nf_nat,xt_conntrack,xt_state,xt_CT,nf_conntrack_ipv6,nf_conntrack_ipv4
wmi 19256 0
nf_defrag_ipv4 12729 1 nf_conntrack_ipv4
mac_hid 13253 0
lp 17799 0
parport 42466 1 lp
igb 186020 0
ixgbe 236655 0
hid_generic 12548 0
dca 15232 3 ioatdma,igb,ixgbe
i2c_algo_bit 13564 1 igb
usbhid 53329 0
ahci 30063 4
ptp 18627 2 igb,ixgbe
hid 106315 2 hid_gene...

[Matthew Mosesohn (raytrac3r)]

It looks like the scale lab already has kernel modules on. I don't have any bare metal to reproduce this, so I'm waiting on availability for us to redeploy one of the compute nodes.

[Matthew Mosesohn (raytrac3r)]

I was able to get help from Sergey Galkin to reproduce this on bare metal. It looks like qemu-kvm will add a hook to modprobe kvm on boot. We can just run this harmlessly in any case even if the system doesn't support kvm. It looks like nova puppet module doesn't install qemu-kvm, but our openstack core module itself does. I'll open an upstream bug to report this and see if there are other recommendations, but for now we can just add logic to openstack::compute class to modprobe kvm.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/131677

[Tomasz 'Zen' Napierala (tzn)]

This was also my problem, I couldn't catch the node without module loaded. I will request redeploy.

[Matthew Mosesohn (raytrac3r)]

upstream bug: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1387080

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/131677
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=871c42a94a578b9b8b0dfb45cd3cde96946c929b
Submitter: Jenkins
Branch: master

commit 871c42a94a578b9b8b0dfb45cd3cde96946c929b
Author: Matthew Mosesohn <email address hidden>
Date: Wed Oct 29 13:29:16 2014 +0400

    Ensure qemu-kvm is running when deploying compute on KVM

    qemu-kvm was not being started on ubuntu, which inserts
    kvm kernel modules appropriately.
    This behavior corrects a bug in qemu-kvm package on ubuntu
    where it only loads kvm module after starting the qemu-kvm
    service.
    For CentOS, this functionality is provided in the script
    /etc/sysconfig/modules/kvm.modules, and it is called just
    to provide parity with Ubuntu, even though it does not
    appear to be broken.

    Change-Id: I05017f991593c79dbd6caa60683fd6061dd71626
    Closes-Bug: #1383800

[Artem Panchenko (apanchenko-8)]

verified on iso # 63

api: '1.0'
astute_sha: 97eea90efe0a1f17b4934919d6e459d270c10372
auth_required: true
build_id: 2014-10-30_04-21-22
build_number: '63'
feature_groups:
- mirantis
- techpreview
fuellib_sha: 45b6fc42091a0a33d3e48fbe78b782ce743aedc1
fuelmain_sha: 2ade7c571380a091048d103a6affff634b5b2520
nailgun_sha: 02c6bb2e54bbec76da33167eaf5f2e0b3e2e50a7
ostf_sha: f47fd1d66a7255213ee075d5c11b8f111f922000
production: docker
release: 6.0-techpreview