Systest spawn_three_vms_across_three_virt_nodes failed with error message "Timeout waiting 6 available nodes"

[Egor Kotko]

Systest spawn_three_vms_across_three_virt_nodes failed with error message "Timeout waiting 6 available nodes"
https://product-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.reduced_footprint/17/testReport/%28root%29/spawn_three_vms_across_three_virt_nodes/spawn_three_vms_across_three_virt_nodes/

 Scenario:
            1. Create cluster
            2. Assign compute and virt roles to three slave nodes
            3. Upload VM configuration for one VM to each slave node
            4. Spawn VMs
            5. Wait till VMs become available for allocation

The test failed on creation kvm virtual machine on node-3, puppet log:
http://paste.openstack.org/show/473476/

The /var/log/libvirt/libvirtd.log from node-3 contains permission errors on creation virtual networks:
http://paste.openstack.org/show/473477/

[Bartosz Kupidura (zynzel)]

This is caused by wrong permissions on /dev/kvm.

Failed node:
root@node-3:~# ls -l /dev/kvm
crw------- 1 root root 10, 232 Sep 22 05:41 /dev/kvm

Good node:
root@node-2:~# ls -l /dev/kvm
crw-rw---- 1 root kvm 10, 232 Sep 22 05:40 /dev/kvm

Inside manifests we rely on permissions delivered by packages.

[Matthew Mosesohn (raytrac3r)]

It seems reasonable to ensure the permissions in puppet manifests. I think there should be a Puppet config solution, rather than a packaging one for this issue.

[Egor Kotko (ykotko)]

Was not reproduced on:

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "301"
  build_id: "301"
  nailgun_sha: "4162b0c15adb425b37608c787944d1983f543aa8"
  python-fuelclient_sha: "486bde57cda1badb68f915f66c61b544108606f3"
  fuel-agent_sha: "50e90af6e3d560e9085ff71d2950cfbcca91af67"
  fuel-nailgun-agent_sha: "d7027952870a35db8dc52f185bb1158cdd3d1ebd"
  astute_sha: "6c5b73f93e24cc781c809db9159927655ced5012"
  fuel-library_sha: "5d50055aeca1dd0dc53b43825dc4c8f7780be9dd"
  fuel-ostf_sha: "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c"
  fuelmain_sha: "a65d453215edb0284a2e4761be7a156bb5627677"