[BVT] Agents failed on bootsrapped nodes with ERROR -- : {"message": "deque(['meta', 'numa_topology', 'distances']): None is not of type 'array'", "errors": []}

[Tatyanka]

Deploy ceph HA with RadosGW for objects

    Scenario:

            Create cluster with Neutron
            Add 3 nodes with controller role
            Add 3 nodes with compute and ceph-osd role
            Deploy the cluster
            Check ceph status
            Run OSTF tests
            Check the radosqw daemon is started

Actual result:
Nodes failed to be discovered in nailgun, so test failed with timeout error.
in agent.log on bootsrap:
2016-03-01T20:04:10.230673+00:00 debug: 20:04:10.044867 #5307] DEBUG -- : Found block device: ram15
2016-03-01T20:04:10.230834+00:00 debug: 20:04:10.044906 #5307] DEBUG -- : Block device info: {"size"=>"131072", "removable"=>"0"}
2016-03-01T20:04:10.230994+00:00 debug: 20:04:10.044996 #5307] DEBUG -- : Detailed meta disks: [{:name=>"vda", :model=>nil, :size=>53687091200, :disk=>"vda", :extra=>["disk/by-id/virtio-1e8098f09a4a4bc3b0d5"], :removable=>"0"}, {:name=>"vdb", :model=>nil, :size=>53687091200, :disk=>"vdb", :extra=>["disk/by-id/virtio-e0a18a37fff442d18795"], :removable=>"0"}, {:name=>"vdc", :model=>nil, :size=>53687091200, :disk=>"vdc", :extra=>["disk/by-id/virtio-7c339d8f6225432ca01d"], :removable=>"0"}]
2016-03-01T20:04:10.231157+00:00 debug: 20:04:10.118876 #5307] DEBUG -- : Response: status: 400 body: {"message": "deque(['meta', 'numa_topology', 'distances']): None is not of type 'array'", "errors": []}
2016-03-01T20:04:10.231324+00:00 err: 20:04:10.118951 #5307] ERROR -- : {"message": "deque(['meta', 'numa_topology', 'distances']): None is not of type 'array'", "errors": []}

Looks like a regression after 7a84e00 Merge "Add NUMA topology validation for Node"

https://172.18.160.103/job/9.0-mitaka.all/55/artifact/artifacts/listing-package-changelog.txt

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/286851

[Ihor Kalnytskyi (ikalnytskyi)]

Regression was introduced by this patch

  https://review.openstack.org/#/c/285265/8

where strict validation has been added. So now distance must be alway array while nailgun-agent sends 'nil' as its default value

  https://github.com/openstack/fuel-nailgun-agent/blob/01ec3c5501c4275063e81323116d4b65893a220f/agent#L867

Let's revert the patch with regression for now, and then reapply it when appropriate changes to nailgun-agent is commited.

[Ihor Kalnytskyi (ikalnytskyi)]

Revert - https://review.openstack.org/#/c/286851/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/286851
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=a1d9d3599262433af28642adfb85ec8f60554dec
Submitter: Jenkins
Branch: master

commit a1d9d3599262433af28642adfb85ec8f60554dec
Author: tatyana-leontovich <email address hidden>
Date: Tue Mar 1 21:38:39 2016 +0000

    Revert "Add NUMA topology validation for Node"

    This reverts commit a2e90e6b85f94d89e6fc62acde7de9a1615059a8.

    Change-Id: I227b25edc655b686b461d68cfc541e51e5e02be8
    Closes-Bug: #1551955

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/287054

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/287054
Committed: https://git.openstack.org/cgit/openstack/fuel-nailgun-agent/commit/?id=122f851295e6fb5f8e04b997de7c262527392ceb
Submitter: Jenkins
Branch: master

commit 122f851295e6fb5f8e04b997de7c262527392ceb
Author: Artur Svechnikov <email address hidden>
Date: Wed Mar 2 12:25:59 2016 +0300

    Set default NUMA topology distances

    In case of one NUMA node `lstopo` doesn't return
    information about distances, so by default distances
    should be 2d array [["1.0"]]

    Change-Id: I858c93e7f41b1a670bc72a80cd0c5b47fe63ef12
    Partial-Bug: #1551955

[Nastya Urlapova (aurlapova)]

Verified on:
cat /etc/fuel_build_id:
 90
cat /etc/fuel_build_number:
 90
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 liberty-9.0
rpm -qa | egrep 'fuel|astute|network-checker|shotgun':
 fuel-library9.0-9.0.0-1.mos8133.noarch
 fuel-agent-9.0.0-1.mos261.noarch
 fuel-nailgun-9.0.0-1.mos8553.noarch
 fuel-misc-9.0.0-1.mos8133.noarch
 fuel-mirror-9.0.0-1.mos128.noarch
 fuel-bootstrap-cli-9.0.0-1.mos261.noarch
 fuel_plugin_example_v4_hotpluggable-4.0-4.0.0-1.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8553.noarch
 network-checker-9.0.0-1.mos72.x86_64
 shotgun-9.0.0-1.mos84.noarch
 fuel-openstack-metadata-9.0.0-1.mos8553.noarch
 fuel-notify-9.0.0-1.mos8133.noarch
 fuel-setup-9.0.0-1.mos6316.noarch
 python-fuelclient-9.0.0-1.mos291.noarch
 fuelmenu-9.0.0-1.mos261.noarch
 fuel-9.0.0-1.mos6316.noarch
 fuel-utils-9.0.0-1.mos8133.noarch
 fuel-ostf-9.0.0-1.mos918.noarch
 fuel-release-9.0.0-1.mos6316.noarch
 rubygem-astute-9.0.0-1.mos724.noarch
 fuel-ui-9.0.0-1.mos2600.noarch
 fuel-migrate-9.0.0-1.mos8133.noarch