Sahara OSTF test failed without UI notification

[Dennis Dmitriev]

Cluster configuration:
    - Ubuntu/simple,
    - Neutron/GRE,
    - Network without tagging, eth0:admin, eth1:public, eth2:storage,
          eth3+eth4 = ovs-bond0 : management
    - Ceph for volumes
    - Ceph for images
    - Sahara
    - Murano

Nodes configuration:
    - 1 controller (+ ceph) / 4 CPU, 4096Mb RAM
    - 2 compute / 3 CPU, 6144Mb RAM
    - 2 ceph / 1 CPU, 3072Mb RAM

Deployment of the cluster was successful, all OSTF tests were passed except two platform tests for Sahara and Murano.
Murano showed ''success' != u'completed_w_errors' Please refer to OpenStack logs for more details.' error.

Sahara showed no messages in UI at all. Two instances have been running for ~14 hours, but Sahara cluster is still in 'Waiting' status.

Sahara error in OSTF log: http://paste.openstack.org/show/148473/

[Dennis Dmitriev (ddmitriev)]

{"build_id": "2014-12-03_01-07-36", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "48", "auth_required": true, "api": "1.0", "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d", "production": "docker", "fuelmain_sha": "7626c5aeedcde77ad22fc081c25768944697d404", "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce", "feature_groups": ["mirantis"], "release": "5.1.1", "release_versions": {"2014.1.3-5.1.1": {"VERSION": {"build_id": "2014-12-03_01-07-36", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "48", "api": "1.0", "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d", "production": "docker", "fuelmain_sha": "7626c5aeedcde77ad22fc081c25768944697d404", "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce", "feature_groups": ["mirantis"], "release": "5.1.1", "fuellib_sha": "a3043477337b4a0a8fd166dc83d6cd5d504f5da8"}}}, "fuellib_sha": "a3043477337b4a0a8fd166dc83d6cd5d504f5da8"}

[Yaroslav Lobankov (ylobankov)]

Fuel cluster nodes were KVM virtual machines? Or you used VBox scripts to build cluster?

[Dennis Dmitriev (ddmitriev)]

I used qemu/kvm with fuel-devops to create the environment. But resources for slaves were changed manually.

[Dmitry Mescheryakov (dmitrymex)]

That is a medium bug, we are not going to fix it in maintenance releases 5.1.2 and 6.0.1

[Dmitry Mescheryakov (dmitrymex)]

Pushing the bug to 7.0 as we re close to SCF

[Yaroslav Lobankov (ylobankov)]

It looks like the problem was in security groups. In MOS 5.1.1 before we run Sahara test we have to open some security group ports manually, for example 22 (SSH port). If we forget to do this, the Sahara cluster will hang in "Waiting" state, trying to connect to instances via SSH. In logs I see the following:

2014-12-10T07:00:39.738002+00:00 debug: 2014-12-10 07:00:39.709 4234 DEBUG sahara.service.engine [-] Can't login to node ostf-cluster-1378389495-ostf-test-master-001 (10.108.30.152), reason error: [Errno 110] Connection timed out _wait_until_accessible /usr/lib/python2.7/dist-packages/sahara/service/engine.py:95

Anyway, now these ports are opened automatically and this issue is no longer actual since MOS 6.0.1.