[EDP] Job execution fails with "AttributeError: 'module' object has no attribute 'node_groups'" when Sahara tries to get cluster hosts info

[Sofiia Kostiuchenko]

Job fails right after I start it and throws an error to sahara server log:

2014-07-09 05:42:11.999 19264 DEBUG keystoneclient.middleware.auth_token [-] Received request from user: e4667a3380084a79b31fad83cebaf4c2 with project_id : 1e97e1707559452da53d460cef8da9bb and roles: _member_,admin _build_u
ser_headers /root/qa/nightly-build/sahara/.tox/venv/lib/python2.7/site-packages/keystoneclient/middleware/auth_token.py:1021
2014-07-09 05:42:11.999 19264 DEBUG sahara.utils.api [-] Rest.route.decorator.handler, kwargs={'tenant_id': u'1e97e1707559452da53d460cef8da9bb', 'job_id': u'c739783c-1c3a-4145-b963-271ce6a89852'} handler /root/qa/nightly-build/sahara/sahara/utils/api.py:64
2014-07-09 05:42:12.283 19264 INFO sahara.cli.sahara_all [-] 10.10.80.123 - - [09/Jul/2014 05:42:12] "POST /v1.1/1e97e1707559452da53d460cef8da9bb/jobs/c739783c-1c3a-4145-b963-271ce6a89852/execute HTTP/1.1" 202 532 0.286385
2014-07-09 05:42:12.390 19264 ERROR sahara.service.edp.job_manager [-] Can't run job execution 'a4297d7a-a648-45cf-b2db-5e333f9f5c74' (reason: 'module' object has no attribute 'node_groups')
2014-07-09 05:42:12.390 19264 TRACE sahara.service.edp.job_manager Traceback (most recent call last):
2014-07-09 05:42:12.390 19264 TRACE sahara.service.edp.job_manager File "/root/qa/nightly-build/sahara/sahara/service/edp/job_manager.py", line 149, in run_job
2014-07-09 05:42:12.390 19264 TRACE sahara.service.edp.job_manager _run_job(job_execution_id)
2014-07-09 05:42:12.390 19264 TRACE sahara.service.edp.job_manager File "/root/qa/nightly-build/sahara/sahara/service/edp/job_manager.py", line 177, in _run_job
2014-07-09 05:42:12.390 19264 TRACE sahara.service.edp.job_manager h.configure_cluster_for_hdfs(cluster, data_source)
2014-07-09 05:42:12.390 19264 TRACE sahara.service.edp.job_manager File "/root/qa/nightly-build/sahara/sahara/service/edp/hdfs_helper.py", line 88, in configure_cluster_for_hdfs
2014-07-09 05:42:12.390 19264 TRACE sahara.service.edp.job_manager etc_hosts_information = _get_cluster_hosts_information(host, cluster)
2014-07-09 05:42:12.390 19264 TRACE sahara.service.edp.job_manager File "/root/qa/nightly-build/sahara/sahara/service/edp/hdfs_helper.py", line 78, in _get_cluster_hosts_information
2014-07-09 05:42:12.390 19264 TRACE sahara.service.edp.job_manager for i in u.get_instances(c):
2014-07-09 05:42:12.390 19264 TRACE sahara.service.edp.job_manager File "/root/qa/nightly-build/sahara/sahara/plugins/general/utils.py", line 33, in get_instances
2014-07-09 05:42:12.390 19264 TRACE sahara.service.edp.job_manager nodes = get_node_groups(cluster, node_process)
2014-07-09 05:42:12.390 19264 TRACE sahara.service.edp.job_manager File "/root/qa/nightly-build/sahara/sahara/plugins/general/utils.py", line 23, in get_node_groups
2014-07-09 05:42:12.390 19264 TRACE sahara.service.edp.job_manager return [ng for ng in cluster.node_groups
2014-07-09 05:42:12.390 19264 TRACE sahara.service.edp.job_manager AttributeError: 'module' object has no attribute 'node_groups'
2014-07-09 05:42:12.390 19264 TRACE sahara.service.edp.job_manager

[Robert Levas (rlevas)]

It appears that there is an error in hdfs_helper.py:

~~~~~~
def _get_cluster_hosts_information(host, cluster):
    for clust in conductor.cluster_get_all(context.ctx()):
        if clust.id == cluster.id:
            continue

        for i in u.get_instances(***c***):
            if i.instance_name == host:
                return g.generate_etc_hosts(***c***)

    return None
~~~~~~

Should probably be:

~~~~~~
def _get_cluster_hosts_information(host, cluster):
    for clust in conductor.cluster_get_all(context.ctx()):
        if clust.id == cluster.id:
            continue

        for i in u.get_instances(***clust***):
            if i.instance_name == host:
                return g.generate_etc_hosts(***clust***)

    return None
~~~~~~

[Andrew Lazarev (alazarev)]

yes, but was introduced in pep8 fix: https://review.openstack.org/#/c/101845/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/106305

[Sergey Lukjanov (slukjanov)]

Sofiia, please, don't forget to assign issues on yourself when you're starting working on them.

[Sergey Lukjanov (slukjanov)]

Fixed in https://review.openstack.org/#/c/105778/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sergey Lukjanov (<email address hidden>) on branch: master
Review: https://review.openstack.org/106305
Reason: Fixed in https://review.openstack.org/#/c/105778/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/105778
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=485310e58d1f24c8cc383690dc75ffb3cb0c187d
Submitter: Jenkins
Branch: master

commit 485310e58d1f24c8cc383690dc75ffb3cb0c187d
Author: Sofiia Kostiuchenko <email address hidden>
Date: Wed Jul 9 19:07:18 2014 +0300

    Fixed EDP job execution failure

    Changed name of variable to fix an attribute error
    which Sahara throws right after you start an EDP job

    Closes-Bug: #1339743

    Change-Id: Idefaad6185ef5cd92d3c0ae2fbda1548473f24c6