Sahara

[EDP][Oozie] Configuration of cluster for external hdfs missed for URLs in job_configs

Bug #1425731 reported by Trevor McKay
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Sahara
Fix Released
High
lu huichun
Sahara 3.0.0 "liberty"

Bug Description

If a job is run on data sources that reference an hdfs host not listed in the /etc/hosts file of cluster instances, the Oozie EDP engine will update the /etc/hosts files accordingly.

However, this only works for hdfs URLs referenced via data sources. If for example a Java job is run with manually specified input or output URLs that reference an external hdfs, the Oozie EDP engine will not update the /etc/hosts files.

The engine should look for hdfs URLs contained in job_configs and check for necessary host file updates.

Trevor McKay (tmckay)
summary: - [EDP] Configuration of cluster for external hdfs missed for URLs in
- job_configs
+ [EDP][Oozie] Configuration of cluster for external hdfs missed for URLs
+ in job_configs
lu huichun (lhcxx0508)
Changed in sahara:
assignee: nobody → lu huichun (lhcxx0508)
Sergey Lukjanov (slukjanov)
Changed in sahara:
milestone: none → liberty-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to sahara (master)

Fix proposed to branch: master
Review: https://review.openstack.org/192038

Changed in sahara:
status: New → In Progress
Sergey Lukjanov (slukjanov)
Changed in sahara:
milestone: liberty-1 → liberty-2
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to sahara (master)

Reviewed: https://review.openstack.org/192038
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=9d1dead63d9d26fa0dc979f09a210e2f6ab98a31
Submitter: Jenkins
Branch: master

commit 9d1dead63d9d26fa0dc979f09a210e2f6ab98a31
Author: luhuichun <email address hidden>
Date: Tue Jun 16 10:03:06 2015 +0800

    [EDP][Oozie] external hdfs missed for URLs in job_configs

    If a job is run on data sources that reference an hdfs host not
    listed in the /etc/hosts file of cluster instances, the Oozie EDP
    engine will update the /etc/hosts files accordingly.However, this
    only works for hdfs URLs referenced via data sources. If for
    example a Java job is run with manually specified input or output
    URLs that reference an external hdfs, the Oozie EDP engineu will
    not update the /etc/hosts files.

    Change-Id: I42a89a347cdb1bd6cc9d8b8c2b03b8a09d2e17b9
    Closes-bug: 1425731

Changed in sahara:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to sahara (master)

Fix proposed to branch: master
Review: https://review.openstack.org/203528

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to sahara (master)

Reviewed: https://review.openstack.org/203528
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=d2213c9b9f4876b21673ea97693fa7b66ba65e7c
Submitter: Jenkins
Branch: master

commit d2213c9b9f4876b21673ea97693fa7b66ba65e7c
Author: luhuichun <email address hidden>
Date: Mon Jul 20 16:30:39 2015 +0800

    Add unit test for external hdfs missed for URLs

    Change-Id: Ibcdec64ab206248984db375be370e0f643501707
    Closes-bug: 1425731

Thierry Carrez (ttx)
Changed in sahara:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in sahara:
milestone: liberty-2 → 3.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.