If you are using Oozie, you need to use the Nameservice URI instead of the NameNode URI in your workflow files. For example, where the Nameservice ID is mycluster:
From sahara/sahara/plugins/hdp/edp_engine.py:
def get_name_node_uri(self, cluster):
return cluster['info']['HDFS']['NameNode']
It seems that we've succeeded in setting up a highly available cluster, but we're hard-coding ourselves into only using Oozie through one of the nodes, rather than using the nameservice; as the [info][HDFS][NameNode] is hardcoded as of cluster creation time, and the feature did not build any sense of nameservice into Sahara.
From http:// docs.hortonwork s.com/HDPDocume nts/HDP1/ HDP-1.3. 3/bk_using_ Ambari_ book/content/ install- ha_2x.html:
If you are using Oozie, you need to use the Nameservice URI instead of the NameNode URI in your workflow files. For example, where the Nameservice ID is mycluster:
<workflow-app xmlns=" uri:oozie: workflow: 0.2" name="map- reduce- wf">
<map- reduce>
<job-tracker> ${jobTracker} </job-tracker>
<name-node> hdfs:// mycluster< /name-node>
<start to="mr-node"/>
<action name="mr-node">
From http:// 172.24. 4.230:8080/ #/main/ hosts/my- hdp2ha- b090a511- master- ha-common- f2ebfbc6- 001.novalocal/ configs (which pulls from hdfs-site.xml):
dfs.nameservices: my-hdp2ha-b090a511
From your cluster definition (which should be valid): //172.24. 4.229:8020' , u'Web UI': | 172.24. 4.229:50070'}, u'JobFlow': | 172.24. 4.230:11000'}, | 172.24. 4.230:19888', u'History | 4.230:10020' }, u'Yarn': | 172.24. 4.230:8088', | 4.230:8050' }, | 172.24. 4.230:8080'}} |
| info | {u'HDFS': {u'NameNode': |
| | u'hdfs:
| | u'http://
| | {u'Oozie': u'http://
| | u'MapReduce2': {u'Web UI': |
| | u'http://
| | Server': u'172.24.
| | {u'Web UI': u'http://
| | u'ResourceManager': u'172.24.
| | u'Ambari Console': {u'Web UI': |
| | u'http://
From sahara/ sahara/ service/ edp/oozie/ engine. py: name_node_ uri(self. cluster)
job_parameters = {
"jobTracke r": rm_path,
"nameNode" : nn_path,
"user. name": hdfs_user,
oozie_ libpath_ key: oozie_libpath,
app_ path: "%s%s" % (nn_path, path_to_workflow),
"oozie. use.system. libpath" : "true"}
nn_path = self.get_
...
From sahara/ sahara/ plugins/ hdp/edp_ engine. py: node_uri( self, cluster): 'info'] ['HDFS' ]['NameNode' ]
def get_name_
return cluster[
It seems that we've succeeded in setting up a highly available cluster, but we're hard-coding ourselves into only using Oozie through one of the nodes, rather than using the nameservice; as the [info][ HDFS][NameNode] is hardcoded as of cluster creation time, and the feature did not build any sense of nameservice into Sahara.