[EDP] Sahara makes 8 round-trips to VM to create hdfs dir

[Andrew Lazarev]

Steps to repro:
1. Run EDP job

Logs:
2014-08-13 13:53:13.017 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] _execute_command took 5.4 seconds to complete _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:13.017 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] Executing "sudo rm -f /tmp/example.pig.6507c3cb-8720-4dd7-a8c5-946621c025a7" _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:14.075 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] _execute_command took 1.1 seconds to complete _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:14.182 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] Executing "sudo su - -c "hadoop dfs -test -e /" hdfs" _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:18.399 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] _execute_command took 4.2 seconds to complete _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:18.399 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] Executing "sudo su - -c "hadoop dfs -test -e /user/" hdfs" _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:22.982 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] _execute_command took 4.6 seconds to complete _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:22.983 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] Executing "sudo su - -c "hadoop dfs -test -e /user/hdfs/" hdfs" _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:27.795 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] _execute_command took 4.8 seconds to complete _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:27.795 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] Executing "sudo su - -c "hadoop dfs -test -e /user/hdfs/test/" hdfs" _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:32.319 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] _execute_command took 4.5 seconds to complete _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:32.319 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] Executing "sudo su - -c "hadoop dfs -test -e /user/hdfs/test/1048e374-b4e4-402e-a57f-631c57e5b0fe/" hdfs" _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:36.492 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] _execute_command took 4.2 seconds to complete _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:36.493 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] Executing "sudo su - -c "hadoop dfs -test -e /user/hdfs/test/1048e374-b4e4-402e-a57f-631c57e5b0fe/lib/" hdfs" _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:40.920 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] _execute_command took 4.4 seconds to complete _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:40.921 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] Executing "sudo su - -c "hadoop dfs -mkdir /user/hdfs/test/1048e374-b4e4-402e-a57f-631c57e5b0fe/lib/" hdfs" _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459

[Andrew Lazarev (alazarev)]

There is https://issues.apache.org/jira/browse/HADOOP-8873
Mkdir behavior changed between 1.2.1 and 2.0.0. In 1x "-mkdir -p" will create "-p" dir. In 2x "-mkdir -p" is a flag to create recursively.

[Andrew Lazarev (alazarev)]

More direct link: https://issues.apache.org/jira/browse/HADOOP-8551

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/115115

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/115120

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/115120
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=44c00a6e95db3573c25396b4627495f83613edda
Submitter: Jenkins
Branch: master

commit 44c00a6e95db3573c25396b4627495f83613edda
Author: Andrew Lazarev <email address hidden>
Date: Mon Aug 18 14:32:05 2014 -0700

    Removed one round trip to server for HDFS put

    Change-Id: Ie0fc0d486ef351cf78513b398c810ac8133d35be
    Partial-Bug: #1356582

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/115115
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=c1db199530c3adeecc9f2e5c379089c76d07b50f
Submitter: Jenkins
Branch: master

commit c1db199530c3adeecc9f2e5c379089c76d07b50f
Author: Andrew Lazarev <email address hidden>
Date: Mon Aug 18 13:45:36 2014 -0700

    Added create_hdfs_dir method to oozie edp engine

    Adding abstract create_hdfs_dir to base oozie edp engine allows to
    differenciate hadoop1 and hadoop2 implementations and use different
    code for different hadoops. This is done to remove ugly hacks around
    hdfs communication made to work with any hdfs implementation.

    Change-Id: I8b5f59e6bfba6d5ac24cf53e9cba1de8bd034c90
    Closes-Bug: #1355114
    Closes-Bug: #1356582
    Partial-Bug: #1357512