Steps to repro:
1. Run EDP job
Logs:
2014-08-13 13:53:13.017 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] _execute_command took 5.4 seconds to complete _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:13.017 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] Executing "sudo rm -f /tmp/example.pig.6507c3cb-8720-4dd7-a8c5-946621c025a7" _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:14.075 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] _execute_command took 1.1 seconds to complete _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:14.182 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] Executing "sudo su - -c "hadoop dfs -test -e /" hdfs" _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:18.399 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] _execute_command took 4.2 seconds to complete _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:18.399 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] Executing "sudo su - -c "hadoop dfs -test -e /user/" hdfs" _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:22.982 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] _execute_command took 4.6 seconds to complete _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:22.983 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] Executing "sudo su - -c "hadoop dfs -test -e /user/hdfs/" hdfs" _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:27.795 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] _execute_command took 4.8 seconds to complete _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:27.795 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] Executing "sudo su - -c "hadoop dfs -test -e /user/hdfs/test/" hdfs" _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:32.319 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] _execute_command took 4.5 seconds to complete _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:32.319 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] Executing "sudo su - -c "hadoop dfs -test -e /user/hdfs/test/1048e374-b4e4-402e-a57f-631c57e5b0fe/" hdfs" _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:36.492 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] _execute_command took 4.2 seconds to complete _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:36.493 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] Executing "sudo su - -c "hadoop dfs -test -e /user/hdfs/test/1048e374-b4e4-402e-a57f-631c57e5b0fe/lib/" hdfs" _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:40.920 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] _execute_command took 4.4 seconds to complete _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
2014-08-13 13:53:40.921 5305 DEBUG sahara.utils.ssh_remote [-] [al-test-master-001] Executing "sudo su - -c "hadoop dfs -mkdir /user/hdfs/test/1048e374-b4e4-402e-a57f-631c57e5b0fe/lib/" hdfs" _log_command /Users/andrew/Sahara/sahara/sahara/utils/ssh_remote.py:459
There is https:/ /issues. apache. org/jira/ browse/ HADOOP- 8873
Mkdir behavior changed between 1.2.1 and 2.0.0. In 1x "-mkdir -p" will create "-p" dir. In 2x "-mkdir -p" is a flag to create recursively.