Parallel running EDP job failed

[Sergey Reshetnyak]

Step to reproduce:
1. Run 4 EDP job in one time.
2. Get an error:

2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager Traceback (most recent call last):
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/service/edp/job_manager.py", line 114, in run_job
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager _run_job(job_execution_id)
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/service/edp/job_manager.py", line 105, in _run_job
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager jid = eng.run_job(job_execution)
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/service/edp/oozie/engine.py", line 93, in run_job
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager wf_xml, hdfs_user)
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/service/edp/oozie/engine.py", line 57, in _upload_workflow_file
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager h.put_file_to_hdfs(r, wf_xml, "workflow.xml", job_dir, hdfs_user)
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/service/edp/hdfs_helper.py", line 32, in put_file_to_hdfs
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager hdfs_user)
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/service/edp/hdfs_helper.py", line 46, in move_from_local
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager '%s %s" %s' % (source, target, hdfs_user))
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/utils/ssh_remote.py", line 400, in execute_command
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager get_stderr, raise_when_error)
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/utils/ssh_remote.py", line 469, in _run_s
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager return self._run_with_log(func, timeout, *args, **kwargs)
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/utils/ssh_remote.py", line 366, in _run_with_log
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager return self._run(func, *args, **kwargs)
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/utils/ssh_remote.py", line 466, in _run
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager return procutils.run_in_subprocess(self.proc, func, args, kwargs)
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/utils/procutils.py", line 52, in run_in_subprocess
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager raise SubprocessException(result['exception'])
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager SubprocessException: RemoteCommandException: Error during command execution: "sudo su - -c "hadoop dfs -copyFromLocal /tmp/workflow.xml /user/hadoop/Edp-test-job-7d42cf42/f49dbbc1-84db-4554-be1a-8ea06dd1fad0/workflow.xml" hadoop"
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager Return code: 1
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager STDERR:
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager DEPRECATED: Use of this script to execute hdfs command is deprecated.
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager Instead use the hdfs command for it.
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager copyFromLocal: `/tmp/workflow.xml': No such file or directory
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/111247

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/111268

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/111247
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=9a9b14d8a068c6b32a6b335cc1b006e8cb23c41d
Submitter: Jenkins
Branch: master

commit 9a9b14d8a068c6b32a6b335cc1b006e8cb23c41d
Author: Sergey Reshetnyak <email address hidden>
Date: Fri Aug 1 15:31:31 2014 +0400

    Fix put_file_to_hdfs method in hdfs_helper

    Use random file name for temp file

    Change-Id: I4be62d7c9da6948449f446c6410cc46a14d64021
    Closes-bug: #1351276

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/111268
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=31089652a8780d016763440e872655692556078e
Submitter: Jenkins
Branch: stable/icehouse

commit 31089652a8780d016763440e872655692556078e
Author: Sergey Reshetnyak <email address hidden>
Date: Fri Aug 1 15:31:31 2014 +0400

    Fix put_file_to_hdfs method in hdfs_helper

    Use random file name for temp file

    Change-Id: I4be62d7c9da6948449f446c6410cc46a14d64021
    Closes-bug: #1351276
    (cherry picked from commit 9a9b14d8a068c6b32a6b335cc1b006e8cb23c41d)