Parallel running EDP job failed

Bug #1351276 reported by Sergey Reshetnyak
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Sahara
Fix Released
Critical
Sergey Reshetnyak
Icehouse
Fix Released
High
Sergey Reshetnyak

Bug Description

Step to reproduce:
1. Run 4 EDP job in one time.
2. Get an error:

2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager Traceback (most recent call last):
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/service/edp/job_manager.py", line 114, in run_job
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager _run_job(job_execution_id)
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/service/edp/job_manager.py", line 105, in _run_job
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager jid = eng.run_job(job_execution)
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/service/edp/oozie/engine.py", line 93, in run_job
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager wf_xml, hdfs_user)
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/service/edp/oozie/engine.py", line 57, in _upload_workflow_file
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager h.put_file_to_hdfs(r, wf_xml, "workflow.xml", job_dir, hdfs_user)
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/service/edp/hdfs_helper.py", line 32, in put_file_to_hdfs
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager hdfs_user)
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/service/edp/hdfs_helper.py", line 46, in move_from_local
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager '%s %s" %s' % (source, target, hdfs_user))
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/utils/ssh_remote.py", line 400, in execute_command
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager get_stderr, raise_when_error)
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/utils/ssh_remote.py", line 469, in _run_s
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager return self._run_with_log(func, timeout, *args, **kwargs)
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/utils/ssh_remote.py", line 366, in _run_with_log
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager return self._run(func, *args, **kwargs)
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/utils/ssh_remote.py", line 466, in _run
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager return procutils.run_in_subprocess(self.proc, func, args, kwargs)
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager File "/usr/local/lib/python2.7/dist-packages/sahara/utils/procutils.py", line 52, in run_in_subprocess
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager raise SubprocessException(result['exception'])
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager SubprocessException: RemoteCommandException: Error during command execution: "sudo su - -c "hadoop dfs -copyFromLocal /tmp/workflow.xml /user/hadoop/Edp-test-job-7d42cf42/f49dbbc1-84db-4554-be1a-8ea06dd1fad0/workflow.xml" hadoop"
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager Return code: 1
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager STDERR:
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager DEPRECATED: Use of this script to execute hdfs command is deprecated.
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager Instead use the hdfs command for it.
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager copyFromLocal: `/tmp/workflow.xml': No such file or directory
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager
2014-08-01 09:31:46.058 3378 TRACE sahara.service.edp.job_manager

Changed in sahara:
assignee: nobody → Sergey Reshetnyak (sreshetniak)
Changed in sahara:
status: New → Confirmed
importance: Undecided → Critical
milestone: none → juno-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to sahara (master)

Fix proposed to branch: master
Review: https://review.openstack.org/111247

Changed in sahara:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to sahara (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/111268

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to sahara (master)

Reviewed: https://review.openstack.org/111247
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=9a9b14d8a068c6b32a6b335cc1b006e8cb23c41d
Submitter: Jenkins
Branch: master

commit 9a9b14d8a068c6b32a6b335cc1b006e8cb23c41d
Author: Sergey Reshetnyak <email address hidden>
Date: Fri Aug 1 15:31:31 2014 +0400

    Fix put_file_to_hdfs method in hdfs_helper

    Use random file name for temp file

    Change-Id: I4be62d7c9da6948449f446c6410cc46a14d64021
    Closes-bug: #1351276

Changed in sahara:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to sahara (stable/icehouse)

Reviewed: https://review.openstack.org/111268
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=31089652a8780d016763440e872655692556078e
Submitter: Jenkins
Branch: stable/icehouse

commit 31089652a8780d016763440e872655692556078e
Author: Sergey Reshetnyak <email address hidden>
Date: Fri Aug 1 15:31:31 2014 +0400

    Fix put_file_to_hdfs method in hdfs_helper

    Use random file name for temp file

    Change-Id: I4be62d7c9da6948449f446c6410cc46a14d64021
    Closes-bug: #1351276
    (cherry picked from commit 9a9b14d8a068c6b32a6b335cc1b006e8cb23c41d)

Thierry Carrez (ttx)
Changed in sahara:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in sahara:
milestone: juno-3 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.