EDP does not accept Swift container as collection of job binaries

[Dmitry Mescheryakov]

We have code which accepts Swift container as a storage of job binaries, here:
https://github.com/openstack/sahara/blob/93b79027acd659bf007cdd7f5f79785c0c86adf3/sahara/service/edp/binary_retrievers/internal_swift.py#L54

It is expected that we will take all the files inside container as job binaries. But the code does not work. The following exception occurs at a later stage:

2014-04-11 13:33:39.680 61774 TRACE sahara.context Traceback (most recent call last):
2014-04-11 13:33:39.680 61774 TRACE sahara.context File "/Users/dmitryme/projects/elastic-hadoop/sahara/sahara/context.py", line 124, in _wrapper
2014-04-11 13:33:39.680 61774 TRACE sahara.context func(*args, **kwargs)
2014-04-11 13:33:39.680 61774 TRACE sahara.context File "/Users/dmitryme/projects/elastic-hadoop/sahara/sahara/service/edp/job_manager.py", line 144, in run_job
2014-04-11 13:33:39.680 61774 TRACE sahara.context upload_job_files(oozie_server, wf_dir, job, hdfs_user)
2014-04-11 13:33:39.680 61774 TRACE sahara.context File "/Users/dmitryme/projects/elastic-hadoop/sahara/sahara/service/edp/job_manager.py", line 194, in upload_job_files
2014-04-11 13:33:39.680 61774 TRACE sahara.context hdfs_user)
2014-04-11 13:33:39.680 61774 TRACE sahara.context File "/Users/dmitryme/projects/elastic-hadoop/sahara/sahara/service/edp/hdfs_helper.py", line 30, in put_file_to_hdfs
2014-04-11 13:33:39.680 61774 TRACE sahara.context r.write_file_to('/tmp/%s' % file_name, file)
2014-04-11 13:33:39.680 61774 TRACE sahara.context File "/Users/dmitryme/projects/elastic-hadoop/sahara/sahara/utils/ssh_remote.py", line 371, in write_file_to
2014-04-11 13:33:39.680 61774 TRACE sahara.context self._run_s(_write_file_to, timeout, remote_file, data, run_as_root)
2014-04-11 13:33:39.680 61774 TRACE sahara.context File "/Users/dmitryme/projects/elastic-hadoop/sahara/sahara/utils/ssh_remote.py", line 428, in _run_s
2014-04-11 13:33:39.680 61774 TRACE sahara.context return self._run_with_log(func, timeout, *args, **kwargs)
2014-04-11 13:33:39.680 61774 TRACE sahara.context File "/Users/dmitryme/projects/elastic-hadoop/sahara/sahara/utils/ssh_remote.py", line 334, in _run_with_log
2014-04-11 13:33:39.680 61774 TRACE sahara.context return self._run(func, *args, **kwargs)
2014-04-11 13:33:39.680 61774 TRACE sahara.context File "/Users/dmitryme/projects/elastic-hadoop/sahara/sahara/utils/ssh_remote.py", line 425, in _run
2014-04-11 13:33:39.680 61774 TRACE sahara.context return procutils.run_in_subprocess(self.proc, func, args, kwargs)
2014-04-11 13:33:39.680 61774 TRACE sahara.context File "/Users/dmitryme/projects/elastic-hadoop/sahara/sahara/utils/procutils.py", line 52, in run_in_subprocess
2014-04-11 13:33:39.680 61774 TRACE sahara.context raise SubprocessException(result['exception'])
2014-04-11 13:33:39.680 61774 TRACE sahara.context SubprocessException: TypeError: Expected unicode or bytes, got {'edp-lib.jar': '<binary data here>'}

I think we should consider enhancing getting binaries from container to allow getting all binaries with some prefix.

[Trevor McKay (tmckay)]

There is a secondary issue here, too, which maybe can be handled in the same bug. If the path does not exist, Sahara gets an exception when it tries to make the HEAD request to swift. An invalid path should be handled and the job execution should be moved to "KILLED".

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/132344

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/132344
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=7054a5089f206ff2fdb336b8f2498af59d089f83
Submitter: Jenkins
Branch: master

commit 7054a5089f206ff2fdb336b8f2498af59d089f83
Author: Michael McCune <email address hidden>
Date: Fri Oct 31 16:00:26 2014 -0400

    Removing Swift container support for job binaries

    This change removes the ability to request an entire container as a
    source for a job binary. The reason for this removal is that the
    internal code paths for retrieving job binary raw data from Swift will
    need to be refactored to support this functionality. The scope of the
    changes needed will impact the REST API and the concepts surrounding how
    job binaries are aquired. Allowing full containers should be discussed
    in a new specification to add the feature.

    Changes
    * removing container support from get_raw_data.
    * moving the functions to acquire a Swift client into a more appropriate
      place.
    * removing context argument from internal_swift.get_raw_data
    * creating tests for internal swift get_raw_data function

    Change-Id: I666873038a323e63d17b8a24b4213ed0240a4c59
    Closes-Bug: #1306505