Fail to execute spark job on xenial vanilla image

[Shu Yingya]

env: xenial ubuntu vanilla image built by SIE master branch
step:
1. run command in openstack controller node
----
openstack --debug dataprocessing job execute --job-template spark-job-template --cluster old --configs edp.java.main_class:sahara.edp.spark.SparkWordCount edp.spark.adapt_for_swift:true fs.swift.service.sahara.username:admin fs.swift.service.sahara.password:admin --args swift://edp-example/sample_input.txt swift://edp-example/output
----

2. The job will finish in status "Done with Error"
3. ssh into the master node and execute:
----
./launch_command sudo -u hadoop /opt/spark/bin/spark-submit --driver-class-path /opt/hadoop/share/hadoop/tools/lib/hadoop-openstack-2.7.1.jar --files spark.xml --class org.openstack.sahara.edp.SparkWrapper --jars builtin-5dfc694d-b447-49cd-95e0-9347e4a7d411.jar --master yarn --deploy-mode cluster /tmp/spark-jb spark.xml sahara.edp.spark.SparkWordCount swift://edp-example.sahara/sample_input.txt swift://edp-example.sahara/output
----

===Error message
ubuntu The program 'python' can be found in the following packages:
===
analysis:
spark job would fail to execute because ubuntu xenial doesn't preinstall python2.7. and the the file located at "https://github.com/openstack/sahara/blob/master/sahara/service/edp/resources/launch_command.py" start with shebang "#!/usr/bin/env python". we should run "apt install python-minimal" to install python.

[Shu Yingya (felixshu)]

not sure "apt install python-minimal" is a good way to solve it.
but at least it is unreasonable to change the shebang in "launch_command.py" script.

[Telles Mota Vidal Nóbrega (tellesmvn)]

We could add installation of python as a step during image creation.

[Luigi Toscano (ltoscano)]

Or make sure that the script works with python3 and, if python3 is available, call the script using python3 (I think that it should override the shebang value).
We would need to have everything working with python3 anyway really soon.

[Shu Yingya (felixshu)]

right now, there is an IndexError while using python3 subprocess package like this:
2017-12-19 11:03:49,841 ERROR list index out of range
Traceback (most recent call last):
  File "./launch_command", line 76, in <module>
    stderr=open("stderr", "w"))
  File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.5/subprocess.py", line 1448, in _execute_child
    executable = args[0]
IndexError: list index out of range

But you are right, we should migrate to python3 ASAP

[Jeremy Freudberg (jfreud)]

https://github.com/openstack/sahara/blob/84c1ea5924024f0c5ed34849bdb4aa4d0145b81f/sahara/service/edp/spark/engine.py#L111

https://github.com/openstack/sahara/blob/8fec6571a1438399f89431bb3c3e18177a7c1703/sahara/utils/ssh_remote.py#L361

we could dynamically edit the script based on what python is available (using that existing helper function).

or we could just modify the images.

btw, i think this affects storm plugin on xenial as well.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/537228

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/537228
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=d02e61aa68ca569ecc465c47eab803a68fe97033
Submitter: Zuul
Branch: master

commit d02e61aa68ca569ecc465c47eab803a68fe97033
Author: Shu Yingya <email address hidden>
Date: Wed Jan 24 10:41:22 2018 +0800

    Dynamically add python version into launch_command

    Ubuntu Xenial or later server won't install Python2 anymore by default.
    Sahara should have the ability to dynamically edit the remotely executed
    python script based on what python is available.

    Change-Id: Ie0fdd829d1b0ff019329957fbdbbfd150320b8ab
    Closes-Bug: #1739009

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/sahara 8.0.0.0b3 development milestone.