fail to start vanilla cluster with image built by latest sahara-image-element script

[Shu Yingya]

using latest sahara-image-elements project(lastest commit is f3f5613238a58a43281c917a49ca23eae65db16e) to build a vanilla image, the cluster can't successfully start.

sahara-engine.log:
Error ID: a777b7bc-2940-4c0a-838d-d6480f9fa866
Error ID: 19960099-615b-40b1-8267-6bd7732626ea
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/sahara/context.py", line 167, in _wrapper
    func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/sahara/utils/cluster_progress_ops.py", line 139, in handler
    add_fail_event(instance, e)
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/site-packages/sahara/utils/cluster_progress_ops.py", line 136, in handler
    value = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/sahara/plugins/vanilla/hadoop2/run_scripts.py", line 61, in _start_processes
    'sudo su - -c "hadoop-daemon.sh start datanode" hadoop')
  File "/usr/lib/python2.7/site-packages/sahara/utils/ssh_remote.py", line 855, in execute_command
    cmd, run_as_root, get_stderr, raise_when_error)
  File "/usr/lib/python2.7/site-packages/sahara/utils/ssh_remote.py", line 955, in _run_s
    return self._run_with_log(func, timeout, description, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/sahara/utils/ssh_remote.py", line 766, in _run_with_log
    return self._run(func, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/sahara/utils/ssh_remote.py", line 951, in _run
    return procutils.run_in_subprocess(self.proc, func, args, kwargs)
  File "/usr/lib/python2.7/site-packages/sahara/utils/procutils.py", line 54, in run_in_subprocess
    raise exceptions.SubprocessException(result['exception'])
SubprocessException: RemoteCommandException: Error during command execution: "sudo su - -c "hadoop-daemon.sh start datanode" hadoop"
Return code: 1
STDOUT:
starting datanode, logging to /mnt/log/hadoop/hadoop/hadoop-hadoop-datanode-origin-vanilla-vw-1.novalocal.out

and the log in :/mnt/log/hadoop/hadoop/hadoop-hadoop-datanode-origin-vanilla-vw-1.novalocal.log
is http://paste.openstack.org/show/624724/

[Shi Yan (yanshi-403)]

I have the exact same issue when using the latest script to build the vanilla image.

Further, if using the old vanilla image(ocata), it is working fine.

[Shu Yingya (felixshu)]

If you would like to build the latest image, please revert the change "a77a9a978a655044a0b58a299df965c89391090d" util this patch has been fixed.

[Luigi Toscano (ltoscano)]

Uhm, the commit a77a9a978a655044a0b58a299df965c89391090d is "Add S3 jar to Hadoop classpath"; Jeremy, could it be the change to DIB_HDFS_LIB_DIR?

If anyone has some bandwidth to test it, can you please try to only change back the value of DIB_HDFS_LIB_DIR to

export DIB_HDFS_LIB_DIR="/opt/hadoop/share/hadoop/tools/lib"

in diskimage-create/diskimage-create.sh, keeping the rest of the change?

[Shu Yingya (felixshu)]

Thanks for your hint, Luigi. Let me try it on my env.

[Shi Yan (yanshi-403)]

Hi, Luigi and Yingya

After I rebuild the vanilla image with the only change to DIB_HDFS_LIB_DIR, as Luigi indicated here, I test it and it works well!

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/522990
Committed: https://git.openstack.org/cgit/openstack/sahara-image-elements/commit/?id=3ee20cbc09fc0fcfb91c81382747470de7050a03
Submitter: Zuul
Branch: master

commit 3ee20cbc09fc0fcfb91c81382747470de7050a03
Author: Jeremy Freudberg <email address hidden>
Date: Sun Nov 26 21:47:27 2017 +0000

    Revise s3_hadoop

    * Handle Hadoop classpath better
    * Include proper support for Spark classpath
    * Formally limit element's use to Vanilla and Spark

    Change-Id: I65abd7e375dba11599a4ab943d24f878235cd71d
    Closes-Bug: #1727757
    Closes-Bug: #1728061

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/542369

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/542369
Committed: https://git.openstack.org/cgit/openstack/sahara-image-elements/commit/?id=779c2f57047a9e326ef4c8175f319f8940eef489
Submitter: Zuul
Branch: stable/pike

commit 779c2f57047a9e326ef4c8175f319f8940eef489
Author: Jeremy Freudberg <email address hidden>
Date: Sun Nov 26 21:47:27 2017 +0000

    Revise s3_hadoop

    * Handle Hadoop classpath better
    * Include proper support for Spark classpath
    * Formally limit element's use to Vanilla and Spark

    Change-Id: I65abd7e375dba11599a4ab943d24f878235cd71d
    Closes-Bug: #1727757
    Closes-Bug: #1728061

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/sahara-image-elements 8.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/sahara-image-elements 7.0.2 release.