HDP image failures/warnings in boot.log for VM

[Robert Nettleton]

This bug describes a minor issue in the HDP images in sahara-image-elements.

Some error messages and warnings are occurring in a VM instance based on either of the HDP images (HDP 1.3.2 or HDP 2.0.6).

The warnings are as follows:

"Starting ambari-server
ERROR: Exiting with exit code 1.
REASON: Unable to detect a system user for Ambari Server.
- If this is a new setup, then run the "ambari-server setup" command to create the user
- If this is an upgrade of an existing setup, run the "ambari-server upgrade" command.
Refer to the Ambari documentation for more information on setup and upgrade.
Starting atd: [ OK ]
/etc/rc3.d/S95hadoop-mapreduce-historyserver: line 40: /usr/lib/bigtop-utils/bigtop-detect-javahome: No such file or directory
Starting Hadoop historyserver:[ OK ]
Error: JAVA_HOME is not set and could not be found.
/etc/rc3.d/S95hadoop-yarn-nodemanager: line 40: /usr/lib/bigtop-utils/bigtop-detect-javahome: No such file or directory
Starting Hadoop nodemanager:[ OK ]
Error: JAVA_HOME is not set and could not be found.
/etc/rc3.d/S95hadoop-yarn-proxyserver: line 40: /usr/lib/bigtop-utils/bigtop-detect-javahome: No such file or directory
Starting Hadoop proxyserver:[ OK ]
Error: JAVA_HOME is not set and could not be found.
/etc/rc3.d/S95hadoop-yarn-resourcemanager: line 40: /usr/lib/bigtop-utils/bigtop-detect-javahome: No such file or directory
Starting Hadoop resourcemanager:[ OK ]
Error: JAVA_HOME is not set and could not be found."

The ambari-server service starts up upon first boot, and runs into errors, since it has not been setup yet with the "ambari-server setup" command. The setup command is usually run by the HDP plugin during cluster startup. This problem occurs with both versions of HDP in these images.

The other errors are only specific to the HDP 2.0.6 image. Some of the Hadoop 2.x package installs now include automatic startup of a set of Hadoop services. These services attempt to startup during the first boot of the instance. This startup fails, since the Hadoop services have not yet been configured by Ambari.

These errors/warnings are really only a cosmetic issue. Once HDP plugin configures ambari-server, which in turn configures the Hadoop services, these problems do not actually occur during cluster startup/runtime. This issue only occurs upon the first boot of the image.

Even though this is a cosmetic issue, it should be resolved, since these errors can be confusing to users as they try to debug failing cluster startups.

I have a fix for this implemented, and will be submitting a patch soon to resolve this.

[Robert Nettleton (rnettleton)]

More comments about the fix required:

The ambari-server and ambari-agent services will be turned off by default, using the chkconfig command.

The hadoop services listed in these warnings/errors will be turned off by default, using the chkconfig command.

Turning these services off by default will only affect the first boot, and will not modify the behavior of cluster startups.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/92715

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/92715
Committed: https://git.openstack.org/cgit/openstack/sahara-image-elements/commit/?id=b97a026e450701c4b21f15ab9ecfa911ac4268c3
Submitter: Jenkins
Branch: master

commit b97a026e450701c4b21f15ab9ecfa911ac4268c3
Author: Bob Nettleton <email address hidden>
Date: Wed May 7 17:19:11 2014 -0400

    Turns off ambari and hadoop2 services for initial VM boot in HDP images

    This patch resolves a minor bug in the HDP image generation scripts.

    Some warning and error messages show up in the boot.log for VM
      instances based on the HDP images (v1.3.2 and v2.0.6). The
      warnings involve the ambari-server startup, as well as the
      startup of some Hadoop 2 services in the v2.0.6 image.

    The basic problem is that these services should not be starting
      upon the first boot of the image, since the HDP plugin has
      not yet configured the VM instances for a cluster startup.

    The ambari-server and ambari-agent services have been turned off
      by default in this patch, by using the chkconfig command. This keeps
      the ambari services from starting at first boot. This is desired,
      since the Ambari services don't have the configuration required
      during initial boot to function properly.

    The same issue occurs with the Hadoop 2 services, and these are
      turned off using the same chkconfig mechanism.

    Closes-Bug: #1317270

    Change-Id: I5fdd45a41b2e6bb8f9b632594bbd6ee538e09512