[Sahara] Cluster fails to deploy, CDH 5.4.0, Ubuntu

[Fabrizio Soppelsa]

Detailed bug description: CDH 5.4.0 with Ubuntu cluster fails to deploy due to: Provision Hadoop Cluster: Command aborted because of exception: Command timed-out after 90 seconds. It was observed that with CentOS, cluster got created (but not reproduced in Mirantis labs)
Steps to reproduce: Create typical 5.4.0 Cluster Template, and create a cluster with Ubuntu baseimage
Expected results: Cluster deploys
Actual result: Cluster fails to deploy
Reproducibility: Yes
Workaround: Unknown yet
Impact: Impossible to deploy a cluster
Description of the environment:
- Operating system: Ubuntu 14.04
- Versions of components: MOS 7.0
- Reference architecture: HA, Neutron DVR, l2pop (if that matters)
- Network model: VXLAN (if that matters)
- Related projects installed: Sahara
Additional information:

2016-03-31 10:00:00.074 2219 INFO sahara.utils.general [-] Cluster status has been changed: id=c2d1cfe3-d1a0-4d9f-82f4-4df086e9b960, New status=Starting
2016-03-31 10:05:16.352 2219 ERROR sahara.service.ops [-] Error during operating on cluster test-ubuntu (reason: Failed to Provision Hadoop Cluster: Command aborted because of exception: Command timed-out after 90 seconds
Error ID: d3d03a37-b6f3-40bb-b32e-ea13fa9de5ee)
2016-03-31 10:05:17.078 2219 INFO sahara.utils.general [-] Cluster status has been changed: id=c2d1cfe3-d1a0-4d9f-82f4-4df086e9b960, New status=Error

We have a hardware lab with this error reproduced and logs, contact me for access.

[Nikita Konovalov (nkonovalov)]

This is a customer-found issue

[masafumi_ohta (masafumi-ohta)]

see the comment Cloudera employee added.

http://community.cloudera.com/t5/Cloudera-Manager-Installation/CDH-cluster-command-timed-out-after-90-seconds/td-p/39167

[masafumi_ohta (masafumi-ohta)]

I have been looking into the issue perhaps I have found the rootcause might be /etc/hosts couldn't be unchanged launching Ubuntu instances according to the instance name though /etc/hostname is changed.
the workaround is using cloud-init to force changing /etc/hosts/ using 'manage_etc_hosts: true'

for more detail http://community.cloudera.com/t5/Cloudera-Manager-Installation/CDH-cluster-command-timed-out-after-90-seconds/td-p/39167

[Nikita Konovalov (nkonovalov)]

Sahara does set up a valid /etc/hosts file on all instances.
Please see https://github.com/openstack/sahara/blob/e6ff7d17eae426fa04f1d859d3ae0ea237adb32d/sahara/utils/cluster.py#L135

[Vitaly Sedelnik (vsedelnik)]

Removed from 7.0-mu-4 as per feedback from Sahara team that solution from Cloudera forum doesn't actually resolve this issue. More time for reproduction and producing a fix is needed.

[Fabrizio Soppelsa (fsoppelsa)]

Guys, any news on this, can we target it for MU5?

[Nikita Konovalov (nkonovalov)]

This issue occurred on the customer's environment with custom internal DNS.
The issue reproduced repeatedly when the Ubuntu-based guest image was used.

The root-cause was never found since the timeout and actual requests made by Cloudera Manager are in the proprietary codebase.

There is a workaround build specially for the described case. It temporarily disables all DNS resolution on VMs by modifying /etc/resolve.conf for the provisioning time. Then is returns the file to its original state.

The workaround is available here https://review.fuel-infra.org/#/c/19099/2

[Denis Meltsaykin (dmeltsaykin)]

Closing as Won't Fix as there is workaround and no fix.

[Fabrizio Soppelsa (fsoppelsa)]

Nikita, is the workaround intended to stay there with -2, or is there an upcoming discussion in community on this DNS thing?