[Ambari] 'service ambari-server start' never finishes

[Peter Nordquist]

In my Sahara deployment (Mitaka), it seems like my Ambari server takes just long enough to start that it hits a deadlock. I've updated the ssh_timeout_common from 300 to 900 and it never finishes. The odd part is that when the ssh session times out the server finishes starting properly. My current running theory is that since they flush to stdout and it doesn't look like the output is being captured this is causing the script to wait for the ability to write to the OS stdout buffer but since it's not being read the OS blocks when trying to flush. I've changed the line here [0] to 'service ambari-server start >/dev/null' and it seemed to fix my installation. This change feels like a hack though so maybe the system should be piping the output somehow? The related change in Ambari is here [1] and [2]. I'm not sure a change in Ambari caused this to happen though as I was having intermittent issues with this before the Mitaka upgrade.

[0]: https://github.com/openstack/sahara/blob/master/sahara/plugins/ambari/deploy.py#L58
[1]: https://issues.apache.org/jira/browse/AMBARI-6447
[2]: https://reviews.apache.org/r/23383/

[Vitalii Gridnev (vgridnev)]

Can you explain what image was used?

[Vitalii Gridnev (vgridnev)]

Actually I saw such kind of behavior on Centos7 images, anyway moving to Confirmed

[Peter Nordquist (pnnl-plnordquist)]

Yeah I was using the Centos 7 images being built by the Sahara image elements repo from the stable/mitaka branch there. It might only be an issue on Centos 7 since there's some special glue there for SystemD to execute old init.d scripts or something along those lines.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/315024

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/324416

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/315024
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=6329a0aa736383f095d61527583e65b904e90b8e
Submitter: Jenkins
Branch: master

commit 6329a0aa736383f095d61527583e65b904e90b8e
Author: Vitaly Gridnev <email address hidden>
Date: Wed May 11 16:07:47 2016 +0300

    workaround to fix ambari start on centos7

    Change-Id: I1a0383861db50ae5136e8dff9de4119b566606f1
    Closes-bug: 1579187

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/324416
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=f4ee3d0e1694c0c5f5cc4fb46080fc40e5af163e
Submitter: Jenkins
Branch: stable/mitaka

commit f4ee3d0e1694c0c5f5cc4fb46080fc40e5af163e
Author: Vitaly Gridnev <email address hidden>
Date: Wed May 11 16:07:47 2016 +0300

    workaround to fix ambari start on centos7

    Change-Id: I1a0383861db50ae5136e8dff9de4119b566606f1
    Closes-bug: 1579187

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/sahara 5.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/sahara 4.1.0 release.