overcloud deploy fails with ou must add a [grafana-server] group and add at least one node

Bug #1838460 reported by John Fulton
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
chandan kumar

Bug Description

0. Stein deployment fails in overcloud deploy step and emits:

       "fatal: [controller-0]: FAILED! => changed=false ",
        " msg: you must add a [grafana-server] group and add at least one node.",
        "fatal: [controller-1]: FAILED! => changed=false ",
        "fatal: [controller-2]: FAILED! => changed=false ",
        "fatal: [ceph-0]: FAILED! => changed=false ",
        "fatal: [ceph-1]: FAILED! => changed=false ",
        "fatal: [ceph-2]: FAILED! => changed=false ",
        "fatal: [compute-0]: FAILED! => changed=false ",

1. Root Cause:

ceph-ansible 4.0.0rc13 enables ceph-dashboard by default but the earlier version didn't:

 https://github.com/ceph/ceph-ansible/commit/93826e061d3d025fa631d726ada1c7a7d77c12b1

2. Workaround:
Deploy with the following:

parameter_defaults:
  CephAnsibleExtraConfig:
    dashboard_enabled: False

3. Proposed fix:
Fix will be to make dashboard_enabled=False in THT (overriding the new default in ceph-ansible).

Revision history for this message
John Fulton (jfulton-org) wrote :

This issue only affects Stein. Master (the future Train) already has the relevant parameter, CephEnableDashboard, and it defaults it to false. This the fix for this bug could go directly to Stein.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/673675

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/673675
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=d89fe28b3ede2b02066de66fb4555423993827fc
Submitter: Zuul
Branch: stable/stein

commit d89fe28b3ede2b02066de66fb4555423993827fc
Author: John Fulton <email address hidden>
Date: Tue Jul 30 15:52:54 2019 -0400

    Disable ceph-dashboard by default in Stein

    ceph-ansible 4 recently changed defaults and now
    deploys dashboard by default. This will cause a
    TripleO Stein deployment to fail unless additional
    parameters are added which Stein does not support.
    Opt to override this new ceph-ansible default in
    Stein (master, the future Train) will support it
    however.

    This patch is going directly into Stein instead of
    being added to master and then being backported,
    since master has a CephEnableDashboard parameter
    in addition to plenty of code to correctly set up
    the dashboard which would be too much to backport.
    Instead of misleading the user with a non-working
    CephEnableDashboard parameter, we just directly
    pass a flag to ceph-ansible to disable it.

    Closes-Bug: #1838460

    Change-Id: I718cd33513607efefb2cef3ebe110c523f6c6c6a

tags: added: in-stable-stein
Revision history for this message
chandan kumar (chkumar246) wrote :

In standalone scenario1, we are seeing the same issue:
https://zuul.opendev.org/t/openstack/build/ca8e1f55314941a4941d616cb1521a1b/log/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#4922

2019-08-23 05:20:46 | "Friday 23 August 2019 05:20:45 +0000 (0:00:00.120) 0:00:27.869 ********* ",

2019-08-23 05:20:46 | "TASK [ceph-validate : fail if [grafana-server] group doesn't exist] ************",

2019-08-23 05:20:46 | "Friday 23 August 2019 05:20:46 +0000 (0:00:00.354) 0:00:28.224 ********* ",

2019-08-23 05:20:46 | "fatal: [standalone]: FAILED! => changed=false ",

2019-08-23 05:20:46 | " msg: you must add a [grafana-server] group and add at least one node.",

2019-08-23 05:20:46 | "NO MORE HOSTS LEFT ************************************************************

Probable cause: https://review.opendev.org/#/c/677705/

Revision history for this message
chandan kumar (chkumar246) wrote :
tags: added: alert promotion-blocker
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/678186

Changed in tripleo:
assignee: John Fulton (jfulton-org) → chandan kumar (chkumar246)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (master)

Change abandoned by Chandan Kumar (raukadah) (<email address hidden>) on branch: master
Review: https://review.opendev.org/678186
Reason: in favor of https://review.opendev.org/#/c/674556

Revision history for this message
chandan kumar (chkumar246) wrote :
Ronelle Landy (rlandy)
Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 10.6.1

This issue was fixed in the openstack/tripleo-heat-templates 10.6.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.