Deployments with a Ceph node fail from the GUI

[Udi Kalifon]

I set CephStorageCount to 1 and included the "storage environment" setting. Deployment failed in ControllerNodesPostDeployment and ComputeNodesPostDeployment with this error on the failed resources:

Error: authentication_type cephx requires either key or keyring to be set but both are undef at /etc/puppet/modules/ceph/manifests/mon.pp:108 on node default-controller-0.localdomain

It looks like the GUI doesn't pass all needed parameters to the stack creation. I don't get this problem when deploying from the CLI.

[Udi Kalifon (ukalifon)]

Missing parameters were: CephClusterFSID, CephMonKey, CephAdminKey and CephClientKey. The GUI should fill them in with randomly generated values if the user didn't provide them.

[Julie Pichon (jpichon)]

Trying out Newton, I'm still seeing an issue with the CephClusterFSID parameter (web UI deployment). I assigned 1 node to Ceph Storage, enabled the Storage environment, and the deployment failed with:

Resource CREATE failed: resources[1]: resources.CephStorageServiceChain.resources.ServiceChain.Property error: resources.CephBase.properties: Property CephClusterFSID not assigned

Looking at the Mistral environment, I see the following Ceph related values:

| | "CephMonKey": "AABqHgpYAAAAABAAlQErZrdGZhrrhfuqEH+jHB==", |
| | "CephAdminKey": "AQBqHgpYAAAAABAA8NLc8Y/tNnE4fuY/eOMCHy==", |
| | "CephRgwKey": "AABqHgpYAAAAABAAkjtVCDFZB0YZbFmm6W+iMA==", |
| | "CephClientKey": "AABqHgpYAAAAABAAfl83c2b5pfofg+nYzzyHOw==", |

It doesn't seem to be in the list of automatically generated passwords: https://github.com/openstack/tripleo-common/blob/stable/newton/tripleo_common/constants.py#L57 . IIUC the CephClusterFSID needs to be set manually in some cases (external pre-existing Ceph) and be generated automatically in others (new Ceph deployment?).

Unfortunately at the moment I can't work around the problem because I can't figure out where to set this particular property. I looked in the parameters for the overall deployment configuration and in the parameters/services for the Ceph Storage role itself. I also tried setting it as JSON config as part of CephStorageExtraConfig, ExtraConfig, ServiceConfigSettings in the Ceph Storage parameters but with no luck (using {"ceph::profile::params::fsid":"4b5c8c0a-ff60-454b-a1b4-9747aa737d19"}).

[Julie Pichon (jpichon)]

I got the same error deploying from the CLI with the new plan deployment command:

$ openstack overcloud plan deploy overcloud
Starting to deploy plan: overcloud
Started Mistral Workflow. Execution ID: 99da985b-615e-43c4-9691-2d80f3bc2ad8
...
2016-10-24 13:32:04Z [overcloud.CephStorageServiceChain.ServiceChain.1]: CREATE_FAILED resources[1]: Property error: resources.CephBase.properties: Property CephClusterFSID not assigned
...

I see this is set at deploy time in the CLI when using the legacy overcloud deploy command though:

https://github.com/openstack/python-tripleoclient/blob/1969cc/tripleoclient/v1/overcloud_deploy.py#L129

[Julie Pichon (jpichon)]

When deploying with overcloud deploy --templates, a "CephClusterFSID" key is added to the "parameters_default" in the Mistral environment and the deployment doesn't get stuck on this.

As far as I can tell there are two bugs:

1. CephClusterFSID should be generated automatically when not set, like the other credentials.

2. The GUI should allow the user to set this parameter manually.

Perhaps documentation on how to use the 'ExtraConfig' options esp. from the UI would be useful as well, I'm not sure if the format I tried was correct.

[Julie Pichon (jpichon)]

I created bug 1636555 for the first bug. We can keep this one to track the changes required in tripleo-ui.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/390612

[Jiri Tomasek (jtomasek)]

At first look on the 2nd part of the bug, the parameters are defined in ceph-base.yaml template [1] which is not directly included in any environment, but is referenced from templates which are in storage-environment.yaml [2]

So when this environment is enabled, the parameters defined in ceph-base.yaml should be available to set under one of OS::TripleO::Services::CephMon, OS::TripleO::Services::CephOSD, OS::TripleO::Services::CephClient services. For some reason those are not included there.

[1] https://github.com/openstack/tripleo-heat-templates/blob/master/environments/storage-environment.yaml
[2] https://github.com/openstack/tripleo-heat-templates/blob/master/environments/storage-environment.yaml

[Jiri Tomasek (jtomasek)]

Ok, so after discussing this with shardy, the parameters defined in ceph-base.yaml are probably included in another nested resource under those specific services. I am going to investigate further and try to find out how to include those nested parameters in service parameters output.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/390946

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/390946
Committed: https://git.openstack.org/cgit/openstack/tripleo-ui/commit/?id=b2ae0ea2d2b584066ffe53c7f7736523f58724f6
Submitter: Jenkins
Branch: master

commit b2ae0ea2d2b584066ffe53c7f7736523f58724f6
Author: Jiri Tomasek <email address hidden>
Date: Wed Oct 26 20:13:43 2016 +0200

    Include nested parameters in service parameters

    This change flattens any nested parameters inside a service
    resource and merges them with direct service parameters

    Closes-Bug: 1626426
    Change-Id: I1d36ce676b07e63ceef369c0fa5a80681a89fcce

[Jason E. Rist (jason-rist)]

Does this need to be backported?

[Julie Pichon (jpichon)]

It likely does, or there'll be deployment issues with Ceph on Newton.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/392474

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/392474
Committed: https://git.openstack.org/cgit/openstack/tripleo-ui/commit/?id=588bdbfabfabc28537f8606caaf90ccea234efc3
Submitter: Jenkins
Branch: stable/newton

commit 588bdbfabfabc28537f8606caaf90ccea234efc3
Author: Jiri Tomasek <email address hidden>
Date: Wed Oct 26 20:13:43 2016 +0200

    Include nested parameters in service parameters

    This change flattens any nested parameters inside a service
    resource and merges them with direct service parameters

    Closes-Bug: 1626426
    Change-Id: I1d36ce676b07e63ceef369c0fa5a80681a89fcce
    (cherry picked from commit b2ae0ea2d2b584066ffe53c7f7736523f58724f6)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/390612
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=49d23e0821ed8662e4b4ef43368251f3efb7fe0a
Submitter: Jenkins
Branch: master

commit 49d23e0821ed8662e4b4ef43368251f3efb7fe0a
Author: Julie Pichon <email address hidden>
Date: Tue Oct 25 17:23:46 2016 +0100

    Add CephClusterFSID to generated passwords

    Deployments with a Ceph Storage component fail without this parameter.
    Currently it's generated automatically when deploying with the CLI,
    this re-uses the same code.

    Change-Id: Icb93d336a35ce6ec393023b88cea34df4810f391
    Closes-Bug: #1636555
    Related-Bug: #1626426

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/394493

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/394493
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=79bd89a0a1a46bebd06e0e8ed68e96f4345a52cb
Submitter: Jenkins
Branch: stable/newton

commit 79bd89a0a1a46bebd06e0e8ed68e96f4345a52cb
Author: Julie Pichon <email address hidden>
Date: Tue Oct 25 17:23:46 2016 +0100

    Add CephClusterFSID to generated passwords

    Deployments with a Ceph Storage component fail without this parameter.
    Currently it's generated automatically when deploying with the CLI,
    this re-uses the same code.

    Change-Id: Icb93d336a35ce6ec393023b88cea34df4810f391
    Closes-Bug: #1636555
    Related-Bug: #1626426
    (cherry picked from commit 49d23e0821ed8662e4b4ef43368251f3efb7fe0a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-ui 1.0.5 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-ui 2.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-ui 1.0.5 release.