It is impossible to deploy a Fuel cluster with Ceph RadosGW for objects (Swift API)

[Yaroslav Lobankov]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.0"
  api: "1.0"
  build_number: "47"
  build_id: "2014-12-09_03-28-22"
  astute_sha: "16b252d93be6aaa73030b8100cf8c5ca6a970a91"
  fuellib_sha: "c8df66ac50f90ed7708ecd5ee4cf1cefe9fd1b70"
  ostf_sha: "a9afb68710d809570460c29d6c3293219d3624d4"
  nailgun_sha: "22bd43b89a17843f9199f92d61fc86cb0f8772f1"
  fuelmain_sha: "2040303660ca19ae7aba1ab16cc75485d9c944d6"

ENVIRONMENT:
HA mode, Ubuntu, Neutron with GRE segmentation, Ceph RBD for volumes (Cinder), Ceph RBD for images (Glance), Ceph RadosGW for objects (Swift API), 3 controllers, 1 (compute + ceph)

After deployment failed I could see the following errors in logs:
2014-12-09 14:57:39 ERR
/usr/bin/puppet:4
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:91:in `execute'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/util.rb:478:in `exit_on_fail'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/application.rb:470:in `plugin_hook'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:146:in `run_command'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:218:in `main'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:268:in `apply_catalog'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `run'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:124:in `apply_catalog'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `benchmark'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:125:in `apply_catalog'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:163:in `apply'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:108:in `as_logging_destination'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:164:in `apply'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:108:in `evaluate'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:106:in `traverse'
2014-12-09 14:57:39 ERR
/usr/lib/ruby/vendor_ruby/puppet/graph/simple_graph.rb:242:in `report_cycles_in_graph'
2014-12-09 14:57:39 ERR
Try the '--graph' option and opening the resulting '.dot' file in OmniGraffle or GraphViz
2014-12-09 14:57:39 ERR
(Ceph_conf[client.radosgw.gateway/host] => File[/root/ceph.conf] => Class[Ceph::Conf] => Class[Ceph::Mon] => Exec[ceph-deploy mon create] => Class[Ceph::Mon] => Class[Ceph::Radosgw] => Ceph_conf[client.radosgw.gateway/host])
2014-12-09 14:57:39 ERR
Could not apply complete catalog: Found 1 dependency cycle:

[Vladimir Kuklin (vkuklin)]

verified on {"build_id": "2014-12-08_22-41-06", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "46", "auth_required": true, "api": "1.0", "nailgun_sha": "73a2fa3aa669ca8d95788e80dc052dfa16eca24a", "production": "docker", "fuelmain_sha": "2040303660ca19ae7aba1ab16cc75485d9c944d6", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis", "experimental"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-08_22-41-06", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "46", "api": "1.0", "nailgun_sha": "73a2fa3aa669ca8d95788e80dc052dfa16eca24a", "production": "docker", "fuelmain_sha": "2040303660ca19ae7aba1ab16cc75485d9c944d6", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "c8df66ac50f90ed7708ecd5ee4cf1cefe9fd1b70"}}}, "fuellib_sha":
"c8df66ac50f90ed7708ecd5ee4cf1cefe9fd1b70"}

We were not able to reproduce the issue. It seems, you have customized manifests.

[Vladimir Kuklin (vkuklin)]

Sorry, I only checked primary controller. Waiting for secondary ones.

[Vladimir Kuklin (vkuklin)]

Also, I see the problem that failed nodes are marked as ready even after catalogue compilation failure

[Dmitry Borodaenko (angdraug)]

Looks like the loop was introduced in https://review.openstack.org/136516, I'm trying to figure out a way around it.

[Aleksey Kasatkin (alekseyk-ru)]

It was deployed successfully on VBox with the same config but one controller: ISO #47,

HA mode, Ubuntu, Neutron with GRE segmentation, Ceph RBD for volumes (Cinder), Ceph RBD for images (Glance), Ceph RadosGW for objects (Swift API), 1 controller, 1 (compute + ceph)

[Dmitry Borodaenko (angdraug)]

Comment #7 confirms my theory, this change is only relevant for secondary controllers:
https://review.openstack.org/#/c/136516/4/deployment/puppet/ceph/manifests/conf.pp

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/140418

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/140437

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/140418
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=72ec49f536ff6337a77084c8f5f477bcae259c5e
Submitter: Jenkins
Branch: master

commit 72ec49f536ff6337a77084c8f5f477bcae259c5e
Author: Dmitry Borodaenko <email address hidden>
Date: Tue Dec 9 09:54:47 2014 -0800

    use specific ceph_conf resources to avoid dependency loop

    Spaceship operator matching all ceph_conf resources creates a dependency
    loop because it includes ceph_conf resources defined in ceph::radosgw
    which in turn depends on ceph::conf.

    Change-Id: I60773d1c419b2f1cbf5b5e54b74684cc66b986a9
    Closes-Bug: #1400744

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/140437
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=2c99931072d951301d395ebd5bf45c8d401301bb
Submitter: Jenkins
Branch: stable/6.0

commit 2c99931072d951301d395ebd5bf45c8d401301bb
Author: Dmitry Borodaenko <email address hidden>
Date: Tue Dec 9 09:54:47 2014 -0800

    use specific ceph_conf resources to avoid dependency loop

    Spaceship operator matching all ceph_conf resources creates a dependency
    loop because it includes ceph_conf resources defined in ceph::radosgw
    which in turn depends on ceph::conf.

    Change-Id: I60773d1c419b2f1cbf5b5e54b74684cc66b986a9
    Closes-Bug: #1400744

[Timur Nurlygayanov (tnurlygayanov)]

Verifed on MOS 6.0 ISO #56:
{"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "auth_required": true, "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}}}, "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}

Steps to Verify:
1. Create OpenStack cloud with Cinder + Ceph, Glance+Ceph and RadosGW for SwiftAPI
2. Deploy this cloud

Observed Result:
User can successfully deploy OpenStack cloud with RadosGW without any errors.