Ceph failed to deploy with bootstrap-osd keyring not found; run 'gatherkeys'

Bug #1394584 reported by Tatyanka
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Stanislav Makar

Bug Description

http://jenkins-product.srt.mirantis.net:8080/job/6.0.ubuntu.promo_bvt/49/consoleFull

http://paste.openstack.org/show/135236/

root@node-1:~# /usr/bin/ceph-deploy osd prepare node-1:/dev/vdb2 node-1:/dev/vdc2
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.9): /usr/bin/ceph-deploy osd prepare node-1:/dev/vdb2 node-1:/dev/vdc2
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks node-1:/dev/vdb2: node-1:/dev/vdc2:
[ceph_deploy][ERROR ] RuntimeError: bootstrap-osd keyring not found; run 'gatherkeys'

root@node-1:~# /usr/bin/ceph-deploy gatherkeys node-1
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.9): /usr/bin/ceph-deploy gatherkeys node-1
[ceph_deploy.gatherkeys][DEBUG ] Checking node-1 for /etc/ceph/ceph.client.admin.keyring
[node-1][DEBUG ] connected to host: node-1
[node-1][DEBUG ] detect platform information from remote host
[node-1][DEBUG ] detect machine type
[node-1][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][WARNIN] Unable to find /etc/ceph/ceph.client.admin.keyring on ['node-1']
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
[ceph_deploy.gatherkeys][DEBUG ] Checking node-1 for /var/lib/ceph/bootstrap-osd/ceph.keyring
[node-1][DEBUG ] connected to host: node-1
[node-1][DEBUG ] detect platform information from remote host
[node-1][DEBUG ] detect machine type
[node-1][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][WARNIN] Unable to find /var/lib/ceph/bootstrap-osd/ceph.keyring on ['node-1']
[ceph_deploy.gatherkeys][DEBUG ] Checking node-1 for /var/lib/ceph/bootstrap-mds/ceph.keyring
[node-1][DEBUG ] connected to host: node-1
[node-1][DEBUG ] detect platform information from remote host
[node-1][DEBUG ] detect machine type
[node-1][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][WARNIN] Unable to find /var/lib/ceph/bootstrap-mds/ceph.keyring on ['node-1']
root@node-1:~# ls /var/lib/ceph/bootstrap-mds/
root@node-1:~#

Tags: ceph verified
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Stanislav Makar (smakar)
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Stanislav Makar (smakar)
Revision history for this message
Stanislav Makar (smakar) wrote :

from logs first fail:

Thu Nov 20 11:38:06 +0000 2014 Puppet (err): Could not apply complete catalog: Found 1 dependency cycle:
(Cinder_config[DEFAULT/host] => Service[cinder-api] => Haproxy::Balancermember[cinder-api] => Concat::Fragment[haproxy_cinder-api_balancermember_cinder-api] => File[/var/lib/puppet/concat/_etc_haproxy_conf.d_070-cinder-api.cfg/fragments/01-cinder-api_haproxy_cinder-api_balancermember_cinder-api] => Concat::Fragment[haproxy_cinder-api_balancermember_cinder-api] => Haproxy::Balancermember[cinder-api] => Openstack::Ha::Haproxy_service[cinder-api] => Exec[wait-for-haproxy-keystone-backend] => Exec[waiting-for-neutron-api] => Neutron_router[router04] => Openstack::Network::Create_router[router04] => Class[Openstack::Controller] => Class[Ceph] => Ceph::Pool[volumes] => Exec[Create volumes pool] => Ceph::Pool[volumes] => Class[Cinder::Volume::Rbd] => Cinder::Backend::Rbd[DEFAULT] => Cinder_config[DEFAULT/host])
Try the '--graph' option and opening the resulting '.dot' file in OmniGraffle or GraphViz

Changed in fuel:
status: New → In Progress
Revision history for this message
Stanislav Makar (smakar) wrote :

this bug is caused by the commit https://github.com/stackforge/fuel-library/commit/4f8e008ecaec8e2acff79108970f34b4ed1f29ef

Strings in the file deployment/puppet/osnailyfacter/manifests/cluster_ha.pp :

      if $::use_neutron {
          Exec<| title=='wait-for-haproxy-keystone-backend' |> -> Exec <| title == 'waiting-for-neutron-api' |>
          Exec<| title=='wait-for-haproxy-keystone-admin-backend' |> -> Exec <| title == 'waiting-for-neutron-api' |>
        }

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/136267

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/136267
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=22cd3fa0c8b2a12a8a4450aa043c4fabe7096580
Submitter: Jenkins
Branch: master

commit 22cd3fa0c8b2a12a8a4450aa043c4fabe7096580
Author: Stanislav Makar <email address hidden>
Date: Fri Nov 21 08:25:14 2014 +0000

    Remove the broken dependencies

    Change-Id: Ie851080219e78454d66749fee3883dadebd58678
    Closes-bug: #1394584

Changed in fuel:
status: In Progress → Fix Committed
tags: added: on-verification
Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Verified on ISO MOS 6.0 #56
{"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "auth_required": true, "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}}}, "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}

Steps To Verify:
1. Configure HA cloud with Ceph+ Cinder, Glance + Swift + Ceilometer, 1 controller and 2 compute.
2. Deploy environment.
3. Login to OpenStack and create Volume in Cinder.

Observed Result:
Deployment finished successfully without any errors, user can create Volume using Cinder + Ceph.

tags: added: verified
removed: on-verification
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.