Ceph request to get keyring / ceph status fails on Nautilus

Bug #1840283 reported by Chris MacNaughton on 2019-08-15
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack ceph-mon charm
Ubuntu Cloud Archive

Bug Description

root@juju-64b09e-ceph-autotune-0:~# ceph -v
ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)

# What's in /var/lib/ceph/mon/ ?

root@juju-64b09e-ceph-autotune-0:~# ls /var/lib/ceph/mon/

# Ask Ceph for the client.osd-upgrade key:

root@juju-64b09e-ceph-autotune-0:~# sudo -u ceph ceph --name mon. --keyring /var/lib/ceph/mon/ceph-juju-64b09e-ceph-autotune-0/keyring auth get client.osd-upgrade
2019-08-15 14:59:33.511 7f6f93b95700 -1 auth: unable to find a keyring on /var/lib/ceph/mon/ceph-/keyring: (2) No such file or directory
2019-08-15 14:59:33.511 7f6f93b95700 -1 AuthRegistry(0x7f6f8c07fc28) no keyring found at /var/lib/ceph/mon/ceph-/keyring, disabling cephx
exported keyring for client.osd-upgrade
        key = AQDJb1VduEKkIhAAe9HREe7yL2w0pzQZ5GbnOQ==
        caps mon = "allow command "config-key"; allow command "osd tree"; allow command "config-key list"; allow command "config-key put"; allow command "config-key get"; allow command "config-key exists"; allow command "osd out"; allow command "osd in"; allow command "osd rm"; allow command "auth del""

# Ask for ceph status:

root@juju-64b09e-ceph-autotune-0:~# ceph -s
[errno 2] error connecting to the cluster

# Specify keyring details:

root@juju-64b09e-ceph-autotune-0:~# sudo -u ceph ceph --name mon. --keyring /var/lib/ceph/mon/ceph-juju-64b09e-ceph-autotune-0/keyring status
2019-08-15 15:20:43.606 7f0593520700 -1 auth: unable to find a keyring on /var/lib/ceph/mon/ceph-/keyring: (2) No such file or directory
2019-08-15 15:20:43.606 7f0593520700 -1 AuthRegistry(0x7f058c07fc28) no keyring found at /var/lib/ceph/mon/ceph-/keyring, disabling cephx
    id: c6e2d994-bf6a-11e9-ae54-fa163e703d27
    health: HEALTH_WARN
            no active mgr
            3 monitors have not enabled msgr2

    mon: 3 daemons, quorum juju-64b09e-ceph-autotune-1,juju-64b09e-ceph-autotune-0,juju-64b09e-ceph-autotune-2 (age 35m)
    mgr: no daemons active
    osd: 3 osds: 3 up (since 35m), 3 in (since 35m)

    pools: 0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage: 0 B used, 0 B / 0 B avail

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: ceph 14.2.1-0ubuntu3~cloud0 [origin: Canonical]
ProcVersionSignature: Ubuntu 4.15.0-58.64-generic 4.15.18
Uname: Linux 4.15.0-58-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.7
Architecture: amd64
                "impl": "launchpad",
                "project": "cloud-archive",
                "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
Date: Thu Aug 15 15:18:58 2019
Ec2AMI: ami-00000734
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.blue
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
 PATH=(custom, no user)
SourcePackage: ceph
UpgradeStatus: No upgrade log present (probably fresh install)

Frode Nordahl (fnordahl) wrote :

While the error message the ceph tool prints is a bit confusing the tool still does what it should.

I believe the issue here is a change in the bootstrap process [0] and our charms need to be updated to cope with the new workflow.

The smoking gun is this:

2019-08-19 05:29:00 DEBUG mon-relation-changed /usr/sbin/ceph-create-keys: This tool is obsolete; mons now create these keys on their own, and
2019-08-19 05:29:00 DEBUG mon-relation-changed /usr/sbin/ceph-create-keys: this tool does nothing except print this message.
2019-08-19 05:29:00 DEBUG mon-relation-changed /usr/sbin/ceph-create-keys: It will be removed in the next release. Please fix your script/tool.
2019-08-19 05:29:00 INFO juju-log mon:0: Retrying 'add_keyring_to_ceph' 1 more times (delay=15)

Excerpt from Nautilus release notes [0]:
The ceph-create-keys tool is now obsolete. The monitors automatically create these keys on their own. For now the script prints a warning message and exits, but it will be removed in the next release. Note that ceph-create-keys would also write the admin and bootstrap keys to /etc/ceph and /var/lib/ceph, but this script no longer does that. Any deployment tools that relied on this behavior should instead make use of the ceph auth export <entity-name> command for whichever key(s) they need.

0: https://docs.ceph.com/docs/master/releases/nautilus/

Frode Nordahl (fnordahl) wrote :

Running these manually on the mon units got me a step further:
ceph --name mon. --keyring /var/lib/ceph/mon/ceph-$(hostname -s)/keyring auth export client.admin > /etc/ceph/ceph.client.admin.keyring

ceph --name mon. --keyring /var/lib/ceph/mon/ceph-$(hostname -s)/keyring auth export client.osd-upgrade > /var/lib/ceph/osd/ceph.client.osd-upgrade.keyring

ceph --name mon. --keyring /var/lib/ceph/mon/ceph-$(hostname -s)/keyring auth export client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring

Frode Nordahl (fnordahl) on 2019-08-19
Changed in cloud-archive:
status: New → Invalid

This seems to be resolved in charms.ceph, and just needs syncing into ceph-mon

Changed in charm-ceph-mon:
status: New → Fix Committed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers