Ceph request to get keyring / ceph status fails on Nautilus

[Chris MacNaughton]

root@juju-64b09e-ceph-autotune-0:~# ceph -v
ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)

# What's in /var/lib/ceph/mon/ ?

root@juju-64b09e-ceph-autotune-0:~# ls /var/lib/ceph/mon/
ceph-juju-64b09e-ceph-autotune-0

# Ask Ceph for the client.osd-upgrade key:

root@juju-64b09e-ceph-autotune-0:~# sudo -u ceph ceph --name mon. --keyring /var/lib/ceph/mon/ceph-juju-64b09e-ceph-autotune-0/keyring auth get client.osd-upgrade
2019-08-15 14:59:33.511 7f6f93b95700 -1 auth: unable to find a keyring on /var/lib/ceph/mon/ceph-/keyring: (2) No such file or directory
2019-08-15 14:59:33.511 7f6f93b95700 -1 AuthRegistry(0x7f6f8c07fc28) no keyring found at /var/lib/ceph/mon/ceph-/keyring, disabling cephx
exported keyring for client.osd-upgrade
[client.osd-upgrade]
        key = AQDJb1VduEKkIhAAe9HREe7yL2w0pzQZ5GbnOQ==
        caps mon = "allow command "config-key"; allow command "osd tree"; allow command "config-key list"; allow command "config-key put"; allow command "config-key get"; allow command "config-key exists"; allow command "osd out"; allow command "osd in"; allow command "osd rm"; allow command "auth del""

# Ask for ceph status:

root@juju-64b09e-ceph-autotune-0:~# ceph -s
[errno 2] error connecting to the cluster

# Specify keyring details:

root@juju-64b09e-ceph-autotune-0:~# sudo -u ceph ceph --name mon. --keyring /var/lib/ceph/mon/ceph-juju-64b09e-ceph-autotune-0/keyring status
2019-08-15 15:20:43.606 7f0593520700 -1 auth: unable to find a keyring on /var/lib/ceph/mon/ceph-/keyring: (2) No such file or directory
2019-08-15 15:20:43.606 7f0593520700 -1 AuthRegistry(0x7f058c07fc28) no keyring found at /var/lib/ceph/mon/ceph-/keyring, disabling cephx
  cluster:
    id: c6e2d994-bf6a-11e9-ae54-fa163e703d27
    health: HEALTH_WARN
            no active mgr
            3 monitors have not enabled msgr2

  services:
    mon: 3 daemons, quorum juju-64b09e-ceph-autotune-1,juju-64b09e-ceph-autotune-0,juju-64b09e-ceph-autotune-2 (age 35m)
    mgr: no daemons active
    osd: 3 osds: 3 up (since 35m), 3 in (since 35m)

  data:
    pools: 0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage: 0 B used, 0 B / 0 B avail
    pgs:

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: ceph 14.2.1-0ubuntu3~cloud0 [origin: Canonical]
ProcVersionSignature: Ubuntu 4.15.0-58.64-generic 4.15.18
Uname: Linux 4.15.0-58-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.7
Architecture: amd64
CrashDB:
 {
                "impl": "launchpad",
                "project": "cloud-archive",
                "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
             }
Date: Thu Aug 15 15:18:58 2019
Ec2AMI: ami-00000734
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.blue
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
ProcEnviron:
 TERM=screen-256color
 PATH=(custom, no user)
 LANG=C.UTF-8
 SHELL=/bin/bash
SourcePackage: ceph
UpgradeStatus: No upgrade log present (probably fresh install)

[Frode Nordahl (fnordahl)]

While the error message the ceph tool prints is a bit confusing the tool still does what it should.

I believe the issue here is a change in the bootstrap process [0] and our charms need to be updated to cope with the new workflow.

The smoking gun is this:

2019-08-19 05:29:00 DEBUG mon-relation-changed /usr/sbin/ceph-create-keys: This tool is obsolete; mons now create these keys on their own, and
2019-08-19 05:29:00 DEBUG mon-relation-changed /usr/sbin/ceph-create-keys: this tool does nothing except print this message.
2019-08-19 05:29:00 DEBUG mon-relation-changed /usr/sbin/ceph-create-keys: It will be removed in the next release. Please fix your script/tool.
2019-08-19 05:29:00 INFO juju-log mon:0: Retrying 'add_keyring_to_ceph' 1 more times (delay=15)

Excerpt from Nautilus release notes [0]:
The ceph-create-keys tool is now obsolete. The monitors automatically create these keys on their own. For now the script prints a warning message and exits, but it will be removed in the next release. Note that ceph-create-keys would also write the admin and bootstrap keys to /etc/ceph and /var/lib/ceph, but this script no longer does that. Any deployment tools that relied on this behavior should instead make use of the ceph auth export <entity-name> command for whichever key(s) they need.

0: https://docs.ceph.com/docs/master/releases/nautilus/

[Frode Nordahl (fnordahl)]

Running these manually on the mon units got me a step further:
ceph --name mon. --keyring /var/lib/ceph/mon/ceph-$(hostname -s)/keyring auth export client.admin > /etc/ceph/ceph.client.admin.keyring

ceph --name mon. --keyring /var/lib/ceph/mon/ceph-$(hostname -s)/keyring auth export client.osd-upgrade > /var/lib/ceph/osd/ceph.client.osd-upgrade.keyring

ceph --name mon. --keyring /var/lib/ceph/mon/ceph-$(hostname -s)/keyring auth export client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring

[Chris MacNaughton (chris.macnaughton)]

This seems to be resolved in charms.ceph, and just needs syncing into ceph-mon

[Chris MacNaughton (chris.macnaughton)]

This is resolved in ceph-mon via https://github.com/openstack/charm-ceph-mon/commit/62155d10a8b00a07459f0b2ceff7b264ffdd551c