Ceph Nautilus (train/proposed) fails to bootstrap monitor cluster

[Chris MacNaughton]

2019-08-23 12:45:41 DEBUG mon-relation-changed creating /var/lib/ceph/tmp/juju-d8c880-zaza-34134266e06a-0.mon.keyring
2019-08-23 12:45:41 DEBUG mon-relation-changed added entity mon. auth(key=AQCX319d1WIGEhAAFI3dRJ0iZNpzmbViGITrtw==)
2019-08-23 12:45:41 DEBUG mon-relation-changed added 1 caps to entity mon.
2019-08-23 12:45:53 DEBUG mon-relation-changed Created symlink /<email address hidden> → /lib/systemd/system/ceph-mon@.service.
2019-08-23 12:46:10 DEBUG mon-relation-changed Error ENOENT: failed to find client.admin in keyring
2019-08-23 12:46:10 INFO juju-log mon:0: Retrying '_create_keyrings' 3 more times (delay=5)
2019-08-23 12:46:19 DEBUG mon-relation-changed Error ENOENT: failed to find client.admin in keyring
2019-08-23 12:46:19 INFO juju-log mon:0: Retrying '_create_keyrings' 2 more times (delay=10)
2019-08-23 12:46:32 DEBUG mon-relation-changed Error ENOENT: failed to find client.admin in keyring
2019-08-23 12:46:32 INFO juju-log mon:0: Retrying '_create_keyrings' 1 more times (delay=15)
2019-08-23 12:46:50 DEBUG mon-relation-changed Error ENOENT: failed to find client.admin in keyring
2019-08-23 12:46:50 DEBUG mon-relation-changed Traceback (most recent call last):
2019-08-23 12:46:50 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-mon-0/charm/hooks/mon-relation-changed", line 977, in <module>
2019-08-23 12:46:50 DEBUG mon-relation-changed hooks.execute(sys.argv)
2019-08-23 12:46:50 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-mon-0/charm/hooks/charmhelpers/core/hookenv.py", line 914, in execute
2019-08-23 12:46:50 DEBUG mon-relation-changed self._hooks[hook_name]()
2019-08-23 12:46:50 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-mon-0/charm/hooks/mon-relation-changed", line 424, in mon_relation
2019-08-23 12:46:50 DEBUG mon-relation-changed ceph.bootstrap_monitor_cluster(leader_get('monitor-secret'))
2019-08-23 12:46:50 DEBUG mon-relation-changed File "lib/ceph/utils.py", line 1306, in bootstrap_monitor_cluster
2019-08-23 12:46:50 DEBUG mon-relation-changed _create_keyrings()
2019-08-23 12:46:50 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-mon-0/charm/hooks/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
2019-08-23 12:46:50 DEBUG mon-relation-changed return f(*args, **kwargs)
2019-08-23 12:46:50 DEBUG mon-relation-changed File "lib/ceph/utils.py", line 1374, in _create_keyrings
2019-08-23 12:46:50 DEBUG mon-relation-changed 'auth', 'get', 'client.admin',
2019-08-23 12:46:50 DEBUG mon-relation-changed File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
2019-08-23 12:46:50 DEBUG mon-relation-changed **kwargs).stdout
2019-08-23 12:46:50 DEBUG mon-relation-changed File "/usr/lib/python3.6/subprocess.py", line 438, in run
2019-08-23 12:46:50 DEBUG mon-relation-changed output=stdout, stderr=stderr)
2019-08-23 12:46:50 DEBUG mon-relation-changed subprocess.CalledProcessError: Command '['sudo', '-u', 'ceph', 'ceph', '--name', 'mon.', '--keyring', '/var/lib/ceph/mon/ceph-juju-d8c880-zaza-34134266e06a-0/keyring', 'auth', 'get', 'client.admin']' returned non-zero exit status 2.
2019-08-23 12:46:51 ERROR juju.worker.uniter.operation runhook.go:132 hook "mon-relation-changed" failed: exit status 1

[Chris MacNaughton (chris.macnaughton)]

I see this when using either bionic-train/proposed or bionic-train/staging with the attached bundle

[Corey Bryant (corey.bryant)]

This appears to be fixed by the following commit:

commit 872c670ed6367ff575262f9284273ff60bffd6b7
Author: James Page <email address hidden>
Date: Thu May 30 16:48:17 2019 +0100

    Add support for Nautilus release

    Resync charms.ceph to pickup changes to support new approach to
    admin and bootstrap key generation with in the ceph-mon cluster.

    Update get_mon_hosts to avoid specifying the port number for the
    MON daemons; we use the default so this is not required, and at
    Nautilus the MON daemons run both v1 and v2 messenger ports.

    Specifying the port in the ceph.conf file disables the v2
    messenger port which is not the desired behaviour on upgrade
    or new installation.

    Drop start hook; this has been present in the charm since its
    inception as 'ceph' but is really not needed - re-deploying the
    charm to units which previous hosted ceph-mon is dangerous
    and the ceph-* daemons should still be running anyway.

    Depends-On: I2dfab7404b72e314625ea554ee64595c5e26f3c6
    Change-Id: I340dbf427551e6f9f8cd4ca89128627e497d6097