hook failed: "mon-relation-changed" FileNotFoundError: [Errno 2] No such file or directory: '/etc/ceph/ceph.client.admin.keyring'

[Frode Nordahl]

2018-03-24 12:35:15 INFO juju-log mon:138: Making dir /var/lib/charm/ceph-mon ceph:ceph 555
2018-03-24 12:35:23 DEBUG juju-log mon:138: Writing file /var/lib/charm/ceph-mon/ceph.conf root:root 644
2018-03-24 12:35:29 INFO juju-log mon:138: Making dir /var/run/ceph ceph:ceph 755
2018-03-24 12:35:35 INFO juju-log mon:138: Making dir /var/lib/ceph/mon/ceph-juju-430ba4-auto-osci-sv06-74 ceph:ceph 755
2018-03-24 12:35:35 DEBUG mon-relation-changed creating /var/lib/ceph/tmp/juju-430ba4-auto-osci-sv06-74.mon.keyring
2018-03-24 12:35:35 DEBUG mon-relation-changed added entity mon. auth auth(auid = 18446744073709551615 key=AQCXrnZQwI7KGBAAiPofmKEXKxu5bUzoYLVkbQ== with 0 caps)
2018-03-24 12:35:35 DEBUG mon-relation-changed ceph-mon: renaming mon.noname-b 172.17.106.33:6789/0 to mon.juju-430ba4-auto-osci-sv06-74
2018-03-24 12:35:35 DEBUG mon-relation-changed ceph-mon: set fsid to 6547bd3e-1397-11e2-82e5-53567c8d32dc
2018-03-24 12:35:35 DEBUG mon-relation-changed ceph-mon: created monfs at /var/lib/ceph/mon/ceph-juju-430ba4-auto-osci-sv06-74 for mon.juju-430ba4-auto-osci-sv06-74
2018-03-24 12:35:40 DEBUG mon-relation-changed Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-mon.service to /lib/systemd/system/ceph-mon.service.
2018-03-24 12:35:44 INFO juju-log mon:138: Retrying 'add_keyring_to_ceph' 3 more times (delay=5)
2018-03-24 12:35:49 DEBUG mon-relation-changed creating /var/lib/ceph/tmp/juju-430ba4-auto-osci-sv06-74.mon.keyring
2018-03-24 12:35:49 DEBUG mon-relation-changed added entity mon. auth auth(auid = 18446744073709551615 key=AQCXrnZQwI7KGBAAiPofmKEXKxu5bUzoYLVkbQ== with 0 caps)
2018-03-24 12:35:49 DEBUG mon-relation-changed '/var/lib/ceph/mon/ceph-juju-430ba4-auto-osci-sv06-74' already exists and is not empty: monitor may already exist
2018-03-24 12:35:59 INFO juju-log mon:138: Retrying 'add_keyring_to_ceph' 2 more times (delay=10)
2018-03-24 12:36:09 DEBUG mon-relation-changed creating /var/lib/ceph/tmp/juju-430ba4-auto-osci-sv06-74.mon.keyring
2018-03-24 12:36:09 DEBUG mon-relation-changed added entity mon. auth auth(auid = 18446744073709551615 key=AQCXrnZQwI7KGBAAiPofmKEXKxu5bUzoYLVkbQ== with 0 caps)
2018-03-24 12:36:09 DEBUG mon-relation-changed '/var/lib/ceph/mon/ceph-juju-430ba4-auto-osci-sv06-74' already exists and is not empty: monitor may already exist
2018-03-24 12:36:16 INFO juju-log mon:138: Retrying 'add_keyring_to_ceph' 1 more times (delay=15)
2018-03-24 12:36:31 DEBUG mon-relation-changed creating /var/lib/ceph/tmp/juju-430ba4-auto-osci-sv06-74.mon.keyring
2018-03-24 12:36:31 DEBUG mon-relation-changed added entity mon. auth auth(auid = 18446744073709551615 key=AQCXrnZQwI7KGBAAiPofmKEXKxu5bUzoYLVkbQ== with 0 caps)
2018-03-24 12:36:31 DEBUG mon-relation-changed '/var/lib/ceph/mon/ceph-juju-430ba4-auto-osci-sv06-74' already exists and is not empty: monitor may already exist
2018-03-24 12:36:39 DEBUG mon-relation-changed Traceback (most recent call last):
2018-03-24 12:36:39 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-mon-20/charm/hooks/mon-relation-changed", line 725, in <module>
2018-03-24 12:36:39 DEBUG mon-relation-changed hooks.execute(sys.argv)
2018-03-24 12:36:39 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-mon-20/charm/hooks/charmhelpers/core/hookenv.py", line 800, in execute
2018-03-24 12:36:39 DEBUG mon-relation-changed self._hooks[hook_name]()
2018-03-24 12:36:39 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-mon-20/charm/hooks/mon-relation-changed", line 356, in mon_relation
2018-03-24 12:36:39 DEBUG mon-relation-changed ceph.bootstrap_monitor_cluster(leader_get('monitor-secret'))
2018-03-24 12:36:39 DEBUG mon-relation-changed File "lib/ceph/utils.py", line 1323, in bootstrap_monitor_cluster
2018-03-24 12:36:39 DEBUG mon-relation-changed init_marker)
2018-03-24 12:36:39 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-mon-20/charm/hooks/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
2018-03-24 12:36:39 DEBUG mon-relation-changed return f(*args, **kwargs)
2018-03-24 12:36:39 DEBUG mon-relation-changed File "lib/ceph/utils.py", line 1364, in add_keyring_to_ceph
2018-03-24 12:36:39 DEBUG mon-relation-changed osstat = os.stat(_client_admin_keyring)
2018-03-24 12:36:39 DEBUG mon-relation-changed FileNotFoundError: [Errno 2] No such file or directory: '/etc/ceph/ceph.client.admin.keyring'
2018-03-24 12:36:39 ERROR juju.worker.uniter.operation runhook.go:113 hook "mon-relation-changed" failed: exit status 1

[Frode Nordahl (fnordahl)]

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_amulet_full/openstack/charm-ceph-mon/525024/3/1303/index.html

[Frode Nordahl (fnordahl)]

Investigating CI artifacts from ceph-mon unit 20: /etc/ceph/ceph.client.admin.keyring exists and the ceph-mon itself seems to be running.