2019-03-13 08:49:05 DEBUG install Created symlink /etc/systemd/system/ceph-create-keys.service → /dev/null.
2019-03-13 08:49:09 INFO juju-log still waiting for leader to setup keys
2019-03-13 08:49:13 DEBUG juju-log Hardening function 'config_changed'
2019-03-13 08:49:14 DEBUG juju-log No hardening applied to 'config_changed'
2019-03-13 08:49:14 INFO juju-log Ceph is not bootstrapped, skipping upgrade checks.
2019-03-13 08:49:14 INFO juju-log Monitor hosts are ['10.219.3.224:6789']
2019-03-13 08:49:14 DEBUG juju-log Updating sysctl_file: /etc/sysctl.d/50-ceph-charm.conf values: {'kernel.pid_max': 2097152, 'vm.max_map_count': 524288, 'kernel.threads-max': 2097152}
2019-03-13 08:49:14 DEBUG config-changed sysctl: permission denied on key 'kernel.pid_max'
2019-03-13 08:49:14 DEBUG config-changed sysctl: permission denied on key 'vm.max_map_count'
2019-03-13 08:49:14 DEBUG config-changed sysctl: permission denied on key 'kernel.threads-max'
2019-03-13 08:49:15 INFO juju-log still waiting for leader to setup keys
2019-03-13 08:49:28 INFO juju-log Making dir /var/lib/charm/ceph-mon-b ceph:ceph 555
2019-03-13 08:49:31 DEBUG juju-log Writing file /var/lib/charm/ceph-mon-b/ceph.conf root:root 644
2019-03-13 08:49:31 DEBUG leader-settings-changed update-alternatives: using /var/lib/charm/ceph-mon-b/ceph.conf to provide /etc/ceph/ceph.conf (ceph.conf) in auto mode
2019-03-13 08:49:31 INFO juju-log Not enough mons (1), punting.
2019-03-13 08:49:36 DEBUG juju-log rbd-mirror:20: mon cluster is not in quorum
2019-03-13 08:49:42 DEBUG juju-log rbd-mirror:20: mon cluster is not in quorum
2019-03-13 08:49:49 DEBUG juju-log rbd-mirror:21: mon cluster is not in quorum
2019-03-13 08:49:58 DEBUG juju-log rbd-mirror:20: mon cluster is not in quorum
2019-03-13 08:50:08 DEBUG juju-log rbd-mirror:21: mon cluster is not in quorum
2019-03-13 08:50:17 DEBUG juju-log rbd-mirror:21: mon cluster is not in quorum
2019-03-13 08:50:35 INFO juju-log mon:1: Making dir /var/lib/charm/ceph-mon-b ceph:ceph 555
2019-03-13 08:50:39 DEBUG mon-relation-changed unable to get monitor info from DNS SRV with service name: ceph-mon
2019-03-13 08:50:39 DEBUG mon-relation-changed no monitors specified to connect to.
2019-03-13 08:50:39 DEBUG mon-relation-changed 2019-03-13 08:50:39.323 7f358c7cb700 -1 failed for service _ceph-mon._tcp
2019-03-13 08:50:39 DEBUG mon-relation-changed 2019-03-13 08:50:39.323 7f358c7cb700 -1 monclient: get_monmap_and_config cannot identify monitors to contact
2019-03-13 08:50:39 DEBUG mon-relation-changed [errno 2] error connecting to the cluster
2019-03-13 08:50:39 DEBUG mon-relation-changed Traceback (most recent call last):
2019-03-13 08:50:39 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-mon-b-0/charm/hooks/mon-relation-changed", line 947, in <module>
2019-03-13 08:50:39 DEBUG mon-relation-changed hooks.execute(sys.argv)
2019-03-13 08:50:39 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-mon-b-0/charm/hooks/charmhelpers/core/hookenv.py", line 909, in execute
2019-03-13 08:50:39 DEBUG mon-relation-changed self._hooks[hook_name]()
2019-03-13 08:50:39 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-mon-b-0/charm/hooks/mon-relation-changed", line 408, in mon_relation
2019-03-13 08:50:39 DEBUG mon-relation-changed emit_cephconf()
2019-03-13 08:50:39 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-mon-b-0/charm/hooks/mon-relation-changed", line 211, in emit_cephconf
2019-03-13 08:50:39 DEBUG mon-relation-changed render('ceph.conf', charm_ceph_conf, get_ceph_context(), perms=0o644)
2019-03-13 08:50:39 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-mon-b-0/charm/hooks/mon-relation-changed", line 190, in get_ceph_context
2019-03-13 08:50:39 DEBUG mon-relation-changed rbd_features = get_rbd_features()
2019-03-13 08:50:39 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-mon-b-0/charm/hooks/utils.py", line 219, in get_rbd_features
2019-03-13 08:50:39 DEBUG mon-relation-changed return add_rbd_mirror_features(get_default_rbd_features())
2019-03-13 08:50:39 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-mon-b-0/charm/hooks/utils.py", line 190, in get_default_rbd_features
2019-03-13 08:50:39 DEBUG mon-relation-changed universal_newlines=True)
2019-03-13 08:50:39 DEBUG mon-relation-changed File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
2019-03-13 08:50:39 DEBUG mon-relation-changed **kwargs).stdout
2019-03-13 08:50:39 DEBUG mon-relation-changed File "/usr/lib/python3.6/subprocess.py", line 418, in run
2019-03-13 08:50:39 DEBUG mon-relation-changed output=stdout, stderr=stderr)
2019-03-13 08:50:39 DEBUG mon-relation-changed subprocess.CalledProcessError: Command '['ceph', '-c', '/dev/null', '--show-config']' returned non-zero exit status 1.
2019-03-13 08:50:39 ERROR juju.worker.uniter.operation runhook.go:132 hook "mon-relation-changed" failed: exit status 1
It appears the ``ceph`` command line tool requires a connection to a cluster even when it will not use it.
Alternatives:
1) Install python3-rados and get default like this:
import rados
r = rados.Rados( conffile= '/dev/null' ) get('rbd_ default_ features' ) # = '61'
r.conf_
2) Hard code '61' in the charm
Add unit test that validates the value against latest version of python3-rados so we keep up to date.