When adding additional ceph units to a single ceph deployment, and changing the 'expected_mons' value from '1' to '3' causes 'ceph.client.admin.keyring' to get updated with invalid credentials on the original ceph unit.
Example adding additional ceph units to machines '3' and '4':
$ juju add-unit --to=lxc:3 ceph
$ juju add-unit --to=lxc:4 ceph
$ juju service set ceph monitor-count=3
`ceph status` output from each ceph unit:
________
2018-02-09 18:09:51.294276 7f8fd7d4f700 0 librados: client.admin authentication error (1) Operation not permitted
Error connecting to cluster: PermissionError
2018-02-09 18:09:52.953990 7fe0c54f9700 0 librados: client.admin authentication error (1) Operation not permitted
Error connecting to cluster: PermissionError
2018-02-09 18:09:53.622467 7f8d12177700 0 librados: client.admin authentication error (1) Operation not permitted
Error connecting to cluster: PermissionError
________
When adding additional ceph units to a single ceph deployment, and changing the 'expected_mons' value from '1' to '3' causes 'ceph.client. admin.keyring' to get updated with invalid credentials on the original ceph unit.
Example adding additional ceph units to machines '3' and '4':
$ juju add-unit --to=lxc:3 ceph
$ juju add-unit --to=lxc:4 ceph
$ juju service set ceph monitor-count=3
`ceph status` output from each ceph unit:
________
2018-02-09 18:09:51.294276 7f8fd7d4f700 0 librados: client.admin authentication error (1) Operation not permitted
Error connecting to cluster: PermissionError
2018-02-09 18:09:52.953990 7fe0c54f9700 0 librados: client.admin authentication error (1) Operation not permitted
Error connecting to cluster: PermissionError
2018-02-09 18:09:53.622467 7f8d12177700 0 librados: client.admin authentication error (1) Operation not permitted
Error connecting to cluster: PermissionError
________