charm adds extra port to monitor-hosts in ceph.conf as IP_ADDRESS:6789:6789

[Vern Hart]

Docs for ceph-proxy say monitor-hosts can contain multiple space delimited entries with the form:

  {IP / Hostname}:{port} {IP / Hostname}:{port}

However, in the /etc/ceph/ceph.conf file on the gnocchi units we end up with:

  mon host = 10.a.a.a:6789 10.b.b.b:6789 10.c.c.c:6789:6789

The difference between the behavior when gnocchi is related to ceph-proxy versus ceph-mon is that ceph-mon sends a single address:

  https://github.com/openstack/charm-ceph-mon/blob/master/hooks/ceph_hooks.py#L679

            public_addr = get_public_addr()
            data = {'key': ceph.get_named_key(service_name),
                    'auth': config('auth-supported'),
                    'ceph-public-address': public_addr}

whereas ceph-proxy sends the string of the config as is:

  https://github.com/openstack/charm-ceph-proxy/blob/master/hooks/ceph_hooks.py#L191

            ceph_addrs = config('monitor-hosts')
            data = {'key': ceph.get_named_key(service_name),
                    'auth': config('auth-supported'),
                    'ceph-public-address': ceph_addrs}

The gnocchi charm seems to assume this is an array of hosts.

String versus list aside, I can't actually figure out where the extra port is coming from.

[Vern Hart (vern)]

If I run ceph --id gnocchi mon_status I get:

  server name not found: 10.c.c.c:6789:6789 (Temporary failure in name resolution)
  unable to parse addrs in '10.a.a.a:6789 10.b.b.b:6789 10.c.c.c:6789:6789'
  InvalidArgumentError does not take keyword arguments

If I temporarily change ceph.conf (because juju changes it back) to remove the extra port and run the ceph command I get normal looking mon_status:

{"name":"CephBlockMon03","rank":2,"state":"peon","election_epoc":46,"quorum":[0,1,2],"features":{"required_con":"159873415184952845","required_mon":["kraken","luminous"],"quorum_con":"349823480294924932","quorum_mon":["kraken","luminous"]},"outside_quorum":[],"extra_probe_peers":["10.a.a.a:6789/0","10.b.b.b:6789/0"],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3d1e2586-338f-11e9-929a-02f2b9b313b8","modified":"2019-02-11 14:52:27.531389","created":"2019-02-11 14:52:27.531389","features":{"persistent":["kraken","luminous"],"optional":[]},"mons":[{"rank":0,"name":"CephBlockMon01","addr":"10.a.a.a:6789/0","public_addr":"10.a.a.a:6789/0"},{"rank":1,"name":"CephBlockMon02","addr":"10.b.b.b:6789/0","public_addr":"10.b.b.b:6789/0"},{"rank":2,"name":"CephBlockMon03","addr":"10.c.c.c:6789/0","public_addr":"10.c.c.c:6789/0"}]},"feature_map":{"mon":{"group":{"features":"0x3ffddff8eea4fffb","release":"luminous","num":1}},"osd":{"group":{"features":"0x3ffddff8eea4fffb","release":"luminous","num":132}},"client":{"group":{"features":"0x3ffddff8eea4fffb","release":"luminous","num":1}}}}

[Nobuto Murata (nobuto)]

I've reproduced it as:
unable to parse addrs in '10.0.8.129:6789 10.0.8.169:6789 10.0.8.188:6789:6789'

More details on steps:

## deploy ceph cluster
$ juju deploy --series bionic ceph-mon -n 3
Located charm "cs:ceph-mon-32".
Deploying charm "cs:ceph-mon-32".

$ juju config ceph-mon auth-supported=none

$ juju deploy --series bionic ceph-osd -n 3
Located charm "cs:ceph-osd-276".
Deploying charm "cs:ceph-osd-276".

$ juju config ceph-osd osd-devices=/srv/ceph
$ juju add-relation ceph-osd:mon ceph-mon:osd

$ juju wait -w

## ceph-proxy
$ juju deploy --series bionic ceph-proxy
Located charm "cs:ceph-proxy-17".
Deploying charm "cs:ceph-proxy-17".

$ juju config ceph-proxy auth-supported=none

$ juju config ceph-proxy fsid=$(juju run --unit ceph-mon/0 -- leader-get fsid)
$ juju config ceph-proxy fsid
-> bc8a8a36-3389-11e9-b320-00163ebb364a

$ juju config ceph-proxy admin-key=$(juju run --unit ceph-mon/0 -- leader-get monitor-secret)
$ juju config ceph-proxy admin-key
-> AQCDwWpcswHFMhAA/Jf9ME6pYq1a5e0jfH3Paw==

$ juju config ceph-proxy monitor-hosts="$(juju run --unit ceph-mon/0 -- grep '^mon host' /etc/ceph/ceph.conf | awk -F '= ' '{print $2}')"
$ juju config ceph-proxy monitor-hosts
-> 10.0.8.128:6789 10.0.8.167:6789 10.0.8.171:6789

## gnocchi and dependencies
$ juju deploy --series bionic gnocchi
Located charm "cs:gnocchi-16".
Deploying charm "cs:gnocchi-16".

$ juju deploy --series bionic keystone
$ juju deploy --series bionic percona-cluster mysql
$ juju deploy --series bionic memcached

$ juju add-relation gnocchi mysql
$ juju add-relation keystone mysql
$ juju add-relation gnocchi keystone
$ juju add-relation gnocchi memcached

$ juju ssh gnocchi/0 -- sudo cat /etc/ceph/ceph.conf | grep 'mon host'
mon host = 10.0.8.129:6789 10.0.8.169:6789 10.0.8.188:6789:6789

-> :6789:6789 is added

[Nobuto Murata (nobuto)]

`juju export-bundle` for your reference of the deployed model.

[Nobuto Murata (nobuto)]

Subscribing ~field-critical since Gnocchi deployment with ceph-proxy charm is blocked.

This could be workarounded by omitting the port number from monitor-hosts config in ceph-proxy charm since 6789 is the default anyway. But until it's confirmed there is a workaround, this bug should be treated as critical.

[Nobuto Murata (nobuto)]

Simpler steps:

$ juju deploy ./gnocchi_bundle.yaml
$ juju wait

$ juju deploy --series bionic ceph-proxy
$ juju config ceph-proxy auth-supported=none

$ juju config ceph-proxy fsid=$(juju run --unit ceph-mon/0 -- leader-get fsid)
$ juju config ceph-proxy admin-key=$(juju run --unit ceph-mon/0 -- leader-get monitor-secret)

$ juju config ceph-proxy monitor-hosts="$(juju run --unit ceph-mon/0 -- grep '^mon host' /etc/ceph/ceph.conf | awk -F '= ' '{print $2}')"

$ juju add-relation gnocchi:storage-ceph ceph-proxy:client

$ juju ssh gnocchi/0 -- sudo cat /etc/ceph/ceph.conf | grep 'mon host'

[Nobuto Murata (nobuto)]

Downgrading to ~field-medium since I was told by on-site engineer (Vern) that a workaround as dropping the port number from the charm config worked for now.

It will be escalated again in case the workaround has a side effect to other charms than Gnocchi.

[Vern Hart (vern)]

I was able to get this working by dropping the ports on the ceph-proxy monitor-hosts config:

$ juju config ceph-proxy monitor-hosts="10.a.a.a 10.b.b.b 10.c.c.c"

$ juju ssh gnocchi/0 sudo cat /etc/ceph/ceph.conf | grep mon.host
mon host = 10.a.a.a 10.b.b.b 10.c.c.c:6789

$ juju ssh glance/0 sudo cat /etc/ceph/ceph.conf | grep mon.host
mon host = 10.a.a.a 10.b.b.b 10.c.c.c

This only works because the ports are the default ports.

Theoretically, I could add back in the ports on the first two hosts but leave it off of the third.

[Chris MacNaughton (chris.macnaughton)]

This looks like a bug in the charm-interface-ceph-client project, so will affect all reactive charms that use ceph.

[Vern Hart (vern)]

We redeployed today and, after reconfiguring the ceph cluster and entering the new mon server info, we encountered this bug again.

We removed the ports from the IPs as a workaround.

Again, this workaround only works because 6789 is the default port.

  juju config ceph-proxy monitor-hosts="10.a.a.a 10.b.b.b 10.c.c.c"