cinder-ceph not using data from ceph-proxy relation

[Vance Morris]

$ juju version
2.0.1-xenial-s390x

After deploying and configuring ceph-proxy, I am able to run ceph commands from inside the unit:
root@juju-01984a-0-lxd-6:~# ceph osd pool ls
rbd
libvirt-pool
default.rgw.buckets
default.rgw
default.rgw.root
default.rgw.control
default.rgw.gc
default.rgw.buckets.index
default.rgw.buckets.extra
default.log
default.intent-log
default.usage
default.users
default.users.email
default.users.swift
default.users.uid

I have cinder and cinder-ceph deployed as well, with the following relations:
Relation Provides Consumes Type
ceph ceph-proxy cinder-ceph regular
cluster cinder cinder peer
storage-backend cinder cinder-ceph subordinate

Connecting in to the cinder unit, ceph commands are failing, and it seems that the ceph configuration doesn't make it in from the proxy charm:
root@juju-01984a-0-lxd-0:~# ceph osd pool ls
no monitors specified to connect to.
Error connecting to cluster: ObjectNotFound
root@juju-01984a-0-lxd-0:~# cat /etc/ceph/ceph.conf
###############################################################################
# [ WARNING ]
# cinder configuration file maintained by Juju
# local changes may be overwritten.
###############################################################################
[global]
log to syslog =
err to syslog =
clog to syslog =

[client]

Looking at the debug-log on cinder-ceph, I see where it seeks the mon_hosts, key, and auth data from the relationship, and seems to be satisfied...
http://paste.ubuntu.com/23487994/

[Vance Morris (vmorris)]

juju status
http://paste.ubuntu.com/23488004/

[Felipe Reyes (freyes)]

This is how nova-compute will see the IP of the monitor hosts when
nova-compute and ceph (real ceph, not ceph-proxy) are related:

$ juju run --unit nova-compute/0 "relation-get -r ceph:20 - ceph/0"
auth: cephx
ceph-public-address: 10.5.156.36
key: AQCe7JhY/J7BKRAANcBJdNIMZXiYJA5KqRi0Nw==
private-address: 10.5.156.36
$ juju run --unit nova-compute/0 "relation-get -r ceph:20 - ceph/1"
auth: cephx
ceph-public-address: 10.5.156.37
key: AQCe7JhY/J7BKRAANcBJdNIMZXiYJA5KqRi0Nw==
private-address: 10.5.156.37
$ juju run --unit nova-compute/0 "relation-get -r ceph:20 - ceph/2"
auth: cephx
ceph-public-address: 10.5.156.38
key: AQCe7JhY/J7BKRAANcBJdNIMZXiYJA5KqRi0Nw==
private-address: 10.5.156.38

The code to generate ceph.conf looks for ceph-public-address and falls back to
use the private-address[0], the content of ceph-public-address shouldn't be a
random string (i.e. inject the IPs of 3 hosts concatenated) because
format_ipv6_addr()[1] will fail and ipv6 addresses won't be properly
formatted.

In an environment using ceph-proxy, the value for ceph-public-address is the
same of the private-address:

$ juju run --unit nova-compute/0 "relation-get -r ceph:35 - ceph-proxy/0"
auth: cephx
ceph-public-address: 10.5.156.34
key: AQDhm5hYV/+bNRAAR7U7zVirnmlAz2E9GmcdbA==
private-address: 10.5.156.34

ceph-proxy will populate the local ceph.conf (local to where ceph-proxy is
running) with the mon hosts IPs[2] and it will set
ceph-public-address=get_public_addr() in the 'client' relation with other
applications[3]

The change needed is that ceph-proxy sets in the relation the values in monitor-hosts, but it will be able to put a single IP, so the choices I see are:

1) change charmhelpers.contrib.openstack.context.CephContext to split the value of ceph-public-address and iterate that list
2) add a new field in the relation to contain a list of IPs, for example ceph-mon-hosts

Thoughts?

[0] http://bazaar.launchpad.net/~charm-helpers/charm-helpers/devel/view/head:/charmhelpers/contrib/openstack/context.py#L478
[1] http://bazaar.launchpad.net/~charm-helpers/charm-helpers/devel/view/head:/charmhelpers/contrib/openstack/context.py#L489
[2] https://github.com/openstack/charm-ceph-proxy/blob/master/hooks/ceph_hooks.py#L78
[3] https://github.com/openstack/charm-ceph-proxy/blob/master/hooks/ceph_hooks.py#L174

[Felipe Reyes (freyes)]

Adding charm-helpers task, because charmhelpers.contrib.openstack.context.CephContext needs to be fixed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/441214

[OpenStack Infra (hudson-openstack)]

Change abandoned by Felipe Reyes (<email address hidden>) on branch: master
Review: https://review.openstack.org/441216

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/441242
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-radosgw/commit/?id=d4c7935583dd138a246a0661338cb938acb80214
Submitter: Jenkins
Branch: master

commit d4c7935583dd138a246a0661338cb938acb80214
Author: Felipe Reyes <email address hidden>
Date: Fri Mar 3 14:21:07 2017 -0300

    charm-helpers sync to pickup changes to CephContext

    CephContext now is capable of dealing with a space separated list of IPs
    in the ceph-public-address field of the relation with ceph. This allows
    ceph-proxy to pass a list of monitor hosts instead of a single one.

    Change-Id: Iff04f93911bd8fcecf5afeb71feda853b30d74d6
    Partial-Bug: 1642430

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/441214
Committed: https://git.openstack.org/cgit/openstack/charm-cinder-ceph/commit/?id=644849982e50d16012f59c45b4ae3e1d87b44393
Submitter: Jenkins
Branch: master

commit 644849982e50d16012f59c45b4ae3e1d87b44393
Author: Felipe Reyes <email address hidden>
Date: Fri Mar 3 13:18:00 2017 -0300

    charm-helpers sync to pickup changes to CephContext

    CephContext now is capable of dealing with a space separated list of IPs
    in the ceph-public-address field of the relation with ceph. This allows
    ceph-proxy to pass a list of monitor hosts instead of a single one.

    Change-Id: I295d2d837b3864a4222007288de5ed28c9d2ce54
    Partial-Bug: 1642430

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/441237
Committed: https://git.openstack.org/cgit/openstack/charm-glance/commit/?id=896ea18db03aef92f0d4bfaf9885c6fdcde38c5a
Submitter: Jenkins
Branch: master

commit 896ea18db03aef92f0d4bfaf9885c6fdcde38c5a
Author: Felipe Reyes <email address hidden>
Date: Fri Mar 3 14:15:34 2017 -0300

    charm-helpers sync to pickup changes to CephContext

    CephContext now is capable of dealing with a space separated list of IPs
    in the ceph-public-address field of the relation with ceph. This allows
    ceph-proxy to pass a list of monitor hosts instead of a single one.

    Change-Id: I83abc2c151636aea0b5dffc32f2e3c91acb7558b
    Partial-Bug: 1642430
    Closes-Bug: 1670223

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/450470
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-radosgw/commit/?id=d9ee86297eb8ccbb1ac921d76869c1c0d44d96ab
Submitter: Jenkins
Branch: master

commit d9ee86297eb8ccbb1ac921d76869c1c0d44d96ab
Author: Felipe Reyes <email address hidden>
Date: Mon Mar 27 18:47:51 2017 -0300

    Handle list of addresses in ceph-public-address

    charmhelpers' CephContext class is capable of handling a list of IP
    addresses in the ceph-public-address attribute, this allows
    ceph-proxy to hand the list of monitor hosts.

    Closes-Bug: 1642430
    Change-Id: I2bbff167fce2c75a3a619b658f0c569c6d5be3d5

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/436950
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-proxy/commit/?id=f250c2274a52707bba60576f5c85d120f79d65ca
Submitter: Jenkins
Branch: master

commit f250c2274a52707bba60576f5c85d120f79d65ca
Author: Felipe Reyes <email address hidden>
Date: Fri Feb 17 14:25:47 2017 -0300

    Set ceph-public-address with monitor-hosts config

    charms capable of relating to ceph build the list of monitor hosts
    from the ceph-public-address in the relation data or simply using the
    private-address field provided by juju.

    This patch populates ceph-public-address in the relations with the
    list of monitor hosts provided in the configuration, so clients connect
    directly to the non juju managed ceph and not to ceph-proxy IP.

    Closes-Bug: #1642430
    Change-Id: Ib2e3c6a8b0d9fc0b6d729d845a802945b64fc74c

[Felipe Reyes (freyes)]

glance and nova-compute received the fix via a charm-helpers sync

[Florian Haas (fghaas)]

It is now over a month since these fixes were committed. As far as i can tell, this bug completely breaks the charm's advertised functionality — if I want ceph-proxy to use its own address rather than the mon addresses it is configured to use, why would I even want to use this charm and not ceph-mon/ceph-osd directly?

This charm has been released to the charm store, and nothing in https://jujucharms.com/ceph-proxy/ indicates that is is alpha, beta, or otherwise not ready for production use, so you must assume that people will want to use this in production. So, a complete functionality breakage would normally warrant a new charm release outside the normal "milestone" cycle. Can you please consider giving this charm a release so it is functionally useful to people?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/17.02
Review: https://review.openstack.org/468569

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/17.02
Review: https://review.openstack.org/468570

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/17.02
Review: https://review.openstack.org/468571

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/17.02
Review: https://review.openstack.org/468572

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/17.02
Review: https://review.openstack.org/468573

[Felipe Reyes (freyes)]

To test the patches that fix this bug, I used the following bundles and scripts

- gen.py http://paste.ubuntu.com/24671884/
- standalone-ceph.yaml http://paste.ubuntu.com/24671885/
- openstack-with-ceph-proxy.yaml.tpl http://paste.ubuntu.com/24671887/

Usage:

0) get copies of the charms with the proposed backport
  - https://review.openstack.org/#/c/468569/
  - https://review.openstack.org/#/c/468570/
  - https://review.openstack.org/#/c/468571/
  - https://review.openstack.org/#/c/468572/
  - https://review.openstack.org/#/c/468573/
1) juju deploy ./standalone-ceph.yaml
2) .. wait until ceph is done
3) ./gen.py
  - This script will read openstack-with-ceph-proxy.yaml.tpl, populate the admin-key and monitor-hosts fields for the ceph-proxy charm and write openstack-with-ceph-proxy.yaml
4) juju deploy ./openstack-with-ceph-proxy.yaml

Verification

- juju ssh ceph/0 sudo rados lspools # pool for cinder, glance, rgw and rbd need to be created
- mon hosts is properly configured in the related units:
  juju run --application ceph-proxy "sudo cat /etc/ceph/ceph.conf"
  juju run --application nova-compute "sudo cat /etc/ceph/ceph.conf"
  juju run --application cinder-ceph "sudo cat /etc/ceph/ceph.conf"
  juju run --application ceph-radosgw "sudo cat /etc/ceph/ceph.conf"
  juju run --application glance "sudo cat /etc/ceph/ceph.conf"

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/468569
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-radosgw/commit/?id=a56708973e70e6c038246ad447871579d1bea7ac
Submitter: Jenkins
Branch: stable/17.02

commit a56708973e70e6c038246ad447871579d1bea7ac
Author: Felipe Reyes <email address hidden>
Date: Mon Mar 27 18:47:51 2017 -0300

    Handle list of addresses in ceph-public-address

    charmhelpers' CephContext class is capable of handling a list of IP
    addresses in the ceph-public-address attribute, this allows
    ceph-proxy to hand the list of monitor hosts.

    Revision 708 from lp:charm-helpers needed to allow functional tests execute
    'juju actions'.

    Closes-Bug: 1642430
    Change-Id: I2bbff167fce2c75a3a619b658f0c569c6d5be3d5
    (cherry picked from commit d9ee86297eb8ccbb1ac921d76869c1c0d44d96ab)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/468572
Committed: https://git.openstack.org/cgit/openstack/charm-nova-compute/commit/?id=396d4a49ea89e341164b3c834c55312aece42e8a
Submitter: Jenkins
Branch: stable/17.02

commit 396d4a49ea89e341164b3c834c55312aece42e8a
Author: Felipe Reyes <email address hidden>
Date: Fri May 26 16:02:37 2017 -0400

    Allow ceph-public-address to contain a list of IPs

    Ceph relation allows units to hand off an IP different from the
    public-address or private-address using the ceph-public-address key, that
    key can hold a single IP address, this patch extends CephContext to be
    able to handle a list of space separated addresses.

    This is needed to let ceph-proxy hand off the IPs of the monitor hosts from
    an external Ceph cluster. Backport from lp:charm-helpers revision 703

    Revision 708 from lp:charm-helpers needed to allow functional tests execute
    'juju actions'.

    Change-Id: I01a5ca030783364075e58a2a2b34e9a1997f20d6
    Closes-Bug: 1642430

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/468570
Committed: https://git.openstack.org/cgit/openstack/charm-glance/commit/?id=d456104952168d3b1009a9fb929c554f2625387e
Submitter: Jenkins
Branch: stable/17.02

commit d456104952168d3b1009a9fb929c554f2625387e
Author: Felipe Reyes <email address hidden>
Date: Fri May 26 16:39:28 2017 -0400

    Allow ceph-public-address to contain a list of IPs

    Ceph relation allows units to hand off an IP different from the
    public-address or private-address using the ceph-public-address key, that
    key can hold a single IP address, this patch extends CephContext to be
    able to handle a list of space separated addresses.

    This is needed to let ceph-proxy hand off the IPs of the monitor hosts from
    an external Ceph cluster. Backport from lp:charm-helpers revision 703

    Revision 708 from lp:charm-helpers needed to allow functional tests execute
    'juju actions'.

    Change-Id: Id8bc88e0276bd17917aba0d553fde7ec05f44726
    Closes-Bug: 1642430

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/468573
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-proxy/commit/?id=a4ceb5c8000f0f5fe9b5649150b579f1d9fa1acb
Submitter: Jenkins
Branch: stable/17.02

commit a4ceb5c8000f0f5fe9b5649150b579f1d9fa1acb
Author: Felipe Reyes <email address hidden>
Date: Fri Feb 17 14:25:47 2017 -0300

    Set ceph-public-address with monitor-hosts config

    charms capable of relating to ceph build the list of monitor hosts
    from the ceph-public-address in the relation data or simply using the
    private-address field provided by juju.

    This patch populates ceph-public-address in the relations with the
    list of monitor hosts provided in the configuration, so clients connect
    directly to the non juju managed ceph and not to ceph-proxy IP.

    Closes-Bug: #1642430
    Change-Id: Ib2e3c6a8b0d9fc0b6d729d845a802945b64fc74c
    (cherry picked from commit f250c2274a52707bba60576f5c85d120f79d65ca)

[Felipe Reyes (freyes)]

All the fixes needed landed in stable/17.02, marking the tasks as "fix released" https://review.openstack.org/#/q/topic:bug/1642430+branch:stable/17.02