broken authentication when using multiple ceph backends on a single cinder application

[Peter Sabaini]

When removing a cinder-ceph storage backend relation we ended up with a env var override for the cinder service that referenced the non-existing ceph authx key from cinder-ceph

Detail:

1. Pre-existing cinder service configured for rbd backend

2. Added a cinder-ceph service foo-storage and relate to cinder

3. Removed relation cinder - cinder-ceph

This reconfigured the original rbd backend in cinder.conf, however it left an override for ceph authx behind:

cat /etc/init/cinder-volume.override
env CEPH_ARGS="--id foo-storage"

As the foo-storage key doesn't exist at this point the cinder-volume service errored out with (redacted):

2018-05-04 14:22:19.576 4010160 ERROR cinder.volume.manager [req-fd6ff4fd-f041-47c7-a2da-144ea353fef9 8bb46780e82b40c594476a7609b1b88f 5f182a7b537a4aa7b40832ed4bbbfbed - - -] Driver initialize connection failed (error: Unexpected error while running command.
Command: ceph mon dump --format=json --id cinder --cluster ceph
Exit code: 1
Stdout: u''
Stderr: u'2018-05-04 14:22:19.561548 7ff990749700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.foo-storage.keyring: (2) No such file or directory\n2018-05-04 14:22:19.561560 7ff990749700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication\n2018-05-04 14:22:19.5
61562 7ff990749700 0 librados: client.foo-storage initialization error (2) No such file or directory\nError connecting to cluster: ObjectNotFound\n').

Afaict those env settings are done on ceph-relation-changed in set_ceph_env_variables()

Version info: 17.08 charms on a trusty/mitaka cloud.

[James Page (james-page)]

I think this is side-effect of the history of this deployment - the ceph support directly in the cinder charm also writes this override, but I'm pretty sure that for any recent openstack release its simply not required so we could drop it from the charm and tidy up in a commit.

However that probably does not help the 'how do I move to cinder-ceph' use case that you have here.

[James Page (james-page)]

Confirmed:

        return {
            "cinder": {
                "/etc/cinder/cinder.conf": {
                    "sections": {
                        service: [
                            ('volume_backend_name', service),
                            ('volume_driver', volume_driver),
                            ('rbd_pool', service),
                            ('rbd_user', service),
                            ('rbd_secret_uuid', leader_get('secret-uuid')),
                        ]
                    }
                }
            }
        }

rbd_user is set.

[James Page (james-page)]

We should probably also be writing out rbd_ceph_conf to allow multiple ceph clusters to be used from the same cinder-volume service.

[James Page (james-page)]

That option is supported since mitaka at least.

[James Page (james-page)]

Infact all the way back to icehouse; this needs a refresh across both charms to avoid this conflict and to allow migration to a cinder-ceph configured backend.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/567554

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/567556

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/567554
Committed: https://git.openstack.org/cgit/openstack/charm-cinder/commit/?id=3451c1c4985359a9a1c4c1c7527808fd84a380e6
Submitter: Zuul
Branch: master

commit 3451c1c4985359a9a1c4c1c7527808fd84a380e6
Author: James Page <email address hidden>
Date: Thu May 10 11:38:59 2018 +0100

    Tidy ceph backend configuration

    Drop generation of upstart override file and /etc/environment and
    scrub any existing charm configuration in these locations from
    an existing install.

    These where required way back in the dawn of time when ceph support
    was alpha/beta in cinder.

    Provide backend specific configuration file path, allowing multiple
    ceph clusters to be used with a single cinder application.

    Change-Id: I7adba0d35fb7406afa40f047b79a9ab51a6a333d
    Closes-Bug: 1769196

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/567556
Committed: https://git.openstack.org/cgit/openstack/charm-cinder-ceph/commit/?id=b1829e983ae3b56e016cf9de437fde35b1f6a83f
Submitter: Zuul
Branch: master

commit b1829e983ae3b56e016cf9de437fde35b1f6a83f
Author: James Page <email address hidden>
Date: Thu May 10 11:48:38 2018 +0100

    Tidy ceph backend configuration

    Drop generation of upstart override file and /etc/environment
    and scrub any existing charm configuration in these locations
    from an existing install.

    These where required way back in the dawn of time when ceph
    support was alpha/beta in cinder.

    Provide backend specific configuration file path, allowing
    multiple ceph clusters to be used with a single cinder
    application.

    Change-Id: I8a097e4de1c5c980f118a587a1a64792fad2fa05
    Closes-Bug: 1769196