Wrong ceph auth client when disable telemetry

[Dimitri Savineau]

Description
===========
Deploying with ceph-ansible and disable-telemetry environment enabled causes a bad ceph auth osd capabilities for client.openstack.
This is a regression introduced by https://github.com/openstack/tripleo-heat-templates/commit/959cb6c5391bae657113ec6f69abe1a7cc277ee5
Because the GnocchiRbdPoolName parameter is set to '', then the osd capabilities gets an empty value for this pool.
As a result glance is not able to upload an image (and probably the other services using rbd too like nova and cinder).

Steps to reproduce
==================
* Deploy the overcloud with environments/ceph-ansible/ceph-ansible.yaml and environments/disable-telemetry.yaml
  $ openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-rgw.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml -e /home/stack/templates/tripleo.yaml -t 120
* Try to upload an image into glance.
  $ openstack image create --public --file cirros-0.3.5-x86_64-disk.img cirros-0.3.5-x86_64-disk.img

Expected result
===============
The image is correctly uploaded into glance and stored in ceph

Actual result
=============
A RBD permissions error in glance logs (see logs below)

# cat /etc/ceph/ceph.client.openstack.keyring
[client.openstack]
        key = AQApqCJbAAAAABAA4FZqczmue3pb+TW5DBmjhg==
        caps mds = ""
        caps mgr = "allow *"
        caps mon = "allow r"
        caps osd = "allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool="

As you can see at the end of the osd auth, there's an empty value after pool=
Efter updating the client.openstack osd capabilities by removing the extra ", allow rwx pool=" I don't see the error anymore

Environment
===========
os: RHEL 7.5
release: Rocky
storage: Ceph 12.2.4-6 (for nova, glance, cinder, rgw)
network: Openvswitch 2.9.0-19
nodes: 1 controler, 1 compute, 1 ceph
containers: enabled

Logs & Configs
==============
http://paste.openstack.org/show/723510/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/575571

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/575571
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=9746e2f9bb7926c6a5ff3028d5ae496ba4836a49
Submitter: Zuul
Branch: master

commit 9746e2f9bb7926c6a5ff3028d5ae496ba4836a49
Author: Giulio Fidente <email address hidden>
Date: Mon Jun 18 12:13:05 2018 +0200

    Do not grant caps if pool name is empty

    The openstack_keys map can have permissions for an empty pool
    which results in an invalid kerying.

    Co-Authored-By: Giulio Fidente <email address hidden>
    Change-Id: Ic5ae53d9ab52ea5e7c3f75a240a7a7f4bb5632ba
    Closes-Bug: 1776987

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/579591

[OpenStack Infra (hudson-openstack)]

Change abandoned by John Fulton (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/579591
Reason: messed up the cherrypick

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/579600

[OpenStack Infra (hudson-openstack)]

Change abandoned by Emilien Macchi (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/579600
Reason: The gate is suffering of timeouts, we need to clear it. Please do not restore or recheck this patch, I'll take care of it when gate is stable again.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/579600
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=70271cd94dee7020eaacb1e83036f66d2383f179
Submitter: Zuul
Branch: stable/queens

commit 70271cd94dee7020eaacb1e83036f66d2383f179
Author: Giulio Fidente <email address hidden>
Date: Mon Jun 18 12:13:05 2018 +0200

    Do not grant caps if pool name is empty

    The openstack_keys map can have permissions for an empty pool
    which results in an invalid kerying.

    Co-Authored-By: Giulio Fidente <email address hidden>
    Change-Id: Ic5ae53d9ab52ea5e7c3f75a240a7a7f4bb5632ba
    Closes-Bug: 1776987
    (cherry picked from commit 9746e2f9bb7926c6a5ff3028d5ae496ba4836a49)

[Dimitri Savineau (dsavineau)]

The fix proposed doesn't work and we still have the same result:

# on the controller node
# cat /etc/ceph/ceph.client.openstack.keyring
[client.openstack]
        key = AQAtu0xbAAAAABAAxnhiCioJG8F3FMX6xXe0dw==
        caps mgr = "allow *"
        caps mon = "profile rbd"
        caps osd = "profile rbd pool=volumes, profile rbd pool=backups, profile rbd pool=vms, profile rbd pool=images, profile rbd pool="

# on the undercloud
# cat /var/lib/mistral/config-download-latest/ceph-ansible/group_vars/all.yml
(...)
openstack_keys:
- caps:
        mgr: allow *
        mon: profile rbd
        osd: profile rbd pool=volumes, profile rbd pool=backups, profile rbd pool=vms,
            profile rbd pool=images, profile rbd pool=
    key: AQAtu0xbAAAAABAAxnhiCioJG8F3FMX6xXe0dw==
    mode: '0600'
    name: client.openstack

[John Fulton (jfulton-org)]

Dimitri,

Yes, you're right. Despite the following changes, I am still setting the same problem [1] on my queens overcloud.

https://review.openstack.org/#/c/579600/
https://review.openstack.org/#/c/571196/

  John

[1]

(undercloud) [stack@undercloud cephmetrics-osp]$ ansible -i tripleo-inventory.yaml all -b -m shell -a "cat /etc/ceph/ceph.client.openstack.keyring"
 [WARNING]: Skipping unexpected key (hostvars) in group (_meta), only "vars", "children" and "hosts" are valid

192.168.24.17 | FAILED | rc=1 >>
cat: /etc/ceph/ceph.client.openstack.keyring: No such file or directorynon-zero return code

192.168.24.14 | SUCCESS | rc=0 >>
[client.openstack]
 key = AQBNnFdbAAAAABAANfWmwQfho/H9i1vn2mVopQ==
 caps mds = "''"
 caps mgr = "'allow *'"
 caps mon = "'profile rbd'"
 caps osd = "'profile rbd pool=volumes, profile rbd pool=backups, profile rbd pool=vms, profile rbd pool=images, profile rbd pool='"

192.168.24.13 | SUCCESS | rc=0 >>
[client.openstack]
 key = AQBNnFdbAAAAABAANfWmwQfho/H9i1vn2mVopQ==
 caps mds = ""
 caps mgr = "allow *"
 caps mon = "profile rbd"
 caps osd = "profile rbd pool=volumes, profile rbd pool=backups, profile rbd pool=vms, profile rbd pool=images, profile rbd pool="

192.168.24.16 | SUCCESS | rc=0 >>
[client.openstack]
 key = AQBNnFdbAAAAABAANfWmwQfho/H9i1vn2mVopQ==
 caps mds = ""
 caps mgr = "allow *"
 caps mon = "profile rbd"
 caps osd = "profile rbd pool=volumes, profile rbd pool=backups, profile rbd pool=vms, profile rbd pool=images, profile rbd pool="

192.168.24.11 | SUCCESS | rc=0 >>
[client.openstack]
 key = AQBNnFdbAAAAABAANfWmwQfho/H9i1vn2mVopQ==
 caps mds = "''"
 caps mgr = "'allow *'"
 caps mon = "'profile rbd'"
 caps osd = "'profile rbd pool=volumes, profile rbd pool=backups, profile rbd pool=vms, profile rbd pool=images, profile rbd pool='"

localhost | FAILED | rc=1 >>
cat: /etc/ceph/ceph.client.openstack.keyring: No such file or directorynon-zero return code

192.168.24.7 | FAILED | rc=1 >>
cat: /etc/ceph/ceph.client.openstack.keyring: No such file or directorynon-zero return code

192.168.24.9 | FAILED | rc=1 >>
cat: /etc/ceph/ceph.client.openstack.keyring: No such file or directorynon-zero return code

(undercloud) [stack@undercloud cephmetrics-osp]$

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/585911

[John Fulton (jfulton-org)]

https://review.openstack.org/585911 in it's current state fixes it for me.

(undercloud) [stack@undercloud cephmetrics-osp]$ ansible -i inventory all -b -m shell -a "cat /etc/ceph/ceph.client.openstack.keyring"
overcloud-cephstorage-0 | FAILED | rc=1 >>
cat: /etc/ceph/ceph.client.openstack.keyring: No such file or directorynon-zero return code

overcloud-cephstorage-1 | FAILED | rc=1 >>
cat: /etc/ceph/ceph.client.openstack.keyring: No such file or directorynon-zero return code

overcloud-controller-2 | SUCCESS | rc=0 >>
[client.openstack]
        key = AQBNnFdbAAAAABAANfWmwQfho/H9i1vn2mVopQ==
        caps mds = ""
        caps mgr = "allow *"
        caps mon = "profile rbd"
        caps osd = "profile rbd pool=volumes, profile rbd pool=backups, profile rbd pool=vms, profile rbd pool=images, profile rbd pool=metrics"

overcloud-controller-1 | SUCCESS | rc=0 >>
[client.openstack]
        key = AQBNnFdbAAAAABAANfWmwQfho/H9i1vn2mVopQ==
        caps mds = ""
        caps mgr = "allow *"
        caps mon = "profile rbd"
        caps osd = "profile rbd pool=volumes, profile rbd pool=backups, profile rbd pool=vms, profile rbd pool=images, profile rbd pool=metrics"

overcloud-cephstorage-2 | FAILED | rc=1 >>
cat: /etc/ceph/ceph.client.openstack.keyring: No such file or directorynon-zero return code

overcloud-controller-0 | SUCCESS | rc=0 >>
[client.openstack]
        key = AQBNnFdbAAAAABAANfWmwQfho/H9i1vn2mVopQ==
        caps mds = "''"
        caps mgr = "'allow *'"
        caps mon = "'profile rbd'"
        caps osd = "'profile rbd pool=volumes, profile rbd pool=backups, profile rbd pool=vms, profile rbd pool=images, profile rbd pool=metrics'"

overcloud-novacompute-0 | SUCCESS | rc=0 >>
[client.openstack]
        key = AQBNnFdbAAAAABAANfWmwQfho/H9i1vn2mVopQ==
        caps mds = "''"
        caps mgr = "'allow *'"
        caps mon = "'profile rbd'"
        caps osd = "'profile rbd pool=volumes, profile rbd pool=backups, profile rbd pool=vms, profile rbd pool=images, profile rbd pool=metrics'"

(undercloud) [stack@undercloud cephmetrics-osp]$

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/586233

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.0.0.0b4 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.0.5 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/604734

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/604735

[OpenStack Infra (hudson-openstack)]

Change abandoned by Giulio Fidente (<email address hidden>) on branch: master
Review: https://review.openstack.org/586233
Reason: this is fixed by https://review.openstack.org/585911

[OpenStack Infra (hudson-openstack)]

Change abandoned by Juan Antonio Osorio Robles (<email address hidden>) on branch: master
Review: https://review.openstack.org/585911
Reason: Purging the gate to free up resources and address the timeout issues

[OpenStack Infra (hudson-openstack)]

Change abandoned by Juan Antonio Osorio Robles (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/604735
Reason: Purging the gate to free up resources and address the timeout issues

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/604734
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=5a66f6ce0e257ee631c1469a89a5b34a8bd8405a
Submitter: Zuul
Branch: stable/rocky

commit 5a66f6ce0e257ee631c1469a89a5b34a8bd8405a
Author: Giulio Fidente <email address hidden>
Date: Thu Sep 20 14:33:25 2018 +0200

    Stop cap granting to empty pool when telemetry disabled

    Attempt to not create GnocchiRbdPool pool when it is set to empty
    string [1] resulted in related bug which was not resolved by
    originally proposed fix [2].

    1. https://review.openstack.org/#/c/575571
    2. https://review.openstack.org/#/c/570043

    Change-Id: Ie7a42822be89cced480302d40180b9972d191004
    Closes-Bug: 1776987
    (cherry picked from commit e0b52904c05b8c2df0471f97ebc374af7f51324b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/604735
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=17dec684dc0bc64709700ba663fab48e9a50e153
Submitter: Zuul
Branch: stable/queens

commit 17dec684dc0bc64709700ba663fab48e9a50e153
Author: Giulio Fidente <email address hidden>
Date: Thu Sep 20 14:33:25 2018 +0200

    Stop cap granting to empty pool when telemetry disabled

    Attempt to not create GnocchiRbdPool pool when it is set to empty
    string [1] resulted in related bug which was not resolved by
    originally proposed fix [2].

    1. https://review.openstack.org/#/c/575571
    2. https://review.openstack.org/#/c/570043

    Change-Id: Ie7a42822be89cced480302d40180b9972d191004
    Closes-Bug: 1776987
    (cherry picked from commit e0b52904c05b8c2df0471f97ebc374af7f51324b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/585911
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=e0b52904c05b8c2df0471f97ebc374af7f51324b
Submitter: Zuul
Branch: master

commit e0b52904c05b8c2df0471f97ebc374af7f51324b
Author: Giulio Fidente <email address hidden>
Date: Thu Sep 20 14:33:25 2018 +0200

    Stop cap granting to empty pool when telemetry disabled

    Attempt to not create GnocchiRbdPool pool when it is set to empty
    string [1] resulted in related bug which was not resolved by
    originally proposed fix [2].

    1. https://review.openstack.org/#/c/575571
    2. https://review.openstack.org/#/c/570043

    Change-Id: Ie7a42822be89cced480302d40180b9972d191004
    Closes-Bug: 1776987

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.1.0 release.