ceph-rgw installation/configuration lacks idempotency, configuration changes break the setup
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| OpenStack-Ansible |
Fix Released
|
Medium
|
Unassigned | ||
Bug Description
Deploying a Ceph radosgw with the ceph-rgw-
Steps to reproduce:
(1) Create a user_variables.yml configuration that includes the following items:
ceph_conf_
"client.rgw.{{ ansible_hostname }}":
host: "{{ ansible_hostname }}"
rgw keystone accepted roles: "Member, member, _member_, admin, swiftoperator"
rgw keystone admin domain: default
rgw keystone admin password: "{{ radosgw_
rgw keystone admin project: "{{ radosgw_
rgw keystone admin tenant: "{{ radosgw_
rgw keystone admin user: "{{ radosgw_admin_user }}"
rgw keystone api version: 3
rgw keystone revocation interval: 900
rgw keystone token cache size: 10000
rgw keystone url: "{{ keystone_
rgw swift account in url: "true"
rgw enable apis: swift
rgw swift url prefix: ""
rgw swift versioning enabled: true
(2) Run setup-everythin
(3) Shell into (or attach to) one of the rgw containers.
You'll see a ceph.conf similar to this:
[client.
host = daisy-ceph-
keyring = /var/lib/
log file = /var/log/
rgw frontends = civetweb port=192.
[client.
host = eric-ceph-
keyring = /var/lib/
log file = /var/log/
rgw frontends = civetweb port=192.
[client.
host = frank-ceph-
keyring = /var/lib/
log file = /var/log/
rgw frontends = civetweb port=192.
[client.
host = daisy-ceph-
rgw enable apis = swift
rgw keystone accepted roles = Member, member, _member_, admin, swiftoperator
rgw keystone admin domain = default
rgw keystone admin password = <password>
rgw keystone admin project = service
rgw keystone admin tenant = service
rgw keystone admin user = radosgw
rgw keystone api version = 3
rgw keystone revocation interval = 900
rgw keystone token cache size = 10000
rgw keystone url = http://
rgw swift account in url = true
rgw swift url prefix =
rgw swift versioning enabled = True
(4) Change your user_variables.yml:
ceph_conf_
"client.rgw.{{ ansible_hostname }}":
host: "{{ ansible_hostname }}"
rgw keystone accepted roles: "Member, member, _member_, admin, swiftoperator"
rgw keystone admin domain: default
rgw keystone admin password: "{{ radosgw_
rgw keystone admin project: "{{ radosgw_
rgw keystone admin tenant: "{{ radosgw_
rgw keystone admin user: "{{ radosgw_admin_user }}"
rgw keystone api version: 3
rgw keystone revocation interval: 900
rgw keystone token cache size: 10000
rgw keystone url: "{{ keystone_
rgw swift account in url: "true"
rgw enable apis: swift
rgw swift url prefix: "/"
rgw swift versioning enabled: true
(The only change is "rgw swift url prefix" going from "" to "/".)
(5) Run the ceph-rgw-
(6) Again, look into the generated ceph.conf in one of the rgw containers:
[client.rgw.]
host =
keyring = /var/lib/
log file = /var/log/
log file = /var/log/
log file = /var/log/
rgw frontends = civetweb port=192.
rgw frontends = civetweb port=192.
rgw frontends = civetweb port=192.
[client.
host = daisy-ceph-
rgw enable apis = swift
rgw keystone accepted roles = Member, member, _member_, admin, swiftoperator
rgw keystone admin domain = default
rgw keystone admin password = 306f87d512e8b36
rgw keystone admin project = service
rgw keystone admin tenant = service
rgw keystone admin user = radosgw
rgw keystone api version = 3
rgw keystone revocation interval = 900
rgw keystone token cache size = 10000
rgw keystone url = http://
rgw swift account in url = true
rgw swift url prefix = /
rgw swift versioning enabled = True
# Ansible managed
[global]
cluster network = 192.168.155.0/24
fsid = 8c4846f1-
mon host = 192.168.
osd pool default min size = 1
osd pool default size = 2
public network = 192.168.155.0/24
Observe the configuration differences:
- [client.rgw.] section (as if the hostname was empty)
- keyring option pointing to a non-existing key file
- log file options for all 3 rgw hosts squished together
- rgw frontends options now completely useless
- correctly created host-specific section now lacks the rgw frontends option, meaning it's now listening on port 7480, not 8080.
- all haproxy backends are now dead, because haproxy still expects rgw to listen on port 8080
- the radosgw service, and hence the cloud's Swift API endpoint, is now non-functional.
(7) Try rolling back the change, by restoring "rgw swift url prefix" to "" in user_variables.yml and rerunning ceph-rgw-
(8) Observe that the only change this generates is in the host-specific section, whereas the useless [client.rgw.] section remains, and the service is still non-functional.
[client.rgw.]
host =
keyring = /var/lib/
log file = /var/log/
log file = /var/log/
log file = /var/log/
rgw frontends = civetweb port=192.
rgw frontends = civetweb port=192.
rgw frontends = civetweb port=192.
[client.
host = daisy-ceph-
rgw enable apis = swift
rgw keystone accepted roles = Member, member, _member_, admin, swiftoperator
rgw keystone admin domain = default
rgw keystone admin password = 306f87d512e8b36
rgw keystone admin project = service
rgw keystone admin tenant = service
rgw keystone admin user = radosgw
rgw keystone api version = 3
rgw keystone revocation interval = 900
rgw keystone token cache size = 10000
rgw keystone url = http://
rgw swift account in url = true
rgw swift url prefix =
rgw swift versioning enabled = True
So, it seems that making any change to the ceph_config_
| OpenStack Infra (hudson-openstack) wrote Related fix proposed to openstack-ansible (master) | #1 |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to openstack-ansible (stable/rocky) | #2 |
| OpenStack Infra (hudson-openstack) wrote Related fix merged to openstack-ansible (master) | #3 |
| OpenStack Infra (hudson-openstack) wrote Fix merged to openstack-ansible (stable/rocky) | #4 |
| tags: | added: in-stable-rocky |
| Changed in openstack-ansible: | |
| status: | New → Confirmed |
| status: | Confirmed → Fix Released |
| importance: | Undecided → Medium |
| OpenStack Infra (hudson-openstack) wrote Fix included in openstack/openstack-ansible 18.1.0 | #5 |
Related fix proposed to branch: master
Review: https:/