ceph-mon does not process broker requests when it should be
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ceph Monitor Charm |
Fix Released
|
Critical
|
Dmitrii Shcherbakov |
Bug Description
A xenial-mitaka deployment got blocked on an unexpected state of the radosgw service on a radosgw unit: the service is masked while it should not be.
https:/
https:/
juju ssh ceph-radosgw/0
root@juju-
2021-08-31 13:34:17 WARNING unit.ceph-
2021-08-31 13:34:23 WARNING unit.ceph-
2021-08-31 13:35:33 WARNING unit.ceph-
ubuntu@
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.7 LTS
Release: 16.04
Codename: xenial
ubuntu@
ii radosgw 10.2.11-
ubuntu@
● radosgw.service
Loaded: masked (/dev/null; bad)
Active: inactive (dead)
ubuntu@
● radosgw.service
Loaded: masked (/dev/null; bad)
Active: inactive (dead)
The radosgw unit determines whether systemd should be used or not correctly by the looks of it at the time of a manual check so the service name is "radosgw", however, this service got previously masked somehow which is only possible from this charm if it thinks that the systemd code path should be used.
https:/
https:/
root@juju-
egress-subnets: 10.5.1.50/32
ingress-address: 10.5.1.50
private-address: 10.5.1.50
egress-subnets: 10.5.3.121/32
ingress-address: 10.5.3.121
private-address: 10.5.3.121
auth: cephx
ceph-public-
egress-subnets: 10.5.2.242/32
fsid: 33bc4e50-
ingress-address: 10.5.2.242
private-address: 10.5.2.242
radosgw_key: AQBLMC5hyoihCxA
(Pdb) debug systemd_
ENTERING RECURSIVE DEBUGGER
> <string>
((Pdb)) s
--Call--
> /var/lib/
-> def systemd_
((Pdb)) relation_
{'private-address': '10.5.1.50', 'ingress-address': '10.5.1.50', 'egress-subnets': '10.5.1.50/32'}
((Pdb)) relation_
{'private-address': '10.5.3.121', 'ingress-address': '10.5.3.121', 'egress-subnets': '10.5.3.121/32'}
((Pdb)) relation_
{'private-address': '10.5.2.242', 'ingress-address': '10.5.2.242', 'egress-subnets': '10.5.2.242/32', 'radosgw_key': 'AQBLMC5hyoihCx
2', 'fsid': '33bc4e50-
((Pdb)) relation_
((Pdb))
# ...
> /var/lib/
-> for rid in relation_
((Pdb)) n
> /var/lib/
-> return False
Changed in charm-ceph-mon: | |
importance: | Undecided → Critical |
assignee: | nobody → Dmitrii Shcherbakov (dmitriis) |
milestone: | none → 21.10 |
Changed in charm-ceph-mon: | |
status: | Fix Committed → Fix Released |
The radosgw service is masked earlier at the install hook execution time where it is paused:
https:/ /github. com/openstack/ charm-ceph- radosgw/ blob/c97fced794 7e55870240557ae 41dad7872a3c9d2 /hooks/ hooks.py# L174-L181
.format( service_ name()) , level=DEBUG) pause(service_ name())
def install():
# ...
log('Disable service "{}" until we have keys for it.'
service_
https:/ /github. com/openstack/ charm-ceph- radosgw/ blob/c97fced794 7e55870240557ae 41dad7872a3c9d2 /hooks/ charmhelpers/ core/host. py#L178- L200 (service_pause) pause(service_ name, init_dir= "/etc/init" , initd_dir= "/etc/init. d",
**kwargs) : systemd( service_ name=service_ name):
service( 'disable' , service_name)
service( 'mask', service_name)
def service_
# ...
if init_is_
https:/ /github. com/openstack/ charm-ceph- radosgw/ blob/c97fced794 7e55870240557ae 41dad7872a3c9d2 /hooks/ charmhelpers/ core/host. py#L316- L326 (init_is_systemd)
(Pdb) from charmhelpers. core.host import init_is_systemd
(Pdb) init_is_systemd()
True