"juju refresh ceph-osd" triggers ceph-osd@.service restart in all the cluster at the same time
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Ceph OSD Charm | Status tracked in Trunk | |||||
Quincy.2 |
Fix Released
|
High
|
Unassigned | |||
Reef |
Fix Committed
|
High
|
Unassigned | |||
Squid-jammy |
Fix Committed
|
High
|
Unassigned | |||
Trunk |
Fix Released
|
High
|
Unassigned |
Bug Description
* An overview of your environment
The environment is a Focal-Yoga cloud running Ceph Quincy 17.2.7. There are 15 ceph-osd units hosting 6 disks per unit.
* A description of the problem and how it differs from your expected outcome
A "juju refresh ceph-osd" executed in the cloud (upgrading from revision 564 to 585 (quincy/stable channel)) causes all the ceph-osd@ services of all the ceph-osd units to be restarted at the same time. This fact leads to PGs going from Active to Inactive/Down at the same time, causing a performance disruption in the Ceph cluster.
During some earlier tests upgrading the ceph-osd charm from revision 576 to 585 (quincy/stable) did not show this behavior (ceph-osd@ services were not restarted).
The expected outcome while upgrading the ceph-osd charm would be no impact in the cluster or a minimal impact (e.g. rolling restart of ceph-osd services).
* Software versions for: Ubuntu, OpenStack, Juju, and MAAS
Ubuntu 20.04.6 LTS
OpenStack Yoga (yoga/stable)
Juju 2.9.44
MAAS 3.3
* Command output errors or log messages directly related to the problem
In the /var/log/
```
2024-06-04 07:30:33 INFO unit.ceph-
2024-06-04 07:30:33 INFO unit.ceph-
2024-06-04 07:30:34 INFO unit.ceph-
2024-06-04 07:30:34 WARNING unit.ceph-
2024-06-04 07:30:34 WARNING unit.ceph-
2024-06-04 07:30:34 WARNING unit.ceph-
2024-06-04 07:30:34 INFO unit.ceph-
2024-06-04 07:30:34 INFO unit.ceph-
2024-06-04 07:30:34 INFO unit.ceph-
2024-06-04 07:31:22 INFO juju.worker.
```
* Steps to reproduce the problem
Running "juju refresh ceph-osd" in the cloud, upgrading from revision 564 to 585 (quincy/stable channel) triggers the massive ceph-osd@
tags: | added: sts |
Changed in charm-ceph-osd: | |
status: | New → Confirmed |
importance: | Undecided → High |
Related fix proposed to branch: master /review. opendev. org/c/openstack /charm- ceph-osd/ +/921704
Review: https:/