microceph cluster forming takes a very long time

[Marian Gasparovic]

When running Sunbeam deployment it times out on microceph application not to be ready.
Turns out to form 6 nodes cluster takes over two hours.

charm
microceph reef/beta 59

snap
microceph 18.2.0+snap891b397dd3 1039 reef/beta

Attached are logs from snap and charm

solqa-lab1-server-37 is the leader

Model Controller Cloud/Region Version SLA Timestamp
openstack-machines silo5-controller silo5/default 3.4.3 unsupported 16:18:53Z

App Version Status Scale Charm Channel Rev Exposed Message
microceph active 6 microceph reef/beta 59 no
microk8s active 4 microk8s legacy/stable 121 no
sunbeam-machine active 9 sunbeam-machine 2024.1/beta 25 no

Unit Workload Agent Machine Public address Ports Message
microceph/0* active idle 0 10.246.164.143
microceph/1 active idle 3 10.246.164.163
microceph/2 active idle 4 10.246.165.14
microceph/3 active idle 5 10.246.165.13
microceph/4 active idle 7 10.246.165.16
microceph/5 active idle 8 10.246.164.162
microk8s/0* active idle 1 10.246.164.144 16443/tcp
microk8s/1 active idle 5 10.246.165.13 16443/tcp
microk8s/2 active idle 6 10.246.164.155 16443/tcp
microk8s/3 active idle 7 10.246.165.16 16443/tcp
sunbeam-machine/0* active idle 0 10.246.164.143
sunbeam-machine/1 active idle 1 10.246.164.144
sunbeam-machine/2 active idle 2 10.246.164.156
sunbeam-machine/3 active idle 3 10.246.164.163
sunbeam-machine/4 active idle 4 10.246.165.14
sunbeam-machine/5 active idle 5 10.246.165.13
sunbeam-machine/6 active idle 6 10.246.164.155
sunbeam-machine/7 active idle 7 10.246.165.16
sunbeam-machine/8 active idle 8 10.246.164.162

Machine State Address Inst id Base AZ Message
0 started 10.246.164.143 solqa-lab1-server-37 ubuntu@22.04 zone1 Deployed
1 started 10.246.164.144 solqa-lab1-server-38 ubuntu@22.04 zone2 Deployed
2 started 10.246.164.156 solqa-lab1-server-40 ubuntu@22.04 zone1 Deployed
3 started 10.246.164.163 solqa-lab1-server-41 ubuntu@22.04 zone2 Deployed
4 started 10.246.165.14 solqa-lab1-server-42 ubuntu@22.04 zone3 Deployed
5 started 10.246.165.13 solqa-lab1-server-46 ubuntu@22.04 zone1 Deployed
6 started 10.246.164.155 solqa-lab1-server-47 ubuntu@22.04 zone2 Deployed
7 started 10.246.165.16 solqa-lab1-server-48 ubuntu@22.04 zone3 Deployed
8 started 10.246.164.162 solqa-lab1-server-51 ubuntu@22.04 zone3 Deployed

[Marian Gasparovic (marosg)]

Adding also microceph_status.log which shows microceph status every 30 seconds with timestamps so it is easily visible when new members joined. There are at least 15 minutes delays between each new member

[Hemanth Nakkina (hemanth-n)]

Charm microceph bugs are tracked now in github.

Raised a corresponding issue in charm-microceph
https://github.com/canonical/charm-microceph/issues/91

[utkarsh bhatt (utkarshbhatthere)]

Thanks for the bug, taking a look. For brevity, we will continue this discussion on the GitHub issue mentioned by Hemanth.

[Hemanth Nakkina (hemanth-n)]

https://github.com/canonical/charm-microceph/pull/92