microceph cluster forming takes a very long time
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Juju Charmed Operator - MicroCeph |
Fix Released
|
Critical
|
Hemanth Nakkina | ||
OpenStack Snap |
Fix Released
|
Critical
|
Unassigned |
Bug Description
When running Sunbeam deployment it times out on microceph application not to be ready.
Turns out to form 6 nodes cluster takes over two hours.
charm
microceph reef/beta 59
snap
microceph 18.2.0+
Attached are logs from snap and charm
solqa-lab1-
Model Controller Cloud/Region Version SLA Timestamp
openstack-machines silo5-controller silo5/default 3.4.3 unsupported 16:18:53Z
App Version Status Scale Charm Channel Rev Exposed Message
microceph active 6 microceph reef/beta 59 no
microk8s active 4 microk8s legacy/stable 121 no
sunbeam-machine active 9 sunbeam-machine 2024.1/beta 25 no
Unit Workload Agent Machine Public address Ports Message
microceph/0* active idle 0 10.246.164.143
microceph/1 active idle 3 10.246.164.163
microceph/2 active idle 4 10.246.165.14
microceph/3 active idle 5 10.246.165.13
microceph/4 active idle 7 10.246.165.16
microceph/5 active idle 8 10.246.164.162
microk8s/0* active idle 1 10.246.164.144 16443/tcp
microk8s/1 active idle 5 10.246.165.13 16443/tcp
microk8s/2 active idle 6 10.246.164.155 16443/tcp
microk8s/3 active idle 7 10.246.165.16 16443/tcp
sunbeam-machine/0* active idle 0 10.246.164.143
sunbeam-machine/1 active idle 1 10.246.164.144
sunbeam-machine/2 active idle 2 10.246.164.156
sunbeam-machine/3 active idle 3 10.246.164.163
sunbeam-machine/4 active idle 4 10.246.165.14
sunbeam-machine/5 active idle 5 10.246.165.13
sunbeam-machine/6 active idle 6 10.246.164.155
sunbeam-machine/7 active idle 7 10.246.165.16
sunbeam-machine/8 active idle 8 10.246.164.162
Machine State Address Inst id Base AZ Message
0 started 10.246.164.143 solqa-lab1-
1 started 10.246.164.144 solqa-lab1-
2 started 10.246.164.156 solqa-lab1-
3 started 10.246.164.163 solqa-lab1-
4 started 10.246.165.14 solqa-lab1-
5 started 10.246.165.13 solqa-lab1-
6 started 10.246.164.155 solqa-lab1-
7 started 10.246.165.16 solqa-lab1-
8 started 10.246.164.162 solqa-lab1-
Changed in charm-microceph: | |
importance: | Undecided → Critical |
assignee: | nobody → Hemanth Nakkina (hemanth-n) |
tags: | added: open-2220 |
Changed in charm-microceph: | |
status: | In Progress → Fix Committed |
Changed in snap-openstack: | |
status: | Confirmed → Fix Committed |
Changed in snap-openstack: | |
importance: | Undecided → Critical |
Changed in snap-openstack: | |
status: | Fix Committed → Fix Released |
Changed in charm-microceph: | |
status: | Fix Committed → Fix Released |
Adding also microceph_ status. log which shows microceph status every 30 seconds with timestamps so it is easily visible when new members joined. There are at least 15 minutes delays between each new member