juju run remove-disk Fails with RADOS object not found (error connecting to the cluster)
Add disk succeeded but remove disk fails. Thankyou for the support !
Remove Disk:
-----------
root@c1:~# juju run ceph-osd/1 remove-disk osd-ids=osd.3 purge=true
Running operation 7 with 1 task
- task 8 on unit-ceph-osd-1
Waiting for task 8...
2023-09-25T19:32:25.179+0000 7fed1d95b640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.osd-removal.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
2023-09-25T19:32:25.179+0000 7fed1d95b640 -1 AuthRegistry(0x7fed1805fcf0) no keyring found at /etc/ceph/ceph.client.osd-removal.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx
2023-09-25T19:32:25.183+0000 7fed1d95b640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.osd-removal.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
2023-09-25T19:32:25.183+0000 7fed1d95b640 -1 AuthRegistry(0x7fed18065788) no keyring found at /etc/ceph/ceph.client.osd-removal.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx
2023-09-25T19:32:25.183+0000 7fed1d95b640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.osd-removal.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
2023-09-25T19:32:25.183+0000 7fed1d95b640 -1 AuthRegistry(0x7fed1d959f80) no keyring found at /etc/ceph/ceph.client.osd-removal.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx
[errno 2] RADOS object not found (error connecting to the cluster)
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-ceph-osd-1/charm/actions/remove-disk", line 367, in <module>
main()
File "/var/lib/juju/agents/unit-ceph-osd-1/charm/actions/remove-disk", line 347, in main
action_osd.remove(purge, timeout, force)
File "/var/lib/juju/agents/unit-ceph-osd-1/charm/actions/remove-disk", line 213, in remove
reweight_osd(self.osd_id)
File "/var/lib/juju/agents/unit-ceph-osd-1/charm/actions/remove-disk", line 122, in reweight_osd
subprocess.check_call(['ceph', '--id', 'osd-removal',
File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ceph', '--id', 'osd-removal', 'osd', 'crush', 'reweight', 'osd.3', '0']' returned non-zero exit status 1.
ERROR the following task failed:
- id "8" with return code 1
----
root@c1:~# juju status
Model Controller Cloud/Region Version SLA Timestamp
ceph manual-default manual/default 3.2.3 unsupported 19:38:50Z
App Version Status Scale Charm Channel Rev Exposed Message
ceph-mon 17.2.6 active 3 ceph-mon quincy/stable 183 no Unit is ready and clustered
ceph-osd 17.2.6 active 3 ceph-osd quincy/stable 564 no Unit is ready (1 OSD)
Add disk: Success
----------
root@c1:~# juju run ceph-osd/1 add-disk osd-devices=/dev/sdc
Running operation 5 with 1 task
- task 6 on unit-ceph-osd-1
juju run remove-disk Fails with RADOS object not found (error connecting to the cluster)
Add disk succeeded but remove disk fails. Thankyou for the support !
Remove Disk:
-----------
root@c1:~# juju run ceph-osd/1 remove-disk osd-ids=osd.3 purge=true
Running operation 7 with 1 task
- task 8 on unit-ceph-osd-1
Waiting for task 8... 25T19:32: 25.179+ 0000 7fed1d95b640 -1 auth: unable to find a keyring on /etc/ceph/ ceph.client. osd-removal. keyring, /etc/ceph/ ceph.keyring, /etc/ceph/ keyring, /etc/ceph/ keyring. bin: (2) No such file or directory 25T19:32: 25.179+ 0000 7fed1d95b640 -1 AuthRegistry( 0x7fed1805fcf0) no keyring found at /etc/ceph/ ceph.client. osd-removal. keyring, /etc/ceph/ ceph.keyring, /etc/ceph/ keyring, /etc/ceph/ keyring. bin, disabling cephx 25T19:32: 25.183+ 0000 7fed1d95b640 -1 auth: unable to find a keyring on /etc/ceph/ ceph.client. osd-removal. keyring, /etc/ceph/ ceph.keyring, /etc/ceph/ keyring, /etc/ceph/ keyring. bin: (2) No such file or directory 25T19:32: 25.183+ 0000 7fed1d95b640 -1 AuthRegistry( 0x7fed18065788) no keyring found at /etc/ceph/ ceph.client. osd-removal. keyring, /etc/ceph/ ceph.keyring, /etc/ceph/ keyring, /etc/ceph/ keyring. bin, disabling cephx 25T19:32: 25.183+ 0000 7fed1d95b640 -1 auth: unable to find a keyring on /etc/ceph/ ceph.client. osd-removal. keyring, /etc/ceph/ ceph.keyring, /etc/ceph/ keyring, /etc/ceph/ keyring. bin: (2) No such file or directory 25T19:32: 25.183+ 0000 7fed1d95b640 -1 AuthRegistry( 0x7fed1d959f80) no keyring found at /etc/ceph/ ceph.client. osd-removal. keyring, /etc/ceph/ ceph.keyring, /etc/ceph/ keyring, /etc/ceph/ keyring. bin, disabling cephx juju/agents/ unit-ceph- osd-1/charm/ actions/ remove- disk", line 367, in <module> juju/agents/ unit-ceph- osd-1/charm/ actions/ remove- disk", line 347, in main osd.remove( purge, timeout, force) juju/agents/ unit-ceph- osd-1/charm/ actions/ remove- disk", line 213, in remove osd(self. osd_id) juju/agents/ unit-ceph- osd-1/charm/ actions/ remove- disk", line 122, in reweight_osd check_call( ['ceph' , '--id', 'osd-removal', python3. 10/subprocess. py", line 369, in check_call ror(retcode, cmd) CalledProcessEr ror: Command '['ceph', '--id', 'osd-removal', 'osd', 'crush', 'reweight', 'osd.3', '0']' returned non-zero exit status 1.
2023-09-
2023-09-
2023-09-
2023-09-
2023-09-
2023-09-
[errno 2] RADOS object not found (error connecting to the cluster)
Traceback (most recent call last):
File "/var/lib/
main()
File "/var/lib/
action_
File "/var/lib/
reweight_
File "/var/lib/
subprocess.
File "/usr/lib/
raise CalledProcessEr
subprocess.
ERROR the following task failed:
- id "8" with return code 1
----
root@c1:~# juju status
Model Controller Cloud/Region Version SLA Timestamp
ceph manual-default manual/default 3.2.3 unsupported 19:38:50Z
App Version Status Scale Charm Channel Rev Exposed Message
ceph-mon 17.2.6 active 3 ceph-mon quincy/stable 183 no Unit is ready and clustered
ceph-osd 17.2.6 active 3 ceph-osd quincy/stable 564 no Unit is ready (1 OSD)
Add disk: Success
----------
root@c1:~# juju run ceph-osd/1 add-disk osd-devices= /dev/sdc
Running operation 5 with 1 task
- task 6 on unit-ceph-osd-1
Waiting for task 6... dd93-4c3c- 9594-7dd847e17b 08" successfully created b12e1e3b- dd93-4c3c- 9594-7dd847e17b 08" created. ceph-authtool --gen-print-key bootstrap- osd --keyring /var/lib/ ceph/bootstrap- osd/ceph. keyring -i - osd new b12e1e3b- dd93-4c3c- 9594-7dd847e17b 08 ceph-authtool --gen-print-key ceph/osd/ ceph-3 juju/tools/ unit-ceph- osd-1:/ usr/local/ sbin:/usr/ local/bin: /usr/sbin: /usr/bin: /sbin:/ bin:/snap/ bin b12e1e3b- dd93-4c3c- 9594-7dd847e17b 08/osd- block-b12e1e3b- dd93-4c3c- 9594-7dd847e17b 08 b12e1e3b- dd93-4c3c- 9594-7dd847e17b 08/osd- block-b12e1e3b- dd93-4c3c- 9594-7dd847e17b 08 /var/lib/ ceph/osd/ ceph-3/ block bootstrap- osd --keyring /var/lib/ ceph/bootstrap- osd/ceph. keyring mon getmap -o /var/lib/ ceph/osd/ ceph-3/ activate. monmap 25T19:10: 48.123+ 0000 7f77e5be0640 -1 auth: unable to find a keyring on /etc/ceph/ ceph.client. bootstrap- osd.keyring, /etc/ceph/ ceph.keyring, /etc/ceph/ keyring, /etc/ceph/ keyring. bin: (2) No such file or directory 25T19:10: 48.123+ 0000 7f77e5be0640 -1 AuthRegistry( 0x7f77e005fe00) no keyring found at /etc/ceph/ ceph.client. bootstrap- osd.keyring, /etc/ceph/ ceph.keyring, /etc/ceph/ keyring, /etc/ceph/ keyring. bin, disabling cephx ceph/osd/ ceph-3/ keyring ceph/osd/ ceph-3/ ceph/osd/ ceph-3/ activate. monmap --keyfile - --osd-data /var/lib/ ceph/osd/ ceph-3/ --osd-uuid b12e1e3b- dd93-4c3c- 9594-7dd847e17b 08 --setuser ceph --setgroup ceph 25T19:10: 48.527+ 0000 7fc140651800 -1 bluestore( /var/lib/ ceph/osd/ ceph-3/ ) _read_fsid unparsable uuid dd93-4c3c- 9594-7dd847e17b 08/osd- block-b12e1e3b- dd93-4c3c- 9594-7dd847e17b 08 ceph/osd/ ceph-3 ceph-bluestore- tool --cluster=ceph prime-osd-dir --dev /dev/ceph- b12e1e3b- dd93-4c3c- 9594-7dd847e17b 08/osd- block-b12e1e3b- dd93-4c3c- 9594-7dd847e17b 08 --path /var/lib/ ceph/osd/ ceph-3 --no-mon-config b12e1e3b- dd93-4c3c- 9594-7dd847e17b 08/osd- block-b12e1e3b- dd93-4c3c- 9594-7dd847e17b 08 /var/lib/ ceph/osd/ ceph-3/ block ceph/osd/ ceph-3/ block ceph/osd/ ceph-3 lvm-3-b12e1e3b- dd93-4c3c- 9594-7dd847e17b 08 system/ ceph-volume@ .service. system/ ceph-osd. target. wants/ceph- osd@3.service → /lib/systemd/ system/ ceph-osd@ .service. dd93-4c3c- 9594-7dd847e17b 08/osd- block-b12e1e3b- dd93-4c3c- 9594-7dd847e17b 08
Physical volume "/dev/sdc" successfully created.
Volume group "ceph-b12e1e3b-
Logical volume "osd-block-
partx: /dev/sdc: failed to read partition table
Failed to find physical volume "/dev/sdc".
Failed to find physical volume "/dev/sdc".
Running command: /usr/bin/
Running command: /usr/bin/ceph --cluster ceph --name client.
Running command: /usr/bin/
Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/
--> Executable selinuxenabled not in PATH: /var/lib/
Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
Running command: /usr/bin/ln -s /dev/ceph-
Running command: /usr/bin/ceph --cluster ceph --name client.
stderr: 2023-09-
2023-09-
stderr: got monmap epoch 1
--> Creating keyring file for osd.3
Running command: /usr/bin/chown -R ceph:ceph /var/lib/
Running command: /usr/bin/chown -R ceph:ceph /var/lib/
Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 3 --monmap /var/lib/
stderr: 2023-09-
--> ceph-volume lvm prepare successful for: ceph-b12e1e3b-
Running command: /usr/bin/chown -R ceph:ceph /var/lib/
Running command: /usr/bin/
Running command: /usr/bin/ln -snf /dev/ceph-
Running command: /usr/bin/chown -h ceph:ceph /var/lib/
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
Running command: /usr/bin/chown -R ceph:ceph /var/lib/
Running command: /usr/bin/systemctl enable ceph-volume@
stderr: Created symlink /<email address hidden> → /lib/systemd/
Running command: /usr/bin/systemctl enable --runtime ceph-osd@3
stderr: Created symlink /run/systemd/
Running command: /usr/bin/systemctl start ceph-osd@3
--> ceph-volume lvm activate successful for osd ID: 3
--> ceph-volume lvm create successful for: ceph-b12e1e3b-
root@c1:~# juju ssh ceph-mon/leader sudo ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.07794 root default
-5 0.01949 host vm1
2 hdd 0.01949 osd.2 up 1.00000 1.00000
-7 0.03897 host vm2
1 hdd 0.01949 osd.1 up 1.00000 1.00000
3 hdd 0.01949 osd.3 up 1.00000 1.00000
-3 0.01949 host vm3
0 hdd 0.01949 osd.0 up 1.00000 1.00000