I think I found the issue. In the log for failed OSD initialization we can see "_aio_start io_setup(2) failed with EAGAIN; try increasing /proc/sys/fs/aio-max-nr". So I checked both aio-max-nr and aio-nr and realized the system is close to the limit. When I increased the limit and manually run ceph-volume, it succeeded.
# Read the limit
root@storage-1:~# cat /proc/sys/fs/aio-max-nr
65536
# Read the current value
root@storage-1:~# cat /proc/sys/fs/aio-nr
63490
# Then I increased /proc/sys/fs/aio-max-nr to 1048576
root@storage-1:~# echo 1048576 > /proc/sys/fs/aio-max-nr
root@storage-1:~# cat /proc/sys/fs/aio-max-nr
1048576
After this I was able to successfully initialize new OSD.
I'm wondering if this is something that charm could calculate and adjust?
I think I found the issue. In the log for failed OSD initialization we can see "_aio_start io_setup(2) failed with EAGAIN; try increasing /proc/sys/ fs/aio- max-nr" . So I checked both aio-max-nr and aio-nr and realized the system is close to the limit. When I increased the limit and manually run ceph-volume, it succeeded.
# Read the limit fs/aio- max-nr
root@storage-1:~# cat /proc/sys/
65536
# Read the current value
root@storage-1:~# cat /proc/sys/fs/aio-nr
63490
# Then I increased /proc/sys/ fs/aio- max-nr to 1048576 fs/aio- max-nr fs/aio- max-nr
root@storage-1:~# echo 1048576 > /proc/sys/
root@storage-1:~# cat /proc/sys/
1048576
After this I was able to successfully initialize new OSD.
I'm wondering if this is something that charm could calculate and adjust?