pacific -> quincy upgrade - rados-classes loading broken

[James Page]

Ubuntu 20.04
UCA Xena -> Yoga upgrade

Deployment was functional before upgrade; post upgrade a number of issues occured:

a) the devicehealth mgr module failed to startup with a device I/O error
b) the radosgw instances in the deployment failed to startup with and error about not being able to load the bucket list

The ceph-osd processes where reporting:

ceph-osd.9.log:2022-06-20T13:44:36.201+0000 7f0ad9feb700 0 _load_class could not stat class lib/x86_64-linux-gnu/rados-classes/libcls_rbd.so: (2) No such file or directory

for various libcls_*.so files despite the fact that then where installed albeit under /usr/lib...

A new deployment using Yoga/Quincy does not exhibit this symptom.

Setting

  osd class dir = /usr/lib/x86_64-linux-gnu/rados-classes

worked around this issue in the deployment.

[James Page (james-page)]

Struggling to reproduce this on a new deployment - tested new and upgrade paths

focal + quincy: OK
focal + pacific -> quincy: OK
focal + octopus -> pacific -> quincy: OK

I might not be going far enough back in time - the deployment that reported this issue was been in service for some time and started off as a bionic deployment.

[Dan Hill (hillpd)]

This is the same issue that was seen in bug 1986747. Upgrading a cluster from bionic is required to reproduced the problem.