cephfs-nfs gate fails with luminous 12.1.1

[Tom Barron]

Since upgrade from 12.1.0 to 12.1.1 the cephfs nfs gate job has been failing because manila cannot create shares. Examination of the scheduler log indicates that cephfs-nfs backend is reporting no free space. The manila driver code hasn't changed: it just calls volume_client.rados.get_cluster_stats() to get available space.

[Ramana Raja (rraja)]

Thanks for filing this, Tom. As you already figured, the get_cluster_stats() no longer returns the correct ceph cluster usage details. It's worth mentioning that quite a few Ceph usability API changes made it into Ceph 12.1.1. But I'm unable to pin point the culprit commit.

Meanwhile, what do you think of using Ceph Kraken stable packages instead of Ceph luminous RC packages in the gate?
https://review.openstack.org/#/c/488358/

[Tom Barron (tpb)]

> Meanwhile, what do you think of using Ceph Kraken stable packages instead of Ceph luminous RC packages in the gate?

Make sense.

[Tom Barron (tpb)]

https://review.openstack.org/#/c/488358/ backs devstack install of ceph to kraken and fixes gate breakage:

2017-08-04 14:19:46.493 | ceph-base x86_64 1:11.2.0-718.g11d5c2b.el7 ceph 4.9 M
2017-08-04 14:19:46.493 | ceph-common x86_64 1:11.2.0-718.g11d5c2b.el7 ceph 19 M
2017-08-04 14:19:46.493 | ceph-mds x86_64 1:11.2.0-718.g11d5c2b.el7 ceph 3.0 M
2017-08-04 14:19:46.493 | ceph-mgr x86_64 1:11.2.0-718.g11d5c2b.el7 ceph 1.9 M
2017-08-04 14:19:46.493 | ceph-mon x86_64 1:11.2.0-718.g11d5c2b.el7 ceph 3.1 M
2017-08-04 14:19:46.493 | ceph-osd x86_64 1:11.2.0-718.g11d5c2b.el7 ceph 10 M
2017-08-04 14:19:46.493 | ceph-selinux x86_64 1:11.2.0-718.g11d5c2b.el7 ceph 19 k
...
2017-08-04 14:19:46.493 | libcephfs2 x86_64 1:11.2.0-718.g11d5c2b.el7 ceph 1.9 M

[Tom Barron (tpb)]

Since gate tests are unblocked I'm downgrading Importance but leaving the bug open. We should get the cephfs-nfs driver to work with both kraken and luminous (again) but we no longer need the fix urgently since luminous is brand new and will be getting bug fixes for some time.

[Ramana Raja (rraja)]

> Thanks for filing this, Tom. As you already figured, the get_cluster_stats() no longer returns the correct ceph cluster usage details. It's worth mentioning that quite a few Ceph usability API changes made it into Ceph 12.1.1. But I'm unable to pin point the culprit commit.

Looks like it's not an issue with the rados API, but an issue with how the Ceph cluster is setup/configured in the cephfs-nfs gate.

[Ramana Raja (rraja)]

The issue was with how Ceph was setup in the gate. This was fixed a while back with
https://github.com/openstack/devstack-plugin-ceph/commit/68185a82942bff49550517b27bcdacc311c32032

It wasn't a Manila bug.