Increased schedule time for non thin_provisioned backends

[Jose Castro Leon]

We have been seeing a continuous increase of schedule time in creating new manila shares.
Currently we have 1400 shares in 4 different cephfs backends.

When analyzing the scheduling time, the scheduler needs to retrieve information of all backends.
CephFS backend does not support thin_provisioning and it does also not provide 'provisioned_capacity_gb'. If a backend pool does not have this information it falls back to an expensive set of SQL queries (1 per backend + 1 per share in each backend).

https://opendev.org/openstack/manila/src/branch/master/manila/scheduler/host_manager.py#L433-L436

[Jose Castro Leon (jose-castro-leon)]

The idea is to disable this calculation if the backend pool does not provide thin_provisioned volumes.

Just by doing that the schedule time goes from 45 seconds now to less than a second.

[Goutham Pacha Ravi (gouthamr)]

Jose: Thank you for this bug report, and for bringing this to our attention on IRC [1]. As we discussed, when the request doesn't need support for thin provisioning and the backend storage doesn't provide support for thin provisioning, we don't need this calculation.

Also thanks for assigning this bug to yourself. We'll look forward to the fix, and review. Do you have a timeline in mind? I'm asking to target this bug, for now I'll set it to Ussuri rc-1, which is Apr 23, 2020.

[1] http://eavesdrop.openstack.org/irclogs/%23openstack-manila/%23openstack-manila.2020-03-26.log.html#t2020-03-26T16:01:33

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/716265

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716265
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=cd9292b931a0f74079512cd43652129a9c595ee8
Submitter: Zuul
Branch: master

commit cd9292b931a0f74079512cd43652129a9c595ee8
Author: Jose Castro Leon <email address hidden>
Date: Tue Mar 31 13:33:07 2020 +0200

    Remove provisioned calculation on non thin provision backends

    On those backends, there is no need to calculate provisioned_capacity_gb
    as it is not used during the scheduling. This calculation was not
    scaling properly on big environments as it implies many database
    queries.

    Change-Id: If1ea4ceedc495dc6c6e247feccfbdc4899ad725c
    Closes-Bug: #1869712

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/716458

[Goutham Pacha Ravi (gouthamr)]

https://review.opendev.org/716265 has been reverted from manila's master branch and https://review.opendev.org/716458 (backport to stable/train) has been abandoned. This bug's re-opened to include the fix to "capability lists" as discussed on the review for https://review.opendev.org/#/c/716552/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Goutham Pacha Ravi (<email address hidden>) on branch: stable/train
Review: https://review.opendev.org/716458
Reason: This bugfix is being reverted from master: https://review.opendev.org/#/c/716683/

A subsequent fix will address the issue in https://launchpad.net/bugs/1869712 and we'll pursue a backport then

[Vida Haririan (vhariria)]

Additional comments http://eavesdrop.openstack.org/meetings/manila/2020/manila.2020-04-02-15.01.log.html

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/718016

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/718016
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=2c30fed486320ec5ba88f21ae3df14b14da6eb35
Submitter: Zuul
Branch: master

commit 2c30fed486320ec5ba88f21ae3df14b14da6eb35
Author: Jose Castro Leon <email address hidden>
Date: Tue Apr 7 11:24:00 2020 +0200

    Remove provisioned calculation on non thin provision backends

    On those backends, there is no need to calculate provisioned_capacity_gb
    as it is not used during the scheduling. This calculation was not
    scaling properly on big environments as it implies many database
    queries.

    Closes-Bug: #1869712

    Change-Id: Icb8947991723a2a1cc334a72276a35cf50fc1b7d

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/718401

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/720099

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/718401
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=89aa8939aa53e52254e4f98da0d80a76ef12b9e7
Submitter: Zuul
Branch: stable/train

commit 89aa8939aa53e52254e4f98da0d80a76ef12b9e7
Author: Jose Castro Leon <email address hidden>
Date: Tue Apr 7 11:24:00 2020 +0200

    Remove provisioned calculation on non thin provision backends

    On those backends, there is no need to calculate provisioned_capacity_gb
    as it is not used during the scheduling. This calculation was not
    scaling properly on big environments as it implies many database
    queries.

    Closes-Bug: #1869712

    Change-Id: Icb8947991723a2a1cc334a72276a35cf50fc1b7d
    (cherry picked from commit 2c30fed486320ec5ba88f21ae3df14b14da6eb35)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/723615

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/720099
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=920cd8e6df8a9b553a77a3a029c9d282c73e1634
Submitter: Zuul
Branch: stable/stein

commit 920cd8e6df8a9b553a77a3a029c9d282c73e1634
Author: Jose Castro Leon <email address hidden>
Date: Tue Apr 7 11:24:00 2020 +0200

    Remove provisioned calculation on non thin provision backends

    On those backends, there is no need to calculate provisioned_capacity_gb
    as it is not used during the scheduling. This calculation was not
    scaling properly on big environments as it implies many database
    queries.

    Closes-Bug: #1869712

    Change-Id: Icb8947991723a2a1cc334a72276a35cf50fc1b7d
    (cherry picked from commit 2c30fed486320ec5ba88f21ae3df14b14da6eb35)
    (cherry picked from commit 89aa8939aa53e52254e4f98da0d80a76ef12b9e7)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/723656

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/723615
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=49deb1ed312ad9b89e573d84fedf31b0e7894dd2
Submitter: Zuul
Branch: stable/rocky

commit 49deb1ed312ad9b89e573d84fedf31b0e7894dd2
Author: Jose Castro Leon <email address hidden>
Date: Tue Apr 7 11:24:00 2020 +0200

    Remove provisioned calculation on non thin provision backends

    On those backends, there is no need to calculate provisioned_capacity_gb
    as it is not used during the scheduling. This calculation was not
    scaling properly on big environments as it implies many database
    queries.

    Closes-Bug: #1869712

    Change-Id: Icb8947991723a2a1cc334a72276a35cf50fc1b7d
    (cherry picked from commit 2c30fed486320ec5ba88f21ae3df14b14da6eb35)
    (cherry picked from commit 89aa8939aa53e52254e4f98da0d80a76ef12b9e7)
    (cherry picked from commit 920cd8e6df8a9b553a77a3a029c9d282c73e1634)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/723656
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=dd50f6be5a5b1847354790e2fe1f426185e4abff
Submitter: Zuul
Branch: stable/queens

commit dd50f6be5a5b1847354790e2fe1f426185e4abff
Author: Jose Castro Leon <email address hidden>
Date: Tue Apr 7 11:24:00 2020 +0200

    Remove provisioned calculation on non thin provision backends

    On those backends, there is no need to calculate provisioned_capacity_gb
    as it is not used during the scheduling. This calculation was not
    scaling properly on big environments as it implies many database
    queries.

    Closes-Bug: #1869712

    Change-Id: Icb8947991723a2a1cc334a72276a35cf50fc1b7d
    (cherry picked from commit 2c30fed486320ec5ba88f21ae3df14b14da6eb35)
    (cherry picked from commit 89aa8939aa53e52254e4f98da0d80a76ef12b9e7)
    (cherry picked from commit 920cd8e6df8a9b553a77a3a029c9d282c73e1634)
    (cherry picked from commit 49deb1ed312ad9b89e573d84fedf31b0e7894dd2)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila rocky-eol release.