NetApp ONTAP drivers reporting incorrect provisioned_capacity_gb

[Gorka Eguileor]

All NetApp ONTAP drivers are incorrectly reporting "provisioned_capacity_gb", probably because provisioned capacity means something different on the NetApp ONTAP storage arrays themselves.

Reported values are always actual physical used size [1][2][3], which is not correct as it should contain the maximum size that all volumes present in the backend could reach [4], total virtual size if you will.

Due to this over provisioning doesn't work as expected. A clear example is if you use thin provisioning with a 1.0 max_over_provisioning ratio, as you'll still be doing over provisioning.

[1] https://github.com/openstack/cinder/blob/55e29347221f1c3c1fca26ab0144ecc1f54b87b6/cinder/volume/drivers/netapp/dataontap/nfs_base.py#L888
[2] https://github.com/openstack/cinder/blob/55e29347221f1c3c1fca26ab0144ecc1f54b87b6/cinder/volume/drivers/netapp/dataontap/block_7mode.py#L344
[3] https://github.com/openstack/cinder/blob/55e29347221f1c3c1fca26ab0144ecc1f54b87b6/cinder/volume/drivers/netapp/dataontap/block_cmode.py#L320
[4] https://review.openstack.org/#/c/490116/2/specs/queens/provisioning-improvements.rst@103

NetApp E-Series drivers may have a similar behavior, reported as a separate bug here: https://bugs.launchpad.net/cinder/+bug/1718739

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/505672

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/505672
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=42b8b7fe60ffdd7a7772dc0ab228265dc83344bc
Submitter: Jenkins
Branch: master

commit 42b8b7fe60ffdd7a7772dc0ab228265dc83344bc
Author: Goutham Pacha Ravi <email address hidden>
Date: Wed Sep 13 23:54:08 2017 -0600

    NetApp ONTAP: Fix reporting of provisioned_capacity_gb

    The ONTAP drivers in Cinder ("7mode" and "cmode") cannot
    reliably and efficiently track provisioned_capacity_gb as expected
    by the Cinder scheduler.

    The driver authors originally assumed that provisioned_capacity_gb
    is consumed space on the backend. This results in miscalculation of
    over subscription in the Cinder scheduler.

    The fix adopted here is to remove this wrong reporting and rely on
    calculation of the provisioned_capacity_gb in the scheduler.

    Change-Id: Ic106dbcae8ceaac265b710756ab1874e445ca826
    Closes-Bug: #1714209

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/509677

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: driverfixes/ocata
Review: https://review.openstack.org/509780

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/509677
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=558571b44d9cd2195993e42539fd2c689b179ee6
Submitter: Jenkins
Branch: stable/pike

commit 558571b44d9cd2195993e42539fd2c689b179ee6
Author: Goutham Pacha Ravi <email address hidden>
Date: Wed Sep 13 23:54:08 2017 -0600

    NetApp ONTAP: Fix reporting of provisioned_capacity_gb

    The ONTAP drivers in Cinder ("7mode" and "cmode") cannot
    reliably and efficiently track provisioned_capacity_gb as expected
    by the Cinder scheduler.

    The driver authors originally assumed that provisioned_capacity_gb
    is consumed space on the backend. This results in miscalculation of
    over subscription in the Cinder scheduler.

    The fix adopted here is to remove this wrong reporting and rely on
    calculation of the provisioned_capacity_gb in the scheduler.

    Change-Id: Ic106dbcae8ceaac265b710756ab1874e445ca826
    Closes-Bug: #1714209
    (cherry picked from commit 42b8b7fe60ffdd7a7772dc0ab228265dc83344bc)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 12.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/509780
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=eef24cfce800c601e5e1e7a21e026ad9b03d4ed1
Submitter: Zuul
Branch: driverfixes/ocata

commit eef24cfce800c601e5e1e7a21e026ad9b03d4ed1
Author: Goutham Pacha Ravi <email address hidden>
Date: Wed Sep 13 23:54:08 2017 -0600

    NetApp ONTAP: Fix reporting of provisioned_capacity_gb

    The ONTAP drivers in Cinder ("7mode" and "cmode") cannot
    reliably and efficiently track provisioned_capacity_gb as expected
    by the Cinder scheduler.

    The driver authors originally assumed that provisioned_capacity_gb
    is consumed space on the backend. This results in miscalculation of
    over subscription in the Cinder scheduler.

    The fix adopted here is to remove this wrong reporting and rely on
    calculation of the provisioned_capacity_gb in the scheduler.

    Change-Id: Ic106dbcae8ceaac265b710756ab1874e445ca826
    Closes-Bug: #1714209
    (cherry picked from commit 42b8b7fe60ffdd7a7772dc0ab228265dc83344bc)
    (cherry picked from commit 558571b44d9cd2195993e42539fd2c689b179ee6)
    Conflicts:
     cinder/tests/unit/volume/drivers/netapp/dataontap/test_nfs_7mode.py

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 11.0.1 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/523478

[OpenStack Infra (hudson-openstack)]

Change abandoned by Goutham Pacha Ravi (<email address hidden>) on branch: master
Review: https://review.openstack.org/523478

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/560600

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/560600
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=83a849a78c93ad7a1a7f2c9c0cd3b7ae08a2ff32
Submitter: Zuul
Branch: stable/ocata

commit 83a849a78c93ad7a1a7f2c9c0cd3b7ae08a2ff32
Author: Goutham Pacha Ravi <email address hidden>
Date: Wed Sep 13 23:54:08 2017 -0600

    NetApp ONTAP: Fix reporting of provisioned_capacity_gb

    The ONTAP drivers in Cinder ("7mode" and "cmode") cannot
    reliably and efficiently track provisioned_capacity_gb as expected
    by the Cinder scheduler.

    The driver authors originally assumed that provisioned_capacity_gb
    is consumed space on the backend. This results in miscalculation of
    over subscription in the Cinder scheduler.

    The fix adopted here is to remove this wrong reporting and rely on
    calculation of the provisioned_capacity_gb in the scheduler.

    Change-Id: Ic106dbcae8ceaac265b710756ab1874e445ca826
    Closes-Bug: #1714209
    (cherry picked from commit 42b8b7fe60ffdd7a7772dc0ab228265dc83344bc)
    (cherry picked from commit 558571b44d9cd2195993e42539fd2c689b179ee6)
    Conflicts:
     cinder/tests/unit/volume/drivers/netapp/dataontap/test_nfs_7mode.py

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 10.0.7 release.