NetApp driver is hitting the QoS policy limit due soft deletion

[Felipe Rodrigues]

Description
===========
Customer cannot create a new volume with QoS due the limit of QoS policy on the storage. The ONTAP has a limit of 12,000 policies [1]. Although the customer does not have 12,000 volumes with QoS, he/she is hitting that limit. The customer is creating 1,000 volumes with QoS per hour and then deleting them.

Reproduce
==================

We can hit the same issue by having an ONTAP with 11,000 policies, so it should be able to create until 1,000 volumes with QoS.

A chronological list of steps which will help reproduce the issue you hit:

1. setup a NFS NetApp backend on a storage that has 11,000 QoS policies
2. create 800 volumes with QoS
3. delete 100 volumes (resulting in 700 volumes)
4. After the volumes have been deleted, create more 100 volumes with QoS (resulting in 800 volumes)
5. Keep doing the step "3" and "4"

Expected result
===============
The user should be able to have 800 volumes with QoS, since the system is supporting 1,000 using that configuration.

Actual result
=============
After some create/delete tasks, the user hits the storage QoS policy limit. The error log [2].

Environment
===========
1. Cinder master

2. ONTAP 9.8 (could be later version)

References
==============

[1]https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/What_is_Quality_of_Service_(QoS)_in_ONTAP%3F

[2] https://paste.opendev.org/show/bvToIu5krty5MWryPoD5/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/853450

[Felipe Rodrigues (felipefutty)]

The approach used in the patch [1] was tested faking an environment with some scripts creating and deleting volumes (maximum of 120 volumes would exist together). We count the number of QoS policies on the storage and plot a graphic, comparing the force flag approach (patch [1]) and without the force (current master approach).

The results are showed by the attached image (scatter-plot.png). As we can see with force flag, the number of policies keeps constant, since the number of existing volumes are almost constant, while the old approach increases during a period of time, even the number of volumes does not increase that way.

[1] https://review.opendev.org/c/openstack/cinder/+/853450