Huawei: capacity calculation error in batch creation

[zhaohua]

When create 50 shares at the same time. One operation failed.
To look at the log of manila-share, the reason is:

2016-02-14 08:34:44.939 ERROR oslo_messaging.rpc.dispatcher [req-d20a3d6c-fa15-4dff-96f7-5c485e6a734a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Exception during message handling: Invalid share: Failed to create share share-40ae66ff-1d74-442c-9335-ef14d51cba6e.Reason: Invalid share: Create filesystem error.
result: {u'data': {}, u'error': {u'code': 1077949090, u'description': u'The storage pool does not have enough space for storing data.', u'suggestion': u'Expand the storage pool and try again.'}}....

And to look at the log of manila-scheduler, I found there is something wrong with the calculation of capacity:
(To search "Space information for share creation on host".
At 08:34:36 the avail of the two pools are 199 and 1
At 08:34:44 the avail of the two pools are still 199 and 1
)

2016-02-14 08:34:36.34 DEBUG manila.scheduler.filters.base [req-a219c58d-5486-437d-bd81-a1e2299f269a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Starting with 2 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:81
2016-02-14 08:34:36.35 DEBUG manila.scheduler.filters.base [req-a219c58d-5486-437d-bd81-a1e2299f269a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filter AvailabilityZoneFilter returned 2 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:98
2016-02-14 08:34:36.35 DEBUG manila.scheduler.filters.capacity [req-a219c58d-5486-437d-bd81-a1e2299f269a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Space information for share creation on host ubuntu@generic1#StoragePool004 (requested / avail): 2/199.0 from (pid=3444) host_passes /opt/stack/manila/manila/scheduler/filters/capacity.py:78
2016-02-14 08:34:36.36 DEBUG manila.scheduler.filters.capacity [req-a219c58d-5486-437d-bd81-a1e2299f269a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Space information for share creation on host ubuntu@generic1#StoragePool003 (requested / avail): 2/1.0 from (pid=3444) host_passes /opt/stack/manila/manila/scheduler/filters/capacity.py:78
2016-02-14 08:34:36.37 DEBUG manila.scheduler.filters.base [req-a219c58d-5486-437d-bd81-a1e2299f269a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filter CapacityFilter returned 2 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:98
2016-02-14 08:34:36.38 DEBUG manila.scheduler.filters.base [req-a219c58d-5486-437d-bd81-a1e2299f269a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filter CapabilitiesFilter returned 2 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:98
2016-02-14 08:34:36.38 DEBUG manila.scheduler.filters.base [req-a219c58d-5486-437d-bd81-a1e2299f269a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filter ConsistencyGroupFilter returned 2 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:98
2016-02-14 08:34:36.39 DEBUG manila.scheduler.drivers.filter [req-a219c58d-5486-437d-bd81-a1e2299f269a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filtered share [host: 'ubuntu@generic1#StoragePool004', free_capacity_gb: 199.0, pools: None, host: 'ubuntu@generic1#StoragePool003', free_capacity_gb: 1.0, pools: None] from (pid=3444) _schedule_share /opt/stack/manila/manila/scheduler/drivers/filter.py:195
2016-02-14 08:34:36.40 DEBUG manila.scheduler.drivers.filter [req-a219c58d-5486-437d-bd81-a1e2299f269a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Choosing for share: WeighedHost [host: ubuntu@generic1#StoragePool003, weight: 1.0] from (pid=3444) _schedule_share /opt/stack/manila/manila/scheduler/drivers/filter.py:202
2016-02-14 08:34:36.116 DEBUG oslo_messaging._drivers.amqpdriver [req-a219c58d-5486-437d-bd81-a1e2299f269a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] CAST unique_id: 6665911e40d14f329d50d22933618eb7 exchange 'openstack' topic 'manila-share' from (pid=3444) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-02-14 08:34:38.503 DEBUG oslo_messaging._drivers.amqpdriver [-] received message msg_id: None reply to None from (pid=3444) __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:196
2016-02-14 08:34:38.519 DEBUG manila.scheduler.filters.base [req-9b09a672-77bc-40cf-bd4a-6ac0772d5761 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Starting with 2 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:81
2016-02-14 08:34:38.523 DEBUG manila.scheduler.filters.base [req-9b09a672-77bc-40cf-bd4a-6ac0772d5761 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filter AvailabilityZoneFilter returned 2 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:98
2016-02-14 08:34:38.525 DEBUG manila.scheduler.filters.capacity [req-9b09a672-77bc-40cf-bd4a-6ac0772d5761 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Space information for share creation on host ubuntu@generic1#StoragePool004 (requested / avail): 2/199.0 from (pid=3444) host_passes /opt/stack/manila/manila/scheduler/filters/capacity.py:78
2016-02-14 08:34:38.526 DEBUG manila.scheduler.filters.capacity [req-9b09a672-77bc-40cf-bd4a-6ac0772d5761 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Space information for share creation on host ubuntu@generic1#StoragePool003 (requested / avail): 2/-1.0 from (pid=3444) host_passes /opt/stack/manila/manila/scheduler/filters/capacity.py:78
2016-02-14 08:34:38.527 DEBUG manila.scheduler.filters.base [req-9b09a672-77bc-40cf-bd4a-6ac0772d5761 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filter CapacityFilter returned 1 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:98
2016-02-14 08:34:38.528 DEBUG manila.scheduler.filters.base [req-9b09a672-77bc-40cf-bd4a-6ac0772d5761 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filter CapabilitiesFilter returned 1 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:98
2016-02-14 08:34:38.529 DEBUG manila.scheduler.filters.base [req-9b09a672-77bc-40cf-bd4a-6ac0772d5761 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filter ConsistencyGroupFilter returned 1 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:98
2016-02-14 08:34:38.530 DEBUG manila.scheduler.drivers.filter [req-9b09a672-77bc-40cf-bd4a-6ac0772d5761 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filtered share [host: 'ubuntu@generic1#StoragePool004', free_capacity_gb: 199.0, pools: None] from (pid=3444) _schedule_share /opt/stack/manila/manila/scheduler/drivers/filter.py:195
2016-02-14 08:34:38.531 DEBUG manila.scheduler.drivers.filter [req-9b09a672-77bc-40cf-bd4a-6ac0772d5761 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Choosing for share: WeighedHost [host: ubuntu@generic1#StoragePool004, weight: 0.0] from (pid=3444) _schedule_share /opt/stack/manila/manila/scheduler/drivers/filter.py:202
2016-02-14 08:34:38.653 DEBUG oslo_messaging._drivers.amqpdriver [req-9b09a672-77bc-40cf-bd4a-6ac0772d5761 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] CAST unique_id: a24fda287eb4438e975a4c212c505bf8 exchange 'openstack' topic 'manila-share' from (pid=3444) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-02-14 08:34:41.443 DEBUG oslo_messaging._drivers.amqpdriver [-] received message msg_id: None reply to None from (pid=3444) __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:196
2016-02-14 08:34:41.456 DEBUG manila.scheduler.filters.base [req-906cce64-f5ef-4402-b581-65930c2e9382 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Starting with 2 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:81
2016-02-14 08:34:41.457 DEBUG manila.scheduler.filters.base [req-906cce64-f5ef-4402-b581-65930c2e9382 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filter AvailabilityZoneFilter returned 2 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:98
2016-02-14 08:34:41.458 DEBUG manila.scheduler.filters.capacity [req-906cce64-f5ef-4402-b581-65930c2e9382 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Space information for share creation on host ubuntu@generic1#StoragePool004 (requested / avail): 2/197.0 from (pid=3444) host_passes /opt/stack/manila/manila/scheduler/filters/capacity.py:78
2016-02-14 08:34:41.458 DEBUG manila.scheduler.filters.capacity [req-906cce64-f5ef-4402-b581-65930c2e9382 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Space information for share creation on host ubuntu@generic1#StoragePool003 (requested / avail): 2/-1.0 from (pid=3444) host_passes /opt/stack/manila/manila/scheduler/filters/capacity.py:78
2016-02-14 08:34:41.459 DEBUG manila.scheduler.filters.base [req-906cce64-f5ef-4402-b581-65930c2e9382 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filter CapacityFilter returned 1 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:98
2016-02-14 08:34:41.460 DEBUG manila.scheduler.filters.base [req-906cce64-f5ef-4402-b581-65930c2e9382 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filter CapabilitiesFilter returned 1 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:98
2016-02-14 08:34:41.462 DEBUG manila.scheduler.filters.base [req-906cce64-f5ef-4402-b581-65930c2e9382 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filter ConsistencyGroupFilter returned 1 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:98
2016-02-14 08:34:41.462 DEBUG manila.scheduler.drivers.filter [req-906cce64-f5ef-4402-b581-65930c2e9382 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filtered share [host: 'ubuntu@generic1#StoragePool004', free_capacity_gb: 197.0, pools: None] from (pid=3444) _schedule_share /opt/stack/manila/manila/scheduler/drivers/filter.py:195
2016-02-14 08:34:41.463 DEBUG manila.scheduler.drivers.filter [req-906cce64-f5ef-4402-b581-65930c2e9382 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Choosing for share: WeighedHost [host: ubuntu@generic1#StoragePool004, weight: 0.0] from (pid=3444) _schedule_share /opt/stack/manila/manila/scheduler/drivers/filter.py:202
2016-02-14 08:34:41.558 DEBUG oslo_messaging._drivers.amqpdriver [req-906cce64-f5ef-4402-b581-65930c2e9382 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] CAST unique_id: 43af2f7527ea432fafec5ee5d6795855 exchange 'openstack' topic 'manila-share' from (pid=3444) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-02-14 08:34:42.942 DEBUG oslo_messaging._drivers.amqpdriver [-] received message msg_id: None reply to None from (pid=3444) __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:196
2016-02-14 08:34:42.945 DEBUG manila.scheduler.host_manager [req-48de2f0d-4a1f-4f60-873b-e68e8de37890 None None] Received share service update from ubuntu@generic1: {u'qos': True, u'snapshot_support': True, u'share_backend_name': u'GENERIC1', u'driver_handles_share_servers': False, u'driver_version': u'1.2', u'total_capacity_gb': 0.0, u'pools': [{u'huawei_smartpartition': True, u'pool_name': u'StoragePool003', u'qos': True, u'compression': True, u'provisioned_capacity_gb': 199.0, u'allocated_capacity_gb': 199.0, u'free_capacity_gb': 1.0, u'total_capacity_gb': 200.0, u'huawei_smartcache': True, u'reserved_percentage': 0, u'max_over_subscription_ratio': 20.0, u'dedupe': True, u'thin_provisioning': True}, {u'huawei_smartpartition': True, u'pool_name': u'StoragePool004', u'qos': True, u'compression': False, u'provisioned_capacity_gb': 1.0, u'allocated_capacity_gb': 1.0, u'free_capacity_gb': 199.0, u'total_capacity_gb': 200.0, u'huawei_smartcache': True, u'reserved_percentage': 0, u'max_over_subscription_ratio': 20.0, u'dedupe': False, u'thin_provisioning': False}], u'vendor_name': u'Huawei', u'reserved_percentage': 0, u'free_capacity_gb': 0.0, u'storage_protocol': u'NFS_CIFS'} from (pid=3444) update_service_capabilities /opt/stack/manila/manila/scheduler/host_manager.py:482
2016-02-14 08:34:44.417 DEBUG oslo_messaging._drivers.amqpdriver [-] received message msg_id: None reply to None from (pid=3444) __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:196
2016-02-14 08:34:44.433 DEBUG manila.scheduler.filters.base [req-d20a3d6c-fa15-4dff-96f7-5c485e6a734a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Starting with 2 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:81
2016-02-14 08:34:44.435 DEBUG manila.scheduler.filters.base [req-d20a3d6c-fa15-4dff-96f7-5c485e6a734a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filter AvailabilityZoneFilter returned 2 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:98
2016-02-14 08:34:44.436 DEBUG manila.scheduler.filters.capacity [req-d20a3d6c-fa15-4dff-96f7-5c485e6a734a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Space information for share creation on host ubuntu@generic1#StoragePool004 (requested / avail): 2/199.0 from (pid=3444) host_passes /opt/stack/manila/manila/scheduler/filters/capacity.py:78
2016-02-14 08:34:44.436 DEBUG manila.scheduler.filters.capacity [req-d20a3d6c-fa15-4dff-96f7-5c485e6a734a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Space information for share creation on host ubuntu@generic1#StoragePool003 (requested / avail): 2/1.0 from (pid=3444) host_passes /opt/stack/manila/manila/scheduler/filters/capacity.py:78
2016-02-14 08:34:44.437 DEBUG manila.scheduler.filters.base [req-d20a3d6c-fa15-4dff-96f7-5c485e6a734a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filter CapacityFilter returned 2 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:98
2016-02-14 08:34:44.438 DEBUG manila.scheduler.filters.base [req-d20a3d6c-fa15-4dff-96f7-5c485e6a734a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filter CapabilitiesFilter returned 2 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:98
2016-02-14 08:34:44.439 DEBUG manila.scheduler.filters.base [req-d20a3d6c-fa15-4dff-96f7-5c485e6a734a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filter ConsistencyGroupFilter returned 2 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:98
2016-02-14 08:34:44.439 DEBUG manila.scheduler.drivers.filter [req-d20a3d6c-fa15-4dff-96f7-5c485e6a734a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filtered share [host: 'ubuntu@generic1#StoragePool004', free_capacity_gb: 199.0, pools: None, host: 'ubuntu@generic1#StoragePool003', free_capacity_gb: 1.0, pools: None] from (pid=3444) _schedule_share /opt/stack/manila/manila/scheduler/drivers/filter.py:195
2016-02-14 08:34:44.440 DEBUG manila.scheduler.drivers.filter [req-d20a3d6c-fa15-4dff-96f7-5c485e6a734a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Choosing for share: WeighedHost [host: ubuntu@generic1#StoragePool003, weight: 1.0] from (pid=3444) _schedule_share /opt/stack/manila/manila/scheduler/drivers/filter.py:202
2016-02-14 08:34:44.546 DEBUG oslo_messaging._drivers.amqpdriver [req-d20a3d6c-fa15-4dff-96f7-5c485e6a734a 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] CAST unique_id: 18c1cec4f8b243f3a35192e4bc6af288 exchange 'openstack' topic 'manila-share' from (pid=3444) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-02-14 08:34:46.869 DEBUG oslo_messaging._drivers.amqpdriver [-] received message msg_id: None reply to None from (pid=3444) __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:196
2016-02-14 08:34:46.889 DEBUG manila.scheduler.filters.base [req-02746394-ae4c-4fac-b3c6-7a7f9872fedb 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Starting with 2 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:81
2016-02-14 08:34:46.890 DEBUG manila.scheduler.filters.base [req-02746394-ae4c-4fac-b3c6-7a7f9872fedb 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filter AvailabilityZoneFilter returned 2 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:98
2016-02-14 08:34:46.891 DEBUG manila.scheduler.filters.capacity [req-02746394-ae4c-4fac-b3c6-7a7f9872fedb 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Space information for share creation on host ubuntu@generic1#StoragePool004 (requested / avail): 2/199.0 from (pid=3444) host_passes /opt/stack/manila/manila/scheduler/filters/capacity.py:78
2016-02-14 08:34:46.891 DEBUG manila.scheduler.filters.capacity [req-02746394-ae4c-4fac-b3c6-7a7f9872fedb 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Space information for share creation on host ubuntu@generic1#StoragePool003 (requested / avail): 2/-1.0 from (pid=3444) host_passes /opt/stack/manila/manila/scheduler/filters/capacity.py:78
2016-02-14 08:34:46.892 DEBUG manila.scheduler.filters.base [req-02746394-ae4c-4fac-b3c6-7a7f9872fedb 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filter CapacityFilter returned 1 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:98
2016-02-14 08:34:46.893 DEBUG manila.scheduler.filters.base [req-02746394-ae4c-4fac-b3c6-7a7f9872fedb 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filter CapabilitiesFilter returned 1 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:98
2016-02-14 08:34:46.894 DEBUG manila.scheduler.filters.base [req-02746394-ae4c-4fac-b3c6-7a7f9872fedb 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filter ConsistencyGroupFilter returned 1 host(s) from (pid=3444) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:98
2016-02-14 08:34:46.894 DEBUG manila.scheduler.drivers.filter [req-02746394-ae4c-4fac-b3c6-7a7f9872fedb 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Filtered share [host: 'ubuntu@generic1#StoragePool004', free_capacity_gb: 199.0, pools: None] from (pid=3444) _schedule_share /opt/stack/manila/manila/scheduler/drivers/filter.py:195
2016-02-14 08:34:46.895 DEBUG manila.scheduler.drivers.filter [req-02746394-ae4c-4fac-b3c6-7a7f9872fedb 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] Choosing for share: WeighedHost [host: ubuntu@generic1#StoragePool004, weight: 0.0] from (pid=3444) _schedule_share /opt/stack/manila/manila/scheduler/drivers/filter.py:202
2016-02-14 08:34:46.985 DEBUG oslo_messaging._drivers.amqpdriver [req-02746394-ae4c-4fac-b3c6-7a7f9872fedb 4ae019e9d86242038e4369d08be1d233 086743ad5cb04fc6876c1fe04d4fc16c] CAST unique_id: 7190caf73a3441fea8bed5f075e5c652 exchange 'openstack' topic 'manila-share' from (pid=3444) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-02-14 08:34:49.316 DEBUG oslo_messaging._drivers.amqpdriver [-] received message msg_id: None reply to None from (pid=3444) __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:196

[Ben Swartzlander (bswartz)]

I'm not sure what's going on here. I've done tests with multiple pools of equal size and a burst of share creation requests, and the shares were distributed across the pools like you would expect.

I'm wondering if something about the share type or capabilities caused the scheduler to always pick 1 pool in this case.

[zhaohua (zhaohua-zhao)]

I think I found the reason :

at 08:34:36.36 ,the capacity of the pool is 1
at 08:34:38.526 ,the capacity of the pool is -1
at 08:34:44.436, the capacity of the pool is 1

There is a share_update_stat called at 08:34:42.945, backend reported the capacity is 1.

So here is the case:
1. At 08:34:36.36 ,thread1 choosed pool one to create share,and changed the capacity of pool one from 1 to -1.
2. There are so many threads waiting to create shares, so thread1's rest is waiting to send to backend storage system.
3. At 08:34:42.945, the update_share_state of the backend been called. Beacuse thread1's rest had not been send, the backend still report the capacity of pool one is 1.
4. At 08:34:44.436, thread2 choosed pool one, because the capacity of pool one is 1.
5.Thread1 and thread2's rest send to the backend storage system. And the capacity is not enough for two shares.

But, I still don't know how to fix it.

[Ben Swartzlander (bswartz)]

Unless someone from Huawei wants to fix this, we need a way to reproduce the bug on an open source backend so someone can track down the behavior. I'm going to retarget out of Mitaka because the bug has no owner and I'm not able to reproduce it myself.