Cinder doesn't allow to create several volumes in parallel: ERROR: Request Entity Too Large (HTTP 413)

[Timur Nurlygayanov]

Steps To Reproduce:
1. Deploy OpenStack cluster with Cinder
2. Login to OpenStack controller node
3. Run the following script:
#________________________________
for i in `seq 1 10`
do
    cinder create --display-name test$i 2 &
done
#________________________________

4. Get list of volumes

Expected Results:
All 10 volumes should be created successfully, without any errors. User should see all new 10 volumes in the list.

Observed Results:
Some volumes failed to create, and we get "ERROR: Request Entity Too Large (HTTP 413) (Request-ID: req-0f12b6d5-b45a-4681-b8f4-d89e48efd1c4)" error in 'glance list' command.

Example:
http://paste.mosi.mirantis.net/show/599/

Note:
Looks like it is Cinder API performance issue, cinder failed because of "many" API requests. We will get more errors if we will increase the count of iterations in script.

This bug was reproduced on MOS 6.1 RC3 (#525) ISO. Deployment configuration: CentOS, HA with 3 controllers and 1 compute, with Ceph backend for Cinder, 1 ceph-osd node (on compute node). Snapshot is attached.
Looks like this issue not reproduced on backends with Cinder LVM, only with Cinder+Ceph deployments.

[Timur Nurlygayanov (tnurlygayanov)]

Diagnostic snapshot is attached ^^

We can see in the log the following errors:
 VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: error connecting to ceph cluster

[Oleksiy Butenko (obutenko)]

Timur, on you cluster Quatas Volumes = 10.
Then you run script, you already have 4 volumes, that's why you got 4 error "ERROR: Request Entity Too Large (HTTP 413) (Request-ID: req-0f12b6d5-b45a-4681-b8f4-d89e48efd1c4)"
cat /var/log/cinder-all.log |grep TRACE
2015-06-20 14:37:07.627 23611 TRACE cinder.volume.api Traceback (most recent call last):
2015-06-20 14:37:07.627 23611 TRACE cinder.volume.api File "/usr/lib/python2.6/site-packages/taskflow/engines/action_engine/executor.py", line 35, in _execute_task
2015-06-20 14:37:07.627 23611 TRACE cinder.volume.api result = task.execute(**arguments)
2015-06-20 14:37:07.627 23611 TRACE cinder.volume.api File "/usr/lib/python2.6/site-packages/cinder/volume/flows/api/create_volume.py", line 598, in execute
2015-06-20 14:37:07.627 23611 TRACE cinder.volume.api raise exception.VolumeLimitExceeded(allowed=quotas['volumes'])
2015-06-20 14:37:07.627 23611 TRACE cinder.volume.api VolumeLimitExceeded: Maximum number of volumes allowed (10) exceeded
2015-06-20 14:37:07.627 23611 TRACE cinder.volume.api

I changed value Quotas Volumes to 100 and run your script again with 30 iteration:
all 30 volumes was create successfully.

[Timur Nurlygayanov (tnurlygayanov)]

Oleksiy, thank you for investigation of the issue!