Comment 4 for bug 1224608

Revision history for this message
Xing Yang (xing-yang) wrote :

This problem was caused by too many requests trying to access db at the same time. We ran into this problem when trying to delete 100 volumes at the same time.

The TimeoutError "QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30" means:
pool size: 5
overflow: 10
timeout: 30

The current connections reached 15 (5+10). The new request will wait 30 seconds (timeout) to acquire connection from the pool or raise the TimeoutError.

This problem was seen in projects other than Cinder:

https://bugs.launchpad.net/nova/+bug/1181372
https://bugs.launchpad.net/ceilometer/+bug/1243251

Here is a solution for this problem.

Add the following options in /etc/cinder.conf:

# Changing the following options max_pool_size and max_overflow
# to allow more db connections
[database]
# Maximum number of SQL connections to keep open in a pool
# (integer value)
max_pool_size=20

# If set, use this value for max_overflow with sqlalchemy
# (integer value)
max_overflow=30

Note: Need to add [database] before max_pool_size and max_overflow.
If you have multi-backend options in cinder.conf, add these options after the multi-backend options, otherwise, cinder-volume will be confused and can't read the multi-backend options.
The following config worked for us:
max_pool_size=25, max_overflow=-1 (-1 means infinite)
max_pool_size=20, max_overflow=30

After making cinder.conf changes, do the following:

cinder-manage db sync
Restart cinder-api, cinder-scheduler, and cinder-volume.