Cinder fails to create volume with Gateway time-out error under high load

[Mikhail Chernik]

When running rally task http://paste.openstack.org/show/593624/ some iterations failed during volume creation with Gateway time-out error.

The reason is that volume creation took more than 2 min, so haproxy dropped connection

cinder-api.log: http://paste.openstack.org/show/593625/

haproxy.log: http://paste.openstack.org/show/593626/

[Ivan Kolodyazhny (e0ne)]

Correct rally task json https://paste.mirantis.net/show/2922/

[Timur Nurlygayanov (tnurlygayanov)]

Could you please attach diagnostic snapshot?

[Mikhail Chernik (mchernik)]

Diagnostic snapshot: http://mos-scale-share.mirantis.com/env_10_scale_run_build_688_snapshot/

[Vitaly Sedelnik (vsedelnik)]

Need more time to investigate and fix the issue - retargeted to 9.3

[Vitaly Sedelnik (vsedelnik)]

Raising to Critical and target back to 9.2 as it looks like this is degradation comparing to 9.1. Waiting for more details from scale team.

[Ivan Kolodyazhny (e0ne)]

According to 9.1 Rally Report [1] it's not degradation. Issue is valid for 9.1 too

[1] http://mos-scale.vm.mirantis.net/test_results/build_9.0.0-495/jenkins-10_env_run_rally_custom-260-ubuntu-MSK-2016-09-29-01:28:41-2016-10-04-01:41:09/rally_report.html#/CinderVolumes.create_volume_backup/overview

[Ivan Kolodyazhny (e0ne)]

Looking throw logs, I've marked this bug as duplicated of #1550192.

Switch to PyMySQL help to resolve this issue on my env.

As a workaround, we can use also:
- cinder api + apache/mod_wsgi
- increase http timeout in HAproxy

IMO, both of proposed workarounds are not very good for 9.2 at this moment. We can test and apply it on customer's envs if needed.

My env:
- all nodes are Vms
- 1 controller
- 1 compute
- 3 osd nodes
- rally scenario config:
-- times: 10
-- concurrency: 5