'cinder list' ERROR: Gateway Timeout (HTTP 504) when we have more than 50k volumes

[Oleksiy Butenko]

MOS 6.1 ISO 525
I used fake driver for cinder.
Steps:
1. Deploy Env(my env: Controller - 3, Compute + Storage-Cinder - 2, Neutron VLAN, Ubuntu)
2. Create 50-60k volumes
3. Enter In cli on controller node - cinder list --all_tenant

Actual result:
I received 504 error time to time

root@node-1:~# . openrc
root@node-1:~# cinder list --all_tenant wc -l
ERROR: Gateway Timeout (HTTP 504)
root@node-1:~# cinder list --all_tenant | wc -l
1004
root@node-1:~# cinder list --all_tenant wc -l
ERROR: Gateway Timeout (HTTP 504)

[Oleksiy Butenko (obutenko)]

{u'build_id': u'2015-06-19_13-02-31', u'build_number': u'525', u'auth_required': True, u'fuel-ostf_sha': u'8fefcf7c4649370f00847cc309c24f0b62de718d', u'fuel-library_sha': u'2e7a08ad9792c700ebf08ce87f4867df36aa9fab', u'nailgun_sha': u'dbd54158812033dd8cfd7e60c3f6650f18013a37', u'openstack_version': u'2014.2.2-6.1', u'production': u'docker', u'api': u'1.0', u'python-fuelclient_sha': u'4fc55db0265bbf39c369df398b9dc7d6469ba13b', u'astute_sha': u'1ea8017fe8889413706d543a5b9f557f5414beae', u'fuelmain_sha': u'a3998372183468f56019c8ce21aa8bb81fee0c2f', u'feature_groups': [u'mirantis'], u'release': u'6.1', u'release_versions': {u'2014.2.2-6.1': {u'VERSION': {u'build_id': u'2015-06-19_13-02-31', u'build_number': u'525', u'fuel-library_sha': u'2e7a08ad9792c700ebf08ce87f4867df36aa9fab', u'nailgun_sha': u'dbd54158812033dd8cfd7e60c3f6650f18013a37', u'fuel-ostf_sha': u'8fefcf7c4649370f00847cc309c24f0b62de718d', u'production': u'docker', u'api': u'1.0', u'python-fuelclient_sha': u'4fc55db0265bbf39c369df398b9dc7d6469ba13b', u'astute_sha': u'1ea8017fe8889413706d543a5b9f557f5414beae', u'fuelmain_sha': u'a3998372183468f56019c8ce21aa8bb81fee0c2f', u'feature_groups': [u'mirantis'], u'release': u'6.1', u'openstack_version': u'2014.2.2-6.1'}}}}

[Oleksiy Butenko (obutenko)]

https://goo.gl/lvsZpX - snapshot

[Alexander Nevenchannyy (anevenchannyy)]

Please add time before cinder list --all_tenant wc -l.
Error 504 look like timeout returned by HaProxy

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Anton Arefiev <email address hidden>
Review: https://review.fuel-infra.org/10643

[Anton Arefiev (aarefiev)]

There are two problems here:
 1) API v1(default in mos 7.0): Currently when there is no limit set on a volume list query we retrieve all volumes and then limit them locally using osapi_max_limit. Similar thing happens when we are using the marker for next pages, we get all
volumes from that marker until the last volume and then limit it locally.
We should be limiting it on the DB side so we only retrieve the data we are actually going to return to the API caller.
v1 returns only osapi_max_limit number of volumes.

2) API v2: In difference from v1, v2 API returns all volumes, but makes volume_number/osapi_max_limit requests(e.g. in current bug we have 50k volumes and osapi_max_limit=1000 by default, so we make 50 requests to cinder API ). In this case we should increase osapi_max_limit in cinder.conf to reduce retrieving data.

Proposed approaches doesn't solve the problem, it will appear again on 100k, 150k ...? volumes. The one of possible solutions here is introduce indexes in db.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/10643
Submitter: mos-infra-ci <>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: cd85bb085a84dbab0f1aa9f66bdfc8012685609b
Author: Anton Arefiev <email address hidden>
Date: Wed Aug 26 08:12:45 2015

Volume list gets only needed data from db

Currently when there is no limit set on a volume list query we retrieve
all volumes and then limit them locally using osapi_max_limit. Similar
thing happens when we are using the marker for next pages, we get all
volumes from that marker until the last volume and then limit it
locally.

We should be limiting it on the DB side so we only retrieve the data we
are actually going to return to the API caller.

Also some minor changes was made:
 - remove loggining each volume from v1 api;
 - set osapi_max_limit in methos body instead of initialize method's args;

Closes-bug: #1471273
Change-Id: Ie903e546074fe118299e8e1acfb9c88c8a10d78c

[Oleksiy Butenko (obutenko)]

verified on ISO 241
{"build_id": "2015-08-27_03-02-45", "build_number": "241", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "2015-08-27_03-02-45", "build_number": "241", "api": "1.0", "fuel-library_sha": "d1f86c7dfec8478c31752f5c8153350a9a6b7a96", "nailgun_sha": "2d04787440e592a2c119df0fd721445ee95a5553", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "8844dc5964fb197b64525f04b6ec15cbbcfe44f0", "production": "docker", "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "dec66ae87b3e5f8b84670c6b4dbd184b0a3c7770", "release": "7.0", "fuelmain_sha": "d6553ba195f2cb8f49b0f9b9c59359c38f5bdd0c"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "d1f86c7dfec8478c31752f5c8153350a9a6b7a96", "nailgun_sha": "2d04787440e592a2c119df0fd721445ee95a5553", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "8844dc5964fb197b64525f04b6ec15cbbcfe44f0", "production": "docker", "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "dec66ae87b3e5f8b84670c6b4dbd184b0a3c7770", "release": "7.0", "fuelmain_sha": "d6553ba195f2cb8f49b0f9b9c59359c38f5bdd0c"}

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Anton Arefiev <email address hidden>
Review: https://review.fuel-infra.org/11644

[Alexey Khivin (akhivin)]

"won't fix" for the old branches due to complexity and customer's request absence

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Alexey Khivin <email address hidden> on branch: openstack-ci/fuel-6.1/2014.2
Review: https://review.fuel-infra.org/11644
Reason: "won't fix" for the old branches due to complexity and customer's request absence