Mirantis OpenStack

Cinder/nova API doesn't support quota managent via keystone v3 api

Bug #1585422 reported by Nikita Koshikov on 2016-05-25

This bug affects 2 people

	Status	Importance	Assigned to	Milestone
Mirantis OpenStack	Status tracked in 10.0.x
10.0.x	Invalid	High	MOS Oslo	Mirantis OpenStack 10.0
7.0.x	Fix Released	High	Anton Chevychalov	Mirantis OpenStack 7.0-mu-6
8.0.x	Invalid	Undecided	Unassigned
9.x	Invalid	High	Dmitry Mescheryakov	Mirantis OpenStack 9.0

Bug Description

This is MOS7+MU3

Steps to reproduce:
1)Create environment with ceph backend and nova-quotas enabled
2)Install keystone-ldap plagin, that will enable additional domain, except 'default'
3)Deploy environment.

After deployment create openrc_v3_admin file with:
#!/bin/sh
source /root/openrc
export OS_IDENTITY_API_VERSION='3'
export OS_USER_DOMAIN_NAME='default'
export OS_AUTH_URL=${OS_AUTH_URL/v2.0/v3}
unset OS_PROJECT_ID

Try to update quotas:
// SRIX-UNIX - is project created inside non-default keystone domain:
# openstack quota show SRIX-UNIX
+----------------------+-----------+
| Field | Value |
+----------------------+-----------+
| backup_gigabytes | 1000 |
| backups | 10 |
| cores | 100 |
| fixed-ips | -1 |
| floating-ips | 50 |
| gigabytes | 1000 |
| injected-file-size | 102400 |
| injected-files | 50 |
| injected-path-size | 4096 |
| instances | 100 |
| key-pairs | 11 |
| network | 10 |
| port | 50 |
| project | SRIX-UNIX |
| properties | 1024 |
| ram | 51200 |
| router | 10 |
| secgroup-rules | 100 |
| secgroups | 10 |
| server_group_members | 10 |
| server_groups | 10 |
| snapshots | 10 |
| subnet | 10 |
| volumes | 10 |
+----------------------+-----------+

#openstack --debug quota set SRIX-UNIX --cores 198
DEBUG: keystoneclient.auth.identity.v3.base Making authentication request to http://10.200.153.31:5000/v3/auth/tokens
INFO: urllib3.connectionpool Starting new HTTP connection (1): 10.200.153.31
DEBUG: urllib3.connectionpool "POST /v3/auth/tokens HTTP/1.1" 201 9561
DEBUG: openstackclient.common.quota.SetQuota take_action(Namespace(cores=198, fixed_ips=None, floating_ips=None, gigabytes=None, injected_file_size=None, injected_files=None, injected_path_size=None, instances=None, key_pairs=None, project='SRIX-UNIX', properties=None, quota_class=False, ram=None, secgroup_rules=None, secgroups=None, snapshots=None, volumes=None))
DEBUG: openstackclient.compute.client Instantiating compute client: <class 'novaclient.v2.client.Client'>
DEBUG: openstackclient.volume.client Instantiating volume client: <class 'cinderclient.v1.client.Client'>
DEBUG: keystoneclient.auth.identity.v3.base Making authentication request to http://10.200.153.31:5000/v3/auth/tokens
INFO: urllib3.connectionpool Resetting dropped connection: 10.200.153.31
DEBUG: urllib3.connectionpool "POST /v3/auth/tokens HTTP/1.1" 201 9561
DEBUG: keystoneclient.session REQ: curl -g -i -X PUT https://public.fuel.local:8774/v2/0e31323105ec41a7ac263bb6ab5c8930/os-quota-sets/SRIX-UNIX -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}029d165407f60be480bd57d6af30e21273ff9801" -d '{"quota_set": {"tenant_id": "SRIX-UNIX", "cores": 198}}'
INFO: urllib3.connectionpool Starting new HTTPS connection (1): public.fuel.local
DEBUG: urllib3.connectionpool "PUT /v2/0e31323105ec41a7ac263bb6ab5c8930/os-quota-sets/SRIX-UNIX HTTP/1.1" 200 335
DEBUG: keystoneclient.session RESP: [200] date: Wed, 25 May 2016 00:33:15 GMT connection: close content-type: application/json content-length: 335 x-compute-request-id: req-8341015b-71db-4a61-87db-4ac33f7ccd91
RESP BODY: {"quota_set": {"injected_file_content_bytes": 102400, "metadata_items": 1024, "server_group_members": 10, "server_groups": 10, "ram": 51200, "floating_ips": 100, "key_pairs": 11, "instances": 100, "security_group_rules": 20, "injected_files": 50, "cores": 198, "fixed_ips": -1, "injected_file_path_bytes": 4096, "security_groups": 10}}

DEBUG: keystoneclient.session REQ: curl -g -i -X PUT https://public.fuel.local:8776/v1/0e31323105ec41a7ac263bb6ab5c8930/os-quota-sets/SRIX-UNIX -H "User-Agent: python-cinderclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}029d165407f60be480bd57d6af30e21273ff9801" -d '{"quota_set": {"gigabytes": null, "tenant_id": "SRIX-UNIX", "snapshots": null, "volumes": null}}'
INFO: urllib3.connectionpool Starting new HTTPS connection (1): public.fuel.local
DEBUG: urllib3.connectionpool "PUT /v1/0e31323105ec41a7ac263bb6ab5c8930/os-quota-sets/SRIX-UNIX HTTP/1.1" 400 141
DEBUG: keystoneclient.session RESP:
ERROR: openstack The server could not comply with the request since it is either malformed or otherwise incorrect. (HTTP 400) (Request-ID: req-62c9db6a-6e23-4d50-8fdf-618c3d554efb)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 303, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/cliff/command.py", line 53, in run
    self.take_action(parsed_args)
  File "/usr/lib/python2.7/dist-packages/openstackclient/common/quota.py", line 124, in take_action
    **volume_kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/v1/quotas.py", line 47, in update
    result = self._update('/os-quota-sets/%s' % (tenant_id), body)
  File "/usr/lib/python2.7/dist-packages/cinderclient/base.py", line 170, in _update
    resp, body = self.api.client.put(url, body=body)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 103, in put
    return self._cs_request(url, 'PUT', **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 94, in _cs_request
    return self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 87, in request
    raise exceptions.from_response(resp, body)
BadRequest: The server could not comply with the request since it is either malformed or otherwise incorrect. (HTTP 400) (Request-ID: req-62c9db6a-6e23-4d50-8fdf-618c3d554efb)
DEBUG: openstackclient.shell clean_up SetQuota
DEBUG: openstackclient.shell got an error: The server could not comply with the request since it is either malformed or otherwise incorrect. (HTTP 400) (Request-ID: req-62c9db6a-6e23-4d50-8fdf-618c3d554efb)
ERROR: openstackclient.shell Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/openstackclient/shell.py", line 176, in run
    return super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 238, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 303, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/cliff/command.py", line 53, in run
    self.take_action(parsed_args)
  File "/usr/lib/python2.7/dist-packages/openstackclient/common/quota.py", line 124, in take_action
    **volume_kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/v1/quotas.py", line 47, in update
    result = self._update('/os-quota-sets/%s' % (tenant_id), body)
  File "/usr/lib/python2.7/dist-packages/cinderclient/base.py", line 170, in _update
    resp, body = self.api.client.put(url, body=body)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 103, in put
    return self._cs_request(url, 'PUT', **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 94, in _cs_request
    return self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 87, in request
    raise exceptions.from_response(resp, body)
BadRequest: The server could not comply with the request since it is either malformed or otherwise incorrect.

For some reason nova sends data to cinder, even just for updating cores.
Here is related cinder error:

<158>May 25 00:23:39 node-26 cinder-api 2016-05-25 00:23:39.376 29302 INFO cinder.api.openstack.wsgi [req-cba03135-5899-47b8-9a53-dc4a2a016ada 69368ca9a1b8423794a53a75b77ff613 0e31323105ec41a7ac2
63bb6ab5c8930 - - -] PUT http://public.fuel.local:8776/v1/0e31323105ec41a7ac263bb6ab5c8930/os-quota-sets/SRIX-UNIX
<155>May 25 00:23:39 node-26 cinder-api 2016-05-25 00:23:39.415 29302 ERROR cinder.api.openstack.wsgi [req-c717c430-7323-43cb-a793-7aad5a3e0a2b - - - - -] Exception handling resource: int() argum
ent must be a string or a number, not 'NoneType'
2016-05-25 00:23:39.415 29302 TRACE cinder.api.openstack.wsgi Traceback (most recent call last):
2016-05-25 00:23:39.415 29302 TRACE cinder.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py", line 1027, in _process_stack
2016-05-25 00:23:39.415 29302 TRACE cinder.api.openstack.wsgi action_result = self.dispatch(meth, request, action_args)
2016-05-25 00:23:39.415 29302 TRACE cinder.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py", line 1107, in dispatch
2016-05-25 00:23:39.415 29302 TRACE cinder.api.openstack.wsgi return method(req=request, **action_args)
2016-05-25 00:23:39.415 29302 TRACE cinder.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/cinder/api/contrib/quotas.py", line 129, in update
2016-05-25 00:23:39.415 29302 TRACE cinder.api.openstack.wsgi value = self._validate_quota_limit(body['quota_set'][key])
2016-05-25 00:23:39.415 29302 TRACE cinder.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/cinder/api/contrib/quotas.py", line 61, in _validate_quota_limit
2016-05-25 00:23:39.415 29302 TRACE cinder.api.openstack.wsgi limit = int(limit)
2016-05-25 00:23:39.415 29302 TRACE cinder.api.openstack.wsgi TypeError: int() argument must be a string or a number, not 'NoneType'
2016-05-25 00:23:39.415 29302 TRACE cinder.api.openstack.wsgi

Bug quota has been updated:
# openstack quota show SRIX-UNIX
+----------------------+-----------+
| Field | Value |
+----------------------+-----------+
| backup_gigabytes | 1000 |
| backups | 10 |
| cores | 198 |
| fixed-ips | -1 |
| floating-ips | 50 |
| gigabytes | 1000 |
| injected-file-size | 102400 |
| injected-files | 50 |
| injected-path-size | 4096 |
| instances | 100 |
| key-pairs | 11 |
| network | 10 |
| port | 50 |
| project | SRIX-UNIX |
| properties | 1024 |
| ram | 51200 |
| router | 10 |
| secgroup-rules | 100 |
| secgroups | 10 |
| server_group_members | 10 |
| server_groups | 10 |
| snapshots | 10 |
| subnet | 10 |
| volumes | 10 |
+----------------------+-----------+

And horizon shows non-correct(default) values - screenshot will be attached.

python-openstackclient 1.0.3-1~u14.04+mos577
cinder-api 1:2015.1.1-1~u14.04+mos3058
cinder-backup 1:2015.1.1-1~u14.04+mos3058
cinder-common 1:2015.1.1-1~u14.04+mos3058
cinder-scheduler 1:2015.1.1-1~u14.04+mos3058
cinder-volume 1:2015.1.1-1~u14.04+mos3058
python-cinder 1:2015.1.1-1~u14.04+mos3058
python-cinderclient 1:1.1.2-1~u14.04+mos322
nova-api 1:2015.1.1-1~u14.04+mos19676
nova-cert 1:2015.1.1-1~u14.04+mos19676
nova-common 1:2015.1.1-1~u14.04+mos19676
nova-conductor 1:2015.1.1-1~u14.04+mos19676
nova-consoleauth 1:2015.1.1-1~u14.04+mos19676
nova-consoleproxy 1:2015.1.1-1~u14.04+mos19676
nova-objectstore 1:2015.1.1-1~u14.04+mos19676
nova-scheduler 1:2015.1.1-1~u14.04+mos19676
python-nova 1:2015.1.1-1~u14.04+mos19676
python-novaclient 1:2.23.0-1~u14.04+mos1247

# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "301"
  build_id: "301"
  nailgun_sha: "4162b0c15adb425b37608c787944d1983f543aa8"
  python-fuelclient_sha: "486bde57cda1badb68f915f66c61b544108606f3"
  fuel-agent_sha: "50e90af6e3d560e9085ff71d2950cfbcca91af67"
  fuel-nailgun-agent_sha: "d7027952870a35db8dc52f185bb1158cdd3d1ebd"
  astute_sha: "6c5b73f93e24cc781c809db9159927655ced5012"
  fuel-library_sha: "5d50055aeca1dd0dc53b43825dc4c8f7780be9dd"
  fuel-ostf_sha: "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c"
  fuelmain_sha: "a65d453215edb0284a2e4761be7a156bb5627677"

See original description

Tags:

Revision history for this message

Nikita Koshikov (nkoshikov) wrote on 2016-05-25:

Screen Shot 2016-05-24 at 17.27.31.png Edit (103.3 KiB, image/png)

description:

updated

Revision history for this message

Nikita Koshikov (nkoshikov) wrote on 2016-05-25:

cinder.conf Edit (2.6 KiB, text/plain)

Revision history for this message

Nikita Koshikov (nkoshikov) wrote on 2016-05-25:

nova.conf Edit (4.0 KiB, text/plain)

Revision history for this message

Nikita Koshikov (nkoshikov) wrote on 2016-05-25:

Here is the same bug, but for Juno https://bugzilla.redhat.com/show_bug.cgi?id=1217735

Dmitry Pyzhov (dpyzhov) on 2016-05-25

no longer affects:

fuel

Revision history for this message

Dina Belova (dbelova) wrote on 2016-05-25:

As this is new feature request in fact, adding enhancement tag and marking as wishlist.

tags:	added: area-nova enhancement removed: nova
Changed in mos:
status:	New → Confirmed
importance:	Undecided → Wishlist
assignee:	nobody → MOS Nova (mos-nova)
milestone:	none → 10.0

Revision history for this message

Nikita Koshikov (nkoshikov) wrote on 2016-05-25:

@dbelova, This is not a new feature - so please remove confusing tags and MOS10 is seemed not good milestone for it at all.

Ivan Kolodyazhny (e0ne) on 2016-05-25

Changed in mos:
assignee:	MOS Nova (mos-nova) → Ivan Kolodyazhny (e0ne)

Ivan Kolodyazhny (e0ne) on 2016-05-26

Changed in mos:
assignee:	Ivan Kolodyazhny (e0ne) → nobody
importance:	Wishlist → Medium

Revision history for this message

Ivan Kolodyazhny (e0ne) wrote on 2016-05-26:

Few comments from Cinder's side:

1) Cinder's CLI works good with Keystone v3 after exporting few environment variables: http://paste.openstack.org/show/505678/

2) 'openstack quota set SRIX-UNIX --cores 198' fails with updating Cinder quotas because OpenStack Client (OSC) passes incorrect params to cinder [1]. OSC should not change Cinder quotas if only Nova params passed. If user specifies all params for Nova and Cinder quotas, everything will work good.

According to items above I think that is's a bug with at least Medium priority.

[1] https://github.com/openstack/python-openstackclient/blob/stable/mitaka/openstackclient/common/quota.py#L134

Revision history for this message

Dina Belova (dbelova) wrote on 2016-05-26:

Due to Ivan's comments moving back to high and 9.0

tags:

removed: enhancement

Revision history for this message

Javier Diaz Jr (javierdiazcharles) wrote on 2016-05-26:

Request to patch issue in MOS 7.0 MU5. Issue is impacting customer. Please advise.

Revision history for this message

Davanum Srinivas (DIMS) (dims-v) wrote on 2016-05-27:

#10

The python-openstackclient of 1.0.3 is way too old.

If this is needed for kilo, we should be at least 1.5.0 (python-openstackclient===1.5.0)
https://raw.githubusercontent.com/openstack/requirements/stable/kilo/upper-constraints.txt

This was fixed way back in 1.1.0:
https://github.com/openstack/python-openstackclient/commit/8bd8a8dfd7d330573740f7610ea28ae291d1e4e2

Revision history for this message

Davanum Srinivas (DIMS) (dims-v) wrote on 2016-05-27:

#11

Specifically there is a check for None in this line:
https://github.com/openstack/python-openstackclient/commit/8bd8a8dfd7d330573740f7610ea28ae291d1e4e2#diff-c61185f38e32c91febc9aa9a5e8d32f5R106

which i believe fixes this problem.

Revision history for this message

Dmitry Mescheryakov (dmitrymex) wrote on 2016-05-30:

#12

I have checked the code and the bug is already fixed starting from 8.0. Hence marking this milestone as everything above as invalid.

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2016-05-30: Fix proposed to openstack/python-openstackclient (openstack-ci/fuel-7.0/2015.1.0)

#13

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Steve Martinelli <email address hidden>
Review: https://review.fuel-infra.org/21367

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2016-05-30: Related fix proposed to openstack/python-openstackclient (openstack-ci/fuel-7.0/2015.1.0)

#14

Related fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Jamie Lennox <email address hidden>
Review: https://review.fuel-infra.org/21368

Revision history for this message

Javier Diaz Jr (javierdiazcharles) wrote on 2016-06-13:

#15

I notice this bug listed as MOS7.0-updates. Any idea as to which MU we are targeting here? Customer is looking for an update / ETA on the fix.

Revision history for this message

Anton Chevychalov (achevychalov) wrote on 2016-09-14:

#16

https://review.fuel-infra.org/21367 has been merged into openstack-ci/fuel-7.0/2015.1.0

https://review.fuel-infra.org/21368 has been merged into openstack-ci/fuel-7.0/2015.1.0

Ekaterina Shutova (eshutova) on 2016-09-20

tags:

added: on-verification

Revision history for this message

Ekaterina Shutova (eshutova) wrote on 2016-09-30:

#17

Verified on MOS 7.0 + MU6 updates.

Before fix we got next message after changing quota for SRIX-UNIX:
BadRequest: The server could not comply with the request since it is either malformed or otherwise incorrect. (HTTP 400) (Request-ID: req-879f96aa-ee3e-42d7-914b-ff57c7dae375)