Neutron is requesting too many tokens

Bug #1191159 reported by Hrushikesh
72
This bug affects 14 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
In Progress
Medium
Drew Thorstensen

Bug Description

ACTUAL BEHAVIOR: I have a 14 compute nodes environment with a separate compute controller. With system being idle, i see quantum modules (OVS plugin used here) are requesting to many tokens (approx 2/sec). In a day, it piles up to 150,000 tokens. This behavior adds chattiness and slower performance on the entire OpenStack module's keystone authentication/authorization process. Here is the dump of the count just for a day's run:

select user_id, count(*) from token group by user_id
"2efad4b253f64b4dae65a28f45438d93";10341 <-- admin user
"a1fa17a31a4246518ab3acbf04ff448a";114769 <--quantum user

EXPECTED BEHAVIOR: Though the expiration of the tokens are set for 24 hrs, the quantum is requesting new tokens now n then. Either a missing configuration or code issue that must be causing this.

here is how api-paste.ini looks like under /etc/quantum
[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
auth_host = 192.168.123.12
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = quantum
admin_password = service_pass

HOW-TO-REPRODUCE:
Install openstack using https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/OVS_MultiNode/OpenStack_Grizzly_Install_Guide.rst
Keep the system idle. Note that the count of token being issue is proportional to no. of compute nodes you have.
--attached conf files
--keystone and quantum logs (from compute, controller+network node)

Revision history for this message
Hrushikesh (hrushikesh-gangur) wrote :
description: updated
Revision history for this message
Aaron Rosen (arosen) wrote :

I agree. I think this issue should be solved in the python-quantumclient to cache tokens similar to how the novaclient does. In the meantime this patch might help you.

https://github.com/openstack/nova/commit/dd9c27f999221001bae9faa03571645824d2a681

Revision history for this message
Gary Kotton (garyk) wrote :

Are you using pki or uuid tokens?

Revision history for this message
Hrushikesh (hrushikesh-gangur) wrote :

PKI tokens

Revision history for this message
Jordan Pittier (jordan-pittier) wrote :

I am also using Grizzly + PKI tokens. I'am also affected by this bug. The patch pointed by Aaron doesn't help on this subject.

Changed in neutron:
status: New → Confirmed
assignee: nobody → yong sheng gong (gongysh)
importance: Undecided → Medium
affects: neutron → nova-project
Revision history for this message
Drew Thorstensen (thorst) wrote :

This seems like a critical bug that will affect long running installations. Since it has not have movement since 6/28 I will take it and propose a patch

Changed in nova-project:
assignee: yong sheng gong (gongysh) → Drew Thorstensen (thorst)
Revision history for this message
Hrushikesh (hrushikesh-gangur) wrote :

I can surely update my findings with H3 bits in a while.

tags: added: havana-rc-potential
Revision history for this message
Matt Riedemann (mriedem) wrote :

Drew's patch is here:

https://review.openstack.org/#/c/49836/

There is another patch from dims here:

https://review.openstack.org/#/c/49836/

summary: - Quantum is requesting too many tokens
+ Neutron is requesting too many tokens
affects: nova-project → nova
Changed in nova:
status: Confirmed → In Progress
Thierry Carrez (ttx)
tags: added: havana-backport-potential
removed: havana-rc-potential
Revision history for this message
Brad Pokorny (bpokorny) wrote :

The patch from dims is this one, but it's been abandoned.
https://review.openstack.org/#/c/50146/

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.