Contrail 3.2.5: Keystone Mitaka: Disabling a project taking a long time (>45 sec) if admin user is associated with the project

Bug #1720448 reported by Deepak Jeyaraman on 2017-09-29
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Won't Fix
High
Ignatious Johnson Christopher
R3.2
Won't Fix
High
Ignatious Johnson Christopher
R4.0
Won't Fix
High
Ignatious Johnson Christopher
R4.1
Won't Fix
High
Ignatious Johnson Christopher
Trunk
Won't Fix
High
Ignatious Johnson Christopher

Bug Description

With contrail 3.2.5, we are noticing the project disable is taking a long time > 45 sec if admin user is associated with the project.

In 3.1 contrail, it is taking forever to delete/disable the project.

This is breaking CSO workflow and this is a must fix for COX (CSO+Contrail )

===

root@ccra-03:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 3.2.5.0-51 51
contrail-config 3.2.5.0-51 51
contrail-config-openstack 3.2.5.0-51 51
contrail-control 3.2.5.0-51 51
contrail-database-common 3.2.5.0-51 51
contrail-dns 3.2.5.0-51 51
contrail-docs 3.2.5.0-51 51
contrail-f5 3.2.5.0-51 51
contrail-fabric-utils 3.2.5.0-51 51
contrail-heat 3.2.5.0-51 51
contrail-install-packages 3.2.5.0-51~mitaka 51

Steps to reproduce:

1. create a project in OS dashboard
2. Add the "admin" user to the project
3. Disable the project -- the operations takes > 45 sec to complete.
4. Or instead of 3, you can remove the admin user from the project, it will take > 45 secs to complete.

5. Now if a project is created and deleted (without the adding of admin user,) it takes 1 sec.

In 3.1, the behavior is worse. It takes a very long time(some time hanging) when the same operation is tried.

information type: Proprietary → Public
description: updated
vivekananda shenoy (vshenoy83) wrote :

Hi Deepak,

Which is the openstack flavor ?

Changed in juniperopenstack:
assignee: nobody → Sachin Bansal (sbansal)
milestone: none → r3.2.6.0
importance: Undecided → High
importance: High → Critical
Sachin Bansal (sbansal) on 2017-10-02
Changed in juniperopenstack:
assignee: Sachin Bansal (sbansal) → Ignatious Johnson Christopher (ijohnson-x)

From: Ignatious Johnson
Sent: Monday, October 2, 2017 11:42 PM
To: Viswanath KJ; Deepak Jayaraman; Rathinasabapathy Sakthivel
Cc: Sachin Bansal
Subject: Re: CXU-9070 - External keystone

Keystone as part of project delete invalidates the used tokens.

Invalidating the user tokens in memcache is taking long time ( will discuss with Sachin on this and get back)
I changed the token backend to sql in keystone.conf and restarted keystone.

[token]
#driver = keystone.token.persistence.backends.memcache.Token
driver = keystone.token.persistence.backends.sql.Token

Now the delete time is small.

Thanks,
Ignatious

From: Deepak Jayaraman <email address hidden>
Date: Monday, October 2, 2017 at 11:50 PM
To: Ignatious Johnson <email address hidden>, Viswanath KJ <email address hidden>, Rathinasabapathy Sakthivel <email address hidden>
Cc: Sachin Bansal <email address hidden>, PrasannaKumar Anandan Gajendran <email address hidden>
Subject: Re: CXU-9070 - External keystone

Thanks Ignatious , yes, the delete time is small now.

Can you please update the other thread too. Also, is this move back to sql instead of memcache having other scale issues ? Have we qualified scale for contrail with keystone set to sql ?

Thanks
Deepak

From: Ignatious Johnson
Sent: Tuesday, October 03, 2017 11:19 PM
To: Deepak Jayaraman <email address hidden>; Viswanath KJ <email address hidden>; Rathinasabapathy Sakthivel <email address hidden>
Cc: Sachin Bansal <email address hidden>; PrasannaKumar Anandan Gajendran <email address hidden>
Subject: Re: CXU-9070 - External keystone

Yes sql backend may degrade keystone performance during scale.

Can I get the setup again for couple of hours for debugging the issue with memcache.
I tried reproducing it locally in build "3.2.5.0-51” with memcache, I didn’t see high time during project delete.

Thanks,
Ignatious

I checked with test team, scale testing is done with sql backend in a HA cluster(multiple openstack).

Where we use sql backend with keystone token_flush cron job to periodically flush the revoked tokens, Revoked tokens are preserved in sql for ever, which will degrade performance. So we do token flush periodically.

Jeba Paulaiyan (jebap) on 2017-11-13
tags: added: config
Fawad (fshaikh) on 2018-02-23
tags: added: cumulus
Changed in juniperopenstack:
status: New → Won't Fix
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers