Option for generating API token per task, not per request

[Jakub Skrzypnik]

Hello,

I'm doing a successful benchmarks of our OpenStack infrastructure, but there's a problem with draining out Keystone during testing other components.

We have configured Keystone to make the key last actually long enough to take the whole test on single token, so it would be nice to have an option to use single token during whole task (or, regenerate it on first expiry warning) when we want to test a raw performance of different component (Swift in this example). Right now it loses a few percent of completed actions only because the Keystone server refused connection, or threw 503s.

When I set the concurrency level to 100, it even got the whole database down for a moment, so the problem is real. I would like to know if you're able to fix that, or point me to official workaround for this case.

With regards,
Kuba

[Boris Pavlovic (boris-42)]

This looks more like a feature than bug.

I agree that this can be quite convenient, however I would suggest few things about implementation here:

1) Use token per subtask (not task)
2) Implement Context that generates and caches tokens in context object for every user
3) Modify base OpenStackScenario to check whatever token exists
4) Modify OSClients to accept tokens

So that is possible to implement.
Would you like to try?

[Jakub Skrzypnik (j.skrzypnik)]

> So that is possible to implement.
> Would you like to try?

Sadly, no.
Just got to read the Rally source for a bit, and it seems that I'm not enough experienced in Python to implement this, without destroying every other functionality important for someone else.

But it seems that someone from your team assigned himself to this task, so I'm shamelessly counting on this :)

[Boris Pavlovic (boris-42)]

Yep Amy is working on it.
It would be nice if you could help her at least with testing that everything works.

[Jakub Skrzypnik (j.skrzypnik)]

> Yep Amy is working on it.

Nice to hear that :)

> It would be nice if you could help her at least with testing that everything works.

Sure thing! You can reach me on your IRC channel if more real-time talk is needed during testing.

[Amy Ge (yaoge)]

Hi Jakub, I submitted my code yesterday to the git review. I tried it with bunch of tasks and so far things work okay, but it definitely needs more effort on testing. I will work on it. And it will be great if you can help me with some testing stuffs!
https://review.openstack.org/#/c/479097/3

[Jakub Skrzypnik (j.skrzypnik)]

Hello,

I've pulled it into our company's mirror of Rally repository as a separate branch to set it, and built a Docker container about it. Right now it seems like the Rally functionality isn't broken, but I can't set the "cache_tokens" variable due to JSON schema validation error.

I'll post the details on Monday and continue testing it.

[Boris Pavlovic (boris-42)]

Jakub,

Please share the task that you used first.

[Jakub Skrzypnik (j.skrzypnik)]

I'm away from work machine now, it'll be done on Monday.

[Jakub Skrzypnik (j.skrzypnik)]

Hello,

I've got the patch wrong, the "cache_tokens" variable was to be used in context definition, not deployment configuration (where I tried to put it).

But, right now I'm kinda unsure, how can I apply this variable to context without setting defaults to create temporary users and tenants? I want to use the previously created user defined in users[] array in config.json (not an admin), but the Rally still tries to make new users for it, saying that I should configure an admin account to do so.

[Boris Pavlovic (boris-42)]

Jakub,

Ya I thought about that yesterday.

At the moment you can't use it with existing_users.
We are going to do one more extra change for that.

[Jakub Skrzypnik (j.skrzypnik)]

No problem, I'll wait.

[Boris Pavlovic (boris-42)]

Jakub,

Okay many things happened in master in recent time:

- We removed hidden context called "existing_users" and merged it with "users" context. This allows us to have the same argument for existing_users case and users managed by rally.

- Amy patch was completed and it's here https://review.openstack.org/#/c/479097/ so basically now in your scenarios you should specify {"context": {"users": {"cache_tokens": True}}} and it should work for you

Could you please fetch Amy's patch and test it in your deployment. Thanks!

[Jakub Skrzypnik (j.skrzypnik)]

Hello, sorry for late reply, I've been on vacation.

But, looks like last cherry-pick from Amy's branch broke that token caching you're talking about.
Here's my task JSON:

{
"SwiftObjects.create_object_then_download_object_and_delete_object" :[
{
  "runner": {
    "type": "constant",
    "concurrency": 5,
    "times": 10
  },
  "args": {
    "container_name": "interia-test",
    "object_size": 8192,
    "objects_per_container": 1000
  },
  "context": {
    "users": {
      "cache_tokens": True
    }
  }
}
]
}

Don't bother about that task name - it's a custom task created only to test whole CRUD chain at once, not for public use ATM.

But when I'm trying to run task with "cache_tokens": True, the schema validation fails:

Subtask SwiftObjects.create_object_then_download_object_and_delete_object[0] has wrong configuration
Subtask configuration:
{"runner": {"type": "constant", "concurrency": 5, "times": 10}, "args": {"container_name": "interia-test", "object_size": 8192, "objects_per_container": 1000}, "context": {"users": {"cache_tokens": true}}}

Reason(s):
 {"cache_tokens": true} is valid under each of {'additionalProperties': False, 'description': 'Use existing users and tenants.', 'properties': {'user_choice_method': {'$ref': '#/definitions/user_choice_method'}, 'cache_tokens': {'$ref': '#/definitions/cache_tokens'}}}, {'additionalProperties': False, 'description': 'Create new temporary users and tenants.', 'properties': {'user_domain': {'type': 'string', 'description': 'ID of domain in which users will be created.'}, 'project_domain': {'type': 'string', 'description': 'ID of domain in which projects will be created.'}, 'cache_tokens': {'$ref': '#/definitions/cache_tokens'}, 'user_choice_method': {'$ref': '#/definitions/user_choice_method'}, 'users_per_tenant': {'minimum': 1, 'type': 'integer', 'description': 'The number of users to create per one tenant.'}, 'tenants': {'minimum': 1, 'type': 'integer', 'description': 'The number of tenants to create.'}, 'resource_management_workers': {'minimum': 1, 'type': 'integer', 'description': 'The number of concurrent threads to use for serving users context.'}}}

Failed validating 'oneOf' in schema:
    {'$schema': 'http://json-schema.org/draft-04/schema',
     'definitions': {'cache_tokens': {'description': 'Option to cach...

[Boris Pavlovic (boris-42)]

I updated patch now it should work

[Jakub Skrzypnik (j.skrzypnik)]

Validation works, but it still complains about missing admin users, which I'm not able to provide:

```
rally@bc0f7ac2af66:~$ rally task start --file swift.json
--------------------------------------------------------------------------------
Preparing input task
--------------------------------------------------------------------------------

Task is:
{
"SwiftObjects.create_object_then_download_object_and_delete_object" :[
{
  "runner": {
    "type": "constant",
    "concurrency": 5,
    "times": 10
  },
  "args": {
    "container_name": "interia-test",
    "object_size": 8192,
    "objects_per_container": 1000
  },
  "context": {
    "users": {
      "cache_tokens": True
    }
  }
}
]
} ...

[Jakub Skrzypnik (j.skrzypnik)]

What's going on with this case?

We're getting back to this task, my last message is about 4 months old and it hasn't been answered.

[Boris Pavlovic (boris-42)]

Can we have hangout call and try to debug this?

[Boris Pavlovic (boris-42)]

- Rally works without admin, I am using it internally at my work

In mean while I am going to rebase this patch https://review.openstack.org/#/c/479097/