constant_for_duration fails with 'NoneType' object is not callable

[Matt Simonin]

Version:
+ rally from docker images (0ea304981eee1313dd213e5313b7e9958ae4736b / Tue Oct 23 20:10:18 2018)

Scenario:

  KeystoneBasic.authenticate_user_and_validate_token:
    -
      args:
        sleep: 1
      runner:
        type: "constant_for_duration"
        duration: 100
        concurrency: 1

The execution fails with the following:

2018-10-29 11:02:42.724 4530 ERROR rally.task.engine [-] Unexpected exception during the workload execution: TypeError: 'NoneType' object is not callable
2018-10-29 11:02:42.724 4530 ERROR rally.task.engine Traceback (most recent call last):
2018-10-29 11:02:42.724 4530 ERROR rally.task.engine File "/usr/local/lib/python2.7/dist-packages/rally/task/engine.py", line 515, in _run_workload
2018-10-29 11:02:42.724 4530 ERROR rally.task.engine workload["args"])
2018-10-29 11:02:42.724 4530 ERROR rally.task.engine File "/usr/local/lib/python2.7/dist-packages/rally/task/runner.py", line 168, in run
2018-10-29 11:02:42.724 4530 ERROR rally.task.engine self._run_scenario(scenario_plugin, "run", context, args)
2018-10-29 11:02:42.724 4530 ERROR rally.task.engine File "/usr/local/lib/python2.7/dist-packages/rally/plugins/common/runners/constant.py", line 340, in _run_scenario
2018-10-29 11:02:42.724 4530 ERROR rally.task.engine result = iter_result.next(timeout)
2018-10-29 11:02:42.724 4530 ERROR rally.task.engine File "/usr/lib/python2.7/multiprocessing/pool.py", line 668, in next
2018-10-29 11:02:42.724 4530 ERROR rally.task.engine raise value
2018-10-29 11:02:42.724 4530 ERROR rally.task.engine TypeError: 'NoneType' object is not callable

Trying to narrow down the problem, it seems that multiprocessing.imap doesn't like that is passed

[Andriy Kurilin (andreykurilin)]

hi Matt!

First of all, the following task should not pass validation, since `KeystoneBasic.authenticate_user_and_validate_token` scenario plugin doesn't accept 'sleep' argument.

If it is an error, please give me more information: At which moment the exception had occurred? Does it happen before any iteration or when the execution of workload had been started?

[Matt Simonin (matthieu-simonin)]

Andrey,

Let me complete the stack trace:

Task is:
---
  KeystoneBasic.authenticate_user_and_validate_token:
    -
      runner:
        type: "constant_for_duration"
        duration: 100
        concurrency: 1

Task syntax is correct :)
Running Rally version 1.1.0
--------------------------------------------------------------------------------
Task 644fde0c-82a4-46e0-9d7d-c2082bd71f8f: started
--------------------------------------------------------------------------------
                                                                                                                                                                                           ...

[Andriy Kurilin (andreykurilin)]

Strange. Can you try more recent image? 1.3.0 tag for example or 'latest'

[Matt Simonin (matthieu-simonin)]

I got the same problem with:

- xrally/xrally-openstack:1.3.0 (commit id: 3855100cc2db45a872b7717f062a890fb7ca487e / 9 oct)
- xrally/xrally-openstack:latest (commit id: 0ea304981eee1313dd213e5313b7e9958ae4736b / 23 oct)

Could it be a problem with the imap function of multiprocessing ?

[Andriy Kurilin (andreykurilin)]

I do not know what it can be :( I tried to execute Dummy.dummy with constant_for_duration runner and everything works with recent images. If it works for me, it means that the issue not in code of rally itself or multiprocessing. The issue can be in input data. How you are running Rally? What and where volumes/directories/files are mounted? What is version of docker?

[Matt Simonin (matthieu-simonin)]

Hum :(
Let me give you some background/context information.
But first, note that the "constant" runner works perfectly fine in my environment.

- I'm running rally from this ansible role:
https://github.com/BeyondTheClouds/enos/blob/19ea494fb3e6c61fa16896cd453c76e0435fd68b/enos/ansible/roles/rally/tasks/bench.yml

which means that /root/rally_home is mounted in /home/rally/data

- docker version is
ii docker-engine 17.05.0~ce-0~debian-jessie amd64

- the input file is the following yaml file:

---
  KeystoneBasic.authenticate_user_and_validate_token:
    -
      runner:
        type: "constant_for_duration"
        duration: {{ duration }}
        concurrency: {{ concurrency }}

template variables are set in a separate file and specified in the command line using the
--task-args-file option.

Additionnaly here is what I naively tested in the same environment:

- running tox -e samples / tox -e py27 -> went fine (it seems there are there some unit tests with constant_for_duration)

- add a break point here : https://github.com/openstack/rally/blob/a19cc57391d2e19e95d4c13187e354f9df825d55/rally/plugins/common/runners/constant.py#L335
as shown the error is thrown from the multiprocessing library.

https://github.com/python/cpython/blob/2.7/Lib/multiprocessing/pool.py#L674

item is (False, NoneType isn't callable),actually it seems the same for every item in self._items.

Hope that helps !

[Matt Simonin (matthieu-simonin)]

Today I've tested outside docker. This unfortunately led to the same problem.

What I did:

```
$) python --version
Python 2.7.15

$) pip install rally-openstack
$) source admin-openrc
$) rally deployment create --fromenv --name discovery
$) cat workload/scenarios/cfd_authenticate_user_and_validate_token.yaml

---
  KeystoneBasic.authenticate_user_and_validate_token:
    -
      runner:
        type: "constant_for_duration"
        duration: 100
        concurrency: 1

$) rally task start workload/scenarios/cfd_authenticate_user_and_validate_token.yaml

--------------------------------------------------------------------------------
Preparing input task
--------------------------------------------------------------------------------

Task is:
---
  KeystoneBasic.authenticate_user_and_validate_token:
    -
      runner:
        type: "constant_for_duration"
        duration: 100
        concurrency: 1

Task syntax is correct :)
Running Rally version 1.2.1
--------------------------------------------------------------------------------
Task 64f80586-354b-495d-aafa-6e12295c8019: started
--------------------------------------------------------------------------------

Running Task... This can take a while...

To track task status use:

        rally task status
        or
        rally task detailed ...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/614576

[Matt Simonin (matthieu-simonin)]

Andrey,
I've submitted a patch that is related to this.
Any advice from any rally expert there would be really appreciated :)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/614576
Committed: https://git.openstack.org/cgit/openstack/rally/commit/?id=cfde22978a8e06bb6d6606dd2641ae647a13dabd
Submitter: Zuul
Branch: master

commit cfde22978a8e06bb6d6606dd2641ae647a13dabd
Author: msimonin <email address hidden>
Date: Wed Oct 31 16:18:48 2018 +0100

    Unify constant runner logic

    Change-Id: I7f9c34e0bc146ac3a8223aff5a42498341a36a88
    Related-Bug: #1800447