Type validation failed when parsing tempest scenario task result

[Yanyan Hu]

When I tried to run the following sample tempest test, a type validation error happened:

http://git.openstack.org/cgit/openstack/rally/tree/samples/tasks/scenarios/tempest-do-not-run-against-production/single_test.yaml

Looks like the 'time' filed of task result for tempest test is string rather than floating and thus failed the type validation.

Part of the output:

======
Totals
======
Ran: 1 tests in 27.0000 sec.
 - Passed: 1
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 0.0913 sec.

==============
Worker Balance
==============
 - Worker 0 (1 tests) => 0:00:00.091320
2016-04-06 04:01:12.274 26534 INFO rally.task.runner [-] Task 5907bddb-3252-4934-aedd-cfc726c487c0 | ITER: 0 END: OK
(1L, <class 'rally.plugins.openstack.scenarios.tempest.tempest.TempestScenario'>, 'single_test', {'task': <rally.common.objects.task.Task object at 0x3c1d310>, 'admin': {'credential': <rally.common.objects.credential.Credential object at 0x3c150d0>}, 'verifier': <rally.verification.tempest.tempest.Tempest object at 0x3c15510>, 'tmp_results_dir': u'/tmp/5907bddb-3252-4934-aedd-cfc726c487c0-results', 'user': {'credential': <rally.common.objects.credential.Credential object at 0x3c15710>, 'id': u'4479c28d9b6a4274970f979247571c15', 'tenant_id': u'b358465bf671484f9480ecaf385bc08b'}, 'scenario_name': 'TempestScenario.single_test', 'config': {'tempest': {}, 'users': {'project_domain': 'default', 'users_per_tenant': 1, 'tenants': 1, 'resource_management_workers': 20, 'user_domain': 'default'}}, 'tenant': {'id': u'b358465bf671484f9480ecaf385bc08b', 'name': u'c_rally_5907bddb_UnIap1WN'}}, {'tempest_conf': '/etc/tempest/tempest.conf', 'test_name': 'tempest.api.image.v1.test_images.ListImagesTest.test_index_no_params'})
2016-04-06 04:01:12.279 26534 INFO rally.task.runner [-] Task 5907bddb-3252-4934-aedd-cfc726c487c0 | ITER: 1 START
2016-04-06 04:01:12.280 26534 INFO rally.verification.tempest.tempest [-] Tempest config file: /etc/tempest/tempest.conf
2016-04-06 04:01:12.290 26293 INFO rally.plugins.openstack.context.not_for_production.tempest [-] Task 5907bddb-3252-4934-aedd-cfc726c487c0 | Starting: Exit context: `tempest`
2016-04-06 04:01:12.291 26293 INFO rally.plugins.openstack.context.not_for_production.tempest [-] Built-in stress cleanup from Tempest looks like can help to shot yourself in the foot. Sorry, but even Rally can not clean up after Tempest. Deal with it.
2016-04-06 04:01:12.291 26293 INFO rally.plugins.openstack.context.not_for_production.tempest [-] Task 5907bddb-3252-4934-aedd-cfc726c487c0 | Completed: Exit context: `tempest`
2016-04-06 04:01:12.291 26293 INFO rally.plugins.openstack.context.keystone.users [-] Task 5907bddb-3252-4934-aedd-cfc726c487c0 | Starting: Exit context: `users`
2016-04-06 04:01:15.230 26293 INFO rally.plugins.openstack.context.keystone.users [-] Task 5907bddb-3252-4934-aedd-cfc726c487c0 | Completed: Exit context: `users`
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine [-] '12.942' is not of type 'number', 'null'

Failed validating 'type' in schema['properties']['atomic_actions']['patternProperties']['.*']:
    {'type': ['number', 'null']}

On instance['atomic_actions']['test_execution']:
    '12.942'
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine Traceback (most recent call last):
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine File "/home/stack/huyanyan/openstack_repo/openstack/rally/rally/task/engine.py", line 340, in run
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine workload.args)
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine File "/home/stack/huyanyan/openstack_repo/openstack/rally/rally/task/runner.py", line 196, in run
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine self._run_scenario(cls, method_name, context, args)
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine File "/home/stack/huyanyan/openstack_repo/openstack/rally/rally/plugins/common/runners/constant.py", line 210, in _run_scenario
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine self._join_processes(process_pool, result_queue)
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine File "/home/stack/huyanyan/openstack_repo/openstack/rally/rally/task/runner.py", line 243, in _join_processes
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine self._send_result(result_queue.get())
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine File "/home/stack/huyanyan/openstack_repo/openstack/rally/rally/task/runner.py", line 262, in _send_result
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine r = ScenarioRunnerResult(result)
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine File "/home/stack/huyanyan/openstack_repo/openstack/rally/rally/task/runner.py", line 131, in __init__
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine jsonschema.validate(result_list, self.RESULT_SCHEMA)
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine File "/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/jsonschema/validators.py", line 478, in validate
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine cls(schema, *args, **kwargs).validate(instance)
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine File "/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/jsonschema/validators.py", line 123, in validate
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine raise error
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine ValidationError: '12.942' is not of type 'number', 'null'
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine Failed validating 'type' in schema['properties']['atomic_actions']['patternProperties']['.*']:
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine {'type': ['number', 'null']}
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine On instance['atomic_actions']['test_execution']:
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine '12.942'
2016-04-06 04:01:16.368 26293 ERROR rally.task.engine
2016-04-06 04:01:16.627 26293 INFO rally.task.engine [-] Task 5907bddb-3252-4934-aedd-cfc726c487c0 | Completed: Benchmarking.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/302098

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/302098
Committed: https://git.openstack.org/cgit/openstack/rally/commit/?id=dc12401539aff7b3fb36f56a10fa48e82e77e647
Submitter: Jenkins
Branch: master

commit dc12401539aff7b3fb36f56a10fa48e82e77e647
Author: yanyanhu <email address hidden>
Date: Wed Apr 6 04:26:30 2016 -0400

    Convert task execution time to float value

    For tempest scenario test, task execution time extracted from tempest
    log is a string rather than a number value. This causes type validation
    error when task engine parses the task result. This patch fixes this
    issue by converting this value to float type.

    Change-Id: I95efb88c57b2d57609ad9a3934632c118b995301
    Closes-Bug: #1566712

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/rally 0.4.0 release.