Rally

Cancelling task with ctrl-c leaves it in running state

Bug #1326051 reported by Hugh Saunders
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Rally
In Progress
High
Boris Pavlovic

Bug Description

When running a cinder create-delete scenario, ctrl-c does not cause the task to abort cleanly, so the task is still marked as running in the DB.

Expected behaviour: cancelling a task marks it as failed, no traceback.

Log:

(.venv)root@icehouse-dev-infra1:~/demos/sprint2/rally# rally -v task start create-and-delete-volume.yaml
================================================================================
Task 3934dad6-8509-43ad-a23d-3f2de51d2972 is started
--------------------------------------------------------------------------------
2014-06-03 16:31:01.609 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Check cloud.
2014-06-03 16:31:01.891 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Check cloud.
2014-06-03 16:31:01.892 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Task validation.
2014-06-03 16:31:01.902 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Task validation of scenarios names.
2014-06-03 16:31:01.905 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Task validation of scenarios names.
2014-06-03 16:31:01.905 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Task validation of syntax.
2014-06-03 16:31:01.908 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Task validation of syntax.
2014-06-03 16:31:01.908 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Task validation of semantic.
2014-06-03 16:31:01.908 24144 INFO rally.benchmark.context.users [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Enter context: `users`
2014-06-03 16:31:02.381 24144 INFO rally.benchmark.context.users [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Enter context: `users`
2014-06-03 16:31:02.382 24144 INFO rally.benchmark.context.users [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Exit context: `users`
2014-06-03 16:31:03.091 24144 INFO rally.benchmark.context.users [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Exit context: `users`
2014-06-03 16:31:03.091 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Task validation of semantic.
2014-06-03 16:31:03.092 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Task validation.
2014-06-03 16:31:03.092 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Benchmarking.
2014-06-03 16:31:03.100 24144 INFO rally.benchmark.context.users [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Enter context: `users`
2014-06-03 16:31:03.570 24144 INFO rally.benchmark.context.users [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Enter context: `users`
2014-06-03 16:31:03.570 24144 INFO rally.benchmark.context.cleanup.cleanup [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Enter context: `cleanup`
2014-06-03 16:31:03.571 24144 INFO rally.benchmark.context.cleanup.cleanup [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Enter context: `cleanup`
2014-06-03 16:31:03.576 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 0 START
2014-06-03 16:31:08.271 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 0 END: OK
2014-06-03 16:31:08.272 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 1 START
2014-06-03 16:31:12.916 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 1 END: OK
2014-06-03 16:31:12.917 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 2 START
2014-06-03 16:31:17.541 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 2 END: OK
2014-06-03 16:31:17.541 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 3 START
2014-06-03 16:31:22.208 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 3 END: OK
2014-06-03 16:31:22.209 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 4 START
2014-06-03 16:31:26.850 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 4 END: OK
2014-06-03 16:31:26.851 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 5 START
2014-06-03 16:31:31.540 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 5 END: OK

<snip a bit>

2014-06-03 16:40:43.679 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 84 START
2014-06-03 16:40:48.323 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 84 END: OK
2014-06-03 16:40:48.324 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 85 START
2014-06-03 16:40:52.978 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 85 END: OK
2014-06-03 16:40:52.979 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 86 START
^C2014-06-03 16:40:56.964 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 86 END: OK
2014-06-03 16:40:56.964 24144 INFO rally.benchmark.context.cleanup.cleanup [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Exit context: `cleanup`
2014-06-03 16:40:56.964 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 87 START
2014-06-03 16:40:56.964 24144 INFO rally.benchmark.context.cleanup.cleanup [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Cleanup users resources.
2014-06-03 16:40:59.086 24144 INFO rally.benchmark.context.cleanup.cleanup [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Cleanup users resources.
2014-06-03 16:40:59.087 24144 INFO rally.benchmark.context.cleanup.cleanup [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Cleanup admin resources.
2014-06-03 16:40:59.318 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 87 END: Error <class 'rally.exceptions.GetResourceNotFound'>: Resource <Volume: 88f80bca-4879-4964-80bd-88b9998d5934> is not found.
2014-06-03 16:40:59.319 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 88 START
2014-06-03 16:40:59.469 24144 INFO rally.benchmark.context.cleanup.cleanup [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Cleanup admin resources.
2014-06-03 16:40:59.469 24144 INFO rally.benchmark.context.cleanup.cleanup [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Exit context: `cleanup`
2014-06-03 16:40:59.470 24144 INFO rally.benchmark.context.users [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Exit context: `users`
2014-06-03 16:41:00.400 24144 INFO rally.benchmark.context.users [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Exit context: `users`
2014-06-03 16:41:00.401 24144 CRITICAL rally [-] KeyboardInterrupt
2014-06-03 16:41:00.401 24144 TRACE rally Traceback (most recent call last):
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/bin/rally", line 10, in <module>
2014-06-03 16:41:00.401 24144 TRACE rally sys.exit(main())
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/cmd/main.py", line 44, in main
2014-06-03 16:41:00.401 24144 TRACE rally return cliutils.run(sys.argv, categories)
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/cmd/cliutils.py", line 193, in run
2014-06-03 16:41:00.401 24144 TRACE rally ret = fn(*fn_args, **fn_kwargs)
2014-06-03 16:41:00.401 24144 TRACE rally File "<string>", line 2, in start
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/cmd/envutils.py", line 63, in default_from_global
2014-06-03 16:41:00.401 24144 TRACE rally return f(*args, **kwargs)
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/cmd/commands/task.py", line 67, in start
2014-06-03 16:41:00.401 24144 TRACE rally api.start_task(deploy_id, config_dict, task=task)
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/orchestrator/api.py", line 104, in start_task
2014-06-03 16:41:00.401 24144 TRACE rally benchmark_engine.run()
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/utils.py", line 162, in wrapper
2014-06-03 16:41:00.401 24144 TRACE rally result = f(self, *args, **kwargs)
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/benchmark/engine.py", line 181, in run
2014-06-03 16:41:00.401 24144 TRACE rally kw.get("args", {}))
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/benchmark/runners/base.py", line 213, in run
2014-06-03 16:41:00.401 24144 TRACE rally args)
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/benchmark/context/base.py", line 87, in run
2014-06-03 16:41:00.401 24144 TRACE rally return ContextManager._magic(ctxlst, func, *args, **kwargs)
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/benchmark/context/base.py", line 116, in _magic
2014-06-03 16:41:00.401 24144 TRACE rally tmp = ContextManager._magic(ctxlst[1:], func, *args, **kwargs)
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/benchmark/context/base.py", line 116, in _magic
2014-06-03 16:41:00.401 24144 TRACE rally tmp = ContextManager._magic(ctxlst[1:], func, *args, **kwargs)
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/benchmark/context/base.py", line 109, in _magic
2014-06-03 16:41:00.401 24144 TRACE rally return func(*args, **kwargs)
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/benchmark/runners/constant.py", line 91, in _run_scenario
2014-06-03 16:41:00.401 24144 TRACE rally result = iter_result.next(timeout)
2014-06-03 16:41:00.401 24144 TRACE rally File "/usr/lib/python2.7/multiprocessing/pool.py", line 613, in next
2014-06-03 16:41:00.401 24144 TRACE rally self._cond.wait(timeout)
2014-06-03 16:41:00.401 24144 TRACE rally File "/usr/lib/python2.7/threading.py", line 262, in wait
2014-06-03 16:41:00.401 24144 TRACE rally _sleep(delay)
2014-06-03 16:41:00.401 24144 TRACE rally KeyboardInterrupt
2014-06-03 16:41:00.401 24144 TRACE rally

(.venv)root@icehouse-dev-infra1:~/demos/sprint2/rally# rally task list
+--------------------------------------+----------------------------+---------+--------+-----+
| uuid | created_at | status | failed | tag |
+--------------------------------------+----------------------------+---------+--------+-----+
| 3934dad6-8509-43ad-a23d-3f2de51d2972 | 2014-06-03 16:31:01.600537 | running | False | |
| ddcd6005-a5c0-4128-b4a6-f70c7d2abb70 | 2014-06-03 16:27:33.348176 | running | False | |
+--------------------------------------+----------------------------+---------+--------+-----+

See original description
Tags: task
Hugh Saunders (hughsaunders)
description: updated
Swapnil Kulkarni (coolsvap-deactivatedaccount)
Changed in rally:
status: New → In Progress
assignee: nobody → Swapnil Kulkarni (coolsvap)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to rally (master)

Reviewed: https://review.openstack.org/107358
Committed: https://git.openstack.org/cgit/stackforge/rally/commit/?id=0e1c90363b8c84efc46ebd1e80a57cef52d85f63
Submitter: Jenkins
Branch: master

commit 0e1c90363b8c84efc46ebd1e80a57cef52d85f63
Author: Swapnil Kulkarni <email address hidden>
Date: Wed Jul 16 12:51:03 2014 +0000

    Handle keyboard interrupt in task run

    Currently cancelling task does not handle keyboard
    interrupts. This fix is to add the exception and
    abort the task.

    Change-Id: Ia3540c5b396212c0f214d41e51797c584329a482
    Closes-Bug:#1326051

Changed in rally:
status: In Progress → Fix Committed
Swapnil Kulkarni (coolsvap-deactivatedaccount)
Changed in rally:
status: Fix Committed → Fix Released
Boris Pavlovic (boris-42)
Changed in rally:
status: Fix Released → In Progress
Boris Pavlovic (boris-42)
Changed in rally:
importance: Undecided → High
Boris Pavlovic (boris-42)
tags: added: task
Kun Huang (academicgareth)
Changed in rally:
assignee: Swapnil Kulkarni (coolsvap) → Kun Huang (academicgareth)
Kun Huang (academicgareth)
Changed in rally:
assignee: Kun Huang (academicgareth) → nobody
Boris Pavlovic (boris-42)
Changed in rally:
assignee: nobody → Boris Pavlovic (boris-42)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to rally (master)

Fix proposed to branch: master
Review: https://review.openstack.org/234031

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.