Cancelling task with ctrl-c leaves it in running state

Bug #1326051 reported by Hugh Saunders on 2014-06-03
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Rally
High
Boris Pavlovic

Bug Description

When running a cinder create-delete scenario, ctrl-c does not cause the task to abort cleanly, so the task is still marked as running in the DB.

Expected behaviour: cancelling a task marks it as failed, no traceback.

Log:

(.venv)root@icehouse-dev-infra1:~/demos/sprint2/rally# rally -v task start create-and-delete-volume.yaml
================================================================================
Task 3934dad6-8509-43ad-a23d-3f2de51d2972 is started
--------------------------------------------------------------------------------
2014-06-03 16:31:01.609 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Check cloud.
2014-06-03 16:31:01.891 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Check cloud.
2014-06-03 16:31:01.892 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Task validation.
2014-06-03 16:31:01.902 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Task validation of scenarios names.
2014-06-03 16:31:01.905 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Task validation of scenarios names.
2014-06-03 16:31:01.905 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Task validation of syntax.
2014-06-03 16:31:01.908 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Task validation of syntax.
2014-06-03 16:31:01.908 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Task validation of semantic.
2014-06-03 16:31:01.908 24144 INFO rally.benchmark.context.users [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Enter context: `users`
2014-06-03 16:31:02.381 24144 INFO rally.benchmark.context.users [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Enter context: `users`
2014-06-03 16:31:02.382 24144 INFO rally.benchmark.context.users [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Exit context: `users`
2014-06-03 16:31:03.091 24144 INFO rally.benchmark.context.users [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Exit context: `users`
2014-06-03 16:31:03.091 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Task validation of semantic.
2014-06-03 16:31:03.092 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Task validation.
2014-06-03 16:31:03.092 24144 INFO rally.benchmark.engine [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Benchmarking.
2014-06-03 16:31:03.100 24144 INFO rally.benchmark.context.users [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Enter context: `users`
2014-06-03 16:31:03.570 24144 INFO rally.benchmark.context.users [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Enter context: `users`
2014-06-03 16:31:03.570 24144 INFO rally.benchmark.context.cleanup.cleanup [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Enter context: `cleanup`
2014-06-03 16:31:03.571 24144 INFO rally.benchmark.context.cleanup.cleanup [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Enter context: `cleanup`
2014-06-03 16:31:03.576 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 0 START
2014-06-03 16:31:08.271 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 0 END: OK
2014-06-03 16:31:08.272 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 1 START
2014-06-03 16:31:12.916 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 1 END: OK
2014-06-03 16:31:12.917 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 2 START
2014-06-03 16:31:17.541 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 2 END: OK
2014-06-03 16:31:17.541 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 3 START
2014-06-03 16:31:22.208 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 3 END: OK
2014-06-03 16:31:22.209 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 4 START
2014-06-03 16:31:26.850 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 4 END: OK
2014-06-03 16:31:26.851 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 5 START
2014-06-03 16:31:31.540 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 5 END: OK

<snip a bit>

2014-06-03 16:40:43.679 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 84 START
2014-06-03 16:40:48.323 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 84 END: OK
2014-06-03 16:40:48.324 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 85 START
2014-06-03 16:40:52.978 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 85 END: OK
2014-06-03 16:40:52.979 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 86 START
^C2014-06-03 16:40:56.964 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 86 END: OK
2014-06-03 16:40:56.964 24144 INFO rally.benchmark.context.cleanup.cleanup [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Exit context: `cleanup`
2014-06-03 16:40:56.964 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 87 START
2014-06-03 16:40:56.964 24144 INFO rally.benchmark.context.cleanup.cleanup [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Cleanup users resources.
2014-06-03 16:40:59.086 24144 INFO rally.benchmark.context.cleanup.cleanup [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Cleanup users resources.
2014-06-03 16:40:59.087 24144 INFO rally.benchmark.context.cleanup.cleanup [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Cleanup admin resources.
2014-06-03 16:40:59.318 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 87 END: Error <class 'rally.exceptions.GetResourceNotFound'>: Resource <Volume: 88f80bca-4879-4964-80bd-88b9998d5934> is not found.
2014-06-03 16:40:59.319 24289 INFO rally.benchmark.runners.base [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | ITER: 88 START
2014-06-03 16:40:59.469 24144 INFO rally.benchmark.context.cleanup.cleanup [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Cleanup admin resources.
2014-06-03 16:40:59.469 24144 INFO rally.benchmark.context.cleanup.cleanup [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Exit context: `cleanup`
2014-06-03 16:40:59.470 24144 INFO rally.benchmark.context.users [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Starting: Exit context: `users`
2014-06-03 16:41:00.400 24144 INFO rally.benchmark.context.users [-] Task 3934dad6-8509-43ad-a23d-3f2de51d2972 | Completed: Exit context: `users`
2014-06-03 16:41:00.401 24144 CRITICAL rally [-] KeyboardInterrupt
2014-06-03 16:41:00.401 24144 TRACE rally Traceback (most recent call last):
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/bin/rally", line 10, in <module>
2014-06-03 16:41:00.401 24144 TRACE rally sys.exit(main())
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/cmd/main.py", line 44, in main
2014-06-03 16:41:00.401 24144 TRACE rally return cliutils.run(sys.argv, categories)
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/cmd/cliutils.py", line 193, in run
2014-06-03 16:41:00.401 24144 TRACE rally ret = fn(*fn_args, **fn_kwargs)
2014-06-03 16:41:00.401 24144 TRACE rally File "<string>", line 2, in start
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/cmd/envutils.py", line 63, in default_from_global
2014-06-03 16:41:00.401 24144 TRACE rally return f(*args, **kwargs)
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/cmd/commands/task.py", line 67, in start
2014-06-03 16:41:00.401 24144 TRACE rally api.start_task(deploy_id, config_dict, task=task)
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/orchestrator/api.py", line 104, in start_task
2014-06-03 16:41:00.401 24144 TRACE rally benchmark_engine.run()
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/utils.py", line 162, in wrapper
2014-06-03 16:41:00.401 24144 TRACE rally result = f(self, *args, **kwargs)
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/benchmark/engine.py", line 181, in run
2014-06-03 16:41:00.401 24144 TRACE rally kw.get("args", {}))
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/benchmark/runners/base.py", line 213, in run
2014-06-03 16:41:00.401 24144 TRACE rally args)
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/benchmark/context/base.py", line 87, in run
2014-06-03 16:41:00.401 24144 TRACE rally return ContextManager._magic(ctxlst, func, *args, **kwargs)
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/benchmark/context/base.py", line 116, in _magic
2014-06-03 16:41:00.401 24144 TRACE rally tmp = ContextManager._magic(ctxlst[1:], func, *args, **kwargs)
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/benchmark/context/base.py", line 116, in _magic
2014-06-03 16:41:00.401 24144 TRACE rally tmp = ContextManager._magic(ctxlst[1:], func, *args, **kwargs)
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/benchmark/context/base.py", line 109, in _magic
2014-06-03 16:41:00.401 24144 TRACE rally return func(*args, **kwargs)
2014-06-03 16:41:00.401 24144 TRACE rally File "/root/demos/sprint2/rally/.venv/local/lib/python2.7/site-packages/rally/benchmark/runners/constant.py", line 91, in _run_scenario
2014-06-03 16:41:00.401 24144 TRACE rally result = iter_result.next(timeout)
2014-06-03 16:41:00.401 24144 TRACE rally File "/usr/lib/python2.7/multiprocessing/pool.py", line 613, in next
2014-06-03 16:41:00.401 24144 TRACE rally self._cond.wait(timeout)
2014-06-03 16:41:00.401 24144 TRACE rally File "/usr/lib/python2.7/threading.py", line 262, in wait
2014-06-03 16:41:00.401 24144 TRACE rally _sleep(delay)
2014-06-03 16:41:00.401 24144 TRACE rally KeyboardInterrupt
2014-06-03 16:41:00.401 24144 TRACE rally

(.venv)root@icehouse-dev-infra1:~/demos/sprint2/rally# rally task list
+--------------------------------------+----------------------------+---------+--------+-----+
| uuid | created_at | status | failed | tag |
+--------------------------------------+----------------------------+---------+--------+-----+
| 3934dad6-8509-43ad-a23d-3f2de51d2972 | 2014-06-03 16:31:01.600537 | running | False | |
| ddcd6005-a5c0-4128-b4a6-f70c7d2abb70 | 2014-06-03 16:27:33.348176 | running | False | |
+--------------------------------------+----------------------------+---------+--------+-----+

description: updated
Changed in rally:
status: New → In Progress
assignee: nobody → Swapnil Kulkarni (coolsvap)

Reviewed: https://review.openstack.org/107358
Committed: https://git.openstack.org/cgit/stackforge/rally/commit/?id=0e1c90363b8c84efc46ebd1e80a57cef52d85f63
Submitter: Jenkins
Branch: master

commit 0e1c90363b8c84efc46ebd1e80a57cef52d85f63
Author: Swapnil Kulkarni <email address hidden>
Date: Wed Jul 16 12:51:03 2014 +0000

    Handle keyboard interrupt in task run

    Currently cancelling task does not handle keyboard
    interrupts. This fix is to add the exception and
    abort the task.

    Change-Id: Ia3540c5b396212c0f214d41e51797c584329a482
    Closes-Bug:#1326051

Changed in rally:
status: In Progress → Fix Committed
Changed in rally:
status: Fix Committed → Fix Released
Changed in rally:
status: Fix Released → In Progress
Changed in rally:
importance: Undecided → High
tags: added: task
Changed in rally:
assignee: Swapnil Kulkarni (coolsvap) → Kun Huang (academicgareth)
Changed in rally:
assignee: Kun Huang (academicgareth) → nobody
Changed in rally:
assignee: nobody → Boris Pavlovic (boris-42)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers