AdHoc actions with invalid base definitions fail in the engine but leave the task/workflow running

[Bob Haddleton]

When an AdHoc action has an invalid base defined, execution of the action within a workflow will fail but leave the task and workflow in the RUNNING state. The only indication of the failure is the exception in the mistral log file.

The problem is that the DBEntityNotFoundError exception is not caught when the engine tries to retrieve the action definition, so the process fails but leaves the task and workflow running.

The solution is to catch the DBEntityNotFoundError exception when the AdHocAction code is looking up the base action definition, and raise InvalidActionException to stop the task and workflow execution and place them in the ERROR state.

The same problem occurs for nested AdHoc Actions, when any of the nested base actions are invalid.

This is a sample trace from running a test with an invalid adhoc action, without the fix applied:

Traceback (most recent call last):
  File "/home/bobh/git/mistral/mistral/tests/unit/engine/test_adhoc_actions.py", line 247, in test_missing_adhoc_action_definition
    'my_wb.wf6', '')
  File "/home/bobh/git/mistral/mistral/engine/action_queue.py", line 74, in decorate
    res = func(*args, **kw)
  File "/home/bobh/git/mistral/.tox/py35/lib/python3.5/site-packages/osprofiler/profiler.py", line 153, in wrapper
    return f(*args, **kwargs)
  File "/home/bobh/git/mistral/mistral/engine/default_engine.py", line 50, in start_workflow
    params
  File "/home/bobh/git/mistral/.tox/py35/lib/python3.5/site-packages/osprofiler/profiler.py", line 153, in wrapper
    return f(*args, **kwargs)
  File "/home/bobh/git/mistral/mistral/engine/workflow_handler.py", line 48, in start_workflow
    params=params
  File "/home/bobh/git/mistral/.tox/py35/lib/python3.5/site-packages/osprofiler/profiler.py", line 153, in wrapper
    return f(*args, **kwargs)
  File "/home/bobh/git/mistral/mistral/engine/workflows.py", line 108, in start
    wf_ctrl.continue_workflow()
  File "/home/bobh/git/mistral/.tox/py35/lib/python3.5/site-packages/osprofiler/profiler.py", line 153, in wrapper
    return f(*args, **kwargs)
  File "/home/bobh/git/mistral/mistral/engine/dispatcher.py", line 94, in dispatch_workflow_commands
    task_handler.run_task(cmd)
  File "/home/bobh/git/mistral/.tox/py35/lib/python3.5/site-packages/osprofiler/profiler.py", line 153, in wrapper
    return f(*args, **kwargs)
  File "/home/bobh/git/mistral/mistral/engine/task_handler.py", line 62, in run_task
    task.run()
  File "/home/bobh/git/mistral/.tox/py35/lib/python3.5/site-packages/osprofiler/profiler.py", line 153, in wrapper
    return f(*args, **kwargs)
  File "/home/bobh/git/mistral/mistral/engine/tasks.py", line 306, in run
    self._run_new()
  File "/home/bobh/git/mistral/.tox/py35/lib/python3.5/site-packages/osprofiler/profiler.py", line 153, in wrapper
    return f(*args, **kwargs)
  File "/home/bobh/git/mistral/mistral/engine/tasks.py", line 332, in _run_new
    self._schedule_actions()
  File "/home/bobh/git/mistral/mistral/engine/tasks.py", line 393, in _schedule_actions
    action = self._build_action()
  File "/home/bobh/git/mistral/mistral/engine/tasks.py", line 469, in _build_action
    wf_ctx=self.wf_ex.context)
  File "/home/bobh/git/mistral/mistral/engine/actions.py", line 365, in __init__
    self.action_spec.get_base()
  File "/home/bobh/git/mistral/mistral/db/v2/api.py", line 169, in get_action_definition
    return IMPL.get_action_definition(name)
  File "/home/bobh/git/mistral/mistral/db/sqlalchemy/base.py", line 117, in _within_session
    result = func(*args, **kw)
  File "/home/bobh/git/mistral/mistral/db/v2/sqlalchemy/api.py", line 659, in get_action_definition
    "Action definition not found [action_name=%s]" % identifier
mistral.exceptions.DBEntityNotFoundError: Action definition not found [action_name=wrong]

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/500132

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/500132
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=8b2f63cbdf84c908e31408575cf87ad565479b0b
Submitter: Jenkins
Branch: master

commit 8b2f63cbdf84c908e31408575cf87ad565479b0b
Author: Bob Haddleton <email address hidden>
Date: Fri Sep 1 11:18:16 2017 -0500

    Catch DBEntityNotFoundError exceptions for invalid AdHoc Actions

    This patch will catch the DBEntityNotFoundError exception that is
    raised when an AdHoc Action has an invalid base action defined. The
    exception is converted to an InvalidActionException which will
    halt the task/workflow and cause them to enter the ERROR state.

    Change-Id: I115b8243641da100474e8c8b34a724978078978b
    Closes-Bug: 1714548

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/mistral 6.0.0.0b1 development milestone.