with-items does not process ad-hoc action default values

Bug #1597640 reported by Xavier Hardy
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Mistral
Fix Released
High
Dougal Matthews

Bug Description

When executing a task with 'with-items' and an ad-hoc action with default values. If one of the default values is omitted, the task fails to start, saying the key (of the missing default value) cannot be found.

This action and this simple workflow reproduces the error:
---
version: '2.0'

test.http:
    input:
        - url: http://www.example.com
        - method: GET
        - timeout: 10

    output: <% $.content %>

    base: std.http
    base-input:
        url: <% $.url %>
        method: <% $.method %>
        timeout: <% $.timeout %>

---
version: '2.0'

with_items_default_bug:
    description: Re-create the with-items bug with default values
    type: direct

    tasks:
        get_pages:
            with-items: page in <% range(0, 1) %>
            action: test.http
            input:
                url: http://www.example.com
                method: GET
            on-success:
                - well_done

        well_done:
            action: std.echo output="Well done"

Here is task state info after the error:
Failed to run task [wf=WorkflowExecution {'state_info': None, 'accepted': False, 'output': {}, 'name': u'with_items_default_bug', 'workflow_name': u'with_items_default_bug', 'state': 'RUNNING', 'created_at': '2016-06-30 07:21:02.341331', 'runtime_context': {'index': 0}, 'updated_at': '2016-06-30 07:21:02.344926', 'spec': {u'tasks': {u'get_pages': {u'with-items': u'page in <% range(0, 1) %>', u'name': u'get_pages', u'on-success': [u'well_done'], u'version': '2.0', u'action': u'test.http', u'input': {u'url': u'http://www.example.com', u'method': u'GET'}, u'type': u'direct'}, u'well_done': {u'action': u'std.echo output="Well done"', u'version': '2.0', u'type': u'direct', u'name': u'well_done'}}, u'type': u'direct', u'description': u'Re-create the with-items bug with default values', u'version': u'2.0', u'name': u'with_items_default_bug'}, 'task_execution_id': None, 'workflow_id': u'ac1b74ea-32bc-4899-998a-e90822afe4c2', 'params': {}, 'context': {'__execution': {'input': {}, 'params': {}, 'id': u'84ab8b9e-4cd2-4473-bd7b-4fa2ac165f8a', 'spec': {u'tasks': {u'get_pages': {u'with-items': u'page in <% range(0, 1) %>', u'name': u'get_pages', u'on-success': [u'well_done'], u'version': '2.0', u'action': u'test.http', u'input': {u'url': u'http://www.example.com', u'method': u'GET'}, u'type': u'direct'}, u'well_done': {u'action': u'std.echo output="Well done"', u'version': '2.0', u'type': u'direct', u'name': u'well_done'}}, u'type': u'direct', u'description': u'Re-create the with-items bug with default values', u'version': u'2.0', u'name': u'with_items_default_bug'}}}, 'input': {}, 'scope': 'private', 'project_id': '<default-project>', 'type': 'workflow_execution', 'id': u'84ab8b9e-4cd2-4473-bd7b-4fa2ac165f8a', 'description': u''}, task=get_pages]: Can not evaluate YAQL expression: $.timeout , data = {u'url': u'http://www.example.com', u'method': u'GET'}; error: u'timeout'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/mistral/engine/task_handler.py", line 42, in run_task
    task.run()
  File "/usr/local/lib/python2.7/dist-packages/mistral/engine/tasks.py", line 228, in run
    self._run_new()
  File "/usr/local/lib/python2.7/dist-packages/mistral/engine/tasks.py", line 259, in _run_new
    self._schedule_actions()
  File "/usr/local/lib/python2.7/dist-packages/mistral/engine/tasks.py", line 394, in _schedule_actions
    action.schedule(input_dict, target, index=idx)
  File "/usr/local/lib/python2.7/dist-packages/mistral/engine/actions.py", line 212, in schedule
    self._prepare_input(input_dict),
  File "/usr/local/lib/python2.7/dist-packages/mistral/engine/actions.py", line 320, in _prepare_input
    input_dict
  File "/usr/local/lib/python2.7/dist-packages/mistral/expressions.py", line 203, in evaluate_recursively
    data[key] = _evaluate_item(data[key], context)
  File "/usr/local/lib/python2.7/dist-packages/mistral/expressions.py", line 186, in _evaluate_item
    return evaluate(item, context)
  File "/usr/local/lib/python2.7/dist-packages/mistral/expressions.py", line 180, in evaluate
    return _EVALUATOR.evaluate(expression, context)
  File "/usr/local/lib/python2.7/dist-packages/mistral/expressions.py", line 146, in evaluate
    cls).evaluate(trim_expr, data_context)
  File "/usr/local/lib/python2.7/dist-packages/mistral/expressions.py", line 95, in evaluate
    " %s" % (expression, data_context, str(e))
YaqlEvaluationException: Can not evaluate YAQL expression: $.timeout , data = {u'url': u'http://www.example.com', u'method': u'GET'}; error: u'timeout'

Changed in mistral:
milestone: none → newton-2
importance: Undecided → High
Dougal Matthews (d0ugal)
Changed in mistral:
assignee: nobody → Dougal Matthews (d0ugal)
Dougal Matthews (d0ugal)
Changed in mistral:
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to mistral (master)

Fix proposed to branch: master
Review: https://review.openstack.org/337696

Changed in mistral:
status: Confirmed → In Progress
Revision history for this message
Xavier Hardy (xavier-hardy) wrote :

No, do not modify the workflow or the ad-hoc action, I have tested it locally, it reproduces the error. The page variable is just an excuse to use with-items. So yes, the action should be called twice (but it's unrelated to the problem). The problem is that the default values are not given to the system action if they are not defined in the ad-hoc action; and this is not the expected behavior.

Revision history for this message
Dougal Matthews (d0ugal) wrote :

I think you missunderstood my comment, or at least I don't understand your reply. I don't intend of modifying the workflow and I have confirmed the error with a unit tests which you can see linked above to the review.

I'll continue investigating.

Revision history for this message
Dougal Matthews (d0ugal) wrote :

I have a test case which verifies this issue [1] but I have not yet been able to find where the issue is. I've done a bunch of investigating and debugging in the code with no luck yet.

[1]: https://review.openstack.org/337696

Changed in mistral:
milestone: newton-2 → newton-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to mistral (master)

Reviewed: https://review.openstack.org/337696
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=277b08ad9ea0c683c49ff58b13898f7a8d5903c2
Submitter: Jenkins
Branch: master

commit 277b08ad9ea0c683c49ff58b13898f7a8d5903c2
Author: Dougal Matthews <email address hidden>
Date: Tue Jul 5 14:12:19 2016 +0100

    Fix the use of both adhoc actions and "with-items" in workflows

    The WithItemsTask previously skipped the validate_input call.
    Adding this resolves the issues with input and with-items.

    Closes-Bug: 1597640
    Change-Id: Id6a1dad09d6f21e24861bd4292df7152843bd9f3
    Co-authored-by: Dawid Deja <email address hidden>

Changed in mistral:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/mistral 3.0.0.0b3

This issue was fixed in the openstack/mistral 3.0.0.0b3 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.