Use published variables in 'publish'

[Dmitri Zimine]

task1:
  action: foo
  publish:
    foo: "foo"

task2:
   action: bar foo={$foo} # assume it has output.bar
   publish:
      foobar: $.foo + $.bar

As a user, want to
1) use workflow input or previously published variables when publishing
2) mix them with the output of task action

The problem is what is the root that $ points in YAQL.
If it points to task output, than referencing them via something like $.__vars.foo
   - but it sucks because it's inconsistent with the way we reference them in action input.
if it points to context, than user needs to refer $.__self.bar
if we merge the dictionaries

Proposal:
merge the dictionaries; local (task action) overrides the global (workflow input, or published), AND be able to explicitly reference by $.__self.bar, $__task.task2.bar and $.__global.foo

[Renat Akhmerov (rakhmerov)]

Hm... I guess I didn't fully understand the proposal. What's "__global" and what's "__self"?

How about just being able to access action output in "publish" via something like "$.__res" whereas "$" will point to workflow context as in other cases? This key "__res" could be just temporarily available only within "publish" scope.

Need to think about options for variable name itself though. My options: "__res", "__action", "__raw_result", "__raw".

[Dmitri Zimine (i-dz)]

Sample:

ask0:
    action: ....
    publish:
        x: $.....

task1:
    action: foo param={$.global_var} # returns {x:y}
    publish:
        p1: {$.global_var} + {$.x} # local variable, input of action, overrides the global x
        p11: {$.global_var} +{$.__global.x} # explicitly refer to x from global
        p12: {$.global_var} +{$.__action.x} # explicitly refer to x from action
        p2: {$.global_var} + {$.__task.task1.x}

task2:
    action: bar param = {$.__task.task1.x}
    action: bar param = {$.x}

alternative names
1) pointing to the incoming global context:
__wf
__wf_ctx
__ctx
__global

2) pointing to action output of this very task
* __action
* __self
* __result
* __res
* __raw
* __raw_result

[Dmitri Zimine (i-dz)]

Manas came up with an alternative.

1) root ALWAYS points to the "workflow context" which contains workflow input, published variables, environment under __env, and task outputs, under <taskname> keys.
2) task output is ALWAYS under <taskname> key, like {$.my_task.x.y}
3) published variables and workflow input are ALWAYS referenced as {$.my_published_var}

The difference with what we had before is instead of referring the task result via __task, by $.__task.<taskname> we refer it by $.<taskname>

This implies that the workflow can't use same names for task name, and workflow variable (input, or published), within the same workflow. It is in complete control of workflow author, therefore perfectly OK.

The usage will look like:

workflow:
  input:
    - wf_input
  tasks:
    task0:
      action: foo.bar param="xxx" # assume action foo.bar returns {x:{y:z}}
      publish:
        x: {$.task0.x.y} # referncing "self" action by the task name
      on-success: task1

    task1:
      action: foo.bar param="{$.wf_input} + {$.task0.x} + {$.foo.bar}" #"
      publish: # mix and match workflow globals with action output
        p1: {$.task1.x} # local variable x from THIS action output (task1)
        p2: {$.x} # {$.x} x from global context published by task0
        p3: {$.wf_input} + {$.task0.x.y} + {$.task1.x.y} # mix and match

The only concern I have on this is: action names tend to be long and typing them up in "publish" sucks.
But as our usage shows, 'publish' is used when the same variable is used in different places and users are sick of typing direct access syntax. So doing it once may be OK.

I think it is simple, consistent, and clean.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/149619

[OpenStack Infra (hudson-openstack)]

Change abandoned by Renat Akhmerov (<email address hidden>) on branch: master
Review: https://review.openstack.org/149619

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/149623

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/149623
Committed: https://git.openstack.org/cgit/stackforge/mistral/commit/?id=93a33c169dfd39c7c0449a82c99c88dc451aeb7f
Submitter: Jenkins
Branch: master

commit 93a33c169dfd39c7c0449a82c99c88dc451aeb7f
Author: Renat Akhmerov <email address hidden>
Date: Fri Jan 23 19:36:31 2015 +0600

    Changing publishing mechanism to allow referencing context variables

    * All previously published variables can be now access using '$.'
      as in other places. E.g., if we had variable 'var1' in the context
      we can use '$.var1' to access it in 'publish'.
    * Action/workflow result can be referenced using "$.taskName.".
      E.g, if task name is 'task0' and its action/workflow returned {x: y}
      then 'x' can be referenced using '$.task0.x' in 'publish'.
    * Fixed a lot of unit tests
    * Fixed with-items implementation (part related to task output)

    Change-Id: Ib7532572dd677b517efd40a812cd3633b5de756f
    Closes-Bug: #1412635