YAQL function "str" doesn't work properly for lists

[Renat Akhmerov]

If we run the workflow:

test_yaql:
  input:
    - my_list

  tasks:
    task1:
      action: std.noop
      publish:
        val: <% str($.my_list) %>

against the input:

{
  "my_list": [
    {
      "k1": "v1",
      "k2": "v2"
    }
  ]
}

Then we'll get a result of publish that doesn't include square brackets:

{'val': "({'k1': 'v1', 'k2': 'v2'},)"}

It is claimed that in Queens version the result would contain square brackets:

{'val': "[{'k1': 'v1', 'k2': 'v2'}]"}

So, in fact, we have a list of dictionaries here that is not properly represented with the "str" YAQL function.

[Renat Akhmerov (rakhmerov)]

It happens because of the line:

new_ctx['$'] = yaql_utils.convert_input_data(data_context)

in the module mistral/utils/expression_utils.py

(Currently can be seen at https://github.com/openstack/mistral/blob/master/mistral/utils/expression_utils.py#L51)

The patch where this issue was introduced: https://review.openstack.org/#/c/477816/

[Renat Akhmerov (rakhmerov)]

We added this line to fix the but https://bugs.launchpad.net/mistral/+bug/1772864

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/636548
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=a39db2d3dc21afbe8b9e1d6f8dd870da272b9671
Submitter: Zuul
Branch: master

commit a39db2d3dc21afbe8b9e1d6f8dd870da272b9671
Author: Renat Akhmerov <email address hidden>
Date: Wed Feb 13 12:36:49 2019 +0700

    Fix how Mistral prepares data for evaluating a YAQL expression

    * If we use the built-in YAQL function 'str' in a workflow then it
      doesn't represent lists as '[item1, item3, ...]' but instead
      creates '(item1, item2,...). This is because the standard YAQL
      function 'yaql_utils.convert_input_data', which is needed to
      convert a initial user data into an internal YAQL format,
      converts all sequences (except strings) into tuples.
      This patch overrides this behavior for sequences that are not
      strings and tuples so that they now get converted into lists.
      YAQL uses tuples because it needs to obtain a safe immutable
      structure to make calculations upon. But in Mistral list is
      more suitable because lots of users care about string
      representations. Immutability is not so important because
      Mistral code base guarantees that the initial data context
      for an expression won't be changed while an expression is
      being evaluated by YAQL.
    * "str" YAQL function used to work well but it was broken in
      https://review.openstack.org/#/c/477816/ that added additional
      context preparation in order to fix the issue
      https://bugs.launchpad.net/mistral/+bug/1772864

    Change-Id: I69d32f8772418d586d6c414842bb54aada217481
    Closes-Bug: #1815710

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/637130

[OpenStack Infra (hudson-openstack)]

Change abandoned by Renat Akhmerov (<email address hidden>) on branch: stable/rocky
Review: https://review.openstack.org/637130
Reason: The initial patch to master had to be reverted.
https://review.openstack.org/#/c/637170/. Read more in the comments of this patch for more details.

[Renat Akhmerov (rakhmerov)]

We had to revert the patch with the fix because it breaks list comparisons in YAQL.

https://review.openstack.org/#/c/637170

[Renat Akhmerov (rakhmerov)]

It's not clear yet how to make "str()" function return "[..]" for lists w/o breaking anything else.

YAQL lib and YAQLuator return "(..)", if using this function, and it's compliant with how Python works with tuples. Also, YAQL uses tuples pretty much everywhere internally when it comes to processing sequences, and there's no easy way to override it, it seems to be a conceptual thing.

[Renat Akhmerov (rakhmerov)]

So closing the ticket for now.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/mistral 8.0.0.0b2 development milestone.