JSON serialization doesn't always work for YAQL expression results

[Renat Akhmerov]

With disabled YAQL output data conversion, there are cases when we get a result from YAQL and it contains objects of types that are not serializable with the standard json library. For example, FrozenDict (an internal YAQL class), iterators and generators. Those objects may be nested deeply inside the data structure returned by the YAQL engine.

Generally, this issue is not a problem anymore for the internal Mistral code because it was already adjusted to account these types. But whenever we try to serialize this data into a JSON string (e.g. in actions) we'll be getting an issue.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/714416
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=0dbab33c4c1aada5bcf96712d31ab7fdf3e4d1bb
Submitter: Zuul
Branch: master

commit 0dbab33c4c1aada5bcf96712d31ab7fdf3e4d1bb
Author: Renat Akhmerov <email address hidden>
Date: Mon Mar 23 17:42:28 2020 +0700

    Fix serialization of structures that might contain YAQL types

    * When YAQL output data conversion is disabled there's still
      an issue caused by presence of not JSON-compatible types within
      a YAQL result. The internal Mistral code is already able to deal
      with that (due to the previous changes) by checking that and
      converting them to what's needed. However, JSON serialization
      may still not work if it's done via the standard "json" library.
      The library simply doesn't handle those non-standard types and
      raises an exception. We have a sanitizing function that all YAQL
      results go through, however, it doesn't make sense to do recursive
      sanitizing for performance reasons. It does make sense to convert
      data as late as possible to avoid redundant data manipulations. So
      the sanitizing function handles only the root object in the object
      graph. The solution for this problem is to use our own utility
      function based on the "oslo_serialization.jsonutils" that is able
      to deal with at least part of the mentioned types, specifically
      FrozenDict and iterators. Generators are still a problem and this
      new function takes care of that separately, assuming that any
      generator is just a special iterator and hence represents a
      collection, i.e. a list in JSON terms. It works for all the cases
      we've encountered so far working with YAQL.
    * Used the new function "utils.to_json_str()" everywhere for JSON
      serialization, including the action "std.http".
    * Added necessary unit tests.

    Closes-Bug: #1869168
    Depends-On: I1081a44a6f305eb1dfe68a5bad30110385130725
    Change-Id: I9e73ea7cbba215c3e1d174b5189be27c640c4d42

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/716485

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716485
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=1e0013835c10cbbdfb747625d29cbbe9d6852948
Submitter: Zuul
Branch: stable/train

commit 1e0013835c10cbbdfb747625d29cbbe9d6852948
Author: Renat Akhmerov <email address hidden>
Date: Mon Mar 23 17:42:28 2020 +0700

    Fix serialization of structures that might contain YAQL types

    * When YAQL output data conversion is disabled there's still
      an issue caused by presence of not JSON-compatible types within
      a YAQL result. The internal Mistral code is already able to deal
      with that (due to the previous changes) by checking that and
      converting them to what's needed. However, JSON serialization
      may still not work if it's done via the standard "json" library.
      The library simply doesn't handle those non-standard types and
      raises an exception. We have a sanitizing function that all YAQL
      results go through, however, it doesn't make sense to do recursive
      sanitizing for performance reasons. It does make sense to convert
      data as late as possible to avoid redundant data manipulations. So
      the sanitizing function handles only the root object in the object
      graph. The solution for this problem is to use our own utility
      function based on the "oslo_serialization.jsonutils" that is able
      to deal with at least part of the mentioned types, specifically
      FrozenDict and iterators.
    * Used the new function "utils.to_json_str()" everywhere for JSON
      serialization, including the action "std.http".
    * Added necessary unit tests.

    Closes-Bug: #1869168
    Depends-On: I1081a44a6f305eb1dfe68a5bad30110385130725
    Change-Id: I9e73ea7cbba215c3e1d174b5189be27c640c4d42

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/mistral 10.0.0.0b3 development milestone.