Add "convert_output_data" config option for YAQL

Bug #1863857 reported by Renat Akhmerov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mistral
Fix Released
High
Renat Akhmerov

Bug Description

When we call YAQL to evaluate an expression it performs conversion of the expression result before returning it to Mistral. In fact, it creates a copy of the result structure in memory. If the result is a big it takes significant CPU and increases memory footprint. We need to give an option for users to disable this YAQL output data conversion as we did before for input conversion.

Revision history for this message
Renat Akhmerov (rakhmerov) wrote :

The merged commit in master: https://review.opendev.org/#/c/707056/

Changed in mistral:
milestone: none → ussuri-2
assignee: nobody → Renat Akhmerov (rakhmerov)
importance: Undecided → High
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to mistral (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/708582

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to mistral (master)

Fix proposed to branch: master
Review: https://review.opendev.org/708585

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to mistral (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/708683

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to mistral (stable/train)

Reviewed: https://review.opendev.org/708582
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=d22676ef396e0fccc6732b14f8be12f4625e3c79
Submitter: Zuul
Branch: stable/train

commit d22676ef396e0fccc6732b14f8be12f4625e3c79
Author: Renat Akhmerov <email address hidden>
Date: Tue Feb 11 11:18:50 2020 +0700

    Add "convert_output_data" config property for YAQL

    * Adding "convert_output_data" config property gives an opportunity
      to increase overal performance. If YAQL always converts an expression
      result, it often takes significant CPU time and overall workflow
      execution time increases. It is especially important when a workflow
      publishes lots of data into the context and uses big workflow
      environments. It's been tested on a very big workflow (~7k tasks)
      with a big workflow environment (~2.5mb) that often uses the YAQL
      function "<% env() %>". This function basically just returns the
      workflow environment.
    * Created all necessary unit tests.
    * Other style fixes.

    Closes-Bug: #1863857
    Change-Id: Ie3169ec884ec9a0e7e50327dd03cd78dcda0a39b
    Cherry picked from Ie3169ec884ec9a0e7e50327dd03cd78dcda0a39b

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to mistral (master)

Reviewed: https://review.opendev.org/708585
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=253fd978c551a0e55c78f16d2cc2e0103982941f
Submitter: Zuul
Branch: master

commit 253fd978c551a0e55c78f16d2cc2e0103982941f
Author: Renat Akhmerov <email address hidden>
Date: Wed Feb 19 16:12:33 2020 +0700

    Release note for "convert_output_data" config option

    Change-Id: I99d30434c9d337529edae041427de05ffcf2ffd6
    Closes-Bug: #1863857

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to mistral (stable/train)

Reviewed: https://review.opendev.org/708683
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=bec0a01320c6f204ff10c14baa9f1afe40efa1e2
Submitter: Zuul
Branch: stable/train

commit bec0a01320c6f204ff10c14baa9f1afe40efa1e2
Author: Renat Akhmerov <email address hidden>
Date: Wed Feb 19 16:12:33 2020 +0700

    Release note for "convert_output_data" config option

    Change-Id: I99d30434c9d337529edae041427de05ffcf2ffd6
    Closes-Bug: #1863857

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/mistral 10.0.0.0b2

This issue was fixed in the openstack/mistral 10.0.0.0b2 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.