Creation of workflow language specifications is too expensive

[Renat Akhmerov]

When it comes to big workflows (several thousand tasks) then uploading a workflow may take minutes. This is too long and we need to find ways to optimize that. One specific thing that takes ~80% of execution time is validating JSON schema of the workflow parts (tasks, policies, etc.) with jsonschema library. One way to overcome this can be adding a config option to disable validation at all (at least JSON schema). For some use cases, when, for example, workflows are auto-generated and pre-validated it will be OK.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/682024
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=64c4038b2c11170a942c909c314e681f73cf895e
Submitter: Zuul
Branch: master

commit 64c4038b2c11170a942c909c314e681f73cf895e
Author: Renat Akhmerov <email address hidden>
Date: Fri Sep 13 18:11:07 2019 +0700

    Optimize creation of language specs

    * Various simple optimizations for creation of specs. Mostly
      additional caching of non-growing data related to the
      structure of the language spec classes. For huge workflows
      with thousands of tasks these changes reduce execution time
      by dozens of seconds.
    * Minor style changes.

    Partial-bug: #1844242
    Change-Id: Ia700e25752d9d35ece18609f2977e6568062e4bf

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/683344

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/683344
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=ac41f94d11cee8c7ed94152f159d3e0ad2a755a6
Submitter: Zuul
Branch: master

commit ac41f94d11cee8c7ed94152f159d3e0ad2a755a6
Author: Renat Akhmerov <email address hidden>
Date: Fri Sep 20 12:03:46 2019 +0700

    Add an ability to disable workflow text validation

    * For the sake of the service performance, it may make sense to
      disable validation of the workflow language syntax if it is
      affordable for a particular use case. For example, if all
      workflows are auto-generated by a 3rd party system and tested
      thoroughly (either by running them with Mistral or at least
      validating them via the special Mistral endpoint) then we can
      safely disable validation of the language syntax when uploading
      workflow definitions. For production systems it makes a big
      difference if workflow texts are large (thousands of tasks).
      This patch adds the boolean parameter "skip_validation" for API
      requests like "POST /v2/workflows" to disable validation, if
      needed, and the new configuration property "validation_mode"
      to set a desired validation mode.
      The option is an enumeration and has the following valid values:
        1) "enabled" - enabled for all API requests unless it's
           explicitly disabled in the request itself
        2) "mandatory" - enabled for all API requests regardless
           of the flag in the request
        3) "disabled" - disabled for all API requrests regardless
           of the flag in the request
      "mandatory" is choosen as the default value for this new
      property to keep compatibility with the previous versions.
    * Minor style changes.

    Closes-Bug: #1844242

    Change-Id: Ib509653d38254954f8449be3031457e5f636ccf2

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/mistral 9.0.0.0rc1 release candidate.