Mistral

Mistral server consumes more memory over time

Bug #1757966 reported by Renat Akhmerov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mistral
Fix Released
Critical
Renat Akhmerov
Mistral rocky-2 "r2"

Bug Description

If Mistral is running long enough it consumes more and more memory (we've seen 20+GB) and doesn't release it even if the system load is reduced or removed completely.

Renat Akhmerov (rakhmerov)
Changed in mistral:
assignee: nobody → Renat Akhmerov (rakhmerov)
importance: Undecided → Critical
milestone: none → rocky-1
Revision history for this message
Renat Akhmerov (rakhmerov) wrote :

Last findings show that deletion of resources somehow has a huge impact on memory. W/o even running workflows.

Dougal Matthews (d0ugal)
Changed in mistral:
status: New → In Progress
do3meli (d-info-e)
summary: - Mistral server consumes more and memory over time
+ Mistral server consumes more memory over time
Dougal Matthews (d0ugal)
Changed in mistral:
milestone: rocky-1 → rocky-2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to mistral (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/564751

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to mistral (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/564965

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to mistral (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/564966

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to mistral (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/564984

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to mistral (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/564985

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to mistral (stable/pike)

Reviewed: https://review.openstack.org/564966
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=5e56a28608d9b3a37a26a89c4d4bf2a78cfb1683
Submitter: Zuul
Branch: stable/pike

commit 5e56a28608d9b3a37a26a89c4d4bf2a78cfb1683
Author: Renat Akhmerov <email address hidden>
Date: Thu Apr 26 18:10:18 2018 +0700

    Get rid of a extra copy of workflow environment

    * Previously, we had two copies of the workflow environment
      passed by a user: one was in the 'params' fields under ke 'env'
      key and another one was copied into the 'context' field under
      the '__env' key so that we can evaluate expressions involving
      the env() function (YAQL or Jinja).
      This patch removes the copy from the 'context' field in favor
      of using an ad-hoc ContextView structure where we now also
      weave in the environment dictionary under the same key '__env'.

    Related-Bug: #1757966
    Change-Id: I1204b082794b376787d126136a79dd204ec3af07

tags: added: in-stable-pike
tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to mistral (stable/queens)

Reviewed: https://review.openstack.org/564984
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=6c5d56cc781dbd1f5e7c6d5d73c4f42ba26d35a5
Submitter: Zuul
Branch: stable/queens

commit 6c5d56cc781dbd1f5e7c6d5d73c4f42ba26d35a5
Author: Renat Akhmerov <email address hidden>
Date: Fri Apr 27 20:00:36 2018 +0700

    Release note for workflow environment optimizations

    Change-Id: Ieff4887384a4f8fa0eb52a6255f3e4402dbf1d8e
    Related-Bug: #1757966

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to mistral (stable/pike)

Reviewed: https://review.openstack.org/564985
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=3eeb561c1ccc5e0ff0f6f225a7f0388b40ca1f74
Submitter: Zuul
Branch: stable/pike

commit 3eeb561c1ccc5e0ff0f6f225a7f0388b40ca1f74
Author: Renat Akhmerov <email address hidden>
Date: Fri Apr 27 20:00:36 2018 +0700

    Release note for workflow environment optimizations

    Change-Id: Ieff4887384a4f8fa0eb52a6255f3e4402dbf1d8e
    Related-Bug: #1757966

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to mistral (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/565004

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to mistral (stable/pike)

Reviewed: https://review.openstack.org/565004
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=7ec98b457cb83087241fbe5153467946c90af099
Submitter: Zuul
Branch: stable/pike

commit 7ec98b457cb83087241fbe5153467946c90af099
Author: Renat Akhmerov <email address hidden>
Date: Fri Apr 27 17:01:31 2018 +0700

    Do not copy workflow environment into subworkflows

    * We previously always copied a workflow environment of a parent
      workflow into a subworkflow when starting it. However, this is
      redundant because we now have 'root_execution_id' field in the
      the workflow execution model so that we can always get an
      environment of a subworkflow just by accessing the root execution.
      It saves a lot of space in DB and increases performance in cases
      when we have a large workflow environment and many subworkflows.

    Change-Id: Id8f5e917b45fedf09ecf2d28798a6949cf5f5c99
    Related-Bug: #1757966

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to mistral (master)

Reviewed: https://review.openstack.org/564462
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=b77769cf447ee85892da676b77189320f1ad838d
Submitter: Zuul
Branch: master

commit b77769cf447ee85892da676b77189320f1ad838d
Author: Renat Akhmerov <email address hidden>
Date: Thu Apr 26 18:10:18 2018 +0700

    Get rid of a extra copy of workflow environment

    * Previously, we had two copies of the workflow environment
      passed by a user: one was in the 'params' fields under ke 'env'
      key and another one was copied into the 'context' field under
      the '__env' key so that we can evaluate expressions involving
      the env() function (YAQL or Jinja).
      This patch removes the copy from the 'context' field in favor
      of using an ad-hoc ContextView structure where we now also
      weave in the environment dictionary under the same key '__env'.

    Related-Bug: #1757966
    Change-Id: I1204b082794b376787d126136a79dd204ec3af07

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/564717
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=f2a9bd45abf0b08e1a906faf29e4a85579d1ba0b
Submitter: Zuul
Branch: master

commit f2a9bd45abf0b08e1a906faf29e4a85579d1ba0b
Author: Renat Akhmerov <email address hidden>
Date: Fri Apr 27 17:01:31 2018 +0700

    Do not copy workflow environment into subworkflows

    * We previously always copied a workflow environment of a parent
      workflow into a subworkflow when starting it. However, this is
      redundant because we now have 'root_execution_id' field in the
      the workflow execution model so that we can always get an
      environment of a subworkflow just by accessing the root execution.
      It saves a lot of space in DB and increases performance in cases
      when we have a large workflow environment and many subworkflows.

    Related-Bug: #1757966
    Change-Id: I15077240ba53663a6267b886ab7b081a7dde2710

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/564751
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=cb1cabbfe652bcfdda4a116c290d8474c3f1f4e1
Submitter: Zuul
Branch: master

commit cb1cabbfe652bcfdda4a116c290d8474c3f1f4e1
Author: Renat Akhmerov <email address hidden>
Date: Fri Apr 27 20:00:36 2018 +0700

    Release note for workflow environment optimizations

    Change-Id: Ieff4887384a4f8fa0eb52a6255f3e4402dbf1d8e
    Related-Bug: #1757966

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to mistral (stable/queens)

Reviewed: https://review.openstack.org/564965
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=227592530227d23125694f2a6a42628109e360de
Submitter: Zuul
Branch: stable/queens

commit 227592530227d23125694f2a6a42628109e360de
Author: Renat Akhmerov <email address hidden>
Date: Thu Apr 26 18:10:18 2018 +0700

    Get rid of a extra copy of workflow environment

    * Previously, we had two copies of the workflow environment
      passed by a user: one was in the 'params' fields under ke 'env'
      key and another one was copied into the 'context' field under
      the '__env' key so that we can evaluate expressions involving
      the env() function (YAQL or Jinja).
      This patch removes the copy from the 'context' field in favor
      of using an ad-hoc ContextView structure where we now also
      weave in the environment dictionary under the same key '__env'.

    Related-Bug: #1757966
    Change-Id: I1204b082794b376787d126136a79dd204ec3af07

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to mistral (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/565949

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to mistral (stable/queens)

Reviewed: https://review.openstack.org/565949
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=9e00e002a97a628bf3ec293fbb8ca63127d9f84e
Submitter: Zuul
Branch: stable/queens

commit 9e00e002a97a628bf3ec293fbb8ca63127d9f84e
Author: Renat Akhmerov <email address hidden>
Date: Fri Apr 27 17:01:31 2018 +0700

    Do not copy workflow environment into subworkflows

    * We previously always copied a workflow environment of a parent
      workflow into a subworkflow when starting it. However, this is
      redundant because we now have 'root_execution_id' field in the
      the workflow execution model so that we can always get an
      environment of a subworkflow just by accessing the root execution.
      It saves a lot of space in DB and increases performance in cases
      when we have a large workflow environment and many subworkflows.

    Change-Id: Ibd66107b8b8bd92b8f38c299c88903d9dba3d9e8
    Related-Bug: #1757966

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to mistral (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/568566

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to mistral (master)

Reviewed: https://review.openstack.org/568312
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=afb0f584c5f1218ec73b2b6ecd9d7509c1df4594
Submitter: Zuul
Branch: master

commit afb0f584c5f1218ec73b2b6ecd9d7509c1df4594
Author: Renat Akhmerov <email address hidden>
Date: Mon May 14 17:09:29 2018 +0700

    Use "passive_deletes=True" in ORM relationships

    * Using "passive_deletes=True" when configuring relationships
      with SQLAlchemy significantly reduces memory footprint when
      deleting a graph of objects (e.g. root workflow execution
      with all dependent tasks, actions and sub workflows) from DB.
      This happens because in this case SQLAlchemy doesn't load
      those dependent objects that are not yet in the session and
      rather relies on cascading mechanisms of databases which are
      activated by the proper configuration of foreign key columns.
      For example, "ON DELETE CASCADE" clause in the foreign key
      definition in case of for MySQL.
      Particularly, this change reduces memory footprint of API
      processes when deleting large graphs of executions and
      execution expiration policy mechanism where the same happens
      but on the engine side.
    * Added a test to make sure that cascaded deletion works well
      with this new property.

    Related-Bug: #1757966
    Change-Id: I36988817fb8e7695094ef98958b8aa151208fadb

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to mistral (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/568762

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to mistral (master)

Reviewed: https://review.openstack.org/568762
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=696e82702c0deafb4a5f1baf35da8e7987818fd0
Submitter: Zuul
Branch: master

commit 696e82702c0deafb4a5f1baf35da8e7987818fd0
Author: Renat Akhmerov <email address hidden>
Date: Wed May 16 13:58:10 2018 +0700

    Release note for using "passive_deletes=True"

    Related-Bug: #1757966

    Change-Id: I013d4837603839364f8d699f7ce97e4400879d38

Renat Akhmerov (rakhmerov)
Changed in mistral:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to mistral (stable/queens)

Reviewed: https://review.openstack.org/568566
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=00b405b072dd8bdf2e4c80bf61abdd6996395e62
Submitter: Zuul
Branch: stable/queens

commit 00b405b072dd8bdf2e4c80bf61abdd6996395e62
Author: Renat Akhmerov <email address hidden>
Date: Mon May 14 17:09:29 2018 +0700

    Use "passive_deletes=True" in ORM relationships

    * Using "passive_deletes=True" when configuring relationships
      with SQLAlchemy significantly reduces memory footprint when
      deleting a graph of objects (e.g. root workflow execution
      with all dependent tasks, actions and sub workflows) from DB.
      This happens because in this case SQLAlchemy doesn't load
      those dependent objects that are not yet in the session and
      rather relies on cascading mechanisms of databases which are
      activated by the proper configuration of foreign key columns.
      For example, "ON DELETE CASCADE" clause in the foreign key
      definition in case of for MySQL.
      Particularly, this change reduces memory footprint of API
      processes when deleting large graphs of executions and
      execution expiration policy mechanism where the same happens
      but on the engine side.
    * Added a test to make sure that cascaded deletion works well
      with this new property.

    Related-Bug: #1757966
    Change-Id: I36988817fb8e7695094ef98958b8aa151208fadb

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to mistral (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/569081

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/569084

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to mistral (stable/pike)

Reviewed: https://review.openstack.org/569084
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=a619245ce96bf2ad0dab41efb260c961103825fc
Submitter: Zuul
Branch: stable/pike

commit a619245ce96bf2ad0dab41efb260c961103825fc
Author: Renat Akhmerov <email address hidden>
Date: Wed May 16 13:58:10 2018 +0700

    Release note for using "passive_deletes=True"

    Related-Bug: #1757966

    Change-Id: I013d4837603839364f8d699f7ce97e4400879d38
    (cherry picked from commit 696e82702c0deafb4a5f1baf35da8e7987818fd0)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to mistral (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/569101

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to mistral (stable/pike)

Reviewed: https://review.openstack.org/569081
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=ae426bf6fe4eb11431f1008e7065ec6aa0383a33
Submitter: Zuul
Branch: stable/pike

commit ae426bf6fe4eb11431f1008e7065ec6aa0383a33
Author: Renat Akhmerov <email address hidden>
Date: Mon May 14 17:09:29 2018 +0700

    Use "passive_deletes=True" in ORM relationships

    * Using "passive_deletes=True" when configuring relationships
      with SQLAlchemy significantly reduces memory footprint when
      deleting a graph of objects (e.g. root workflow execution
      with all dependent tasks, actions and sub workflows) from DB.
      This happens because in this case SQLAlchemy doesn't load
      those dependent objects that are not yet in the session and
      rather relies on cascading mechanisms of databases which are
      activated by the proper configuration of foreign key columns.
      For example, "ON DELETE CASCADE" clause in the foreign key
      definition in case of for MySQL.
      Particularly, this change reduces memory footprint of API
      processes when deleting large graphs of executions and
      execution expiration policy mechanism where the same happens
      but on the engine side.
    * Added a test to make sure that cascaded deletion works well
      with this new property.

    Related-Bug: #1757966
    Change-Id: I36988817fb8e7695094ef98958b8aa151208fadb
    (cherry picked from commit afb0f584c5f1218ec73b2b6ecd9d7509c1df4594)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to mistral (stable/queens)

Reviewed: https://review.openstack.org/569101
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=53815745c9050274362fb3cbca2f1474a1eb2619
Submitter: Zuul
Branch: stable/queens

commit 53815745c9050274362fb3cbca2f1474a1eb2619
Author: Renat Akhmerov <email address hidden>
Date: Wed May 16 13:58:10 2018 +0700

    Release note for using "passive_deletes=True"

    Related-Bug: #1757966

    Change-Id: I013d4837603839364f8d699f7ce97e4400879d38
    (cherry picked from commit 696e82702c0deafb4a5f1baf35da8e7987818fd0)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.