Task serialization when YAQL used takes too much time

Bug #1572103 reported by Stanislaw Bogatkin on 2016-04-19
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Bulat Gaifullin
Mitaka
High
Bulat Gaifullin

Bug Description

Fuel 10.0, current master

When we use tasks with YAQL conditions on environment with many nodes (200+), it takes really long time to get serialization done.
On machine with Intel(R) Xeon(R) CPU E5-2620 (12 cores) we have next serialization time:

5 nodes:
2016-04-15 14:15:49.834 tasks serialization is started.
2016-04-15 14:16:05.277 tasks serialization is finished.
diff = 16s

50 nodes:
2016-04-15 14:30:29.376 tasks serialization is started.
2016-04-15 14:33:25.227 tasks serialization is finished.
diff: 176s (2:56)

200 nodes:
VSIZE: 1.2G, RSIZE: 488.3M
2016-04-15 14:39:25.297 tasks serialization is started.
2016-04-15 15:09:05.396 tasks serialization is finished.
diff: 1780s (29:40)

500 nodes:
VSIZE:2.2G, RSIZE: 1.6G
2016-04-15 15:33:56.381 tasks serialization is started.
2016-04-15 18:08:02.706 tasks serialization is finished
diff: 9246s (2:34:06)

It's really much, we need to find a way to improve this.

Fix proposed to branch: master
Review: https://review.openstack.org/307740

Changed in fuel:
status: New → In Progress
Changed in fuel:
assignee: Stanislaw Bogatkin (sbogatkin) → Bulat Gaifullin (bgaifullin)

Fix proposed to branch: master
Review: https://review.openstack.org/318206

Changed in fuel:
assignee: Bulat Gaifullin (bgaifullin) → Stanislaw Bogatkin (sbogatkin)

Reviewed: https://review.openstack.org/318206
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=9a806f1d12ed45d7fa623dfb0dec664a8bcb2328
Submitter: Jenkins
Branch: master

commit 9a806f1d12ed45d7fa623dfb0dec664a8bcb2328
Author: Stanislaw Bogatkin <email address hidden>
Date: Wed May 18 19:28:36 2016 +0300

    Use C vairant of YAML dumper at tasks serialization

    To speed up seiralization time, use faster implementation of
    YAML dumping when it is possible.

    Change-Id: I992410328c07d14a1f5903aa91d456b040175a69
    Partial-Bug: #1572103
    Co-Authored-With: V. Kuklin <email address hidden>
    Co-Authored-With: B. Gaifullin <email address hidden>

Changed in fuel:
assignee: Stanislaw Bogatkin (sbogatkin) → Bulat Gaifullin (bgaifullin)

Reviewed: https://review.openstack.org/322217
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=6dea7664c2365bc5fd6ac72e80943d3136a5880f
Submitter: Jenkins
Branch: stable/mitaka

commit 6dea7664c2365bc5fd6ac72e80943d3136a5880f
Author: Stanislaw Bogatkin <email address hidden>
Date: Wed May 18 19:28:36 2016 +0300

    Use C vairant of YAML dumper at tasks serialization

    To speed up seiralization time, use faster implementation of
    YAML dumping when it is possible.

    Change-Id: I992410328c07d14a1f5903aa91d456b040175a69
    Partial-Bug: #1572103
    Co-Authored-With: V. Kuklin <email address hidden>
    Co-Authored-With: B. Gaifullin <email address hidden>
    (cherry picked from commit 9a806f1d12ed45d7fa623dfb0dec664a8bcb2328)

tags: added: in-stable-mitaka

Reviewed: https://review.openstack.org/307740
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=02c4283fe7ac850c0d51e03ee9c05cc2ee3a2f60
Submitter: Jenkins
Branch: master

commit 02c4283fe7ac850c0d51e03ee9c05cc2ee3a2f60
Author: Stanislaw Bogatkin <email address hidden>
Date: Tue Apr 19 14:55:57 2016 +0300

    Multiprocess tasks serialization

    Tasks serialization process take long time in case when environment
    contains many nodes. To reduce time of serialization, make
    serialization process work in parallel (reduces time of
    serialization mostly linear way with increasing workers pool)

    DocImpact
    Change-Id: Id3753dbc6983256d410e69c98ab02b61ab6bfb7f
    Partial-Bug: #1572103
    Co-Authored-With: V. Kuklin <email address hidden>
    Co-Authored-With: B. Gaifullin <email address hidden>

ETA: 1 day

Reviewed: https://review.openstack.org/322960
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=4efe81f5894299f361fede9836c65f4ec4ba7853
Submitter: Jenkins
Branch: stable/mitaka

commit 4efe81f5894299f361fede9836c65f4ec4ba7853
Author: Stanislaw Bogatkin <email address hidden>
Date: Tue Apr 19 14:55:57 2016 +0300

    Multiprocess tasks serialization

    Tasks serialization process take long time in case when environment
    contains many nodes. To reduce time of serialization, make
    serialization process work in parallel (reduces time of
    serialization mostly linear way with increasing workers pool)

    DocImpact
    Change-Id: Id3753dbc6983256d410e69c98ab02b61ab6bfb7f
    Partial-Bug: #1572103
    Co-Authored-With: V. Kuklin <email address hidden>
    Co-Authored-With: B. Gaifullin <email address hidden>
    (cherry picked from commit 02c4283fe7ac850c0d51e03ee9c05cc2ee3a2f60)

no longer affects: fuel/newton
tags: added: on-verification
Kyrylo Galanov (kgalanov) wrote :

Verified on ISO #465 RC1

tags: removed: on-verification
Dmitry Pyzhov (dpyzhov) on 2016-08-04
tags: added: 9.1-proposed
Dmitry Pyzhov (dpyzhov) on 2016-12-22
Changed in fuel:
milestone: 10.0 → 10.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers