Parallel engine is slow with taskflow version upgrade

Bug #1902481 reported by Deepika A
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
taskflow
New
Undecided
Unassigned

Bug Description

We are facing slowness executing workload with parallel engine with python3.
The workflow is similar to:

```class ParallelWorkflow:

    def initflow(self):

        self._preworkflow = uf.Flow(self.name + "#Preworkflow")
        self._preworkflow.add(UnorderedPreWorkflow(self._store['items']))

        self._generatedata = lf.Flow(self.name + "#WorkflowMetadata")
        self._generatedata.add(LinearWorkflowMetadata1(self.name + "#WorkflowMetadata1"))
        self._generatedata.add(LinearWorkflowMetadata2(self.name + "#WorkflowMetadata2"))

        self._operations = uf.Flow(self.name + "#Operations")
        self._operations.add(UnorderedOperation1(self._store['items']))
        self._operations.add(UnorderedOperation2(self._store['items']))
        self._operations.add(UnorderedOperation3(self._store['items']))

        self._postworkflow = lf.Flow(self.name + "#Postworkflow")
        self._postworkflow.add(UnorderedPostWorkflow1(self._store['items']))
        self._postworkflow.add(UnorderedPostWorkflow2(self._store['items']))

        self._flow = lf.Flow(self.name)
        self._flow.add(
            self._preworkflow,
            self._generatedata,
            self._operations,
            self._postworkflow)

    # pyhton2 execute
    def execute(self):
        result = engines.run(
            self._flow,
            engine_conf='parallel',
            backend={'connection': self._store['connection']},
            store=self._store)

    # pyhton3 execute
    def execute(self):
        result = engines.run(
            self._flow,
            engine='parallel',
            engine_conf='parallel',
            backend={'connection': self._store['connection']},
            store=self._store)
```

With python2.7, taskflow version is: 0.1.4.dev51 and with python3.6, it's 4.4.0

We found significant time difference (5 times more with py3) in execution of the workflow with the versions.
It was running with default engine however we got some optimization after passing engine to engines.run (see: execute def)
The application uses Eventlet. Adding executor is not very helpful. Still not optimized enough.

Could you please guide how can we achieve expected performance here? Or are we missing anything?

Deepika A (deepikaa)
description: updated
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.