| 2020-11-02 07:50:40 |
Deepika A |
description |
We are facing slowness executing workload with parallel engine with python3.
The workflow is similar to:
```class ParallelWorkflow:
def initflow(self):
self._preworkflow = uf.Flow(self.name + "#Preworkflow")
self._preworkflow.add(UnorderedPreWorkflow(self._store['items']))
self._generatedata = lf.Flow(self.name + "#WorkflowMetadata")
self._generatedata.add(LinearWorkflowMetadata1(self.name + "#WorkflowMetadata1"))
self._generatedata.add(LinearWorkflowMetadata2(self.name + "#WorkflowMetadata2"))
self._operations = lf.Flow(self.name + "#Operations")
self._operations.add(UnorderedOperation1(self._store['items']))
self._operations.add(UnorderedOperation2(self._store['items']))
self._operations.add(UnorderedOperation3(self._store['items']))
self._postworkflow = lf.Flow(self.name + "#Postworkflow")
self._postworkflow.add(UnorderedPostWorkflow1(self._store['items']))
self._postworkflow.add(UnorderedPostWorkflow2(self._store['items']))
self._flow = lf.Flow(self.name)
self._flow.add(
self._preworkflow,
self._generatedata,
self._operations,
self._postworkflow)
# pyhton2 execute
def execute(self):
result = engines.run(
self._flow,
engine_conf='parallel',
backend={'connection': self._store['connection']},
store=self._store)
# pyhton3 execute
def execute(self):
result = engines.run(
self._flow,
engine='parallel',
engine_conf='parallel',
backend={'connection': self._store['connection']},
store=self._store)
```
With python2.7, taskflow version is: 0.1.4.dev51 and with python3.6, it's 4.4.0
We found significant time difference (5 times more with py3) in execution of the workflow with the versions.
It was running with default engine however we got some optimization after passing engine to engines.run (see: execute def)
The application uses Eventlet. Adding executor is not very helpful. Still not optimized enough.
Could you please guide how can we achieve expected performance here? Or are we missing anything? |
We are facing slowness executing workload with parallel engine with python3.
The workflow is similar to:
```class ParallelWorkflow:
def initflow(self):
self._preworkflow = uf.Flow(self.name + "#Preworkflow")
self._preworkflow.add(UnorderedPreWorkflow(self._store['items']))
self._generatedata = lf.Flow(self.name + "#WorkflowMetadata")
self._generatedata.add(LinearWorkflowMetadata1(self.name + "#WorkflowMetadata1"))
self._generatedata.add(LinearWorkflowMetadata2(self.name + "#WorkflowMetadata2"))
self._operations = uf.Flow(self.name + "#Operations")
self._operations.add(UnorderedOperation1(self._store['items']))
self._operations.add(UnorderedOperation2(self._store['items']))
self._operations.add(UnorderedOperation3(self._store['items']))
self._postworkflow = lf.Flow(self.name + "#Postworkflow")
self._postworkflow.add(UnorderedPostWorkflow1(self._store['items']))
self._postworkflow.add(UnorderedPostWorkflow2(self._store['items']))
self._flow = lf.Flow(self.name)
self._flow.add(
self._preworkflow,
self._generatedata,
self._operations,
self._postworkflow)
# pyhton2 execute
def execute(self):
result = engines.run(
self._flow,
engine_conf='parallel',
backend={'connection': self._store['connection']},
store=self._store)
# pyhton3 execute
def execute(self):
result = engines.run(
self._flow,
engine='parallel',
engine_conf='parallel',
backend={'connection': self._store['connection']},
store=self._store)
```
With python2.7, taskflow version is: 0.1.4.dev51 and with python3.6, it's 4.4.0
We found significant time difference (5 times more with py3) in execution of the workflow with the versions.
It was running with default engine however we got some optimization after passing engine to engines.run (see: execute def)
The application uses Eventlet. Adding executor is not very helpful. Still not optimized enough.
Could you please guide how can we achieve expected performance here? Or are we missing anything? |
|
