Scheduler delay is not configurable
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Mistral |
Fix Released
|
High
|
Renat Akhmerov | ||
Bug Description
When we run Mistral in HA every engine instance runs its own Scheduler thread that processes delayed calls. These calls are important part of the entire workflow processing. All these schedulers often compete each other badly because try to process same calls stored in DB. In case of using MySql it often leads to deadlocks which doesn't mean that we use some bad patterns, it's just Mysql behaviour, it's described in their docs that such situations may occur and developers need to design applications taking this into account.
One simple solution that would help reduce contention between Scheduler instances is to:
1) Make delay between Scheduler iterations configurable so that if it's needed we could make it bigger and reduce number of DB hits
2) Add the random part in Scheduler delay to distribute Scheduler iterations more evenly in time which also should reduce contention and load to DB.
| Changed in mistral: | |
| milestone: | none → queens-1 |
| importance: | Undecided → High |
| status: | New → Confirmed |
| assignee: | nobody → Renat Akhmerov (rakhmerov) |
| Changed in mistral: | |
| status: | Confirmed → In Progress |
| OpenStack Infra (hudson-openstack) wrote Fix merged to mistral (master) | #1 |
| Changed in mistral: | |
| status: | In Progress → Fix Released |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to mistral (stable/pike) | #2 |
| OpenStack Infra (hudson-openstack) wrote Change abandoned on mistral (stable/pike) | #3 |
| OpenStack Infra (hudson-openstack) wrote Fix merged to mistral (stable/pike) | #4 |
| tags: | added: in-stable-pike |
| OpenStack Infra (hudson-openstack) wrote Fix included in openstack/mistral 6.0.0.0b1 | #5 |
| OpenStack Infra (hudson-openstack) wrote Fix included in openstack/mistral 5.2.0 | #6 |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit f6b6f1d20b84853
Author: Renat Akhmerov <email address hidden>
Date: Fri Oct 6 13:16:11 2017 +0700
Make scheduler delay configurable
* Made scheduler delay configurable. It now consists of a fixed
part configured with the 'fixed_delay' property and a random
addition limited by the 'random_delay' config property.
Because of this, using loopingcall from oslo was replaced with
a regular loop in a separate thread becase loopingcall
supports only fixed delays.
Closes-Bug: #1721733
Change-Id: I8f6a15be339e20