[nailgun][astute] Switch PostgresDB transaction isolation level to 'SERIALIZABLE'
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Won't Fix
|
High
|
Miroslav Anashkin |
Bug Description
Nailgun version fuel-nailgun.noarch 9.0.0-1.mos8983 (it is update to MOS 9.2 MU-3)
Fuel version fuel.noarch 9.0.0-1.mos6430 (also update to MOS 9.2 MU-3)
Description:
Currently Nailgun uses default PostgresDB transaction isolation level 'Repeatable Read'.
Since all the Nailgun tasks are running in parallel default transaction isolation level leads to serialization anomaly and even DB deadlocks. Such specific conditions is usually a combination of the fast modern CPU, virtualised environment (master node and OpenStack nodes) and high load, presumably to the disk which is used for PostgresDB data.
It makes the following scenario possible:
1. Astute finishes task serialization and casts the message to Nailgun to store the big list of serialized tasks. This list includes thousands of tasks and is inserted as single big transaction.
2. Astute is unable to commit directly the task list store transaction to Nailgun - it may only cast the messages to Nailgun.
3. Astute immediately starts the task execution - it has the list of tasks in memory.
Due to the fast modern CPU speed or slow disk speed under the Postgres DB - some tasks may be finished before the previous transaction with full list of tasks is committed to the DB. Astute casts one more message to Nailgun - to update the finished task status. This update requires the previous transaction to be finished at this moment. If the transaction with the list of tasks is not finished - Nailgun threads with the concurrent writes to the DB may get into deadlock or Nailgun may miss some object. Nailgun does not stop at such errors, it only casts error message to the app.log and continue accepting the requests in another threads. Such requests may never be finished due to DB error or deadlock.
Due to the size, the logs attached as separate file.
Changed in fuel: | |
assignee: | MOS Maintenance (mos-maintenance) → Oleksiy Molchanov (omolchanov) |
status: | New → In Progress |
Changed in fuel: | |
assignee: | Oleksiy Molchanov (omolchanov) → MOS Maintenance (mos-maintenance) |
status: | In Progress → Confirmed |
sla1 for 9.0-updates