haproxy needs to be configured to avoid deadlock with mysql galera cluster

Bug #1387685 reported by Tom Howley
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
tripleo
Invalid
High
Nicholas Randon

Bug Description

When the overcloud controller is deployed in HA mode and mysql galera cluster is being used, we have a potential deadlock issue when two mysql instances are writing at the same time. One issue is that mysql galera does not support SELECT...FOR UPDATE calls as it has no concept of cross-node locking.

I'm not aware if this issue has changed. If not, the workaround is configure haproxy to effectively use mysql in active-passive mode, which can be achieved by configuring all but one mysql instances as 'backup' nodes.

Changed in tripleo:
assignee: nobody → Nicholas Randon (nicholas-randon)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/132048

Changed in tripleo:
status: New → In Progress
Changed in tripleo:
assignee: Nicholas Randon (nicholas-randon) → Alexis Lee (alexisl)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-incubator (master)

Fix proposed to branch: master
Review: https://review.openstack.org/132059

Changed in tripleo:
assignee: Alexis Lee (alexisl) → Nicholas Randon (nicholas-randon)
Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Marking this as "High" because it involves spurious database errors. Ideally OpenStack would handle rollbacks by retrying, but we know that just isn't the case.

Changed in tripleo:
importance: Undecided → High
Changed in tripleo:
status: In Progress → Invalid
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (master)

Change abandoned by Nicholas Randon (<email address hidden>) on branch: master
Review: https://review.openstack.org/132048

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-incubator (master)

Change abandoned by Nicholas Randon (<email address hidden>) on branch: master
Review: https://review.openstack.org/132059

Revision history for this message
Tom Howley (tom-howley) wrote :

This bug has been marked as invalid as the existing config marks all nodes in the haproxy pool as backup, thereby disabling load-balancing, as haproxy will just always just pick the first available node in the list and only change when that node fails. As a result, the existing config already minimises chance of deadlock in galera cluster.

(Please correct if I'm wrong on this.)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.