Fuel for OpenStack

MySQL OCF RA may fail to find possible masters

Bug #1572601 reported by Bogdan Dobrelya on 2016-04-20

This bug report is a duplicate of: Bug #1583173: MySQL OCF RA action monitor must check if a seed node is running the most recent of known GTIDs. Edit Remove

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Fuel for OpenStack	Fix Committed	High	Bogdan Dobrelya	Fuel for OpenStack 10.0
6.1.x	Won't Fix	High	MOS Maintenance	Fuel for OpenStack 6.0-updates
7.0.x	Won't Fix	High	MOS Maintenance	Fuel for OpenStack 7.0-updates
8.0.x	Won't Fix	High	MOS Maintenance	Fuel for OpenStack 8.0-updates
Mitaka	Fix Released	High	Bogdan Dobrelya	Fuel for OpenStack 9.0
Newton	Fix Committed	High	Bogdan Dobrelya	Fuel for OpenStack 10.0

Bug Description

There is a flaw in chose_master() and get_possible_masters().
By design, it shall:
- Find the most seen GTID among the online nodes. Ignore its SEQNO position for now.
- Filter nodes with the most seen GTID as possible masters.
- From those, pick the one with the latest commit (from GTID:SEQNO values) and chose it as a master to Join to.

But it works another way and may throw errors and behave in unexpected way.

Steps to reproduce were given in the Galera reliability testing https://goo.gl/VHyIIE paper. Briefly: deploy a 5 nodes galera cluster, run the given Jepsen cases to verify its self-heal capabilities.

There is also a way to reproduce it on a Fuel env given that the node-1, node-2, node-3 deployed as controller nodes:
1) https://github.com/bogdando/jepsen/tree/fuel/noop , see "How-to run tests from the Fuel master..."
2) PURGE=true ./vagrant_script/lein_test.sh noop ssh-test
3) docker exec -it jepsen bash -c "TESTPROC=mysqld lein test :only jepsen.noop-test/factors-crashstop-test"

See original description

Tags:

Bogdan Dobrelya (bogdando) on 2016-04-20

Changed in fuel:
importance:	Undecided → High
tags:	added: area-library galera tech-debt
description:	updated

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-04-20: Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/308410

Bogdan Dobrelya (bogdando) on 2016-04-20

description:

updated

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-04-21: Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/308872

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-04-21: Change abandoned on fuel-library (stable/mitaka)

Change abandoned by Sergii Golovatiuk (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/308872

OpenStack Infra (hudson-openstack) on 2016-04-21

Changed in fuel:
assignee:	Bogdan Dobrelya (bogdando) → Sergii Golovatiuk (sgolovatiuk)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-04-22: Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/308410
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=958192b151c4a5cef3537eab0813307ea4cbb898
Submitter: Jenkins
Branch: master

commit 958192b151c4a5cef3537eab0813307ea4cbb898
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Apr 20 16:16:00 2016 +0200

Fix possible masters eval

    Fix validate_gtid to report ones ending with ':' as bad.
    Fix possible masters->master evaluation.
    Merge chose_master() and get_possible_masters() as chose_master()

Closes-bug: #1572601

    Change-Id: Ia5a5319bce147f29ad4aef7eb92ce7744273908d
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    Signed-off-by: Sergii Golovatiuk <email address hidden>

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-04-22: Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/308872
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=1146a009f399b3d96bdf5aa2939288aa62ebbbf6
Submitter: Jenkins
Branch: stable/mitaka

commit 1146a009f399b3d96bdf5aa2939288aa62ebbbf6
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Apr 20 16:16:00 2016 +0200

Fix possible masters eval

    Fix validate_gtid to report ones ending with ':' as bad.
    Fix possible masters->master evaluation.
    Merge chose_master() and get_possible_masters() as chose_master()

Closes-bug: #1572601

    Change-Id: Ia5a5319bce147f29ad4aef7eb92ce7744273908d
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    Signed-off-by: Sergii Golovatiuk <email address hidden>
    (cherry picked from commit 958192b151c4a5cef3537eab0813307ea4cbb898)

Bogdan Dobrelya (bogdando) on 2016-05-05

description:

updated

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-13: Fix proposed to fuel-library (stable/6.1)

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/315989

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-16: Fix proposed to fuel-library (stable/7.0)

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/316802

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-18: Fix proposed to fuel-library (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/317978

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-20: Change abandoned on fuel-library (stable/8.0)

#10

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/317978

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-20: Change abandoned on fuel-library (stable/7.0)

#11

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/316802

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-20: Change abandoned on fuel-library (stable/6.1)

#12

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/315989

Bogdan Dobrelya (bogdando) on 2016-06-20

description:	updated
description:	updated

Bogdan Dobrelya (bogdando) on 2016-06-22

description:

updated

Maksym Strukov (unbelll) on 2016-06-22

tags:

added: on-verification

Revision history for this message

Maksym Strukov (unbelll) wrote on 2016-06-24:

#13

Verified as fixed in 9.0-mos-495

tags:

removed: on-verification

Revision history for this message

Vitaly Sedelnik (vsedelnik) wrote on 2016-09-07:

#14

Won't Fix for 6.1-, 7.0- and 8.0-updates as this is too risky change to be accepted to stable branch

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-21: Fix proposed to fuel-library (stable/7.0)

#15

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/374219

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-26: Fix merged to fuel-library (stable/7.0)

#16

Reviewed: https://review.openstack.org/374219
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f9a2d479f3687157d2b17a927a09ce5f995522d6
Submitter: Jenkins
Branch: stable/7.0

commit f9a2d479f3687157d2b17a927a09ce5f995522d6
Author: Denis Puchkin <email address hidden>
Date: Wed Sep 21 17:38:54 2016 +0300

Backport mysql OCF from stable/mitaka

backport mysql ocf script from stable/mitaka

    Closes-bug: #1524826
    Closes-bug: #1542256
    Closes-bug: #1572239
    Closes-bug: #1572557
    Closes-bug: #1572601
    Closes-bug: #1574747
    Closes-bug: #1574497
    Closes-bug: #1576244
    Closes-bug: #1574999
    Closes-bug: #1578278
    Closes-bug: #1388779
    Closes-bug: #1574999
    Closes-bug: #1576244
    Closes-bug: #1583173
    Closes-bug: #1585125

Change-Id: I1cc6f95884a8fbd5c3418ede89bdf9ec6864bdc8

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-27: Fix proposed to fuel-library (stable/8.0)

#17

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/377597