galera mysql instances are not restarted after an update to the config

Bug #1449958 reported by Serge van Ginderachter
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack-Ansible
Fix Released
High
git-harry
Juno
Won't Fix
High
Unassigned
Kilo
Fix Released
High
Jesse Pretorius
Trunk
Fix Released
High
git-harry

Bug Description

Let the output of ansible speak for itself, my.cnf was updated on the three instances, but none of the mysql servers were restarted.

OSAD version 10.1.3

TASK: [galera_config | Drop mariadb configs] **********************************
changed: [dc2-rk1-ch1-bl1_galera_container-8d20af1b] => (item={'dest': '/etc/mysql/my.cnf', 'src': 'my.cnf'})
changed: [dc2-rk1-ch1-bl2_galera_container-d479d417] => (item={'dest': '/etc/mysql/my.cnf', 'src': 'my.cnf'})
changed: [dc2-rk4-ch1-bl1_galera_container-f2b10932] => (item={'dest': '/etc/mysql/my.cnf', 'src': 'my.cnf'})
ok: [dc2-rk1-ch1-bl1_galera_container-8d20af1b] => (item={'dest': '/etc/mysql/conf.d/cluster.cnf', 'src': 'cluster.cnf'})
ok: [dc2-rk1-ch1-bl2_galera_container-d479d417] => (item={'dest': '/etc/mysql/conf.d/cluster.cnf', 'src': 'cluster.cnf'})
changed: [dc2-rk4-ch1-bl1_galera_container-f2b10932] => (item={'dest': '/etc/mysql/conf.d/cluster.cnf', 'src': 'cluster.cnf'})
ok: [dc2-rk1-ch1-bl1_galera_container-8d20af1b] => (item={'dest': '/etc/mysql/debian.cnf', 'src': 'debian.cnf'})
ok: [dc2-rk1-ch1-bl2_galera_container-d479d417] => (item={'dest': '/etc/mysql/debian.cnf', 'src': 'debian.cnf'})
ok: [dc2-rk4-ch1-bl1_galera_container-f2b10932] => (item={'dest': '/etc/mysql/debian.cnf', 'src': 'debian.cnf'})
ok: [dc2-rk1-ch1-bl1_galera_container-8d20af1b] => (item={'dest': '/etc/init.d/mysql', 'src': 'mysql_init.sh'})
ok: [dc2-rk1-ch1-bl2_galera_container-d479d417] => (item={'dest': '/etc/init.d/mysql', 'src': 'mysql_init.sh'})
ok: [dc2-rk4-ch1-bl1_galera_container-f2b10932] => (item={'dest': '/etc/init.d/mysql', 'src': 'mysql_init.sh'})
ok: [dc2-rk1-ch1-bl1_galera_container-8d20af1b] => (item={'dest': '/etc/default/mysql', 'src': 'mysql_defaults'})
ok: [dc2-rk1-ch1-bl2_galera_container-d479d417] => (item={'dest': '/etc/default/mysql', 'src': 'mysql_defaults'})
ok: [dc2-rk4-ch1-bl1_galera_container-f2b10932] => (item={'dest': '/etc/default/mysql', 'src': 'mysql_defaults'})

PLAY [galera[0]] **************************************************************

TASK: [galera_bootstrap | Check wsrep cluster size] ***************************
changed: [dc2-rk1-ch1-bl1_galera_container-8d20af1b]

TASK: [galera_bootstrap | Stop mariadb] ***************************************
skipping: [dc2-rk1-ch1-bl1_galera_container-8d20af1b]

TASK: [galera_bootstrap | Check if mysql is running] **************************
changed: [dc2-rk1-ch1-bl1_galera_container-8d20af1b]

TASK: [galera_bootstrap | bootstrap cluster (initialise mysql with --wsrep-new-cluster)] ***
skipping: [dc2-rk1-ch1-bl1_galera_container-8d20af1b]

PLAY [galera:!galera[0]] ******************************************************

TASK: [galera_restart | Get incoming addresses from cluster] ******************
changed: [dc2-rk4-ch1-bl1_galera_container-f2b10932]

TASK: [galera_restart | Restart mysql] ****************************************
skipping: [dc2-rk4-ch1-bl1_galera_container-f2b10932]

TASK: [galera_restart | Get incoming addresses from cluster] ******************
changed: [dc2-rk1-ch1-bl2_galera_container-d479d417]

TASK: [galera_restart | Restart mysql] ****************************************
skipping: [dc2-rk1-ch1-bl2_galera_container-d479d417]

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to os-ansible-deployment (master)

Fix proposed to branch: master
Review: https://review.openstack.org/200054

Changed in openstack-ansible:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to os-ansible-deployment (master)

Reviewed: https://review.openstack.org/200054
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=4a860f715c21be26b5b4e92f0d9f75e665015469
Submitter: Jenkins
Branch: master

commit 4a860f715c21be26b5b4e92f0d9f75e665015469
Author: git-harry <email address hidden>
Date: Wed Jul 8 10:31:45 2015 +0100

    Restart mysql when config changed

    Currently mysql is only (re)started when creating a cluster, adding a
    node or the service isn't running. This means changes to configuration
    files are not used.

    This commit moves the restart functionality to a handler and sets
    notifies on the the appropriate tasks so that when the config is changes
    mysql is restarted.

    This commit modifies the galera play to run in serial so that only one
    instance of mysql can be restarted at a time. This is to prevent the
    possibility of them all being restarted in one go. An additional play
    'Check galera cluster status' has been added, this makes an attempt to
    check that the cluster is healthy before proceeding with any possible
    config changes. The checks can be overridden by setting
    'galera_ignore_cluster_state=true' when running the playbook.

    A facts module has been added to gather information about the status of
    the cluster for use by the galera-install playbook.

    DocImpact
    Closes-bug: #1449958

    Change-Id: Id83ba642c114d1f5cac04cc219be151a82a1023f

Changed in openstack-ansible:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to os-ansible-deployment (kilo)

Reviewed: https://review.openstack.org/204545
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=8fb117607064bbb325f92bd33ce2bfaecbb555f6
Submitter: Jenkins
Branch: kilo

commit 8fb117607064bbb325f92bd33ce2bfaecbb555f6
Author: git-harry <email address hidden>
Date: Wed Jul 8 10:31:45 2015 +0100

    Restart mysql when config changed

    Currently mysql is only (re)started when creating a cluster, adding a
    node or the service isn't running. This means changes to configuration
    files are not used.

    This commit moves the restart functionality to a handler and sets
    notifies on the the appropriate tasks so that when the config is changes
    mysql is restarted.

    This commit modifies the galera play to run in serial so that only one
    instance of mysql can be restarted at a time. This is to prevent the
    possibility of them all being restarted in one go. An additional play
    'Check galera cluster status' has been added, this makes an attempt to
    check that the cluster is healthy before proceeding with any possible
    config changes. The checks can be overridden by setting
    'galera_ignore_cluster_state=true' when running the playbook.

    A facts module has been added to gather information about the status of
    the cluster for use by the galera-install playbook.

    DocImpact
    Closes-bug: #1449958

    Change-Id: Id83ba642c114d1f5cac04cc219be151a82a1023f
    (cherry picked from commit 4a860f715c21be26b5b4e92f0d9f75e665015469)

Revision history for this message
Jesse Pretorius (jesse-pretorius) wrote :

As this would involve a large scale change of the current Galera playbooks in the stable Juno branch, this is set to "won't fix" for Juno. Anyone concerned with this issue on Juno is advised to upgrade to Kilo.

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote : Fix included in openstack/openstack-ansible 11.2.14

This issue was fixed in the openstack/openstack-ansible 11.2.14 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Related blueprints

Remote bug watches

Bug watches keep track of this bug in other bug trackers.