galera_server - systemd handler should be first

[Kyle L. Henderson]

While deploying OSA stable/newton on a three controller node cluster using Xenial, a co-worker accidentally pulled the power on one of the controller nodes. I restarted the setup-infrastructure playbook to restart the deployment where the errors started. I hit an issue where galera wouldn't start and gave an error message:

fatal: [XXXXX_galera_container-55342560]: FAILED! => {"changed": false, "failed": true, "msg": "Warning: mysql.service changed on disk. Run 'systemctl daemon-reload' to reload units.\nWarning: mysql.service changed on disk. Run 'systemctl daemon-reload' to reload units.\nJob for mysql.service failed because the control process exited with error c
ode. See \"systemctl status mysql.service\" and \"journalctl -xe\" for details.\n"}

The next task run was to reload systemd. It seems reloading systemd should come before attempting to restart galera.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/386604

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/386604
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-galera_server/commit/?id=678128f13ad22308819c83a208beeec8ea109b0f
Submitter: Jenkins
Branch: master

commit 678128f13ad22308819c83a208beeec8ea109b0f
Author: Kyle L. Henderson <email address hidden>
Date: Thu Oct 13 19:50:57 2016 -0500

    Move systemd handler above restart handler

    Since handlers are executed in the order in which they are defined
    the handler to restart systemd should come before the handlers to
    restart mysql. This avoids the error case where the playbook is run
    multiple times and systemd refuses to restart mysql giving the
    following message:

    "Warning: mysql.service changed on disk. Run 'systemctl
    daemon-reload' to reload units."

    Closes-Bug: #1633472

    Change-Id: I3962c0e8f9f1cf3c01a3160be4c744e9fbd1ecfe

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/386768

[Chris Martin (6-chris-z)]

I experienced the same error on a greenfield OSA deployment with 3 infrastructure nodes and no power issues. stable/newton pulled on Wednesday (commit f1043795cf2bcf234d22a7824e8ac96c1c3fae64).

Order of events:

1. Ran setup-infrastructure.yml for the first time. Completed successfully on infra1, failed on infra2 with the above error reported by kyleh.

2. Applied this patch and re-ran setup-infrastructure.yml. Completed successfully on infra1 and infra2, failed on infra3 with a different error:

fatal: [jet03_galera_container-c0d5f771]: FAILED! => {"changed": false, "failed": true, "msg": "Job for mysql.service failed because the control process exited with error code. See \"systemctl status mysql.service\" and \"journalctl -xe\" for details.\n"}

3. Re-ran setup-infrastructure.yml again, completed successfully on all 3 infrastructure nodes.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/386768
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-galera_server/commit/?id=09276e47e99f52b2cb0e7749051741d702328e71
Submitter: Jenkins
Branch: stable/newton

commit 09276e47e99f52b2cb0e7749051741d702328e71
Author: Kyle L. Henderson <email address hidden>
Date: Thu Oct 13 19:50:57 2016 -0500

    Move systemd handler above restart handler

    Since handlers are executed in the order in which they are defined
    the handler to restart systemd should come before the handlers to
    restart mysql. This avoids the error case where the playbook is run
    multiple times and systemd refuses to restart mysql giving the
    following message:

    "Warning: mysql.service changed on disk. Run 'systemctl
    daemon-reload' to reload units."

    Closes-Bug: #1633472

    Change-Id: I3962c0e8f9f1cf3c01a3160be4c744e9fbd1ecfe
    (cherry picked from commit 678128f13ad22308819c83a208beeec8ea109b0f)

[Kyle L. Henderson (kyleh)]

Chris, I've seen the same thing that you reported in comment #4. It doesn't seem directly related to this bug, so after discussing with Jessie and others on IRC, I opened: https://bugs.launchpad.net/openstack-ansible/+bug/1634562

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-galera_server 14.0.0.0rc4 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-galera_server 15.0.0.0b1 development milestone.