bootstrap-pxc fails on zesty

[Ryan Beisner]

Following this commit to address cgroup issue, percona cluster is failing to bootstrap-pxc on Zesty mojo specs and Zesty lp:openstack-charm-testing stable + next bundles. Prior to this commit, pxc (mysql) units deployed to a ready non-error state.

https://github.com/openstack/charm-percona-cluster/commit/fddc1b78f4251db97129f5246fff92fefb18353f

mysql/0 error idle 1.25.10 10 172.17.108.89 hook failed: "config-changed"

2017-02-28 23:07:26 INFO config-changed Running as unit: run-r5d7d6a4b2da2425da1806e508cfc554b.service
2017-02-28 23:07:26 INFO config-changed Press ^] three times within 1s to disconnect TTY.
2017-02-28 23:07:26 INFO config-changed * Bootstrapping Percona XtraDB Cluster database server mysqld
2017-02-28 23:07:26 INFO config-changed ...done.
2017-02-28 23:07:26 INFO worker.uniter.jujuc server.go:172 running hook tool "status-set" ["blocked" "Bootstrap PXC failed"]
2017-02-28 23:07:26 DEBUG worker.uniter.jujuc server.go:173 hook context id "mysql/0-config-changed-6601568812192966830"; dir "/var/lib/juju/agents/unit-mysql-0/charm"
2017-02-28 23:07:26 DEBUG juju.worker.uniter.context context.go:326 [WORKLOAD-STATUS] blocked: Bootstrap PXC failed
2017-02-28 23:07:27 INFO worker.uniter.jujuc server.go:172 running hook tool "juju-log" ["-l" "ERROR" "Bootstrap PXC failed: Command '['systemd-run', '-t', '--service-type=forking', 'service', 'mysql', 'bootstrap-pxc']' returned non-zero exit status 1"]
2017-02-28 23:07:27 DEBUG worker.uniter.jujuc server.go:173 hook context id "mysql/0-config-changed-6601568812192966830"; dir "/var/lib/juju/agents/unit-mysql-0/charm"
2017-02-28 23:07:27 ERROR juju-log Bootstrap PXC failed: Command '['systemd-run', '-t', '--service-type=forking', 'service', 'mysql', 'bootstrap-pxc']' returned non-zero exit status 1
2017-02-28 23:07:27 INFO config-changed Traceback (most recent call last):
2017-02-28 23:07:27 INFO config-changed File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/config-changed", line 709, in <module>
2017-02-28 23:07:27 INFO config-changed main()
2017-02-28 23:07:27 INFO config-changed File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/config-changed", line 701, in main
2017-02-28 23:07:27 INFO config-changed hooks.execute(sys.argv)
2017-02-28 23:07:27 INFO config-changed File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/charmhelpers/core/hookenv.py", line 731, in execute
2017-02-28 23:07:27 INFO config-changed self._hooks[hook_name]()
2017-02-28 23:07:27 INFO config-changed File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2017-02-28 23:07:27 INFO config-changed return f(*args, **kwargs)
2017-02-28 23:07:27 INFO config-changed File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/config-changed", line 278, in config_changed
2017-02-28 23:07:27 INFO config-changed bootstrap=not bootstrapped)
2017-02-28 23:07:27 INFO config-changed File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/config-changed", line 190, in render_config_restart_on_changed
2017-02-28 23:07:27 INFO config-changed bootstrap_pxc()
2017-02-28 23:07:27 INFO config-changed File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/percona_utils.py", line 401, in bootstrap_pxc
2017-02-28 23:07:27 INFO config-changed raise Exception(error_msg)
2017-02-28 23:07:27 INFO config-changed Exception: Bootstrap PXC failed: Command '['systemd-run', '-t', '--service-type=forking', 'service', 'mysql', 'bootstrap-pxc']' returned non-zero exit status 1
2017-02-28 23:07:27 INFO juju.worker.uniter.context context.go:579 handling reboot
2017-02-28 23:07:27 ERROR juju.worker.uniter.operation runhook.go:107 hook "config-changed" failed: exit status 1

[James Page (james-page)]

Looks like systemd-run behaves a little differently on zesty - dropping the -t flag does the trick.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/439443
Committed: https://git.openstack.org/cgit/openstack/charm-percona-cluster/commit/?id=1cae1942d451e0daf9681b2c823643c42565bc33
Submitter: Jenkins
Branch: master

commit 1cae1942d451e0daf9681b2c823643c42565bc33
Author: James Page <email address hidden>
Date: Wed Mar 1 08:15:24 2017 +0000

    Drop capture of pty for bootstrap-pxc

    Use of the '-t' flag to capture the output of the pty results
    in a non-zero return code in later systemd/Ubuntu releases
    (specifically zesty).

    Drop use of this flag for broader compatibility.

    Change-Id: Ib204c073defdc1977aabf2304d9e518a99976f98
    Closes-Bug: 1668833

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/17.02
Review: https://review.openstack.org/440278

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/440278
Committed: https://git.openstack.org/cgit/openstack/charm-percona-cluster/commit/?id=189e5377e0b2917cb663eec3611071e5853707b5
Submitter: Jenkins
Branch: stable/17.02

commit 189e5377e0b2917cb663eec3611071e5853707b5
Author: James Page <email address hidden>
Date: Tue Feb 28 11:57:43 2017 +0100

    Ensure bootstrap-pxc mysqld not in unit cgroup

    The bootstrap process for percona-xtradb-cluster requires execution
    of a non-standard init.d scrip target to start the mysqld in wsrep
    new cluster mode.

    The processes started by the operation where ending up in the cgroup
    associated with the Juju unit daemon, which on restart (as a result
    of a upgrade to juju for example) would result in the mysql daemon
    being killed and not restarted.

    Use systemd-run to ensure that the bootstrap-pxc operation ends up
    in a distinct cgroup so that this does not happen.

    Drop capture of pty for bootstrap-pxc

    Use of the '-t' flag to capture the output of the pty results
    in a non-zero return code in later systemd/Ubuntu releases
    (specifically zesty).

    Drop use of this flag for broader compatibility.

    Change-Id: Iff998c4c23fcad71cffe9bbee60df7f00d2c9893
    Closes-Bug: 1664025
    Closes-Bug: 1668833
    (cherry picked from commit fddc1b78f4251db97129f5246fff92fefb18353f)
    (cherry picked from commit 1cae1942d451e0daf9681b2c823643c42565bc33)