| 2015-12-15 11:18:15 |
Adam Collard |
bug |
|
|
added bug |
| 2015-12-15 11:18:15 |
Adam Collard |
attachment added |
|
Juju status https://bugs.launchpad.net/bugs/1526271/+attachment/4535024/+files/juju.status |
|
| 2015-12-15 11:18:38 |
Adam Collard |
attachment added |
|
netstat -tunlp https://bugs.launchpad.net/charms/+source/hacluster/+bug/1526271/+attachment/4535025/+files/keystone.netstat |
|
| 2015-12-15 11:19:02 |
Adam Collard |
attachment added |
|
juju run of crm status https://bugs.launchpad.net/charms/+source/hacluster/+bug/1526271/+attachment/4535026/+files/keystone.crm.status |
|
| 2015-12-15 11:28:05 |
Adam Collard |
bug |
|
|
added subscriber Landscape |
| 2015-12-15 11:28:12 |
Adam Collard |
tags |
|
kanban-cross-team |
|
| 2015-12-15 11:51:35 |
🤖 Landscape Builder |
tags |
kanban-cross-team |
|
|
| 2015-12-15 11:54:31 |
Adam Collard |
attachment added |
|
Juju unit log of hacluster-keystone/0 https://bugs.launchpad.net/charms/+source/hacluster/+bug/1526271/+attachment/4535047/+files/hacluster-keystone.log |
|
| 2015-12-15 11:54:50 |
Adam Collard |
attachment added |
|
/var/log/syslog on hacluster-keystone/0 https://bugs.launchpad.net/charms/+source/hacluster/+bug/1526271/+attachment/4535048/+files/hacluster-keystone-syslog.log |
|
| 2015-12-15 12:19:20 |
Adam Collard |
description |
Using Juju 1.24.7, cs:trusty/hacluster-26, with attached Juju status.
hacluster unit log shows that it failed to configure the no-quorum-policy, which then lead to no haproxy running on the unit which was the leader and has the VIP. The symptom I saw was getting "Connection refused" when talking to the VIP.
2015-12-15 00:19:22 DEBUG juju-log hanode:15: Ready to form cluster - informing peers
2015-12-15 00:19:22 DEBUG juju-log hanode:15: Parsing cluster configuration using rid: ha:26, unit: keystone/0
2015-12-15 00:19:23 DEBUG juju-log hanode:15: Configuring and (maybe) restarting corosync
2015-12-15 00:19:23 INFO juju-log hanode:15: Writing file /etc/default/corosync root:root 444
2015-12-15 00:19:23 INFO juju-log hanode:15: Writing file /etc/corosync/uidgid.d/hacluster root:root 444
2015-12-15 00:19:23 INFO juju-log hanode:15: Writing file /etc/corosync/authkey root:root 400
2015-12-15 00:19:23 INFO juju-log hanode:15: Writing file /etc/corosync/corosync.conf root:root 444
2015-12-15 00:19:23 INFO hanode-relation-changed * Restarting corosync daemon corosync
2015-12-15 00:19:23 INFO hanode-relation-changed notice [MAIN ] Corosync Cluster Engine ('2.3.3'): started and ready to provide service.
2015-12-15 00:19:23 INFO hanode-relation-changed info [MAIN ] Corosync built-in features: dbus testagents rdma watchdog augeas pie relro bindnow
2015-12-15 00:19:23 INFO hanode-relation-changed ...done.
2015-12-15 00:19:28 INFO hanode-relation-changed Starting Pacemaker Cluster Manager: [ OK ]
2015-12-15 00:19:28 DEBUG juju-log hanode:15: Applying global cluster configuration
2015-12-15 00:19:28 DEBUG juju-log hanode:15: Configuring no-quorum-policy to stop
2015-12-15 00:19:28 INFO hanode-relation-changed Call cib_apply_diff failed (-206): Application of an update diff failed
2015-12-15 00:19:28 INFO hanode-relation-changed ERROR: could not patch cib (rc=206)
2015-12-15 00:19:28 INFO hanode-relation-changed INFO: offending xml diff: <diff crm_feature_set="3.0.7">
2015-12-15 00:19:28 INFO hanode-relation-changed <diff-removed>
2015-12-15 00:19:28 INFO hanode-relation-changed <cib epoch="22"/>
2015-12-15 00:19:28 INFO hanode-relation-changed </diff-removed>
2015-12-15 00:19:28 INFO hanode-relation-changed <diff-added>
2015-12-15 00:19:28 INFO hanode-relation-changed <cib epoch="23" num_updates="10" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.7" cib-last-written="Tue Dec 15 00:19:29 2015" update-origin="juju-machine-0-lxc-8" update-client="cibadmin" have-quorum="1" dc-uuid="1002"/>
2015-12-15 00:19:28 INFO hanode-relation-changed </diff-added>
2015-12-15 00:19:28 INFO hanode-relation-changed </diff>
2015-12-15 00:19:28 INFO hanode-relation-changed
2015-12-15 00:19:28 INFO hanode-relation-changed
2015-12-15 00:19:29 DEBUG juju-log hanode:15: Checking monitor host configuration
2015-12-15 00:19:29 INFO juju-log hanode:15: Disabling STONITH
2015-12-15 00:19:29 DEBUG juju-log hanode:15: Deleting Resources
2015-12-15 00:19:29 DEBUG juju-log hanode:15: Configuring Resources: {'res_ks_eth0_vip': 'ocf:heartbeat:IPaddr2', 'res_ks_haproxy': 'lsb:haproxy'}
2015-12-15 00:19:29 INFO hanode-relation-changed Removing any system startup links for /etc/init.d/haproxy ...
2015-12-15 00:19:29 INFO hanode-relation-changed /etc/rc0.d/K20haproxy
2015-12-15 00:19:29 INFO hanode-relation-changed /etc/rc1.d/K20haproxy
2015-12-15 00:19:29 INFO hanode-relation-changed /etc/rc2.d/S20haproxy
2015-12-15 00:19:29 INFO hanode-relation-changed /etc/rc3.d/S20haproxy
2015-12-15 00:19:29 INFO hanode-relation-changed /etc/rc4.d/S20haproxy
2015-12-15 00:19:29 INFO hanode-relation-changed /etc/rc5.d/S20haproxy
2015-12-15 00:19:29 INFO hanode-relation-changed /etc/rc6.d/K20haproxy
2015-12-15 00:19:29 INFO hanode-relation-changed * Stopping haproxy haproxy
2015-12-15 00:19:29 INFO hanode-relation-changed ...done.
--
keystone/2 has a running haproxy and is behaving as expected |
Using Juju 1.24.7, cs:trusty/hacluster-26, with attached Juju status.
hacluster unit log shows that it failed to configure the no-quorum-policy, which then lead to no haproxy running on the unit which was the leader and has the VIP. The symptom I saw was getting "Connection refused" when talking to the VIP.
2015-12-15 00:19:22 DEBUG juju-log hanode:15: Ready to form cluster - informing peers
2015-12-15 00:19:22 DEBUG juju-log hanode:15: Parsing cluster configuration using rid: ha:26, unit: keystone/0
2015-12-15 00:19:23 DEBUG juju-log hanode:15: Configuring and (maybe) restarting corosync
2015-12-15 00:19:23 INFO juju-log hanode:15: Writing file /etc/default/corosync root:root 444
2015-12-15 00:19:23 INFO juju-log hanode:15: Writing file /etc/corosync/uidgid.d/hacluster root:root 444
2015-12-15 00:19:23 INFO juju-log hanode:15: Writing file /etc/corosync/authkey root:root 400
2015-12-15 00:19:23 INFO juju-log hanode:15: Writing file /etc/corosync/corosync.conf root:root 444
2015-12-15 00:19:23 INFO hanode-relation-changed * Restarting corosync daemon corosync
2015-12-15 00:19:23 INFO hanode-relation-changed notice [MAIN ] Corosync Cluster Engine ('2.3.3'): started and ready to provide service.
2015-12-15 00:19:23 INFO hanode-relation-changed info [MAIN ] Corosync built-in features: dbus testagents rdma watchdog augeas pie relro bindnow
2015-12-15 00:19:23 INFO hanode-relation-changed ...done.
2015-12-15 00:19:28 INFO hanode-relation-changed Starting Pacemaker Cluster Manager: [ OK ]
2015-12-15 00:19:28 DEBUG juju-log hanode:15: Applying global cluster configuration
2015-12-15 00:19:28 DEBUG juju-log hanode:15: Configuring no-quorum-policy to stop
2015-12-15 00:19:28 INFO hanode-relation-changed Call cib_apply_diff failed (-206): Application of an update diff failed
2015-12-15 00:19:28 INFO hanode-relation-changed ERROR: could not patch cib (rc=206)
2015-12-15 00:19:28 INFO hanode-relation-changed INFO: offending xml diff: <diff crm_feature_set="3.0.7">
2015-12-15 00:19:28 INFO hanode-relation-changed <diff-removed>
2015-12-15 00:19:28 INFO hanode-relation-changed <cib epoch="22"/>
2015-12-15 00:19:28 INFO hanode-relation-changed </diff-removed>
2015-12-15 00:19:28 INFO hanode-relation-changed <diff-added>
2015-12-15 00:19:28 INFO hanode-relation-changed <cib epoch="23" num_updates="10" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.7" cib-last-written="Tue Dec 15 00:19:29 2015" update-origin="juju-machine-0-lxc-8" update-client="cibadmin" have-quorum="1" dc-uuid="1002"/>
2015-12-15 00:19:28 INFO hanode-relation-changed </diff-added>
2015-12-15 00:19:28 INFO hanode-relation-changed </diff>
2015-12-15 00:19:28 INFO hanode-relation-changed
2015-12-15 00:19:28 INFO hanode-relation-changed
2015-12-15 00:19:29 DEBUG juju-log hanode:15: Checking monitor host configuration
2015-12-15 00:19:29 INFO juju-log hanode:15: Disabling STONITH
2015-12-15 00:19:29 DEBUG juju-log hanode:15: Deleting Resources
2015-12-15 00:19:29 DEBUG juju-log hanode:15: Configuring Resources: {'res_ks_eth0_vip': 'ocf:heartbeat:IPaddr2', 'res_ks_haproxy': 'lsb:haproxy'}
2015-12-15 00:19:29 INFO hanode-relation-changed Removing any system startup links for /etc/init.d/haproxy ...
2015-12-15 00:19:29 INFO hanode-relation-changed /etc/rc0.d/K20haproxy
2015-12-15 00:19:29 INFO hanode-relation-changed /etc/rc1.d/K20haproxy
2015-12-15 00:19:29 INFO hanode-relation-changed /etc/rc2.d/S20haproxy
2015-12-15 00:19:29 INFO hanode-relation-changed /etc/rc3.d/S20haproxy
2015-12-15 00:19:29 INFO hanode-relation-changed /etc/rc4.d/S20haproxy
2015-12-15 00:19:29 INFO hanode-relation-changed /etc/rc5.d/S20haproxy
2015-12-15 00:19:29 INFO hanode-relation-changed /etc/rc6.d/K20haproxy
2015-12-15 00:19:29 INFO hanode-relation-changed * Stopping haproxy haproxy
2015-12-15 00:19:29 INFO hanode-relation-changed ...done.
--
keystone/2 has a running haproxy and is behaving as expected
Error looks remarkably similar to https://bugs.launchpad.net/fuel/+bug/1363908 where they moved away from using crm:
"We need to use cibadmin -P instead of crm to avoid such problems as it can lead to cluster in unconfigured state and to following problems with cluster scalability and failover." |
|
| 2015-12-15 14:36:18 |
David Britton |
bug task added |
|
haproxy (Ubuntu) |
|
| 2015-12-15 14:56:27 |
James Page |
nominated for series |
|
Ubuntu Wily |
|
| 2015-12-15 14:56:27 |
James Page |
bug task added |
|
haproxy (Ubuntu Wily) |
|
| 2015-12-15 14:56:27 |
James Page |
nominated for series |
|
Ubuntu Trusty |
|
| 2015-12-15 14:56:27 |
James Page |
bug task added |
|
haproxy (Ubuntu Trusty) |
|
| 2015-12-15 14:56:27 |
James Page |
nominated for series |
|
Ubuntu Xenial |
|
| 2015-12-15 14:56:27 |
James Page |
bug task added |
|
haproxy (Ubuntu Xenial) |
|
| 2015-12-15 14:56:27 |
James Page |
nominated for series |
|
Ubuntu Vivid |
|
| 2015-12-15 14:56:27 |
James Page |
bug task added |
|
haproxy (Ubuntu Vivid) |
|
| 2015-12-15 14:56:33 |
James Page |
haproxy (Ubuntu Xenial): status |
New |
Fix Released |
|
| 2015-12-15 14:56:35 |
James Page |
haproxy (Ubuntu Wily): importance |
Undecided |
Critical |
|
| 2015-12-15 14:56:37 |
James Page |
haproxy (Ubuntu Trusty): importance |
Undecided |
Critical |
|
| 2015-12-15 14:56:40 |
James Page |
haproxy (Ubuntu Vivid): importance |
Undecided |
Critical |
|
| 2015-12-15 14:56:42 |
James Page |
haproxy (Ubuntu Xenial): importance |
Undecided |
Critical |
|
| 2015-12-15 14:58:25 |
James Page |
haproxy (Ubuntu Wily): status |
New |
Invalid |
|
| 2015-12-15 14:58:29 |
James Page |
haproxy (Ubuntu Trusty): status |
New |
Confirmed |
|
| 2015-12-15 15:02:33 |
James Page |
haproxy (Ubuntu Vivid): status |
New |
Invalid |
|
| 2015-12-15 15:08:45 |
James Page |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2015-12-15 15:08:50 |
James Page |
haproxy (Ubuntu Trusty): status |
Confirmed |
In Progress |
|
| 2015-12-15 15:08:52 |
James Page |
haproxy (Ubuntu Trusty): assignee |
|
James Page (james-page) |
|
| 2015-12-15 15:08:57 |
James Page |
hacluster (Juju Charms Collection): status |
New |
Invalid |
|
| 2015-12-15 16:29:32 |
Adam Conrad |
haproxy (Ubuntu Trusty): status |
In Progress |
Fix Committed |
|
| 2015-12-15 16:29:38 |
Adam Conrad |
bug |
|
|
added subscriber SRU Verification |
| 2015-12-15 16:29:41 |
Adam Conrad |
tags |
|
verification-needed |
|
| 2015-12-15 19:45:36 |
David Britton |
tags |
verification-needed |
verification-done |
|
| 2015-12-15 19:55:36 |
Launchpad Janitor |
haproxy (Ubuntu Trusty): status |
Fix Committed |
Fix Released |
|
| 2015-12-15 19:55:40 |
Adam Conrad |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
