distro-proposed(trusty-icehouse-proposed)
=========================================

ubuntu@maas-sailor-keystone-p78p:~$ sudo crm status
Last updated: Thu Sep 25 06:20:06 2014
Last change: Thu Sep 25 06:19:44 2014 via crmd on maas-sailor-keystone-h5t9
Stack: corosync
Current DC: maas-sailor-keystone-p78p (173080679) - partition with quorum
Version: 1.1.10-42f2063
2 Nodes configured
5 Resources configured


Online: [ maas-sailor-keystone-h5t9 maas-sailor-keystone-p78p ]

 Clone Set: cl_ping [ping]
     Started: [ maas-sailor-keystone-h5t9 maas-sailor-keystone-p78p ]
 res_ks_vip     (ocf::heartbeat:IPaddr2):       Started maas-sailor-keystone-p78p 
 Clone Set: cl_ks_haproxy [res_ks_haproxy]
     Started: [ maas-sailor-keystone-h5t9 maas-sailor-keystone-p78p ]


ubuntu@maas-sailor-keystone-h5t9:~$ cat <<"EOF" | tee repeat-standby-online.sh

set -e

count=$1

for i in `seq 1 "$count"`; do
    echo $i
    crm node standby
    crm node online
done

EOF

ubuntu@maas-sailor-keystone-h5t9:~$ sudo sh repeat-standby-online.sh 10000

-> run 761 times
-> get /var/crash/_usr_lib_pacemaker_lrmd.0.crash


upgrade to PPA
==============

## upgrade the second node

ubuntu@maas-sailor-keystone-h5t9:~$ sudo crm node standby

ubuntu@maas-sailor-keystone-h5t9:~$ sudo apt-add-repository ppa:inaddy/lp1368737

ubuntu@maas-sailor-keystone-h5t9:~$ sudo apt-get update

ubuntu@maas-sailor-keystone-h5t9:~$ sudo apt-get install \
    libcib3 libcrmcluster4 libcrmcommon3 libcrmservice1 liblrmd1 libpengine4 libpe-rules2 \
    libpe-status4 libstonithd2 libtransitioner2 pacemaker pacemaker-cli-utils

ubuntu@maas-sailor-keystone-h5t9:~$ sudo reboot

ubuntu@maas-sailor-keystone-h5t9:~$ sudo crm node online

## upgrade the first node

ubuntu@maas-sailor-keystone-p78p:~$ sudo crm node standby

ubuntu@maas-sailor-keystone-p78p:~$ sudo apt-add-repository ppa:inaddy/lp1368737

ubuntu@maas-sailor-keystone-p78p:~$ sudo apt-get update

ubuntu@maas-sailor-keystone-p78p:~$ sudo apt-get install \
    libcib3 libcrmcluster4 libcrmcommon3 libcrmservice1 liblrmd1 libpengine4 libpe-rules2 \
    libpe-status4 libstonithd2 libtransitioner2 pacemaker pacemaker-cli-utils

ubuntu@maas-sailor-keystone-p78p:~$ sudo reboot

ubuntu@maas-sailor-keystone-p78p:~$ sudo crm node online



## repeat again

ubuntu@maas-sailor-keystone-h5t9:~$ sudo rm /var/crash/_*

ubuntu@maas-sailor-keystone-h5t9:~$ sudo crm node standby

-> unexpectedly got /var/crash/_usr_lib_pacemaker_lrmd.0.crash