o/ just spent some time poking at this as my review is blocked on the newton ha at https://review.openstack.org/#/c/463529/

I think it *may* be related to these changes https://review.openstack.org/#/c/422489/ https://review.openstack.org/#/c/422484/ - the puppet-tripleo one i checked and it is in stable/ocata but not stable/newton.

It looks backportable but not sure how else that pacemaker.pp has changed in the meantime. I'll ask bandini to have a look here too.

from emiliens test ci review @ https://review.openstack.org/#/c/463763/ - some pointers + copy/paste for convenience (same in mine at /#/c/463529/)

------------
controller 2
http://logs.openstack.org/63/463763/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha/88ab5c3/logs/controller-2-tripleo-ci-c-baz/var/log/messages

May 10 22:32:47 localhost os-collect-config: [2017-05-10 22:32:47,183] (heat-config) [DEBUG] [2017-05-10 22:31:09,674] (heat-config) [DEBUG] Running FACTER_heat_outputs_path="/var/run/heat-config/heat-config-puppet/b452d1fd-2c09-42f8-8736-8cdba8af5f80"  FACTER_fqdn="controller-2-tripleo-ci-c-baz.localdomain"  FACTER_deploy_config_name="ControllerDeployment_Step1"  puppet apply --detailed-exitcodes --logdest console --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules /var/lib/heat-config/heat-config-puppet/b452d1fd-2c09-42f8-8736-8cdba8af5f80.pp
...
May 10 22:32:47 localhost os-collect-config: Error: unable to push cib
May 10 22:32:47 localhost os-collect-config: Call cib_apply_diff failed (-205): Update was older than existing configuration
May 10 22:32:47 localhost os-collect-config: #033[1;31mError: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20170510-25854-1r1fc8w create failed: #033[0m
May 10 22:32:47 localhost os-collect-config: #033[1;31mError: /Stage[main]/Pacemaker::Stonith/Pacemaker::Property[Disable STONITH]/Pcmk_property[property--stonith-enabled]/ensure: change from absent to present failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20170510-25854-1r1fc8w create failed: #033[0m
May 10 22:32:47 localhost os-collect-config: #033[1;31mWarning: /Firewall[998 log all]: Skipping because of failed dependencies#033[0m

------------



http://logs.openstack.org/63/463763/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha/88ab5c3/logs/controller-1-tripleo-ci-b-bar/var/log/messages

May 10 22:32:46 localhost cib[5815]:   error: epoch went backwards: 7 -> 6 (Opts: 0x0)
May 10 22:32:46 localhost cib[5815]: warning: Bad Op   <cib_command __name__="cib_command" t="cib" cib_async_id="d996a02c-0b8c-4441-95b1-a8baa57caf67" cib_op="cib_apply_diff" cib_callid="2" cib_callopt="0" cib_clientid="d996a02c-0b8c-4441-95b1-a8baa57caf67" cib_clientname="cibadmin" acl_target="root" cib_user="root" src="controller-1-tripleo-ci-b-bar" cib_delegated_from="controller-1-tripleo-ci-b-bar">


May 10 22:32:46 localhost os-collect-config: #033[mNotice: /Firewall[999 drop all]: Dependency Pcmk_property[property--stonith-enabled] has failures: true#033[0m
May 10 22:32:46 localhost os-collect-config: #033[mNotice: Finished catalog run in 110.51 seconds#033[0m
May 10 22:32:46 localhost os-collect-config: [2017-05-10 22:32:46,934] (heat-config) [INFO] exception: connect failed
May 10 22:32:46 localhost os-collect-config: #033[1;31mWarning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.#033[0m
May 10 22:32:46 localhost os-collect-config: Error: unable to push cib
May 10 22:32:46 localhost os-collect-config: Call cib_apply_diff failed (-205): Update was older than existing configuration
May 10 22:32:46 localhost os-collect-config: #033[1;31mError: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20170510-1967-4605wr create failed: #033[0m
May 10 22:32:46 localhost os-collect-config: #033[1;31mError: /Stage[main]/Pacemaker::Stonith/Pacemaker::Property[Disable STONITH]/Pcmk_property[property--stonith-enabled]/ensure: change from absent to present failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20170510-1967-4605wr create failed: #033[0m
May 10 22:32:46 localhost os-collect-config: #033[1;31mWarning: /Firewall[998 log all]: Skipping because of failed dependencies#033[0m
May 10 22:32:46 localhost os-collect-config: #033[1;31mWarning: /Firewall[999 drop all]: Skipping because of failed dependencies#033[0m
May 10 22:32:46 localhost os-collect-config: [2017-05-10 22:32:46,934] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/081584e2-a737-4ca3-a022-c20b83938e71.pp. [6]
May 10 22:32:46 localhost os-collect-config: [2017-05-10 22:32:46,939] (heat-config) [INFO] Completed /usr/libexec/heat-config/hooks/puppet