ovb/ipv6 timeout: Error: unable to push cib

[Emilien Macchi]

OVB HA IPV6 randomly timeouts on Pacemaker setup at step 2:

https://logs.rdoproject.org/52/526152/2/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/Zc82f7d56a5fd452192f6d6426c191787/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2017-12-07_23_36_20

Visible in pacemaker logs:
https://logs.rdoproject.org/52/526152/2/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/Zc82f7d56a5fd452192f6d6426c191787/overcloud-controller-0/var/log/journal.txt.gz#_Dec_07_23_12_33

Watchdog may be enabled but stonith-watchdog-timeout is disabled: (null)
notice: Node overcloud-controller-1 state is now lost

[Michele Baldessari (michele)]

So we absolutely need to add /var/log/cluster to what we collect from the rdo jobs (we used to collect
it in tripleo CI), otherwise things get particularly hard.

In any case this is a networking issue of some sort. From:
https://logs.rdoproject.org/52/526152/2/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/Zc82f7d56a5fd452192f6d6426c191787/overcloud-controller-1/var/log/journal.txt.gz we can see that corosync is losing the totem token that is being passed
around:

Dec 07 22:55:30 overcloud-controller-1 cib[18522]: notice: Node overcloud-controller-0 state is now member
Dec 07 22:55:30 overcloud-controller-1 attrd[18525]: notice: Node overcloud-controller-0 state is now member
Dec 07 22:55:30 overcloud-controller-1 stonith-ng[18523]: notice: Node overcloud-controller-0 state is now member
Dec 07 22:55:30 overcloud-controller-1 corosync[18514]: [TOTEM ] Retransmit List: 1a 1b
Dec 07 22:55:30 overcloud-controller-1 corosync[18514]: [TOTEM ] Retransmit List: 1a 1b 2c 2d 2e 2f 30
Dec 07 22:55:30 overcloud-controller-1 corosync[18514]: [TOTEM ] Retransmit List: 1a 1b 2c 2d 2e 2f 30 34 35 36 37
Dec 07 22:55:30 overcloud-controller-1 corosync[18514]: [TOTEM ] Retransmit List: 1a 1b 2c 2d 2e 2f 30 34 35 36 37
Dec 07 22:55:30 overcloud-controller-1 corosync[18514]: [TOTEM ] Retransmit List: 1a 1b 2c 2d 2e 2f 30 34 35 36 37
Dec 07 22:55:30 overcloud-controller-1 corosync[18514]: [TOTEM ] Retransmit List: 1a 1b 2c 2d 2e 2f 30 34 35 36 37
Dec 07 22:55:30 overcloud-controller-1 corosync[18514]: [TOTEM ] Retransmit List: 1a 1b 2c 2d 2e 2f 30 34 35 36 37
Dec 07 22:55:30 overcloud-controller-1 corosync[18514]: [TOTEM ] Retransmit List: 1a 1b 2c 2d 2e 2f 30 34 35 36 37
Dec 07 22:55:30 overcloud-controller-1 corosync[18514]: [TOTEM ] Retransmit List: 1a 1b 2c 2d 2e 2f 30 34 35 36 37
Dec 07 22:55:30 overcloud-controller-1 corosync[18514]: [TOTEM ] Retransmit List: 1a 1b 2c 2d 2e 2f 30 34 35 36 37
Dec 07 22:55:30 overcloud-controller-1 corosync[18514]: [TOTEM ] Retransmit List: 1a 1b 2c 2d 2e 2f 30 34 35 36 37
Dec 07 22:55:30 overcloud-controller-1 corosync[18514]: [TOTEM ] Retransmit List: 1a 1b 2c 2d 2e 2f 30 34 35 36 37
Dec 07 22:55:30 overcloud-controller-1 corosync[18514]: [TOTEM ] Retransmit List: 1a 1b 2c 2d 2e 2f 30 34 35 36 37
Dec 07 22:55:30 overcloud-controller-1 corosync[18514]: [TOTEM ] Retransmit List: 1a 1b 2c 2d 2e 2f 30 34 35 36 37
Dec 07 22:55:30 overcloud-controller-1 corosync[18514]: [TOTEM ] Retransmit List: 1a 1b 2c 2d 2e 2f 30 34 35 36 37
Dec 07 22:55:30 overcloud-controller-1 corosync[18514]: [TOTEM ] Retransmit List: 1a 1b 2c 2d 2e 2f 30 34 35 36 37
Dec 07 22:55:30 overcloud-controller-1 corosync[18514]: [TOTEM ]...