I hit this on my env locally yesterday and again today so filing this bug. The current pacemaker service ansible tasks set '200' seconds https://github.com/openstack/tripleo-heat-templates/blob/be6a66042e14abfc3a1b649ed144bb755c750424/puppet/services/pacemaker.yaml#L146 for the default start timeout on the PacemakerResources (by default it checks ['rabbitmq','haproxy']). In my env this expired by a few seconds in both cases it failed like:
Feb 21 11:08:27 overcloud-controller-0.localdomain os-collect-config[126779]: [2017-02-21 11:08:27,880] (heat-config) [INFO] {"deploy_stdout": "\nPLAY [localhost] ***************************************************************\n\nTASK [setup] *******************************************************************\nok: [localhost]\n\nTASK [Start pacemaker cluster] *************************************************\nchanged: [localhost]\n\nTASK [Check pacemaker resource] ************************************************\nok: [localhost] => (item=rabbitmq)\nfailed: [localhost] (item=haproxy) => {\"failed\": true, \"item\": \"haproxy\", \"msg\": \"Failed, the resource haproxy is not started\\n\"}\n\tto retry, use: --limit @/var/lib/heat-config/heat-config-ansible/2e011edd-78b5-4ac3-a992-ce83a2f755ff_playbook.retry\n\nPLAY RECAP *********************************************************************\nlocalhost : ok=2 changed=1 unreachable=0 failed=1 \n\n", "deploy_stderr": "", "deploy_status_code": 2}
... and then a few seconds later haproxy starts OK:
Feb 21 11:08:46 host-192-0-2-7 crmd[226256]: notice: Result of start operation for haproxy on overcloud-controller-0: 0 (ok)
The module defaults to 300 at https://github.com/redhat-openstack/ansible-pacemaker/blob/67b8b9e000cacfed33016e1509e3579f42f1f335/modules/pacemaker_resource.py#L47 so perhaps we should use the same?
Will post a review momentarily and point here... it seems the upgrades CI is NOT hitting this issue so it could be my environment (virt overcloud nodes @5GB ram + swap, the virt host only has 4 physical cores :/) but it may be worth bumping to the module default anyway?
Fix proposed to branch: master
Review: https:/