[library] Haproxy resource fails to start timing out the deployment

[Artem Panchenko]

api: '1.0'
astute_sha: 17a1dc816d6d56dda64c2db21b94581472cabefb
build_id: 2014-07-01_12-14-51
build_number: '17'
fuellib_sha: 7f7a7b33a5711b6146e5a11811c68e91ca4761af
fuelmain_sha: e9f2fa5d25c7b7b7bc1df61d510cbac16c78cfbb
mirantis: 'yes'
nailgun_sha: c30c3039f73e362b21d85ecb77a2aac078c07392
ostf_sha: d0fe60e0eba61685008b86d101f459fc2d3bb654
production: docker
release: 5.0.1

Steps to reproduce:

1. Install Fuel 5.0 and upgrade it to 5.0.1 using upgrade script (fuel-5.0.1-upgrade-17-2014-07-01_12-14-51.tar)
2. Create new environment (Centos, HA, Neutron/GRE)
3. Add 3 Controllers and 4 compute+cinder nodes
4. Start deployment

Expected result:

- cluster is successfully deployed

Actual:

- deployment failed with the following error in puppet logs (node-16):

http://paste.openstack.org/show/85377/

Here you can find diagnostic snapshot (couldn't attach it to the bug):

https://yadi.sk/d/fcvoxxaGVpqnp

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/104900

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/104902

[Artem Panchenko (apanchenko-8)]

{"build_id": "2014-07-06_00-31-14", "mirantis": "yes", "build_number": "102", "ostf_sha": "09b6bccf7d476771ac859bb3c76c9ebec9da9e1f", "nailgun_sha": "217ae694e487211fc8e352e4a45c0ef66644e1d8", "production": "docker", "api": "1.0", "fuelmain_sha": "cd72d39823b87aa5b7506a6e4f7d6ab0ed32de7b", "astute_sha": "644d279970df3daa5f5a2d2ccf8b4d22d53386ff", "release": "5.0.1", "fuellib_sha": "869acab37a78d018a0806e6fc6b76aabb2cdf5f0"}

Reproduced the same issue on latest iso for 5.0.1 after upgrade from 5.0 on bare metal. Environment configuration:

HA, Centos, NeutronGRE
3 Controllers+Ceph, 1 Compute+Ceph, 1 Ceph nodes
Ceph for volumes and images

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/104900
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=f5c5f739bc0f9005012f2699a2cefd9187c7b8a8
Submitter: Jenkins
Branch: master

commit f5c5f739bc0f9005012f2699a2cefd9187c7b8a8
Author: Dmitry Ilyin <email address hidden>
Date: Fri Jul 4 16:59:09 2014 +0400

    Add haproxy_stats to the haproxy config

    * Enables statistics page to monitor haproxy status
    * Prevent haproxy from failing to start if there
      are no cfg includes present

    Change-Id: I055811107bf959dc156448f9a5a457fb2d569d3f
    Closes-Bug: 1337261

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/104902
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=3a1056ffa76bcfc5243847afd10f1d0cc98fdfe8
Submitter: Jenkins
Branch: stable/5.0

commit 3a1056ffa76bcfc5243847afd10f1d0cc98fdfe8
Author: Dmitry Ilyin <email address hidden>
Date: Fri Jul 4 16:59:09 2014 +0400

    Add haproxy_stats to the haproxy config

    * Enables statistics page to monitor haproxy status
    * Prevent haproxy from failing to start if there
      are no cfg includes present

    Change-Id: I055811107bf959dc156448f9a5a457fb2d569d3f
    Closes-Bug: 1337261

[Artem Panchenko (apanchenko-8)]

verified

api: '1.0'
astute_sha: a4edb51661f50c66e247e0b8d00f2d01e0658fe6
build_id: 2014-07-08_13-57-45
build_number: '107'
fuellib_sha: d4cb36208efaf51a7c0ca012fa63d596d4ee2e29
fuelmain_sha: b0f5151d12751b9b55dcd69bd1445d0d480012d6
mirantis: 'yes'
nailgun_sha: c0082e3a0e8544bad7bd45c15c5dd8632ea045b5
ostf_sha: 09b6bccf7d476771ac859bb3c76c9ebec9da9e1f
production: docker
release: 5.0.1

[Kirill Omelchenko (komelchenko)]

Reproduced on 5.1 build 11
http://paste.openstack.org/show/113316/

        Scenario:
            1. Create cluster
            2. Add 1 controller node
            3. Deploy the cluster
            4. Add 2 controller nodes
            5. Deploy changes

       Actual: Deployment failed

Next errors appeared on in puppet.log of one of the controller nodes:
http://paste.openstack.org/show/113322/

[Kirill Omelchenko (komelchenko)]

This test was affected
http://jenkins-product.srt.mirantis.net:8080/view/5.1_swarm/job/5.1_fuelmain.system_test.ubuntu.thread_4/8/testReport/%28root%29/ha_flat_scalability/ha_flat_scalability/

[Bogdan Dobrelya (bogdando)]

Patching is an experimental feature in 5.1, so moving to 6.0 with high priority

[Bogdan Dobrelya (bogdando)]

Ok, that was not the patching case, but Fuel upgrade instead. Returning the status back.

[Dmitry Ilyin (idv1985)]

Haproxy stats was fixed before 5.1 release.

Other errors mentioned here are unrelated to this bug.