AutoScalingMultiAZSample.template Haproxy doesnt point to the backend servers

[Jeff Sloyer]

The latest version of AutoScalingMultiAZSample.templat (https://github.com/openstack/heat/blob/master/templates/AutoScalingMultiAZSample.template) does not properly configure Haproxy, the Haproxy config that is layed down does not point properly to the backend servers in the auto scale group.

[root@blah haproxy]# cat /etc/haproxy/haproxy.cfg

    global
        daemon
        maxconn 256
        stats socket /tmp/.haproxy-stats

    defaults
        mode http
        timeout connect 5000ms
        timeout client 50000ms
        timeout server 50000ms

        frontend http
            bind *:80

        default_backend servers

        backend servers
            balance roundrobin
            option http-server-close
            option forwardfor
            option httpchk
            timeout check 5s
            server server1 0.0.0.0:80 check inter 30s fall 5 rise 3

Notice on the last line, its pointing to 0.0.0.0:80, it should be the IP for a server in the auto scale group.

[Steven Hardy (shardy)]

Confirmed, I can reproduce this issue, looking into a fix, thanks for the report!

[Steven Hardy (shardy)]

This is a regression caused by https://review.openstack.org/#/c/23287/ - because we no longer wait for instances to become ACTIVE in the Autoscaling resize() method, we end up racing with nova and requesting network details for instances which are still in the BUILD state, so don't necessarily have IPs assiged yet.

This results in LoadBalancer::_instance_to_ipaddress() returning 0.0.0.0 since nova doesn't give us an IP for the instance.

Fix is not to reload the LoadBalancer config until the instances are all active.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/25540

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/25540
Committed: http://github.com/openstack/heat/commit/2ac3b8b8742c05fc5faf2295b5d00f4f12b4aab9
Submitter: Jenkins
Branch: master

commit 2ac3b8b8742c05fc5faf2295b5d00f4f12b4aab9
Author: Steven Hardy <email address hidden>
Date: Wed Mar 27 16:08:41 2013 +0000

    heat engine : Autoscaling reload Loadbalancer correctly

    Since the change implemented as part of bug #1136148 we no longer
    wait for nova to create the instance before reloading the loadbalancer
    which means we race nova assigning an IP to the instance and typically
    the loadbalancer config ends up with the 0.0.0.0 default value
    which the loadbalancer implementation uses when nova returns no IP.

    This patch moves the LB reload into a separate function, which is called
    after instances are active on group creation or adjustment

    Since our loadbalancer is broken without this fix, this is a candidate
    for backporting to grizzly milestone-proposed.

    fixes bug #1160407

    Change-Id: I50423883dead9a615aa79b8765d5480a9345686d

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/25600

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/25600
Committed: http://github.com/openstack/heat/commit/1ac1f895132f2dcbab67025fbc7309b0efeccd28
Submitter: Jenkins
Branch: milestone-proposed

commit 1ac1f895132f2dcbab67025fbc7309b0efeccd28
Author: Steven Hardy <email address hidden>
Date: Wed Mar 27 16:08:41 2013 +0000

    heat engine : Autoscaling reload Loadbalancer correctly

    Since the change implemented as part of bug #1136148 we no longer
    wait for nova to create the instance before reloading the loadbalancer
    which means we race nova assigning an IP to the instance and typically
    the loadbalancer config ends up with the 0.0.0.0 default value
    which the loadbalancer implementation uses when nova returns no IP.

    This patch moves the LB reload into a separate function, which is called
    after instances are active on group creation or adjustment

    Since our loadbalancer is broken without this fix, this is a candidate
    for backporting to grizzly milestone-proposed.

    fixes bug #1160407

    Change-Id: I50423883dead9a615aa79b8765d5480a9345686d