rabbitmq haproxy should use be all active

[Andrew Woodward]

current haproxy config for rabbit

listen rabbitmq-openstack
  bind 10.119.160.37:5672
  balance roundrobin
  mode tcp
  option tcpka
  timeout client 48h
  timeout server 48h
  server node-3 10.119.160.34:5673 check inter 5000 rise 2 fall 3
  server node-5 10.119.160.35:5673 check inter 5000 rise 2 fall 3 backup
  server node-7 10.119.160.36:5673 check inter 5000 rise 2 fall 3 backup

second and third nodes are in backup, but all nodes can take traffic and probably should.

[Andrew Woodward (xarses)]

Also, the health check doesn't determine if the service is active correctly. We've seen some cases where the socket is open, but it won't transact messages

[Dmitry Borodaenko (angdraug)]

Comment #1 is a separate problem, fixing the primary issue will not impact it in anyway. If there's a way to reliable replicate it, a separate bug should be raised.

Severity of the primary issue is somewhat dampened in 4.1 due to using localhost to connect RabbitMQ from controllers. We may still need to balance RabbitMQ connections across all controllers, but implementing such change is a lot easier than testing it. We need a test that demonstrates that at scale (with a large number of compute nodes), balancing RabbitMQ servers across all controllers actually improves performance and does not reduce reliability.

[Mike Scherbakov (mihgen)]

My suggestion is to close this bug as Wishlist or Won't fix for now. It seems to me more like enhancement, which needs pretty extensive testing according to Dmitry. In this case, if it's still needed, we would rather need to file a blueprint for it.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/84707
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=5b3098c837cc011e66643e22eb1d749bb342f4f7
Submitter: Jenkins
Branch: master

commit 5b3098c837cc011e66643e22eb1d749bb342f4f7
Author: Vladimir Sharshov <email address hidden>
Date: Fri Mar 28 12:53:12 2014 +0400

    RabbitMQ 3 support

    RabbitMQ 3 do not support such big TTL which we used before.
    Big TTL was need to solve problem with node time dissynchronization.
    Now we use NTP to resolve this issue. That is why this value
    can be reduced.

    Related-Bug: #1278336
    Related-Bug: #1276770
    Change-Id: Iee79f14929e9f1ac5e5e7dd07aee37f641d3b2a8

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/86884

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/84717
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=f3ae209e713819da72875e4f2dc2b7e2b0371651
Submitter: Jenkins
Branch: master

commit f3ae209e713819da72875e4f2dc2b7e2b0371651
Author: Vladimir Sharshov <email address hidden>
Date: Mon Mar 31 15:40:45 2014 +0400

    RabbitMQ 3 support

    RabbitMQ 3 do not support such big TTL which we used before.
    Big TTL was need to solve problem with node time dissynchronization.
    Now we use NTP to resolve this issue. That is why this value
    can be reduced.

    Related-Bug: #1278336
    Related-Bug: #1276770
    Change-Id: I0a56b88247d9839bc3282294cdb8592ed2701c3a

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/4.1
Review: https://review.openstack.org/87371

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/4.1
Review: https://review.openstack.org/87379

[Bogdan Dobrelya (bogdando)]

Updated the status according to the Dmitry's backport awaiting review for 4.1.1

[Dmitry Borodaenko (angdraug)]

Bogdan, please revert your changes to this bug. The backport you're referring to is marked as *related*, it doesn't close this bug.

[Bogdan Dobrelya (bogdando)]

Ok, I've reverted my changes for status of this bug.

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/87371
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=af53380a11c2a147081a49a6242e22efb9cbad64
Submitter: Jenkins
Branch: stable/4.1

commit af53380a11c2a147081a49a6242e22efb9cbad64
Author: Vladimir Sharshov <email address hidden>
Date: Fri Mar 28 12:53:12 2014 +0400

    RabbitMQ 3 support

    RabbitMQ 3 do not support such big TTL which we used before.
    Big TTL was need to solve problem with node time dissynchronization.
    Now we use NTP to resolve this issue. That is why this value
    can be reduced.

    Related-Bug: #1278336
    Related-Bug: #1276770
    Change-Id: Iee79f14929e9f1ac5e5e7dd07aee37f641d3b2a8

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/87379
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=3b872b132c22f9014b82862b492ae2889226ae2f
Submitter: Jenkins
Branch: stable/4.1

commit 3b872b132c22f9014b82862b492ae2889226ae2f
Author: Vladimir Sharshov <email address hidden>
Date: Mon Mar 31 15:40:45 2014 +0400

    RabbitMQ 3 support

    RabbitMQ 3 do not support such big TTL which we used before.
    Big TTL was need to solve problem with node time dissynchronization.
    Now we use NTP to resolve this issue. That is why this value
    can be reduced.

    Related-Bug: #1278336
    Related-Bug: #1276770
    Change-Id: I0a56b88247d9839bc3282294cdb8592ed2701c3a

[Vladimir Kuklin (vkuklin)]

we are using multiple rabbit hosts from oslo.messaging. other services can successfully work with master/slave haproxy rabbitmq