Shuffle amqp nodes in Openstack configs

Bug #1320184 reported by Bartosz Kupidura on 2014-05-16
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Bartosz Kupidura
4.1.x
High
Vladimir Kuklin
5.0.x
High
Bartosz Kupidura

Bug Description

Now each Openstack node (compute, cinder, ...) connect to #1 controller,
after failure it reconnects to #2, after that to #3 controller.

In this case, ALL AMQP traffic is served by #1.

We can shuffle 'rabbit_hosts' on each 'not controller' node.

Tags: ha Edit Tag help
Bogdan Dobrelya (bogdando) wrote :
Changed in fuel:
milestone: none → 5.0
assignee: nobody → Bartosz Kupidura (zynzel)
importance: Undecided → High
status: New → In Progress
tags: added: ha
Changed in fuel:
milestone: 5.0 → 5.1
milestone: 5.1 → 5.0
milestone: 5.0 → 5.1
milestone: 5.1 → 5.0
Bogdan Dobrelya (bogdando) wrote :

Basically, the idea of the patch was to provide
1) an idempotent and deterministic way to shuffle rabbit nodes for non controller nodes in order to increase RabbitMQ cluster performance (multiple rabbit nodes should serve the network connections instead if the single one)
 and
2) exclude the node IP address for the given controllers from their rabbit hosts (controllers use localhost as well).
Note: Deterministic way means consequent 'puppet apply' runs must ensure the configuration of rabbit_hosts left intact once it has been shuffled at the deployment stage.

Vladimir Kuklin (vkuklin) wrote :

patch is not yet ready and severity of this bug is not so high. postponing.

Mike Scherbakov (mihgen) wrote :

Postponing to 5.1, as:
a) it is not critical for 5.0 HA story, and adds improvements
b) actually it adds more non-determinism, and should be analyzed from this point of view
c) additional research why ubuntu Fuel CI puts -1 is needed before merge
d) answer to comment from Ihar Hrachyshka is needed

Reviewed: https://review.openstack.org/94632
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=71b112d8442c5a60b9ebe95b99b7b2bf9eb34b2a
Submitter: Jenkins
Branch: master

commit 71b112d8442c5a60b9ebe95b99b7b2bf9eb34b2a
Author: Sergii Golovatiuk <email address hidden>
Date: Wed May 21 14:34:49 2014 +0000

    Replace concat.rb with latest stdlib equivalent

    Sync concat puppet function to fix some bugs

    Related-Bug: #1320184
    Related-Bug: #1321284

    Change-Id: I49a614122469b37b1c4eb2fbc29b0877a4427fbd

Reviewed: https://review.openstack.org/93883
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=695aa4118afcd514d1b6abffe2e83679fd6a1cac
Submitter: Jenkins
Branch: master

commit 695aa4118afcd514d1b6abffe2e83679fd6a1cac
Author: Bartosz Kupidura <email address hidden>
Date: Fri May 16 11:50:22 2014 +0200

    Shuffle AMQP nodes for all OpenStack services.

    - Assign 127.0.0.1 as first server in HA
    - Rotate servers using RR to have better balance
    - delete.rb from upstream to fix bug with deletion

    Related-Bug: 1320184
    Change-Id: I3762dedfa06d593f4347e246b4302852261f82cb

Reviewed: https://review.openstack.org/96858
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=18d81e17cfd13c135bb433a11dcc7d2c69f2c84d
Submitter: Jenkins
Branch: stable/4.1

commit 18d81e17cfd13c135bb433a11dcc7d2c69f2c84d
Author: Sergii Golovatiuk <email address hidden>
Date: Wed May 21 14:34:49 2014 +0000

    Replace concat.rb with latest stdlib equivalent

    Sync concat puppet function to fix some bugs

    Related-Bug: #1320184
    Related-Bug: #1321284

    Change-Id: I49a614122469b37b1c4eb2fbc29b0877a4427fbd

Reviewed: https://review.openstack.org/97222
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=b0e1792e532a62f98620e8ade84f5b18c206bff1
Submitter: Jenkins
Branch: stable/4.1

commit b0e1792e532a62f98620e8ade84f5b18c206bff1
Author: Bartosz Kupidura <email address hidden>
Date: Fri May 16 11:50:22 2014 +0200

    Shuffle AMQP nodes for all OpenStack services.

    - Assign 127.0.0.1 as first server in HA
    - Rotate servers using RR to have better balance
    - delete.rb from upstream to fix bug with deletion

    Related-Bug: 1320184
    Change-Id: I3762dedfa06d593f4347e246b4302852261f82cb
    (cherry picked from commit 695aa4118afcd514d1b6abffe2e83679fd6a1cac)

Meg McRoberts (dreidellhasa) wrote :

Documented as fixed in 4.1.1

Dima Shulyak (dshulyak) wrote :

fyi https://github.com/openstack/oslo.messaging/commit/53b9d741a8f5a2fa6cbaea9e23a3679a6af235a4

there is qpid and rabbit implementation for ha queues shuffle, it is in master, but not released yet

no longer affects: fuel/5.1.x
Changed in fuel:
milestone: 5.0.1 → 5.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers