cloud install fails due to config-changed hook failure in rabbitmq-server service
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Landscape Server |
Fix Released
|
Undecided
|
Unassigned | ||
15.07 |
Won't Fix
|
Undecided
|
Unassigned | ||
Cisco-odl |
Fix Released
|
Undecided
|
Chris Glass | ||
falkor |
Fix Released
|
High
|
Данило Шеган | ||
rabbitmq-server (Juju Charms Collection) |
Fix Released
|
High
|
David Ames |
Bug Description
Sometimes, clusters form split-brained with the min-cluster-size=3 on juju 1.24.6. The end result looks like this:
dpb@helo:~[0]$ juju run --service rabbitmq-server 'sudo rabbitmqctl cluster_status'
- MachineId: 0/lxc/12
Stdout: |
Cluster status of node 'rabbit@10-1-61-43' ...
[{nodes,
{running_
{partition
...done.
UnitId: rabbitmq-server/0
- MachineId: 0/lxc/13
Stdout: |
Cluster status of node 'rabbit@10-1-63-87' ...
[{nodes,
{running_
{partition
...done.
UnitId: rabbitmq-server/1
- MachineId: 0/lxc/14
Stdout: |
Cluster status of node 'rabbit@10-1-63-88' ...
[{nodes,
{running_
{partition
...done.
UnitId: rabbitmq-server/2
Lots of logging and debugging attached, I can attach more if necessary.
Related branches
- Billy Olsen: Approve
-
Diff: 278 lines (+134/-49)3 files modifiedhooks/rabbit_utils.py (+46/-36)
hooks/rabbitmq_server_relations.py (+17/-13)
unit_tests/test_rabbit_utils.py (+71/-0)
- Ryan Beisner (community): Needs Information
- Geoff Teale (community): Approve
- OpenStack Charmers: Pending requested
-
Diff: 204 lines (+52/-39)4 files modifiedMakefile (+1/-1)
hooks/rabbit_utils.py (+23/-23)
hooks/rabbitmq_server_relations.py (+19/-6)
unit_tests/test_rabbit_utils.py (+9/-9)
Changed in falkor: | |
status: | Incomplete → New |
Changed in falkor: | |
importance: | Undecided → High |
status: | New → Triaged |
information type: | Proprietary → Private |
information type: | Private → Public |
description: | updated |
Changed in rabbitmq-server (Juju Charms Collection): | |
importance: | Undecided → High |
Changed in rabbitmq-server (Juju Charms Collection): | |
status: | New → Confirmed |
assignee: | nobody → David Ames (thedac) |
Changed in rabbitmq-server (Juju Charms Collection): | |
status: | Confirmed → In Progress |
tags: | added: kanban-cross-team |
Changed in falkor: | |
status: | Triaged → In Progress |
Changed in rabbitmq-server (Juju Charms Collection): | |
status: | In Progress → Fix Committed |
milestone: | none → 15.10 |
Changed in falkor: | |
milestone: | none → 0.12 |
Changed in rabbitmq-server (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in falkor: | |
milestone: | 0.12 → 0.13 |
Changed in landscape: | |
status: | New → Fix Committed |
Changed in falkor: | |
assignee: | nobody → Chris Glass (tribaal) |
Changed in landscape: | |
milestone: | none → 15.11 |
Changed in falkor: | |
milestone: | 0.13 → 0.14 |
Changed in falkor: | |
assignee: | Chris Glass (tribaal) → nobody |
assignee: | nobody → Данило Шеган (danilo) |
Changed in landscape: | |
status: | Fix Committed → Fix Released |
Changed in falkor: | |
milestone: | 0.14 → 0.15 |
Changed in falkor: | |
milestone: | 0.15 → none |
milestone: | none → 0.16 |
Changed in falkor: | |
status: | In Progress → Fix Released |
milestone: | 0.16 → 0.15 |
Thanks for your bug report. Could you please attach /var/log/juju/* (at least all-machines.log) to this bug report?
Also, what does running "/usr/sbin/ rabbitmqctl set_permissions -p nagios- rabbitmq- server- 2 nagios- rabbitmq- server- 2 .* .* .*" on the failed unit yield?
Thanks.