| 2017-01-17 20:29:24 |
Andreas Hasenack |
bug |
|
|
added bug |
| 2017-01-17 20:29:24 |
Andreas Hasenack |
attachment added |
|
juju-status.txt https://bugs.launchpad.net/bugs/1657245/+attachment/4805491/+files/juju-status.txt |
|
| 2017-01-17 20:30:01 |
Andreas Hasenack |
attachment added |
|
rabbit-units.tar.bz2 https://bugs.launchpad.net/charms/+source/rabbitmq-server/+bug/1657245/+attachment/4805492/+files/rabbit-units.tar.bz2 |
|
| 2017-01-17 20:31:59 |
Andreas Hasenack |
description |
juju 2.1b4
cs:xenial/rabbitmq-server-57
maas 2.1.3
I have an HA openstack deployment done by the autopilot where the 3 rabbit units didn't cluster together. In fact, it looks like units 0 and 1 clustered, but unit 2 went ahead on its own (split brain then I suppose). Also of note is that unit 2 is the leader according to juju.
This was noticed when neutron services couldn't connect to rabbit, getting a 403 error back:
2017-01-17 12:30:22.929 32573 ERROR oslo_service.service AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
This attempt can be confirmed in the rabbit/1 unit logs:
=ERROR REPORT==== 17-Jan-2017::12:30:22 ===
closing AMQP connection <0.16879.0> (10.96.22.27:60030 -> 10.96.22.56:5672):
{handshake_error,starting,0,
{amqp_error,access_refused,
"AMQPLAIN login refused: user 'neutron' - invalid credentials",
'connection.start_ok'}}
In fact, rabbit/0 and /1 show all sorts of refused logins because of invalid credentials.
Meanwhile, logs for rabbit/2 show that it is happily creating those users, like neutron:
=INFO REPORT==== 17-Jan-2017::12:12:10 ===
Creating user 'neutron'
Attached are the logs for all 3 rabbit units, as well as the neutron "victim". I have logs of all nodes participating in this deployment if something else is needed. |
juju 2.1b4
cs:xenial/rabbitmq-server-57
maas 2.1.3
I have an HA openstack deployment done by the autopilot where the 3 rabbit units didn't cluster together. In fact, it looks like units 0 and 1 clustered, but unit 2 went ahead on its own (split brain then I suppose). Also of note is that unit 2 is the leader according to juju.
This was noticed when neutron services couldn't connect to rabbit, getting a 403 error back:
2017-01-17 12:30:22.929 32573 ERROR oslo_service.service AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
This attempt can be confirmed in the rabbit/1 unit logs:
=ERROR REPORT==== 17-Jan-2017::12:30:22 ===
closing AMQP connection <0.16879.0> (10.96.22.27:60030 -> 10.96.22.56:5672):
{handshake_error,starting,0,
{amqp_error,access_refused,
"AMQPLAIN login refused: user 'neutron' - invalid credentials",
'connection.start_ok'}}
In fact, rabbit/0 and /1 show all sorts of refused logins because of invalid credentials.
Meanwhile, logs for rabbit/2 show that it is happily creating those users, like neutron:
=INFO REPORT==== 17-Jan-2017::12:12:10 ===
Creating user 'neutron'
Note that there is suspicion that leader election in juju 2.1b4 broke or changed, see details in https://bugs.launchpad.net/charms/+source/rabbitmq-server/+bug/1654116/comments/11 which was also about rabbit.
Attached are the logs for all 3 rabbit units, as well as the neutron "victim". I have logs of all nodes participating in this deployment if something else is needed. |
|
| 2017-01-17 20:32:54 |
Andreas Hasenack |
attachment added |
|
neutron-gateway-0.tar.bz2 https://bugs.launchpad.net/charms/+source/rabbitmq-server/+bug/1657245/+attachment/4805493/+files/neutron-gateway-0.tar.bz2 |
|
| 2017-01-17 20:40:38 |
Andreas Hasenack |
bug |
|
|
added subscriber Landscape |
| 2017-01-18 15:37:10 |
David Ames |
rabbitmq-server (Juju Charms Collection): status |
New |
In Progress |
|
| 2017-01-18 15:37:12 |
David Ames |
rabbitmq-server (Juju Charms Collection): importance |
Undecided |
Critical |
|
| 2017-01-18 15:37:14 |
David Ames |
rabbitmq-server (Juju Charms Collection): assignee |
|
David Ames (thedac) |
|
| 2017-01-18 15:37:17 |
David Ames |
rabbitmq-server (Juju Charms Collection): milestone |
|
17.01 |
|
| 2017-01-18 15:59:26 |
Ryan Beisner |
bug |
|
|
added subscriber Ryan Beisner |
| 2017-01-18 16:59:37 |
Andreas Hasenack |
bug task added |
|
landscape |
|
| 2017-01-18 16:59:47 |
Andreas Hasenack |
landscape: milestone |
|
17.01 |
|
| 2017-01-19 08:18:47 |
Sandor Zeestraten |
bug |
|
|
added subscriber Sandor Zeestraten |
| 2017-01-24 16:39:02 |
Nobuto Murata |
bug |
|
|
added subscriber Nobuto Murata |
| 2017-01-25 17:20:07 |
David Ames |
rabbitmq-server (Juju Charms Collection): status |
In Progress |
Fix Committed |
|
| 2017-02-10 20:55:12 |
Chad Smith |
landscape: milestone |
17.01 |
17.02 |
|
| 2017-02-23 19:23:01 |
James Page |
charm-rabbitmq-server: importance |
Undecided |
Critical |
|
| 2017-02-23 19:23:01 |
James Page |
charm-rabbitmq-server: status |
New |
Fix Committed |
|
| 2017-02-23 19:23:01 |
James Page |
charm-rabbitmq-server: assignee |
|
David Ames (thedac) |
|
| 2017-02-23 19:23:03 |
James Page |
rabbitmq-server (Juju Charms Collection): status |
Fix Committed |
Invalid |
|
| 2017-02-23 19:36:55 |
James Page |
charm-rabbitmq-server: milestone |
|
17.02 |
|
| 2017-02-23 20:36:29 |
James Page |
charm-rabbitmq-server: status |
Fix Committed |
Fix Released |
|
| 2017-03-16 14:39:29 |
Chad Smith |
landscape: status |
New |
Invalid |
|
| 2017-03-16 14:40:26 |
Eric Snow |
summary |
Malformed 3 unit cluster |
Malformed 3 unit cluster (rabbitmq) |
|
