3 controller environment rabbitmq_init_bundle container only exits successfully on controller-0 during FFU and upgrades
Bug #1753949 reported by
Michele Baldessari
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
High
|
Michele Baldessari |
Bug Description
Seen via https:/
OSP11 -> OSP12 upgrade: major upgrade composable step fails because rabbitmq_
Changed in tripleo: | |
status: | New → Triaged |
importance: | Undecided → High |
assignee: | nobody → Michele Baldessari (michele) |
Changed in tripleo: | |
status: | Triaged → In Progress |
To post a comment you must log in.
And there seems to be a 3rd bug https:/ /bugzilla. redhat. com/show_ bug.cgi? id=1551397 that might be related to this. Damien and I looked at logs of https:/ /bugzilla. redhat. com/show_ bug.cgi? id=1551265 and what puzzled us is that rabbitmqctl list_users timed out even though rabbitmq was clearly up on the node: current[ 364070] : Debug: Executing: '/usr/sbin/ rabbitmqctl -q list_users' current[ 364070] : Debug: Executing: '/usr/sbin/ rabbitmqctl -q list_users'
ctrl-1:
Mar 03 05:34:48 controller-1 dockerd-
...
Mar 03 05:38:08 controller-1 dockerd-
shortly after puppet fails due to timeout
But rabbitmq was up after "05:36:40": rabbitmq- bundle- docker- 1)[386276] : INFO: checking for nsenter, which is required when 'monitor_cmd' is specified rabbitmq- bundle- docker- 1)[386425] : INFO: monitor cmd passed: exit code = 0 rabbitmq- bundle- docker- 1)[386485] : INFO: monitor cmd passed: exit code = 0 rabbitmq- bundle- docker- 1)[387381] : INFO: checking for nsenter, which is required when 'monitor_cmd' is specified rabbitmq- bundle- docker- 1)[387542] : INFO: monitor cmd passed: exit code = 0 rabbitmq- bundle- docker- 1)[387605] : INFO: monitor cmd passed: exit code = 0 cluster( rabbitmq) [388021] : DEBUG: rabbitmq monitor : 7 rabbitmq- bundle- docker- 1)[392477] : INFO: monitor cmd passed: exit code = 0 cluster( rabbitmq) [393245] : DEBUG: rabbitmq monitor : 0 cluster( rabbitmq) [394468] : DEBUG: rabbitmq monitor : 0 cluster( rabbitmq) [395249] : DEBUG: rabbitmq monitor : 0
Mar 03 05:35:17 controller-1 docker(
Mar 03 05:35:17 controller-1 docker(
Mar 03 05:35:17 controller-1 docker(
Mar 03 05:35:39 controller-1 docker(
Mar 03 05:35:40 controller-1 docker(
Mar 03 05:35:40 controller-1 docker(
Mar 03 05:35:45 controller-1 rabbitmq-
Mar 03 05:36:40 controller-1 docker(
Mar 03 05:36:48 controller-1 rabbitmq-
Mar 03 05:37:02 controller-1 rabbitmq-
Mar 03 05:37:16 controller-1 rabbitmq-
So while making sure the rabbitmq_user piece of code gets only triggered on a) bootstrap node and b) potentially on BM only. The real question is why does "rabbitmqctl list_users" not return success if the bundle is up on the node?