Rabbitmq cluster dont become active because of error 'tables_not_present'

[Steve Leon]

When creating a cluster of 3 nodes, I have seen that sometimes the cluster doesnt come to ACTIVE. When I ssh to one of the node, i see the error below. This is an intermittent issue with rabbitmq and only seem to happen when all nodes are being started at the same time. http://rabbitmq.1065348.n5.nabble.com/Rabbitmq-boot-failure-with-quot-tables-not-present-quot-td24494.html

Openstack-infra found the same issue: https://review.openstack.org/#/c/173222/. They fixed it by bootstrapping one node first and then create the other nodes after. We might have to do something similar to ensure the consistency of the cluster creation.

2015-06-20 00:31:35.476 | =INFO REPORT==== 20-Jun-2015::00:07:41 ===
2015-06-20 00:31:35.484 | Error description:
2015-06-20 00:31:35.505 | {case_clause,{error,tables_not_present}}
2015-06-20 00:31:35.508 |
2015-06-20 00:31:35.523 | Log files (may contain more information):
2015-06-20 00:31:35.558 | /var/log/rabbitmq/rabbit-node-0.log
2015-06-20 00:31:35.595 | /var/log/rabbitmq/rabbit-node-0-sasl.log
2015-06-20 00:31:35.597 |
2015-06-20 00:31:35.601 | Stack trace:
2015-06-20 00:31:35.631 | [{rabbit_mnesia,discover_cluster,1,
2015-06-20 00:31:35.662 | [{file,"src/rabbit_mnesia.erl"},{line,619}]},
2015-06-20 00:31:35.689 | {rabbit_mnesia,init_from_config,0,
2015-06-20 00:31:35.719 | [{file,"src/rabbit_mnesia.erl"},{line,132}]},
2015-06-20 00:31:35.787 | {rabbit_mnesia,init,0,[{file,"src/rabbit_mnesia.erl"},{line,98}]},
2015-06-20 00:31:35.824 | {rabbit,'-run_boot_step/1-lc$^1/1-1-',1,
2015-06-20 00:31:35.863 | [{file,"src/rabbit.erl"},{line,484}]},
2015-06-20 00:31:35.966 | {rabbit,run_boot_step,1,[{file,"src/rabbit.erl"},{line,483}]},
2015-06-20 00:31:35.999 | {rabbit,'-start/2-lc$^0/1-0-',1,[{file,"src/rabbit.erl"},{line,450}]},
2015-06-20 00:31:36.017 | {rabbit,start,2,[{file,"src/rabbit.erl"},{line,450}]},
2015-06-20 00:31:36.027 | {application_master,start_it_old,4,
2015-06-20 00:31:36.063 | [{file,"application_master.erl"},{line,269}]}]
2015-06-20 00:31:36.064 |
2015-06-20 00:31:36.067 |
2015-06-20 00:31:36.086 | =INFO REPORT==== 20-Jun-2015::00:07:42 ===
2015-06-20 00:31:36.093 | Error description:
2015-06-20 00:31:36.101 | {could_not_start,rabbit,
2015-06-20 00:31:36.110 | {bad_return,
2015-06-20 00:31:36.121 | {{rabbit,start,[normal,[]]},
2015-06-20 00:31:36.127 | {'EXIT',
2015-06-20 00:31:36.160 | {rabbit,failure_during_boot,
2015-06-20 00:31:36.206 | {case_clause,{error,tables_not_present}}}}}}}

[Min Pae (sputnik13)]

This bug is no longer an issue, it is likely related to two separate issues that were already fixed, the first being using an outdated version of rabbitmq and the second being cloud-config restarting rabbitmq repeatedly. Both issues have already been fixed.