rabbitmq fails during startup

[Steven Dake]

rabbitmq has this log message - not sure if its relevant or not:

[sdake@minime-03 ~]$ docker logs rabbitmq

              RabbitMQ 3.3.5. Copyright (C) 2007-2014 GoPivotal, Inc.
  ## ## Licensed under the MPL. See http://www.rabbitmq.com/
  ## ##
  ########## Logs: /var/log/rabbitmq/rabbit.log
  ###### ## /var/log/rabbitmq/rabbit-sasl.log
  ##########
              Starting broker...

BOOT FAILED
===========

Error description:
   {badmatch,{error,tables_not_present}}

Log files (may contain more information):
   /var/log/rabbitmq/rabbit.log
   /var/log/rabbitmq/rabbit-sasl.log

Stack trace:
   [{rabbit_mnesia,init_from_config,0,
                   [{file,"src/rabbit_mnesia.erl"},{line,132}]},
    {rabbit_mnesia,init,0,[{file,"src/rabbit_mnesia.erl"},{line,98}]},
    {rabbit,'-run_boot_step/1-lc$^1/1-1-',1,
            [{file,"src/rabbit.erl"},{line,509}]},
    {rabbit,run_boot_step,1,[{file,"src/rabbit.erl"},{line,508}]},
    {rabbit,'-start/2-lc$^0/1-0-',1,[{file,"src/rabbit.erl"},{line,475}]},
    {rabbit,start,2,[{file,"src/rabbit.erl"},{line,475}]},
    {application_master,start_it_old,4,
                        [{file,"application_master.erl"},{line,269}]}]

BOOT FAILED
===========

Error description:
   {could_not_start,rabbit,
       {bad_return,
           {{rabbit,start,[normal,[]]},
            {'EXIT',
                {rabbit,failure_during_boot,
                    {badmatch,{error,tables_not_present}}}}}}}

Log files (may contain more information):
   /var/log/rabbitmq/rabbit.log
   /var/log/rabbitmq/rabbit-sasl.log

{"init terminating in do_boot",{rabbit,failure_during_boot,{could_not_start,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{rabbit,failure_during_boot,{badmatch,{error,tables_not_present}}}}}}}}}

Crash dump was written to: erl_crash.dump

[Steven Dake (sdake)]

Workaround from upstream suggests start one node, then the other nodes in staggered fashion:

http://rabbitmq.1065348.n5.nabble.com/Rabbitmq-boot-failure-with-quot-tables-not-present-quot-td24494.html#a24512

[Steven Dake (sdake)]

Latest upstream release:
https://www.rabbitmq.com/install-rpm.html

[Steven Dake (sdake)]

Upstream release locks up during container building:

INFO:__main__:rabbitmq: : percona-release-noarch percona-release-x86_64 systemdcontainer
INFO:__main__:rabbitmq: : updates
INFO:__main__:rabbitmq:Cleaning up everything
INFO:__main__:rabbitmq: ---> 36283c8d0194
INFO:__main__:rabbitmq:Removing intermediate container e5aa047066bd
INFO:__main__:rabbitmq:Step 3 : RUN /usr/lib/rabbitmq/bin/rabbitmq-plugins enable rabbitmq_management
INFO:__main__:rabbitmq: ---> Running in b61572dd9afb

[Steven Dake (sdake)]

If I remove rabbitmq-plugins operation, running 3.5.4 returns this in the logs:
=WARNING REPORT==== 8-Sep-2015::23:20:18 ===
Could not auto-cluster with rabbit@minime-02: {error,tables_not_present}

=WARNING REPORT==== 8-Sep-2015::23:20:18 ===
Could not auto-cluster with rabbit@minime-03: {error,tables_not_present}

=WARNING REPORT==== 8-Sep-2015::23:20:18 ===
Could not find any node for auto-clustering from: ['rabbit@minime-01',
                                                   'rabbit@minime-02',
                                                   'rabbit@minime-03']
Starting blank node...

Running 3.4.4 returns same thing, but doesn't crash. Going to try recommended staggered startup.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/221571

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/221571
Committed: https://git.openstack.org/cgit/stackforge/kolla/commit/?id=327d89bd3833068d62935b605d3affc50ca7aeec
Submitter: Jenkins
Branch: master

commit 327d89bd3833068d62935b605d3affc50ca7aeec
Author: Steven Dake <email address hidden>
Date: Tue Sep 8 18:17:56 2015 -0700

    Fix rabbitmq from failure during startup

    Stagger the start times of the rabbitmq daemon so rabbitmq neither
    crashes and also properly forms a cluster. This problem exists in
    the latest rabbitmq 3.5.4 available on rabbitmq.com for download.

    Change-Id: Iac4186b3c3b586de38467294399c7ed9415244de
    Closes-Bug: #1490780