[14.04 3057] RabbitMQ not listening on 5672 port

[Suresh]

   On on mainline build 3057 ubuntu-14-04 mitaka, In openstack HA setup, RabbitMQ not listening on 5672 port and below services fails to come up.

root@nodec54(controller):/# contrail-status
== Contrail Control ==
contrail-control initializing (Database:RabbitMQ connection down)
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Config ==
contrail-api:0 initializing (Database:RabbitMQ[] connection down)
contrail-config-nodemgr active
contrail-device-manager active
contrail-schema initializing (Database:RabbitMQ[] connection down)
contrail-svc-monitor active

== Contrail Web UI ==
contrail-webui active
contrail-webui-middleware active

root@nodec54(controller):/#

root@nodec54:~# service rabbitmq-server status
Status of node rabbit@nodec54ctrl ...
[{pid,10817},
 {running_applications,[]},
 {os,{unix,linux}},
 {erlang_version,"Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:4:4] [async-threads:30] [kernel-poll:true]\n"},
 {memory,[{total,40177296},
          {connection_readers,0},
          {connection_writers,154128},
          {connection_channels,1717200},
          {connection_other,1054960},
          {queue_procs,2704},
          {queue_slave_procs,0},
          {plugins,0},
          {other_proc,13848784},
          {mnesia,309976},
          {mgmt_db,0},
          {msg_index,47504},
          {other_ets,829240},
          {binary,86928},
          {code,16351158},
          {atom,602729},
          {other_system,5171985}]},
 {alarms,[]},
 {listeners,[{clustering,25672,"::"}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,13484040192},
 {disk_free_limit,50000000},
 {disk_free,415073718272},
 {file_descriptors,[{total_limit,10140},
                    {total_used,2},
                    {sockets_limit,9124},
                    {sockets_used,0}]},
 {processes,[{limit,1048576},{used,594}]},
 {run_queue,0},
 {uptime,12257}]
root@nodec54:~# netstat -anp | grep 5672
tcp 0 0 0.0.0.0:25672 0.0.0.0:* LISTEN 10817/beam.smp
tcp 0 0 22.22.22.54:40337 22.22.22.54:25672 ESTABLISHED 23949/beam.smp
tcp 0 0 22.22.22.54:48122 22.22.22.54:25672 ESTABLISHED 27399/beam.smp
tcp 0 0 22.22.22.54:25672 22.22.22.56:57007 ESTABLISHED 10817/beam.smp
tcp 0 0 22.22.22.54:25672 22.22.22.54:48122 ESTABLISHED 10817/beam.smp
tcp 0 0 22.22.22.54:25672 22.22.22.54:40337 ESTABLISHED 10817/beam.smp
root@nodec54:~# ps -aux | grep 10817
rabbitmq 10817 40.6 0.1 2179180 62320 ? Sl 12:48 83:42 /usr/lib/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@nodec54ctrl -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -rabbit tcp_listeners [{"22.22.22.54",5672}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/<email address hidden>"} -rabbit sasl_error_logger {file,"/<email address hidden>"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@nodec54ctrl-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@nodec54ctrl" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
root 25993 0.0 0.0 10472 2188 pts/10 S+ 16:14 0:00 grep --color=auto 10817
root@nodec54:~#

[Abhay Joshi (abhayj)]

We are not seeing this issue in our setup with Openstack HA. Please provide setup to take a look.

[Jeba Paulaiyan (jebap)]

This is not seen in latest Sanity runs. Suresh, please re-open with testbed info if you see it again.