Juniper Openstack

SM provision failures on the computes due to rabbitmq error

Series trunk
Bug #1687451

Bug #1687451 reported by wenqing liang on 2017-05-01

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Juniper Openstack	Status tracked in Trunk
	Trunk	Fix Committed	High	Ranjeet R	Juniper Openstack r4.0.0.0-fcs "r4.0.0.0"

Bug Description

mainline-3064 mitaka on an openstack/contrail ha setup.

root@servermanager:~# server-manager status server

+--------------------------------------+------------------+---------------------+--------------------------------------+
| id | fixed_ip_address | floating_ip_address | port_id |
+--------------------------------------+------------------+---------------------+--------------------------------------+
| 344338c9-e573-48c3-ae3c-77d295622623 | 10.0.0.13 | 10.87.120.14 | 2a406a4d-973a-4cf5-93a8-46be4cc9f57f |
| 854b2880-68e2-4a28-8b19-df0e88028a20 | 10.0.0.9 | 10.87.120.10 | f38097dc-da3a-4ab6-886e-daeb8ebd2b8b |
| 0cbaf9d9-8ac0-47e8-8b2f-ca65f8af0658 | 10.0.0.11 | 10.87.120.5 | defff82c-01ff-4e90-a54f-6469ca9dd342 |
| 45444a52-bd75-4495-8f7d-31e8b4d4ce1f | 10.0.0.5 | 10.87.120.18 | 3a3e7d18-6491-412a-a074-11b97c01a0f4 |
| 85094b65-4a6c-4af6-bbc0-5312df62e4d0 | 10.0.0.8 | 10.87.120.12 | 50d83db2-b8ed-4602-8315-f6b344353ee0 |
| acc94765-7a4f-4ebf-aa31-f44acf27d618 | 10.0.0.12 | 10.87.120.8 | d1985c63-7a93-40b7-bc7e-fc23deae329f |
| 72d25453-cb95-4833-a850-8621de75ce7c | 10.0.0.4 | 10.87.120.19 | a7e6354b-b371-4bd2-abb1-cef71639ae54 |
| 8b354010-8335-46a4-bd71-d3d5d00de4b7 | 10.0.0.10 | 10.87.120.15 | 1fa519fd-dce2-4925-8d69-51681c5b5003 |
| 69853b1e-6783-4487-bdbd-8e26357b5a82 | 10.0.0.7 | 10.87.120.21 | e391a69a-16d0-4040-a347-5b0c9863deb6 |
| f5d263bf-fb4d-4103-a873-5168a576f20c | 10.0.0.6 | 10.87.120.9 | 4bf239b6-71c5-4659-89d8-c0d912018c00 |
+--------------------------------------+------------------+---------------------+--------------------------------------+

Pls see /cs-shared/bugs/1687451/debug.log.

See original description

Tags:

wenqing liang (wliang) on 2017-05-01

description:

updated

Jeba Paulaiyan (jebap) on 2017-05-01

tags:

added: blocker sanity

Revision history for this message

Ramprakash R (ramprakash) wrote on 2017-05-01:

Download full text (6.9 KiB)

rabbitmq clustering seem to have failures. Looks like "hamon" script is trying to restart but causes the services to go down and cluster is not formed:

=INFO REPORT==== 1-May-2017::17:06:36 ===
accepting AMQP connection <0.1069.0> (192.168.10.22:51604 -> 192.168.10.21:5672)

=ERROR REPORT==== 1-May-2017::17:06:51 ===
** Generic server rabbit_node_monitor terminating
** Last message in was {'DOWN',#Ref<0.0.0.811>,process,
                               {rabbit,rabbit@server3ctrl},
                               normal}
** When Server state == {state,
                            {state,
                                {dict,2,16,16,8,80,48,
                                    {[],[],[],[],[],[],[],[],[],[],[],[],[],
                                     [],[],[]},
                                    {{[],[],[],[],[],[],[],[],[],[],
                                      [[{rabbit,rabbit@server2ctrl}|
                                        #Ref<0.0.0.798>],
                                       [{rabbit,rabbit@server3ctrl}|
                                        #Ref<0.0.0.811>]],
                                      [],[],[],[],[]}}},
                                erlang},
                            [],
                            {state,
                                {dict,0,16,16,8,80,48,
                                    {[],[],[],[],[],[],[],[],[],[],[],[],[],
                                     [],[],[]},
                                    {{[],[],[],[],[],[],[],[],[],[],[],[],[],
                                      [],[],[]}}},
                                erlang},
                            undefined,
                            {erlang,#Ref<0.0.0.48590>},
                            not_healing,
                            <<231,60,193,202,20,180,42,137,93,180,223,195,105,
                              11,7,107>>,
                            [{rabbit@server2ctrl,
                                 <<58,225,231,48,0,209,95,224,239,221,27,58,
                                   246,135,70,143>>},
                             {rabbit@server3ctrl,
                                 <<176,117,24,211,8,44,247,194,82,161,181,
                                   115,170,245,238,63>>}]}
** Reason for termination ==
** {bad_return_value,
       {error,
           {{badmatch,
                {error,
                    [{<<19,139,74,182,204,170,56,54,219,165,24,35,209,214,81,
                        223>>,
                      "consoleauth"},
                     {root,none}],
                    ["server2"]}},
            [{rabbit_exchange_type_topic,follow_down_get_path,2,[]},
             {rabbit_exchange_type_topic,'-remove_bindings/3-lc$^1/1-1-',1,[]},
             {rabbit_exchange_type_topic,remove_bindings,3,[]},
             {rabbit_binding,x_callback,4,[]},
             {rabbit_binding,'-process_deletions/1-fun-0-',2,[]},
             {dict,map_bucket,2,[{file,"dict.erl"},{line,460}]},
             {dict,map_bkt_list,2,[{file,"dict.erl"},{line,456}]},
             {dict,map_bkt_list,2,[{file,"dict.erl"},{line,456}]}]}}}

/<email address hidden>

=CRASH REPORT==== 1-May-2017::17:04:23 ===
cr...

rabbitmq clustering seem to have failures. Looks like "hamon" script is trying to restart but causes the services to go down and cluster is not formed:

=INFO REPORT==== 1-May-2017::17:06:36 ===
accepting AMQP connection <0.1069.0> (192.168.10.22:51604 -> 192.168.10.21:5672)

=ERROR REPORT==== 1-May-2017::17:06:51 ===
** Generic server rabbit_node_monitor terminating 
** Last message in was {'DOWN',#Ref<0.0.0.811>,process,
                               {rabbit,rabbit@server3ctrl},
                               normal}
** When Server state == {state,
                            {state,
                                {dict,2,16,16,8,80,48,
                                    {[],[],[],[],[],[],[],[],[],[],[],[],[],
                                     [],[],[]},
                                    {{[],[],[],[],[],[],[],[],[],[],
                                      [[{rabbit,rabbit@server2ctrl}|
                                        #Ref<0.0.0.798>],
                                       [{rabbit,rabbit@server3ctrl}|
                                        #Ref<0.0.0.811>]],
                                      [],[],[],[],[]}}},
                                erlang},
                            [],
                            {state,
                                {dict,0,16,16,8,80,48,
                                    {[],[],[],[],[],[],[],[],[],[],[],[],[],
                                     [],[],[]},
                                    {{[],[],[],[],[],[],[],[],[],[],[],[],[],
                                      [],[],[]}}},
                                erlang},
                            undefined,
                            {erlang,#Ref<0.0.0.48590>},
                            not_healing,
                            <<231,60,193,202,20,180,42,137,93,180,223,195,105,
                              11,7,107>>,
                            [{rabbit@server2ctrl,
                                 <<58,225,231,48,0,209,95,224,239,221,27,58,
                                   246,135,70,143>>},
                             {rabbit@server3ctrl,
                                 <<176,117,24,211,8,44,247,194,82,161,181,
                                   115,170,245,238,63>>}]}
** Reason for termination == 
** {bad_return_value,
       {error,
           {{badmatch,
                {error,
                    [{<<19,139,74,182,204,170,56,54,219,165,24,35,209,214,81,
                        223>>,
                      "consoleauth"},
                     {root,none}],
                    ["server2"]}},
            [{rabbit_exchange_type_topic,follow_down_get_path,2,[]},
             {rabbit_exchange_type_topic,'-remove_bindings/3-lc$^1/1-1-',1,[]},
             {rabbit_exchange_type_topic,remove_bindings,3,[]},
             {rabbit_binding,x_callback,4,[]},
             {rabbit_binding,'-process_deletions/1-fun-0-',2,[]},
             {dict,map_bucket,2,[{file,"dict.erl"},{line,460}]},
             {dict,map_bkt_list,2,[{file,"dict.erl"},{line,456}]},
             {dict,map_bkt_list,2,[{file,"dict.erl"},{line,456}]}]}}}

/var/log/rabbitmq/rabbit@server1ctrl-sasl.log

=CRASH REPORT==== 1-May-2017::17:04:23 ===
  crasher:
    initial call: gen:init_it/6
    pid: <0.11282.0>
    registered_name: []
    exception exit: {badarg,
                        [{dict,fetch,
                             [<<"None1">>,
                              {dict,0,16,16,8,80,48,
                                  {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                   [],[]},
                                  {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                    [],[]}}}],
                             [{file,"dict.erl"},{line,126}]},
                         {rabbit_channel,consumer_monitor,2,[]},
                         {rabbit_channel,handle_cast,2,[]},
                         {gen_server2,handle_msg,2,[]},
                         {proc_lib,init_p_do_apply,3,
                             [{file,"proc_lib.erl"},{line,239}]}]}
      in function  gen_server2:terminate/3
    ancestors: [<0.11279.0>,<0.11278.0>,<0.11275.0>,<0.11274.0>,
                  rabbit_tcp_client_sup,rabbit_sup,<0.134.0>]
    messages: []
    links: [<0.11279.0>]
    dictionary: [{{xtype_to_module,fanout},rabbit_exchange_type_fanout},
                  {process_name,
                      {rabbit_channel,
                          {<<"192.168.10.23:35216 -> 192.168.10.21:5672">>,
                           1}}},
                  {permission_cache,
                      [{{resource,<<"/">>,queue,<<"svc_monitor.server3">>},
                        read},
                       {{resource,<<"/">>,exchange,
                            <<"vnc_config.object-update">>},
                        read},
                       {{resource,<<"/">>,queue,<<"svc_monitor.server3">>},
                        write},
                       {{resource,<<"/">>,exchange,
                            <<"vnc_config.object-update">>},
                        configure},
                       {{resource,<<"/">>,queue,<<"svc_monitor.server3">>},
                        configure}]}]
    trap_exit: true
    status: running
    heap_size: 1598
    stack_size: 27
    reductions: 4987
  neighbours:

=SUPERVISOR REPORT==== 1-May-2017::17:04:23 ===
     Supervisor: {<0.11279.0>,rabbit_channel_sup}
     Context:    child_terminated
     Reason:     {badarg,
                     [{dict,fetch,
                          [<<"None1">>,
                           {dict,0,16,16,8,80,48,
                               {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                []},
                               {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []}}}],
                          [{file,"dict.erl"},{line,126}]},
                      {rabbit_channel,consumer_monitor,2,[]},
                      {rabbit_channel,handle_cast,2,[]},
                      {gen_server2,handle_msg,2,[]},
                      {proc_lib,init_p_do_apply,3,
                          [{file,"proc_lib.erl"},{line,239}]}]}
     Offender:   [{pid,<0.11282.0>},
                  {name,channel},
                  {mfargs,
                      {rabbit_channel,start_link,
                          [1,<0.11276.0>,<0.11280.0>,<0.11276.0>,
                           <<"192.168.10.23:35216 -> 192.168.10.21:5672">>,
                           rabbit_framing_amqp_0_9_1,
                           {user,<<"guest">>,
                               [administrator],
                               [{rabbit_auth_backend_internal,none}]},
                           <<"/">>,
                           [{<<"connection.blocked">>,bool,true},
                            {<<"consumer_cancel_notify">>,bool,true}],
                           <0.11277.0>,<0.11281.0>]}},
                  {restart_type,intrinsic},
                  {shutdown,4294967295},
                  {child_type,worker}]

Revision history for this message

Ramprakash R (ramprakash) wrote on 2017-05-01:

PLEASE NOTE: the above logs are not from the original setup that is reported by Wenqing, but on another setup where the same issue was observed. So the server names may not match.

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-05-02: A change has been merged

Reviewed: https://review.opencontrail.org/30917
Committed: http://github.com/Juniper/contrail-provisioning/commit/b3d761ff1023030e15deba145bd8ccda1678c73e
Submitter: Zuul (<email address hidden>)
Branch: master

commit b3d761ff1023030e15deba145bd8ccda1678c73e
Author: Ranjeet R <email address hidden>
Date: Mon May 1 19:14:25 2017 -0700

Fixes: SM provision failures on the computes due to rabbi...

Disabling RMQ monitoring script
Also, fixing the RMQ script to check for the
correct hosts in case it is enabled in the future.

Change-Id: I96ac3588d316350cd4b059888152bc26cea1ac65
Closes-Bug: 1687451

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.