Fuel for OpenStack

Bug #1463433
Comment #8

Comment 8 for bug 1463433

Revision history for this message

Davanum Srinivas (DIMS) (dims-v) wrote on 2015-06-09: Re: rabbitmq was down on one of controllers during shaker test but there are multiple "Timed out waiting for reply to ID" events logged by Oslo.messaging after rabbitmq recovered from partitioning

"rabbitmqctl report" shows that total memory is higher than high watermark

{memory,[{total,17770473600},
          {connection_procs,63922904},
          {queue_procs,14427954888},
          {plugins,0},
          {other_proc,0},
          {mnesia,2925304},
          {mgmt_db,0},
          {msg_index,2481160},
          {other_ets,6831936},
          {binary,674800824},
          {code,14650466},
          {atom,2478353},
          {other_system,3655403413}]},
{alarms,[memory]},
{listeners,[{clustering,41055,"::"},{amqp,5673,"::"}]},
{vm_memory_high_watermark,0.4},
{vm_memory_limit,13535861145},
{disk_free_limit,50000000},
{disk_free,52060409856},

Log confirms that's the case
/<email address hidden>:memory resource limit alarm set on node 'rabbit@node-1'.
/<email address hidden>:*** Publishers will be blocked until this alarm clears ***

So connections are blocked per "rabbitmqctl report"

Connections:
pid name port peer_port host peer_host ssl peer_cert_subject peer_cert_issuer peer_cert_validity auth_mechanism ssl_protocol ssl_key_exchange ssl_cipher ssl_hash protocol user vhost timeout frame_max channel_max

client_properties recv_oct recv_cnt send_oct send_cnt send_pend state channels
<email address hidden> 127.0.0.1:35246 -> 127.0.0.1:5673 5673 35246 127.0.0.1 127.0.0.1 false AMQPLAIN {0,9,1} nova / 60 131072 65535 [{"product","py-amqp"},{"product_version","1.4.5"},{"capabilitie
s",[{"connection.blocked",true},{"consumer_cancel_notify",true}]}] 62891 1315 13122 1559 0 blocking 1
<email address hidden> 127.0.0.1:35438 -> 127.0.0.1:5673 5673 35438 127.0.0.1 127.0.0.1 false AMQPLAIN {0,9,1} nova / 60 131072 65535 [{"product","py-amqp"},{"product_version","1.4.5"},{"capabilitie
s",[{"connection.blocked",true},{"consumer_cancel_notify",true}]}] 34560 1835 1809297 2966 0 blocking 1
<email address hidden> 127.0.0.1:35948 -> 127.0.0.1:5673 5673 35948 127.0.0.1 127.0.0.1 false AMQPLAIN {0,9,1} nova / 60 131072 65535 [{"product","py-amqp"},{"product_version","1.4.5"},{"cap
abilities",[{"connection.blocked",true},{"consumer_cancel_notify",true}]}] 69104 3505 3894287 3897 0 blocking 1
<email address hidden> 127.0.0.1:35951 -> 127.0.0.1:5673 5673 35951 127.0.0.1 127.0.0.1 false AMQPLAIN {0,9,1} nova / 60 131072 65535 [{"product","py-amqp"},{"product_version","1.4.5"},{"cap
abilities",[{"connection.blocked",true},{"consumer_cancel_notify",true}]}] 69120 3509 3861730 3901 0 blocking 1

"rabbitmqctl report" shows that total memory is higher than high watermark

{memory,[{total,17770473600},
          {connection_procs,63922904},
          {queue_procs,14427954888},
          {plugins,0},
          {other_proc,0},
          {mnesia,2925304},
          {mgmt_db,0},
          {msg_index,2481160},
          {other_ets,6831936},
          {binary,674800824},
          {code,14650466},
          {atom,2478353},
          {other_system,3655403413}]},
 {alarms,[memory]},
 {listeners,[{clustering,41055,"::"},{amqp,5673,"::"}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,13535861145},
 {disk_free_limit,50000000},
 {disk_free,52060409856},

Log confirms that's the case
/var/log/rabbitmq/rabbit@node-49.log:memory resource limit alarm set on node 'rabbit@node-1'.
/var/log/rabbitmq/rabbit@node-49.log:*** Publishers will be blocked until this alarm clears ***

So connections are blocked per "rabbitmqctl report"

Connections:
pid    name    port    peer_port    host    peer_host    ssl    peer_cert_subject    peer_cert_issuer    peer_cert_validity    auth_mechanism    ssl_protocol    ssl_key_exchange    ssl_cipher    ssl_hash    protocol    user    vhost    timeout    frame_max    channel_max

client_properties    recv_oct    recv_cnt    send_oct    send_cnt    send_pend    state    channels
<rabbit@node-1.2.923.0>    127.0.0.1:35246 -> 127.0.0.1:5673    5673    35246    127.0.0.1    127.0.0.1    false                AMQPLAIN                    {0,9,1}    nova    /    60    131072    65535    [{"product","py-amqp"},{"product_version","1.4.5"},{"capabilitie
s",[{"connection.blocked",true},{"consumer_cancel_notify",true}]}]    62891    1315    13122    1559    0    blocking    1
<rabbit@node-1.2.962.0>    127.0.0.1:35438 -> 127.0.0.1:5673    5673    35438    127.0.0.1    127.0.0.1    false                AMQPLAIN                    {0,9,1}    nova    /    60    131072    65535    [{"product","py-amqp"},{"product_version","1.4.5"},{"capabilitie
s",[{"connection.blocked",true},{"consumer_cancel_notify",true}]}]    34560    1835    1809297    2966    0    blocking    1
<rabbit@node-1.2.1017.0>    127.0.0.1:35948 -> 127.0.0.1:5673    5673    35948    127.0.0.1    127.0.0.1    false                AMQPLAIN                    {0,9,1}    nova    /    60    131072    65535    [{"product","py-amqp"},{"product_version","1.4.5"},{"cap
abilities",[{"connection.blocked",true},{"consumer_cancel_notify",true}]}]    69104    3505    3894287    3897    0    blocking    1
<rabbit@node-1.2.1037.0>    127.0.0.1:35951 -> 127.0.0.1:5673    5673    35951    127.0.0.1    127.0.0.1    false                AMQPLAIN                    {0,9,1}    nova    /    60    131072    65535    [{"product","py-amqp"},{"product_version","1.4.5"},{"cap
abilities",[{"connection.blocked",true},{"consumer_cancel_notify",true}]}]    69120    3509    3861730    3901    0    blocking    1