SM:R3.1-6 :rabbitmq port unable to connect in 5672

[sundarkh]

SM:rabbitmq port unable to connect in 5672

Connection to rabbitmq server not up on target node in port 5672

Logs : http://10.204.216.50/Docs/logs/3.0.0.0-2717_2016_02_25_13_55_38/logs/analyticstestsanity3.log

2016-02-25 03:47:50,341 - ERROR - nodeg37:contrail-api module connection to 10.204.217.77:5672 servers NOT UP
2016-02-25 03:47:51,728 - ERROR - <type 'exceptions.AssertionError'>

Impact : test_verify_process_status_config

[sundarkh (sundar-kh)]

SM : R3.1 Mitaka 6 : unable to connect to the port 5672 of the rabbitmq

Config Nodes : [u'nodec35', u'nodec33']
Control Nodes : [u'nodec35', u'nodec33']
Compute Nodes : [u'nodea4', u'nodec57']
Openstack Node : nodec35
WebUI Node : nodec33
Analytics Nodes : [u'nodec33']

test case fails with

2016-07-28 11:51:55,928 - ERROR - nodec35:contrail-api module connection to 192.168.100.4:5672 servers NOT UP
2016-07-28 11:51:55,929 - ERROR - nodec35:contrail-api module connection to 192.168.100.3:5672 servers NOT UP

/<email address hidden>

=ERROR REPORT==== 27-Jul-2016::13:24:50 ===
closing AMQP connection <0.316.0> (192.168.100.4:47879 -> 192.168.100.3:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'guest' can only connect via localhost",
                             'connection.start_ok'}}

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22745
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22745
Committed: http://github.org/Juniper/contrail-puppet/commit/b1ffb505b35139fec969e1db2526f2535817f845
Submitter: Zuul
Branch: R3.1

commit b1ffb505b35139fec969e1db2526f2535817f845
Author: Dheeraj Gautam <email address hidden>
Date: Mon Aug 1 17:05:15 2016 -0700

SM-Liberty/Mitaka: fix rabbit_hosts for openstack

Partial-Bug: #1550134

1. Appended rabbit_port to rabbit_hosts into config files
2. fixed nova.conf for rabbitmq heartbeat.

PATCH 2: removed nova/api.pp code to check-in as new review

PATCH 3: remove package_sku from call at profile/openstack/nova.pp as well

Change-Id: I88a05890a8954baa6139c4fda640efba5ace490f

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22866
Submitter: Dheeraj Gautam (<email address hidden>)

[sundarkh (sundar-kh)]

Issue seen with build 15

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22866
Committed: http://github.org/Juniper/contrail-puppet/commit/33ae950b47206a5ba31f2246e0d59c346df35c76
Submitter: Zuul
Branch: master

commit 33ae950b47206a5ba31f2246e0d59c346df35c76
Author: Dheeraj Gautam <email address hidden>
Date: Mon Aug 1 17:05:15 2016 -0700

SM-Liberty/Mitaka: fix rabbit_hosts for openstack

Partial-Bug: #1550134

1. Appended rabbit_port to rabbit_hosts into config files
2. fixed nova.conf for rabbitmq heartbeat.

PATCH 2: removed nova/api.pp code to check-in as new review

PATCH 3: remove package_sku from call at profile/openstack/nova.pp as well

Conflicts:
 contrail/environment/modules/contrail/manifests/compute/config.pp

Change-Id: I88a05890a8954baa6139c4fda640efba5ace490f

[sundarkh (sundar-kh)]

Seen with build 17 , kilo

ERROR REPORT==== 5-Aug-2016::21:47:22 ===
closing AMQP connection <0.1001.0> (10.204.217.77:39591 -> 10.204.217.77:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'guest' can only connect via localhost",
                             'connection.start_ok'}}

=INFO REPORT==== 5-Aug-2016::21:47:22 ===
Stopped RabbitMQ application

[sundarkh (sundar-kh)]

rabbitmqctl log

ERROR REPORT==== 5-Aug-2016::21:56:04 ===
Channel error on connection <0.1140.0> (10.204.217.77:43105 -> 10.204.217.77:5672, vhost: '/', user: 'guest'), channel 2:
{amqp_error,not_found,
            "no exchange 'reply_13a19576939646e8ac094362fff536a5' in vhost '/'",
            'exchange.declare'}

[Dheeraj Gautam (dgautam)]

This issue is not seen now. So closing the issue now.

[sundarkh (sundar-kh)]

Seen in R3.0 build 69
contrail-api.log

09/27/2016 09:12:55 PM [contrail-api]: SANDESH: [DROP: NoSession] __default__ [SYS_NOTICE]: VncApiError: Connecting to cassandra on ['127.0.0.1:9160']
09/27/2016 09:12:55 PM [contrail-api]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodeg37 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Discovery name = Collector server_addrs = [ 127.0.0.1:5998, ] status = Down description = Subscribe - Status Code 503 >>, << type = IFMap name = IfMap server_addrs = [ 127.0.0.1:8443, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 127.0.0.1:2181, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 127.0.0.1:9160, ] status = Initializing description = >>, ] description = Collector, Discovery:Collector[Subscribe - Status Code 503], Database:Cassandra[] connection down >>, ] >>