contrail analytics api status stuck in "contrail-analytics-api initializing (UvePartitions:UVE-Aggregation[None] connection down)"

[Soumil Kulkarni]

I provision a fresh cluster with server manager (contrail-version : 3.2.0.0-12~mitaka)

The contrail "analytics api service" status is stuck in "contrail-analytics-api initializing (UvePartitions:UVE-Aggregation[None] connection down)" state for some reason.

root@ubuntu-ssl-rabbit-1:~# contrail-status
== Contrail Control ==
supervisor-control: active
contrail-control active
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Analytics ==
supervisor-analytics: active
contrail-alarm-gen:0 active
contrail-analytics-api initializing (UvePartitions:UVE-Aggregation[None] connection down)
contrail-analytics-nodemgr active
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-config-nodemgr active
contrail-device-manager backup
contrail-discovery active
contrail-schema backup
contrail-svc-monitor backup
ifmap active

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active

== Contrail Database ==
contrail-database: active

== Contrail Supervisor Database ==
supervisor-database: active
contrail-database-nodemgr active
kafka active

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active

root@ubuntu-ssl-rabbit-1:~#
root@ubuntu-ssl-rabbit-1:~#
root@ubuntu-ssl-rabbit-1:~#
root@ubuntu-ssl-rabbit-1:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 3.2.0.0-12 12
contrail-config 3.2.0.0-12 12
contrail-config-openstack 3.2.0.0-12 12
contrail-control 3.2.0.0-12 12
contrail-database-common 3.2.0.0-12 12
contrail-dns 3.2.0.0-12 12
contrail-docs 3.2.0.0-12 12
contrail-f5 3.2.0.0-12 12
contrail-fabric-utils 3.2.0.0-12 12
contrail-heat 3.2.0.0-12 12
contrail-lib 3.2.0.0-12 12
contrail-nodemgr 3.2.0.0-12 12
contrail-nova-networkapi 3.2.0.0-12 12
contrail-openstack 3.2.0.0-12 12
contrail-openstack-analytics 3.2.0.0-12 12
contrail-openstack-config 3.2.0.0-12 12
contrail-openstack-control 3.2.0.0-12 12
contrail-openstack-dashboard 3.2.0.0-12 12
contrail-openstack-database 3.2.0.0-12 12
contrail-openstack-ha 3.2.0.0-12 12
contrail-openstack-webui 3.2.0.0-12 12
contrail-setup 3.2.0.0-12 12
contrail-utils 3.2.0.0-12 12
contrail-web-controller 3.2.0.0-12 12
contrail-web-core 3.2.0.0-12 12
ifmap-python-client 0.1-2 12
ifmap-server 0.3.2-1contrail3 12
neutron-plugin-contrail 3.2.0.0-12 12
nova-api 2:13.0.0-0ubuntu2~cloud0.1contrail 12
nova-common 2:13.0.0-0ubuntu2~cloud0.1contrail 12
nova-conductor 2:13.0.0-0ubuntu2~cloud0.1contrail 12
nova-console 2:13.0.0-0ubuntu2~cloud0.1contrail 12
nova-consoleauth 2:13.0.0-0ubuntu2~cloud0.1contrail 12
nova-novncproxy 2:13.0.0-0ubuntu2~cloud0.1contrail 12
nova-scheduler 2:13.0.0-0ubuntu2~cloud0.1contrail 12
python-contrail 3.2.0.0-12 12
python-neutronclient 1:4.1.1-2~cloud0.2contrail 12
python-nova 2:13.0.0-0ubuntu2~cloud0.1contrail 12

[Soumil Kulkarni (soumilk)]

This is a HA setup with 3 nodes having roles[config,control,openstack,webui,collector] and the other 3 node having compute role.

server-manager ip : 10.87.120.20
config node ip : 10.87.120.4

[Soumil Kulkarni (soumilk)]

Enabling ssl @ rabbitmq server is the root cause of this issue.
Presently, contrail-alarm-gen doesn’t support rabbit_use_ssl option.
Therefore, connection to rabbitmq is stuck and alarm-gen doesn’t register with discovery service.

adding rabbit_use_ssl option in contrail-alarm-gen should be the solution to this.
Also, VncKombuClient should not be stuck if the connection to rabbitmq fails.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27241
Submitter: Sundaresan Rajangam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27242
Submitter: Sundaresan Rajangam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27241
Committed: http://github.org/Juniper/contrail-controller/commit/117d876b7322d3b0596794fb9015cd49beca1313
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 117d876b7322d3b0596794fb9015cd49beca1313
Author: Sundaresan Rajangam <email address hidden>
Date: Tue Dec 13 23:45:43 2016 -0800

Add rabbitmq ssl options in contrail-alarm-gen

At present, contrail-alarm-gen doesn't support rabbitmq ssl options.
Therefore, if ssl is enabled at rabbitmq, then contrail-alarm-gen gets
stuck while connecting to rabbitmq.

Change-Id: I03c485fac5bae1fefa52abfa37a8a524321af210
Closes-Bug: #1649467

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27242
Committed: http://github.org/Juniper/contrail-puppet/commit/51a90929832ec6acc2c9855b813858a04218cc73
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 51a90929832ec6acc2c9855b813858a04218cc73
Author: Sundaresan Rajangam <email address hidden>
Date: Tue Dec 13 23:50:39 2016 -0800

Rename rabbit_use_ssl to rabbitmq_use_ssl

Renamed rabbit_use_ssl to rabbitmq_use_ssl to be consistent with
other rabbitmq options.

Change-Id: I61fdce91a53e65f14e5fdef0b0facbf17c48d565
Related-Bug: #1649467

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27351
Submitter: Sundaresan Rajangam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27354
Submitter: Sundaresan Rajangam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27351
Committed: http://github.org/Juniper/contrail-controller/commit/98dd082e3c9394079f231a30e537db3ae3f80f4e
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit 98dd082e3c9394079f231a30e537db3ae3f80f4e
Author: Sundaresan Rajangam <email address hidden>
Date: Tue Dec 13 23:45:43 2016 -0800

Add rabbitmq ssl options in contrail-alarm-gen

At present, contrail-alarm-gen doesn't support rabbitmq ssl options.
Therefore, if ssl is enabled at rabbitmq, then contrail-alarm-gen gets
stuck while connecting to rabbitmq.

Change-Id: I03c485fac5bae1fefa52abfa37a8a524321af210
Closes-Bug: #1649467
(cherry picked from commit 117d876b7322d3b0596794fb9015cd49beca1313)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27354
Committed: http://github.org/Juniper/contrail-puppet/commit/1f876625a97f4476a79de4b1d7bacc7d9ed3246e
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit 1f876625a97f4476a79de4b1d7bacc7d9ed3246e
Author: Sundaresan Rajangam <email address hidden>
Date: Thu Dec 15 12:23:39 2016 -0800

Add rabbitmq ssl options in contrail-alarm-gen

Change-Id: I9f34e2764f379ead586f477922a387f1fb1031c4
Closes-Bug: #1649467

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27452
Submitter: Sundaresan Rajangam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27453
Submitter: Sundaresan Rajangam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27465
Submitter: Sundaresan Rajangam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27452
Committed: http://github.org/Juniper/contrail-controller/commit/72c2b5ecfba703c8a4cf8cd71d1ec527e4cf8132
Submitter: Zuul (<email address hidden>)
Branch: master

commit 72c2b5ecfba703c8a4cf8cd71d1ec527e4cf8132
Author: Sundaresan Rajangam <email address hidden>
Date: Tue Dec 13 23:45:43 2016 -0800

Add rabbitmq ssl options in contrail-alarm-gen

At present, contrail-alarm-gen doesn't support rabbitmq ssl options.
Therefore, if ssl is enabled at rabbitmq, then contrail-alarm-gen gets
stuck while connecting to rabbitmq.

Change-Id: I03c485fac5bae1fefa52abfa37a8a524321af210
Closes-Bug: #1649467
(cherry picked from commit 117d876b7322d3b0596794fb9015cd49beca1313)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27453
Committed: http://github.org/Juniper/contrail-puppet/commit/d255daa3cce818e41d438408219a352f6f035d33
Submitter: Zuul (<email address hidden>)
Branch: master

commit d255daa3cce818e41d438408219a352f6f035d33
Author: Sundaresan Rajangam <email address hidden>
Date: Tue Dec 13 23:50:39 2016 -0800

Rename rabbit_use_ssl to rabbitmq_use_ssl

Renamed rabbit_use_ssl to rabbitmq_use_ssl to be consistent with
other rabbitmq options.

Change-Id: I61fdce91a53e65f14e5fdef0b0facbf17c48d565
Related-Bug: #1649467
(cherry picked from commit 51a90929832ec6acc2c9855b813858a04218cc73)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27465
Committed: http://github.org/Juniper/contrail-puppet/commit/9da7beab4412020c75befa1f28e256641cea991b
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit 9da7beab4412020c75befa1f28e256641cea991b
Author: Sundaresan Rajangam <email address hidden>
Date: Tue Dec 20 15:12:47 2016 -0800

Revert "Add rabbitmq ssl options in contrail-alarm-gen"

This reverts commit 1f876625a97f4476a79de4b1d7bacc7d9ed3246e.

Change-Id: I73185095a02d44281aebba96c9a416b7c5b22387
Closes-Bug: #1649467