Juniper Openstack

contrail-analytics-api status stuck in "initializing (UvePartitions:UVE-Aggregation[None] connection down)" after upgrades from R2.2x to R3.0.1.0-17 juno

  1. Series trunk
  2. Bug #1564223
Bug #1564223 reported by wenqing liang
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.0
Fix Committed
Medium
Raj Reddy
Juniper Openstack r3.0.1.0
Trunk
Fix Committed
Medium
Raj Reddy
Juniper Openstack r3.1.0.0-fcs

Bug Description

contrail-analytics-api status stuck in '"initializing (UvePartitions:UVE-Aggregation[None] connection down)" was seen after upgrades to R3.0.1.0-17 from R2.22.x-8 as well as R2.21.2-30 juno. The status was "active" prior to the upgrades. The upgrade_contrail log for the path from r2.21.2-30 was uploaded to /cs-shared/bugs/1564223.

After upgrade:

root@cmbu-vse2100-1:/opt/contrail/utils# contrail-status
== Contrail Control ==
supervisor-control: active
contrail-control active
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Analytics ==
supervisor-analytics: active
contrail-analytics-api initializing (UvePartitions:UVE-Aggregation[None] connection down)
contrail-analytics-nodemgr active
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-config-nodemgr active
contrail-device-manager active
contrail-discovery:0 active
contrail-schema active
contrail-svc-monitor active
ifmap active

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active

== Contrail Database ==
contrail-database: active
supervisor-database: active
contrail-database-nodemgr active
kafka active

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active

== Contrail Storage ==
contrail-storage-stats: active
root@cmbu-vse2100-1:/opt/contrail/utils# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 3.0.1.0-17 17
contrail-config 3.0.1.0-17 17
contrail-config-openstack 3.0.1.0-17 17
contrail-control 3.0.1.0-17 17
contrail-dns 3.0.1.0-17 17
contrail-docs 3.0.1.0-17 17
contrail-f5 3.0.1.0-17 17
contrail-fabric-utils 3.0.1.0-17 17
contrail-heat 3.0.1.0-17 17
contrail-install-packages 3.0.1.0-17~juno 17
contrail-lib 3.0.1.0-17 17
contrail-nodemgr 3.0.1.0-17 17
contrail-nova-networkapi 3.0.1.0-17 17
contrail-openstack 3.0.1.0-17 17
contrail-openstack-analytics 3.0.1.0-17 17
contrail-openstack-config 3.0.1.0-17 17
contrail-openstack-control 3.0.1.0-17 17
contrail-openstack-dashboard 3.0.1.0-17 17
contrail-openstack-database 3.0.1.0-17 17
contrail-openstack-webui 3.0.1.0-17 17
contrail-setup 3.0.1.0-17 17
contrail-utils 3.0.1.0-17 17
contrail-web-controller 3.0.1.0-17 17
contrail-web-core 3.0.1.0-17 17
ifmap-python-client 0.1-2 17
ifmap-server 0.3.2-1contrail1 17
neutron-plugin-contrail 3.0.1.0-17 17
nova-api 1:2014.2.3-0ubuntu1~cloud0.3contrail17
nova-common 1:2014.2.3-0ubuntu1~cloud0.3contrail17
nova-conductor 1:2014.2.3-0ubuntu1~cloud0.3contrail17
nova-console 1:2014.2.3-0ubuntu1~cloud0.3contrail17
nova-consoleauth 1:2014.2.3-0ubuntu1~cloud0.3contrail17
nova-novncproxy 1:2014.2.3-0ubuntu1~cloud0.3contrail17
nova-objectstore 1:2014.2.3-0ubuntu1~cloud0.3contrail17
nova-scheduler 1:2014.2.3-0ubuntu1~cloud0.3contrail17
python-contrail 3.0.1.0-17 17
python-neutronclient 1:2.3.8-0ubuntu1~cloud0.2contrail17
python-nova 1:2014.2.3-0ubuntu1~cloud0.3contrail17
root@cmbu-vse2100-1:/opt/contrail/utils#

See original description
wenqing liang (wliang)
summary: - contrail-analytics-api status stuck in '"initializing
- (UvePartitions:UVE-Aggregation[None] connection down)" after upgrade
- from R2.22.x-8 to R3.0.1.0-17 juno
+ contrail-analytics-api status stuck in "initializing (UvePartitions:UVE-
+ Aggregation[None] connection down)" after upgrade from R2.22.x-8 to
+ R3.0.1.0-17 juno
information type: Proprietary → Public
wenqing liang (wliang)
description: updated
summary: contrail-analytics-api status stuck in "initializing (UvePartitions:UVE-
- Aggregation[None] connection down)" after upgrade from R2.22.x-8 to
+ Aggregation[None] connection down)" after upgrades from R2.2x to
R3.0.1.0-17 juno
Revision history for this message
Raj Reddy (rajreddy) wrote : Re: [Bug 1564223] contrail-analytics-api status stuck in "initializing (UvePartitions:UVE-Aggregation[None] connection down)" after upgrade from R2.22.x-8 to R3.0.1.0-17 juno

Hi Wenqing, do you have the setup in the same state, can we take a look..

thanks

On Mar 31, 2016, at 12:39 AM, wenqing liang <<email address hidden><mailto:<email address hidden>>> wrote:

** Description changed:

tags: added: analytics
Revision history for this message
wenqing liang (wliang) wrote :

Hi Raj,
I just hit it after upgrade from R2.20-64 to the latest R3.0.1.0-18 as well.

10.87.129.173-175 (root/n1keenA)

Pls let me know when you are done.

Thanks,
Wenqing

From: Raj Reddy
Sent: Thursday, March 31, 2016 6:37 AM
To: Bug 1564223 <email address hidden>; Wenqing Liang <email address hidden>
Cc: Anish Mehta <email address hidden>
Subject: Re: [Bug 1564223] contrail-analytics-api status stuck in "initializing (UvePartitions:UVE-Aggregation[None] connection down)" after upgrade from R2.22.x-8 to R3.0.1.0-17 juno

Hi Wenqing, do you have the setup in the same state, can we take a look..

thanks

On Mar 31, 2016, at 12:39 AM, wenqing liang <<email address hidden><mailto:<email address hidden>>> wrote:

** Description changed:

Revision history for this message
Ignatious Johnson Christopher (ijohnson-x) wrote :
Download full text (3.6 KiB)

Hi Raj/Anish,

We pass the —force-confnew option to the apt-get during upgrade, we don’t pass —force-confmiss,
So the removed(missing) config file(contrail-alarm-gen.ini) will not be unpacked/installed during upgrade

Reference:
===========
http://manpages.ubuntu.com/manpages/trusty/man1/dpkg.1.html

             confmiss: If a conffile is missing and the version in the
              package did change, always install the missing conffile without
              prompting. This is dangerous, since it means not preserving a
              change (removing) made to the file.

              confnew: If a conffile has been modified and the version in the
              package did change, always install the new version without
              prompting, unless the --force-confdef is also specified, in
              which case the default action is preferred.

Thanks,

Ignatious

From: Anish Mehta <<email address hidden><mailto:<email address hidden>>>
Date: Thursday, March 31, 2016 at 11:32 AM
To: Wenqing Liang <<email address hidden><mailto:<email address hidden>>>, Raj Reddy <<email address hidden><mailto:<email address hidden>>>, Bug 1564223 <<email address hidden><mailto:<email address hidden>>>, Ignatious Johnson <<email address hidden><mailto:<email address hidden>>>
Cc: Jeba Paulaiyan <<email address hidden><mailto:<email address hidden>>>
Subject: Re: [Bug 1564223] contrail-analytics-api status stuck in "initializing (UvePartitions:UVE-Aggregation[None] connection down)" after upgrade from R2.22.x-8 to R3.0.1.0-17 juno

Cc’ing Ignatious

From: Anish Mehta
Date: Thursday, March 31, 2016 at 10:11 AM
To: Wenqing Liang, Raj Reddy, Bug 1564223
Cc: Jeba Paulaiyan
Subject: Re: [Bug 1564223] contrail-analytics-api status stuck in "initializing (UvePartitions:UVE-Aggregation[None] connection down)" after upgrade from R2.22.x-8 to R3.0.1.0-17 juno

Hi Wenqing,

You can have the system back.
It has recovered.

We had removed the /etc/contrail/supervisord_analytics_files/contrail-alarm-gen.ini file during R2.2 installation, since that feature was not enabled by default.
We were expecting the R3 upgrade to restore that file. (that feature is always enabled in R3).
That didn’t happen – the file remained missing after the upgrade. I will investigate why.

Meanwhile, I extracted that file from the package and put it in the right place. This system should work for you if you need to do further testing.

Regards,

Anish

From: Wenqing Liang
Date: Thursday, March 31, 2016 at 8:33 AM
To: Raj Reddy, Bug 1564223
Cc: Anish Mehta, Jeba Paulaiyan
Subject: RE: [Bug 1564223] contrail-analytics-api status stuck in "initializing (UvePartitions:UVE-Aggregation[None] connection down)" after upgrade from R2.22.x-8 to R3.0.1.0-17 juno

Hi Raj,
I just hit it after upgrade from R2.20-64 to the latest R3.0.1.0-18 as well.

10.87.129.173-175 (root/n1keenA)

Pls let me know when you are done.

Thanks,
Wenqing

From: Raj Reddy
Sent: Thursday, March 31, 2016 6:37 AM
To: Bug 1564223 <<email address hidden><mailto:<email address hidden>>>; Wenqing Liang <<email address hidden><mailto:<email address hidden>>>
...

Read more...

Revision history for this message
Raj Reddy (rajreddy) wrote :
Download full text (3.8 KiB)

Hi Ignatious,

What’s the right fix in this case? Should we pass —force-confmiss to the apt-get command?

thanks
Raj

On Mar 31, 2016, at 3:08 PM, Ignatious Johnson <<email address hidden><mailto:<email address hidden>>> wrote:

Hi Raj/Anish,

We pass the —force-confnew option to the apt-get during upgrade, we don’t pass —force-confmiss,
So the removed(missing) config file(contrail-alarm-gen.ini) will not be unpacked/installed during upgrade

Reference:
===========
http://manpages.ubuntu.com/manpages/trusty/man1/dpkg.1.html

             confmiss: If a conffile is missing and the version in the
              package did change, always install the missing conffile without
              prompting. This is dangerous, since it means not preserving a
              change (removing) made to the file.

              confnew: If a conffile has been modified and the version in the
              package did change, always install the new version without
              prompting, unless the --force-confdef is also specified, in
              which case the default action is preferred.

Thanks,

Ignatious

From: Anish Mehta <<email address hidden><mailto:<email address hidden>>>
Date: Thursday, March 31, 2016 at 11:32 AM
To: Wenqing Liang <<email address hidden><mailto:<email address hidden>>>, Raj Reddy <<email address hidden><mailto:<email address hidden>>>, Bug 1564223 <<email address hidden><mailto:<email address hidden>>>, Ignatious Johnson <<email address hidden><mailto:<email address hidden>>>
Cc: Jeba Paulaiyan <<email address hidden><mailto:<email address hidden>>>
Subject: Re: [Bug 1564223] contrail-analytics-api status stuck in "initializing (UvePartitions:UVE-Aggregation[None] connection down)" after upgrade from R2.22.x-8 to R3.0.1.0-17 juno

Cc’ing Ignatious

From: Anish Mehta
Date: Thursday, March 31, 2016 at 10:11 AM
To: Wenqing Liang, Raj Reddy, Bug 1564223
Cc: Jeba Paulaiyan
Subject: Re: [Bug 1564223] contrail-analytics-api status stuck in "initializing (UvePartitions:UVE-Aggregation[None] connection down)" after upgrade from R2.22.x-8 to R3.0.1.0-17 juno

Hi Wenqing,

You can have the system back.
It has recovered.

We had removed the /etc/contrail/supervisord_analytics_files/contrail-alarm-gen.ini file during R2.2 installation, since that feature was not enabled by default.
We were expecting the R3 upgrade to restore that file. (that feature is always enabled in R3).
That didn’t happen – the file remained missing after the upgrade. I will investigate why.

Meanwhile, I extracted that file from the package and put it in the right place. This system should work for you if you need to do further testing.

Regards,

Anish

From: Wenqing Liang
Date: Thursday, March 31, 2016 at 8:33 AM
To: Raj Reddy, Bug 1564223
Cc: Anish Mehta, Jeba Paulaiyan
Subject: RE: [Bug 1564223] contrail-analytics-api status stuck in "initializing (UvePartitions:UVE-Aggregation[None] connection down)" after upgrade from R2.22.x-8 to R3.0.1.0-17 juno

Hi Raj,
I just hit it after upgrade from R2.20-64 to the latest R3.0.1.0-18 as well.

10.87.129.173-175 (root/n1keenA)

Pls let me know when you ar...

Read more...

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/18973
Submitter: Raj Reddy (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/18973
Committed: http://github.org/Juniper/contrail-provisioning/commit/892278925f5d344d0d0912685a70e83b84cd926a
Submitter: Zuul
Branch: R3.0

commit 892278925f5d344d0d0912685a70e83b84cd926a
Author: Raj Reddy <email address hidden>
Date: Thu Mar 31 18:40:00 2016 -0700

We are adding --confmiss option to apt-get, so that the conf files
from packages will get installed even if they are missing/removed.

Change-Id: I65c671a5289f36cc11e450924404a43ddc8032ae
Closes-Bug: #1564223

Revision history for this message
wenqing liang (wliang) wrote : RE: [Bug 1564223] contrail-analytics-api status stuck in "initializing (UvePartitions:UVE-Aggregation[None] connection down)" after upgrade from R2.22.x-8 to R3.0.1.0-17 juno
Download full text (4.4 KiB)

Update:
Looks good after upgrade from r2.22.x-8 to the latest R3.0.1.0-23.
Thanks,
Wenqing

From: Raj Reddy
Sent: Thursday, March 31, 2016 4:21 PM
To: Ignatious Johnson <email address hidden>
Cc: Anish Mehta <email address hidden>; Wenqing Liang <email address hidden>; Bug 1564223 <email address hidden>; Jeba Paulaiyan <email address hidden>
Subject: Re: [Bug 1564223] contrail-analytics-api status stuck in "initializing (UvePartitions:UVE-Aggregation[None] connection down)" after upgrade from R2.22.x-8 to R3.0.1.0-17 juno

Hi Ignatious,

What’s the right fix in this case? Should we pass —force-confmiss to the apt-get command?

thanks
Raj

On Mar 31, 2016, at 3:08 PM, Ignatious Johnson <<email address hidden><mailto:<email address hidden>>> wrote:

Hi Raj/Anish,

We pass the —force-confnew option to the apt-get during upgrade, we don’t pass —force-confmiss,
So the removed(missing) config file(contrail-alarm-gen.ini) will not be unpacked/installed during upgrade

Reference:
===========
http://manpages.ubuntu.com/manpages/trusty/man1/dpkg.1.html

             confmiss: If a conffile is missing and the version in the

              package did change, always install the missing conffile without

              prompting. This is dangerous, since it means not preserving a

              change (removing) made to the file.

              confnew: If a conffile has been modified and the version in the

              package did change, always install the new version without

              prompting, unless the --force-confdef is also specified, in

              which case the default action is preferred.

Thanks,

Ignatious
From: Anish Mehta <<email address hidden><mailto:<email address hidden>>>
Date: Thursday, March 31, 2016 at 11:32 AM
To: Wenqing Liang <<email address hidden><mailto:<email address hidden>>>, Raj Reddy <<email address hidden><mailto:<email address hidden>>>, Bug 1564223 <<email address hidden><mailto:<email address hidden>>>, Ignatious Johnson <<email address hidden><mailto:<email address hidden>>>
Cc: Jeba Paulaiyan <<email address hidden><mailto:<email address hidden>>>
Subject: Re: [Bug 1564223] contrail-analytics-api status stuck in "initializing (UvePartitions:UVE-Aggregation[None] connection down)" after upgrade from R2.22.x-8 to R3.0.1.0-17 juno

Cc’ing Ignatious

From: Anish Mehta
Date: Thursday, March 31, 2016 at 10:11 AM
To: Wenqing Liang, Raj Reddy, Bug 1564223
Cc: Jeba Paulaiyan
Subject: Re: [Bug 1564223] contrail-analytics-api status stuck in "initializing (UvePartitions:UVE-Aggregation[None] connection down)" after upgrade from R2.22.x-8 to R3.0.1.0-17 juno

Hi Wenqing,

You can have the system back.
It has recovered.

We had removed the /etc/contrail/supervisord_analytics_files/contrail-alarm-gen.ini file during R2.2 installation, since that feature was not enabled by default.
We were expecting the R3 upgrade to restore that file. (that feature is always enabled in R3).
That didn’t happen – the file remained missing after the upgrade. I will investigate why.

Meanwhile, I extracted that file from the package and put it in the right place. Thi...

Read more...

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/19004
Submitter: Raj Reddy (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/19004
Committed: http://github.org/Juniper/contrail-provisioning/commit/d094bfa14433c245e40e0a88ff6872a67feac205
Submitter: Zuul
Branch: master

commit d094bfa14433c245e40e0a88ff6872a67feac205
Author: Raj Reddy <email address hidden>
Date: Thu Mar 31 18:40:00 2016 -0700

We are adding --confmiss option to apt-get, so that the conf files
from packages will get installed even if they are missing/removed.

Change-Id: I65c671a5289f36cc11e450924404a43ddc8032ae
Closes-Bug: #1564223
(cherry picked from commit 892278925f5d344d0d0912685a70e83b84cd926a)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.