SM: upgrade from 3.0-1 to 3.0-2719 fails

Bug #1550650 reported by Senthilnathan Murugappan
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.0
Fix Committed
Critical
Nitish Krishna Kaveri
Trunk
Fix Committed
Critical
Nitish Krishna Kaveri

Bug Description

During upgrade from 3.0-1 to 3.0-2719 observed that the seeds list was updated without ',' separated ips in cassandra.yaml file.

Due to which cassandra failed to come up and subsequently other services too.

tags: added: blocker
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/17670
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/17954
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/17954
Submitter: Dheeraj Gautam (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/17954
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/17670
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/17954
Committed: http://github.org/Juniper/contrail-puppet/commit/95e25c7b2dc762c927451d4969453e1ef5b1922c
Submitter: Zuul
Branch: R3.0

commit 95e25c7b2dc762c927451d4969453e1ef5b1922c
Author: nitishkrishna <email address hidden>
Date: Sat Feb 27 01:55:51 2016 -0800

Closes-Bug: #1550650 - Fixed upgrade 3.0-1 -> 3.0-2719

Issue in cassandra seeds being set in multi node setup

Closed-Bug: #1550665 - Wrong wget in upgrade_cassandra script

Issue seen when upgrading from 2.23-121 to 3.0-2719

Patch 2:
=======
changes of yaml as per
https://github.com/Juniper/contrail-puppet/commit/14248f994df6a93e72e4877651398547d29059a6#diff-a3c7d6fbc48680683ab283f5ef343734

this was causing cassasnra startup to fail.

Patch 3:
=======

Changed contrail::contrail_repo_name[0] to contrail::params::contrail_repo_name as done in dpdk checkin
[Fixing failure in sm lite]

Change-Id: I7e3a5fb2eee4a059a96f7c3f731546477a5fa898

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/18054
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/17670
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/17670
Committed: http://github.org/Juniper/contrail-puppet/commit/33316fb86f958531dff5eb361249b9a7dc6cfbb1
Submitter: Zuul
Branch: master

commit 33316fb86f958531dff5eb361249b9a7dc6cfbb1
Author: nitishkrishna <email address hidden>
Date: Tue Feb 23 11:29:21 2016 -0800

Closes-Bug: #1545618 : Upgrade of contrail from R2.23 to mainline fixed

Changes:
I. Cassandra upgrade thru intermediate 2.0 version
Cassandra configured using file_line (not templates)
Cassandra templates removed

II. Python-kafka-python package replaced by python-kafka package

III. Contrail-config package brings new configs, causes dependency issues - fixed

IV. Removed secret key from nova api (metadata services bug)

Closes-Bug: #1543333

Patch 2:
Closes-Bug: #1550650 - Error in Cassandra Seeds
Closed-Bug: #1550665 - Error in wget of upgrade_cassandra script

Patch 4:
Changes to config_cassandra (to remove extra cofigs which cause error in multinode)
Changes to database/install.pp (use contrail::params::contrail_repo_name - smlite CI failure)

Closes-Bug: #1550650 - Previous Upgrade cassandra script has bugs

1. The previous shell script did not loop properly to check for connect status
It was succeeding in cases where cassandra comes up within five seconds only

2. Config cassandra had bug in one particular file_line

3. Added dependency for upgrade such that contrail-database package gets upgraded only if cassandra upgrade succeeds

Note:
upgrade_cassandra.sh can be run independently to upgrade cassandra before provision
Usage is:
./upgrade_cassandra.sh <host_ip> <cassandra_home_dir> <image_id of added 3.0 image>

Change-Id: I1a655b129c9878f499aef58cb1b9057c7769d53c

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/18125
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/18127
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/18310
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/18328
Submitter: Abhay Joshi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/18329
Submitter: Abhay Joshi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/18328
Committed: http://github.org/Juniper/contrail-server-manager/commit/add232a8b6b2b8fd2c1c535b8f872377c174553a
Submitter: Zuul
Branch: master

commit add232a8b6b2b8fd2c1c535b8f872377c174553a
Author: Abhay Joshi <email address hidden>
Date: Thu Mar 10 15:53:48 2016 -0800

Closes-Bug: #1550650
When upgrading from 2.2x code to 3.0, keystone.conf admin_token is changed to newly generated value,
however all other contrail files which used to have admin_token in 2.2x code and were generated with
value from 2.2x code retained the old value. In 3.x code, none of the contrail files use token, but since
in upgrade case, old files were retained and had token, we had token mismatch in config files and this resulted
in openstack neutron commands failing on upgrade.
The change is to use the admin_token value from cluster.json (will be the case when cluster is created with 2.2x SM)
in SM 3.0 when populating hiera data for openstack. If cluster.json does not have admin_token, then only use
generated value.
Tested : Successful upgrade from contrail 2.22b117 provisioned using SM 2.22build117, then upgrade SM to R3.0b2723 and
upgrade contrail to 3.0B2723.
Also tested fresh provision of contrail 3.0b 2723 with SM 3.0b2723.

Change-Id: Ieec1645bf9c340a5db8b47fcbc0771aebc9847eb

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/18127
Committed: http://github.org/Juniper/contrail-puppet/commit/555ae08f8f5b1da1ec1a3074a3a198f36a98701d
Submitter: Zuul
Branch: master

commit 555ae08f8f5b1da1ec1a3074a3a198f36a98701d
Author: nitishkrishna <email address hidden>
Date: Wed Mar 2 14:53:52 2016 -0800

Closes-Bug: #1550650 - Upgrade on multi-node fails due to compute out-of-order

Timing issues cause puppet to run resources out of order in compute multinode
Changes made:
1. class calls brought back to execs
2. include changed to contain
3. appended File_line[‘’] to subscribe in upgrade_kernel.pp because some scripts ran before upgrade in Shashi setup

Change-Id: I2fbbb1377960dabe2ae7f040e5059264e8b59cfc

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/18329
Committed: http://github.org/Juniper/contrail-server-manager/commit/3669b824829e823bb3d7f3de1788beb689a4f978
Submitter: Zuul
Branch: R3.0

commit 3669b824829e823bb3d7f3de1788beb689a4f978
Author: Abhay Joshi <email address hidden>
Date: Thu Mar 10 15:53:48 2016 -0800

Closes-Bug: #1550650
When upgrading from 2.2x code to 3.0, keystone.conf admin_token is changed to newly generated value,
however all other contrail files which used to have admin_token in 2.2x code and were generated with
value from 2.2x code retained the old value. In 3.x code, none of the contrail files use token, but since
in upgrade case, old files were retained and had token, we had token mismatch in config files and this resulted
in openstack neutron commands failing on upgrade.
The change is to use the admin_token value from cluster.json (will be the case when cluster is created with 2.2x SM)
in SM 3.0 when populating hiera data for openstack. If cluster.json does not have admin_token, then only use
generated value.
Tested : Successful upgrade from contrail 2.22b117 provisioned using SM 2.22build117, then upgrade SM to R3.0b2723 and
upgrade contrail to 3.0B2723.
Also tested fresh provision of contrail 3.0b 2723 with SM 3.0b2723.

Change-Id: Ieec1645bf9c340a5db8b47fcbc0771aebc9847eb

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/18971
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/18972
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/18971
Committed: http://github.org/Juniper/contrail-puppet/commit/7ad7683f15dc676075647ca12b7afa02f1cca67c
Submitter: Zuul
Branch: master

commit 7ad7683f15dc676075647ca12b7afa02f1cca67c
Author: nitishkrishna <email address hidden>
Date: Thu Mar 31 17:55:17 2016 -0700

Closes-Bug: #1550650 - Re-entrant Upgrade cassandra script and upgrade fixes

Closes-Bug: #1545618 - Duplicate upgrade bug
This change:
1. Fixes upgrade_cassandra.sh to be re-entrant
2. Changes one augeas_conf_set to file line (needed as value itself has quotes)
3. Delete of additional package: contrail-openstack-analytics needed for old python-kafka upgrade issue

Change-Id: Id40eaf59629f94c26c2a6f0e2b47fa5cff7dbe81

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/18972
Committed: http://github.org/Juniper/contrail-puppet/commit/0e2bbb5f28994bd453ca5ab7da7f2b03a56786bc
Submitter: Zuul
Branch: R3.0

commit 0e2bbb5f28994bd453ca5ab7da7f2b03a56786bc
Author: nitishkrishna <email address hidden>
Date: Thu Mar 31 17:55:17 2016 -0700

Closes-Bug: #1550650 - Re-entrant Upgrade cassandra script and upgrade fixes

Closes-Bug: #1545618 - Duplicate upgrade bug
This change:
1. Fixes upgrade_cassandra.sh to be re-entrant
2. Changes one augeas_conf_set to file line (needed as value itself has quotes)
3. Delete of additional package: contrail-openstack-analytics needed for old python-kafka upgrade issue
4. appended File_line[‘’] to subscribe in upgrade_kernel.pp because some scripts ran before upgrade in Shashi setup
5. Changed one file_line to exec in config_cassandra (already merged in mainline)

Change-Id: Id40eaf59629f94c26c2a6f0e2b47fa5cff7dbe81

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.