ServerManager :: Upgrade to mainline from R2.2 image fails.

[Ritam Gangopadhyay]

Upgrade failure:
From:
R2.2 build 103 juno

To:
R3.0 build 2711 juno

After editing the puppet.conf file on all the nodes to have stringify_facts=false and provisioning with mainline. Openstack upgrade is stuck and I see puppet_failed errors. Setup is in error state.

Regards,
Ritam.

_____________________________________________
From: Thilak Raj Surendrababu
Sent: Sunday, February 07, 2016 11:15 PM
To: Ritam Gangopadhyay <email address hidden>; Abhay Joshi <email address hidden>
Cc: Sudheendra Rao <email address hidden>; dl-contrail-server-manager <email address hidden>
Subject: RE: Upgrade using server-manager R2.21 build 103 juno to R3.0 build 2711 juno :: FAIL.

Ritam,
The kickstart from 3.0 needs to be used.
That will put stringify_facts=false in puppet.conf.
Hence you got this error.

We may have to add this param to puppet.conf for upgrade to happen from pre 3.0.

After clearing that error.
I saw some Augeas_errors,
keeper/conf/environment_setting_ZOO_LOG4J_PROP](provider=augeas): sending command 'set' with params ["/files/etc/zookeeper/conf/environment/ZOO_LOG4J_PROP", "INFO,CONSOLE,ROLLINGFILE"]
Debug: Augeas[/etc/zookeeper/conf/environment_setting_ZOO_LOG4J_PROP](provider=augeas): Closed the augeas connection
Error: /Stage[main]/Contrail::Database::New_config_zk_files_setup/Contrail::Lib::Augeas_conf_set[ZOO_LOG4J_PROP]/Augeas[/etc/zookeeper/conf/environment_setting_ZOO_LOG4J_PROP]: Could not evaluate: Error sending command 'set' with params ["/files/etc/zookeeper/conf/environment/ZOO_LOG4J_PROP", "INFO,CONSOLE,ROLLINGFILE"]/Error sending command 'set' with params ["/files/etc/zookeeper/conf/environment/ZOO_LOG4J_PROP", "INFO,CONSOLE,ROLLINGFILE"]
Debug: Contrail::Lib::Augeas_conf_set[ZOO_LOG4J_PROP]: The container Class[Contrail::Database::New_config_zk_files_setup] will propagate my refresh event
Debug: Class[Contrail::Database::Config]: The container Stage[main] will propagate my refresh event
Info: Class[Contrail::Database::Config]: Scheduling r

While modifying the settings, I mistakenly deleted /var/lib/puppet/ssl on targets instead
Of deleting /etc/puppet/ssl

Because of this puppet doesn’t run as a service.
Can you please re-run the tests.

Thanks
Thilak Raj S

_____________________________________________
From: Ritam Gangopadhyay
Sent: Sunday, February 07, 2016 7:28 AM
To: Abhay Joshi <email address hidden>
Cc: Sudheendra Rao <email address hidden>; dl-contrail-server-manager <email address hidden>
Subject: Upgrade using server-manager R2.21 build 103 juno to R3.0 build 2711 juno :: FAIL.

Hi,

       Upgrade of a target cluster provisioned with R2.2 build 103 juno, using R2.2 SM to target build R3.0 - 2711 juno using SM from R3.0 failed.

Setup:-

SM:- nodej8 – 10.204.217.155
Target cluster is a 5 node setup.

Error’s seen:-

Snip from debug file shows – puppet_failed

"2016-02-07 07:19:55,038-DEBUG-server_mgr_puppet.py:793-modify_server_hiera_data(): role-tuple: nodec58 = haproxy"
"2016-02-07 07:19:55,038-DEBUG-server_mgr_puppet.py:799-modify_server_hiera_data(): role-key: contrail::params::enable_haproxy True"
"2016-02-07 07:20:12,709-DEBUG-server_mgr_status.py:104-put_server_status(): Server status Data {'status': 'puppet_failed', 'id': 'nodeg21'}"
"2016-02-07 07:20:13,185-DEBUG-server_mgr_status.py:104-put_server_status(): Server status Data {'status': 'puppet_failed', 'id': 'nodeg17'}"
"2016-02-07 07:20:16,000-DEBUG-server_mgr_status.py:104-put_server_status(): Server status Data {'status': 'puppet_failed', 'id': 'nodeg33'}"

Snip from syslog file on the target nodes show this error:-

Feb 7 07:20:08 nodeg21 kernel: [14551.933428] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-vhost instead.
Feb 7 07:20:10 nodeg21 kernel: [14554.240849] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-vhost instead.
Feb 7 07:20:12 nodeg21 puppet-agent[25141]: Could not retrieve catalog from remote server: Error 400 on SERVER: delete_values(): First argument must be a Hash. Given an argument of class String. at /etc/puppet/environments/juno2711ubuntu14/modules/contrail/manifests/params.pp:820 on node nodeg21.englab.juniper.net
Feb 7 07:20:12 nodeg21 puppet-agent[25141]: Not using cache on failed catalog
Feb 7 07:20:12 nodeg21 puppet-agent[25141]: Could not retrieve catalog; skipping run
Feb 7 07:20:13 nodeg21 puppet-agent[26602]: Local environment: "production" doesn't match server specified node environment "juno2711ubuntu14", switching agent to "juno2711ubuntu14".
Feb 7 07:20:15 nodeg21 kernel: [14559.307696] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-vhost instead.
Feb 7 07:20:16 nodeg21 kernel: [14560.367667] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-vhost instead.
Feb 7 07:20:18 nodeg21 puppet-agent[26602]: Could not retrieve catalog from remote server: Error 400 on SERVER: delete_values(): First argument must be a Hash. Given an argument of class String. at /etc/puppet/environments/juno2711ubuntu14/modules/contrail/manifests/params.pp:820 on node nodeg21.englab.juniper.net

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17670
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17671
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17697
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17706
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17670
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17671
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17706
Committed: http://github.org/Juniper/contrail-server-manager/commit/734f705cc0f3d8d3080bb91ab851e6cc8c7826ec
Submitter: Zuul
Branch: R3.0

commit 734f705cc0f3d8d3080bb91ab851e6cc8c7826ec
Author: nitishkrishna <email address hidden>
Date: Tue Feb 23 15:14:00 2016 -0800

Closes-Bug: #1545618 - Removing unused openstack hiera param

In latest 3.0 server manager the shared secret is hard coded in openstack.yaml if not set via SM cluster params.
An additional parameter in openstack yaml has been deleted as in interfered with the upgrade

Change-Id: Ieeeb415fc20ec2314cd44b5acb25bd64ce3a9fc4
Closes-Bug: #1543333

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17671
Committed: http://github.org/Juniper/contrail-puppet/commit/54f691e724bfdc4f0306a074eff8adb755bab844
Submitter: Zuul
Branch: R3.0

commit 54f691e724bfdc4f0306a074eff8adb755bab844
Author: nitishkrishna <email address hidden>
Date: Tue Feb 23 11:29:21 2016 -0800

Closes-Bug: #1545618 : Upgrade of contrail from R2.23 to R3.0 fixed

Changes:
I. Cassandra upgrade thru intermediate 2.0 version
Cassandra configured using file_line (not templates)
Cassandra templates removed

II. Python-kafka-python package replaced by python-kafka package

III. Contrail-config package brings new configs, causes dependency issues - fixed

IV. Removed secret key from nova api (metadata services bug)

Closes-Bug: #1543333

Change-Id: I1a655b129c9878f499aef58cb1b9057c7769d53c

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17670
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[Ritam Gangopadhyay (ritam)]

Not fixed till build 2722

checked with R2.22 build 117 to R3.0 build 2722 ubuntu14 juno upgrade, cassandra upgrade fails.

Feb 29 09:20:15 nodec33 puppet-agent[15326]: (/Stage[main]/Contrail::Database::Config/File[/etc/zookeeper/conf/zoo.cfg]) Dependency Exec[Upgrade Cassandra to version 2.1 through intermediate version] has failures: true
Feb 29 09:20:15 nodec33 puppet-agent[15326]: (/Stage[main]/Contrail::Database::Config/File[/etc/zookeeper/conf/zoo.cfg]) Skipping because of failed dependencies
Feb 29 09:20:15 nodec33 puppet-agent[15326]: change_set = set maxSessionTimeout "120000"set autopurge.purgeInterval "3"
Feb 29 09:20:15 nodec33 puppet-agent[15326]: (/Stage[main]/Contrail::Database::New_config_zk_files_setup/Contrail::Lib::Augeas_conf_set[zk_cfg_keys]/Notify[change_set = set maxSessionTimeout "120000"set autopurge.purgeInterval "3"]/message) defined 'message' as 'change_set = set maxSessionTimeout "120000"set autopurge.purgeInterval "3"'
Feb 29 09:20:15 nodec33 puppet-agent[15326]: (/Stage[main]/Contrail::Database::New_config_zk_files_setup/File[/etc/zookeeper/conf/log4j.properties]) Dependency Exec[Upgrade Cassandra to version 2.1 through intermediate version] has failures: true
Feb 29 09:20:15 nodec33 puppet-agent[15326]: (/Stage[main]/Contrail::Database::New_config_zk_files_setup/File[/etc/zookeeper/conf/log4j.properties]) Skipping because of failed dependencies
Feb 29 09:20:15 nodec33 puppet-agent[15326]: (/Stage[main]/Contrail::Database::New_config_zk_files_setup/File[/etc/zookeeper/conf/environment]) Dependency Exec[Upgrade Cassandra to version 2.1 through intermediate version] has failures: true

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17697
Committed: http://github.org/Juniper/contrail-server-manager/commit/a3922b43c7a3448c4d65f1d5e3f3046bcd5ee814
Submitter: Zuul
Branch: master

commit a3922b43c7a3448c4d65f1d5e3f3046bcd5ee814
Author: nitishkrishna <email address hidden>
Date: Tue Feb 23 15:14:00 2016 -0800

Closes-Bug: #1545618 - Removing unused openstack hiera param

In latest 3.0 server manager the shared secret is hard coded in openstack.yaml if not set via SM cluster params.
An additional parameter in openstack yaml has been deleted as in interfered with the upgrade

Change-Id: Ieeeb415fc20ec2314cd44b5acb25bd64ce3a9fc4
Closes-Bug: #1543333

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17670
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17670
Committed: http://github.org/Juniper/contrail-puppet/commit/33316fb86f958531dff5eb361249b9a7dc6cfbb1
Submitter: Zuul
Branch: master

commit 33316fb86f958531dff5eb361249b9a7dc6cfbb1
Author: nitishkrishna <email address hidden>
Date: Tue Feb 23 11:29:21 2016 -0800

Closes-Bug: #1545618 : Upgrade of contrail from R2.23 to mainline fixed

Changes:
I. Cassandra upgrade thru intermediate 2.0 version
Cassandra configured using file_line (not templates)
Cassandra templates removed

II. Python-kafka-python package replaced by python-kafka package

III. Contrail-config package brings new configs, causes dependency issues - fixed

IV. Removed secret key from nova api (metadata services bug)

Closes-Bug: #1543333

Patch 2:
Closes-Bug: #1550650 - Error in Cassandra Seeds
Closed-Bug: #1550665 - Error in wget of upgrade_cassandra script

Patch 4:
Changes to config_cassandra (to remove extra cofigs which cause error in multinode)
Changes to database/install.pp (use contrail::params::contrail_repo_name - smlite CI failure)

Closes-Bug: #1550650 - Previous Upgrade cassandra script has bugs

1. The previous shell script did not loop properly to check for connect status
It was succeeding in cases where cassandra comes up within five seconds only

2. Config cassandra had bug in one particular file_line

3. Added dependency for upgrade such that contrail-database package gets upgraded only if cassandra upgrade succeeds

Note:
upgrade_cassandra.sh can be run independently to upgrade cassandra before provision
Usage is:
./upgrade_cassandra.sh <host_ip> <cassandra_home_dir> <image_id of added 3.0 image>

Change-Id: I1a655b129c9878f499aef58cb1b9057c7769d53c

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/18971
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/18972
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/18971
Committed: http://github.org/Juniper/contrail-puppet/commit/7ad7683f15dc676075647ca12b7afa02f1cca67c
Submitter: Zuul
Branch: master

commit 7ad7683f15dc676075647ca12b7afa02f1cca67c
Author: nitishkrishna <email address hidden>
Date: Thu Mar 31 17:55:17 2016 -0700

Closes-Bug: #1550650 - Re-entrant Upgrade cassandra script and upgrade fixes

Closes-Bug: #1545618 - Duplicate upgrade bug
This change:
1. Fixes upgrade_cassandra.sh to be re-entrant
2. Changes one augeas_conf_set to file line (needed as value itself has quotes)
3. Delete of additional package: contrail-openstack-analytics needed for old python-kafka upgrade issue

Change-Id: Id40eaf59629f94c26c2a6f0e2b47fa5cff7dbe81

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/18972
Committed: http://github.org/Juniper/contrail-puppet/commit/0e2bbb5f28994bd453ca5ab7da7f2b03a56786bc
Submitter: Zuul
Branch: R3.0

commit 0e2bbb5f28994bd453ca5ab7da7f2b03a56786bc
Author: nitishkrishna <email address hidden>
Date: Thu Mar 31 17:55:17 2016 -0700

Closes-Bug: #1550650 - Re-entrant Upgrade cassandra script and upgrade fixes

Closes-Bug: #1545618 - Duplicate upgrade bug
This change:
1. Fixes upgrade_cassandra.sh to be re-entrant
2. Changes one augeas_conf_set to file line (needed as value itself has quotes)
3. Delete of additional package: contrail-openstack-analytics needed for old python-kafka upgrade issue
4. appended File_line[‘’] to subscribe in upgrade_kernel.pp because some scripts ran before upgrade in Shashi setup
5. Changed one file_line to exec in config_cassandra (already merged in mainline)

Change-Id: Id40eaf59629f94c26c2a6f0e2b47fa5cff7dbe81