SM Lite: Mainline 2738 provision fails with config_started state

[sundarkh]

SM Lite: Mainline 2738 provision fails with error in state.yaml file

1) Installation of SM is fine and reimage gets completed fine
2) When the provision starts from config node, it fails with config_started with follwing error snippet

Jun 28 23:02:33 nodeg21 puppet-agent[26864]: (/Stage[config]/Contrail::Config::Neutron/Neutron_config[service_providers/service_provider]) Skipping because of failed dependencies
Jun 28 23:02:33 nodeg21 puppet-agent[26864]: (/Stage[config]/Neutron::Server/Service[neutron-server]) Dependency Exec[exec-provision-control] has failures: true
Jun 28 23:02:33 nodeg21 puppet-agent[26864]: (/Stage[config]/Neutron::Server/Service[neutron-server]) Skipping because of failed dependencies
Jun 28 23:02:33 nodeg21 puppet-agent[26864]: Finished catalog run in 24.19 seconds
Jun 28 23:02:34 nodeg21 puppet-agent[1708]: Checksumfile /var/lib/puppet/state/state.yaml is corrupt ((/var/lib/puppet/state/state.yaml): could not find expected ':' while scanning a simple key at line 1803 column 3); replacing
Jun 28 23:02:34 nodeg21 puppet-agent[1708]: Local environment: "production" doesn't match server specified node environment "image_contrail_packages_3_1_0_0_2738_kilo", switching agent to "image_contrail_packages_3_1_0_0_2738_kilo".gent to "image_contrail_packages_3_1_0_0_2738_kilo".
Jun 28 23:02:36 nodeg21 puppet-master[17903]: Compiled catalog for nodeg17.englab.juniper.net in environment image_contrail_packages_3_1_0_0_2738_kilo in 0.65 seconds
Jun 28 23:02:38 nodeg21 puppet-master[17896]: Compiled catalog for nodec38.englab.juniper.net in environment image_contrail_packages_3_1_0_0_2738_kilo in 0.63 seconds
Jun 28 23:02:41 nodeg21 kernel: [ 1973.759731] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-vhost instead.
Jun 28 23:02:41 nodeg21 kernel: [ 1973.765536] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-vhost instead.
Jun 28 23:02:46 nodeg21 puppet-master[17896]: (Scope(Class[Neutron::Server])) The auth_host parameter is deprecated. Please use auth_uri and identity_uri instead.
Jun 28 23:02:46 nodeg21 puppet-master[17896]: (Scope(Class[Neutron::Server])) The auth_port parameter is deprecated. Please use auth_uri and identity_uri instead.
Jun 28 23:02:46 nodeg21 puppet-master[17896]: (Scope(Class[Neutron::Server])) The auth_protocol parameter is deprecated. Please use auth_uri and identity_uri instead.
Jun 28 23:02:47 nodeg21 puppet-master[4660]: Starting Puppet master version 3.7.3
Jun 28 23:02:47 nodeg21 puppet-master[32745]: Compiled catalog for nodeg17.englab.juniper.net in environment image_contrail_packages_3_1_0_0_2738_kilo in 1.13 seconds
Jun 28 23:02:47 nodeg21 puppet-master[17896]: Compiled catalog for nodec58.englab.juniper.net in environment image_contrail_packages_3_1_0_0_2738_kilo in 2.18 seconds
Jun 28 23:02:47 nodeg21 puppet-master[17903]: (Scope(Class[Neutron::Server])) The auth_host parameter is deprecated. Please use auth_uri and identity_uri instead.
Jun 28 23:02:47 nodeg21 puppet-master[17903]: (Scope(Class[Neutron::Server])) The auth_port parameter is deprecated. Please use auth_uri and identity_uri instead.
Jun 28 23:02:47 nodeg21 puppet-master[17903]: (Scope(Class[Neutron::Server])) The auth_protocol parameter is deprecated. Please use auth_uri and identity_uri instead.
Jun 28 23:02:47 nodeg21 puppet-master[9311]: (Scope(Class[Neutron::Server])) The auth_host parameter is deprecated. Please use auth_uri and identity_uri instead.
Jun 28 23:02:47 nodeg21 puppet-master[9311]: (Scope(Class[Neutron::Server])) The auth_port parameter is deprecated. Please use auth_uri and identity_uri instead.
Jun 28 23:02:47 nodeg21 puppet-master[9311]: (Scope(Class[Neutron::Server])) The auth_protocol parameter is deprecated. Please use auth_uri and identity_uri instead.
Jun 28 23:02:47 nodeg21 puppet-master[17903]: Compiled catalog for nodeg21.englab.juniper.net in environment image_contrail_packages_3_1_0_0_2738_kilo in 2.16 seconds
Jun 28 23:02:48 nodeg21 puppet-master[9311]: Compiled catalog for nodeg33.englab.juniper.net in environment image_contrail_packages_3_1_0_0_2738_kilo in 2.11 seconds
Jun 28 23:02:49 nodeg21 puppet-agent[1708]: host_role = controlwebuiopenstackdatabasecollectorconfig and contrail_roles = {"database"=>true, "openstack"=>true, "config"=>true, "control"=>false, "collector"=>false, "webui"=>false, "compute"=>false}
Jun 28 23:02:49 nodeg21 puppet-agent[1708]: (/Stage[contrail]/Contrail::Params/Notify[host_role = controlwebuiopenstackdatabasecollectorconfig and contrail_roles = {"database"=>true, “opensta

Is this related to https://github.com/Juniper/contrail-server-manager/commit/436e19bf775468da9028fe279d76638bfde851cc

[sundarkh (sundar-kh)]

build 2739 , Non HA setup

Provision from SM LIte gets stuck

----> Provisioning the cluster
{
    "return_code": 9,
    "return_data": null,
    "return_msg": "AddrFormatError('invalid IPNetwork ',)"
}

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21814
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/21814
Committed: http://github.org/Juniper/contrail-server-manager/commit/22d60eceb13923e9e0559a25fb5f0dcdd50db1a4
Submitter: Zuul
Branch: master

commit 22d60eceb13923e9e0559a25fb5f0dcdd50db1a4
Author: nitishkrishna <email address hidden>
Date: Mon Jul 11 11:19:12 2016 -0700

Closes-Bug: #1597190 - Default gateway was mandatory for new params, made optional

For any interface, either default_gateway can be given or single gateway for the server as in old params

Change-Id: I6e8e4e4dcb0434cd8fc50b44cb4a0f8999546cef

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21998
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/21998
Committed: http://github.org/Juniper/contrail-server-manager/commit/65213d5cd8720ec30f472df9c01163fe6be09ee5
Submitter: Zuul
Branch: R3.1

commit 65213d5cd8720ec30f472df9c01163fe6be09ee5
Author: nitishkrishna <email address hidden>
Date: Mon Jul 11 11:19:12 2016 -0700

Closes-Bug: #1597190 - Default gateway was mandatory for new params, made optional

For any interface, either default_gateway can be given or single gateway for the server as in old params

Change-Id: I6e8e4e4dcb0434cd8fc50b44cb4a0f8999546cef

[sundarkh (sundar-kh)]

contrail contrail_exec_provision_control is python exec_provision_control.py --api_server_ip "10.204.217.176" --api_server_port 8082 --host_name_list "nodeg21,nodeg33" --host_ip_list "10.204.217.61,10.204.217.73" --router_asn "64512" --mt_options "None" && echo exec-provision-control >> /etc/contrail/contrail_config_exec.out
Jul 18 01:32:40 nodeg21 puppet-agent[30187]: (/Stage[config]/Contrail::Exec_provision_control/Notify[contrail contrail_exec_provision_control is python exec_provision_control.py --api_server_ip "10.204.217.176" --api_server_port 8082 --host_name_list "nodeg21,nodeg33" --host_ip_list "10.204.217.61,10.204.217.73" --router_asn "64512" --mt_options "None" && echo exec-provision-control >> /etc/contrail/contrail_config_exec.out]/message) defined 'message' as 'contrail contrail_exec_provision_control is python exec_provision_control.py --api_server_ip "10.204.217.176" --api_server_port 8082 --host_name_list "nodeg21,nodeg33" --host_ip_list "10.204.217.61,10.204.217.73" --router_asn "64512" --mt_options "None" && echo exec-provision-control >> /etc/contrail/contrail_config_exec.out'
Jul 18 01:32:41 nodeg21 puppet-master[12448]: Compiled catalog for nodeg17.englab.juniper.net in environment image_contrail_packages_3_1_0_0_2_kilo in 0.68 seconds
Jul 18 01:32:45 nodeg21 puppet-agent[30187]: python exec_provision_control.py --api_server_ip "10.204.217.176" --api_server_port 8082 --host_name_list "nodeg21,nodeg33" --host_ip_list "10.204.217.61,10.204.217.73" --router_asn "64512" --mt_options "None" && echo exec-provision-control >> /etc/contrail/contrail_config_exec.out returned 1 instead of one of [0]
Jul 18 01:32:45 nodeg21 puppet-agent[30187]: (/Stage[config]/Contrail::Exec_provision_control/Exec[exec-provision-control]/returns) change from notrun to 0 failed: python exec_provision_control.py --api_server_ip "10.204.217.176" --api_server_port 8082 --host_name_list "nodeg21,nodeg33" --host_ip_list "10.204.217.61,10.204.217.73" --router_asn "64512" --mt_options "None" && echo exec-provision-control >> /etc/contrail/contrail_config_exec.out returned 1 instead of one of [0]

Jul 18 01:32:46 nodeg21 puppet-agent[4564]: Checksumfile /var/lib/puppet/state/state.yaml is corrupt ((/var/lib/puppet/state/state.yaml): could not find expected ':' while scanning a simple key at line 1809 column 3);

SM Lite provision fails again with issue as mentioned in bug description with ( R3.1 build 2)

[Nitish Krishna Kaveri (nitishk)]

When issue was reflagged, it was due to wrong params in cluster JSON

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22223
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22224
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22223
Committed: http://github.org/Juniper/contrail-server-manager/commit/c4e3a29db77b0c7c71f60f96b8e90f94d6c5cc48
Submitter: Zuul
Branch: R3.1

commit c4e3a29db77b0c7c71f60f96b8e90f94d6c5cc48
Author: nitishkrishna <email address hidden>
Date: Wed Jul 20 11:49:23 2016 -0700

Closes-Bug: #1597190 - Default gateway is being set as empty in SM Lite by default

Previously we were handling case where default gateway key in missing from JSON
But SM Lite generated server JSONS have this key but have value as empty
This scenario needs to be handled

Change-Id: Ibd77c9fcc4fe5361b1ce6138754977dffc329573

[Sudheendra Rao (sudheendra-k)]

problem is still seen on R3.1 build3

[Abhay Joshi (abhayj)]

Build 3 does not have code merged yet. Should be available in next build

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22224
Committed: http://github.org/Juniper/contrail-server-manager/commit/6572f4c0b18d3b3cdf68e59303b5ae356399ba31
Submitter: Zuul
Branch: master

commit 6572f4c0b18d3b3cdf68e59303b5ae356399ba31
Author: nitishkrishna <email address hidden>
Date: Wed Jul 20 11:49:23 2016 -0700

Closes-Bug: #1597190 - Default gateway is being set as empty in SM Lite by default

Previously we were handling case where default gateway key in missing from JSON
But SM Lite generated server JSONS have this key but have value as empty
This scenario needs to be handled

Change-Id: Ibd77c9fcc4fe5361b1ce6138754977dffc329573

[sundarkh (sundar-kh)]

I see the issue again in Build 5, (Fix has gone in build 5)

Jul 23 03:52:20 nodeg37 puppet-agent[15295]: python exec_provision_control.py --api_server_ip "10.204.217.77" --api_server_port 8082 --host_name_list "nodeg37" --host_ip_list "10.204.217.77" --router_asn "64512" --mt_options "None" && echo exec-provision-control >> /etc/contrail/contrail_config_exec.out returned 1 instead of one of [0]
Jul 23 03:52:20 nodeg37 puppet-agent[15295]: (/Stage[config]/Contrail::Exec_provision_control/Exec[exec-provision-control]/returns) change from notrun to 0 failed: python exec_provision_control.py --api_server_ip "10.204.217.77" --api_server_port 8082 --host_name_list "nodeg37" --host_ip_list "10.204.217.77" --router_asn "64512" --mt_options "None" && echo exec-provision-control >> /etc/contrail/contrail_config_exec.out returned 1 instead of one of [0]

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22408
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22408
Committed: http://github.org/Juniper/contrail-puppet/commit/b904264909aa34371f38b14d58eb426520e686dd
Submitter: Zuul
Branch: R3.1

commit b904264909aa34371f38b14d58eb426520e686dd
Author: Dheeraj Gautam <email address hidden>
Date: Mon Jul 25 22:21:34 2016 -0700

SM-Mitaka: fix keystone default domain for kilo/liberty

Closes-Bug: #1597190
Closes-Bug: #1605823

missed patch2 of https://review.opencontrail.org/#/c/22243/

Change-Id: Ibd82d9e73051b3a2a13fbc06feb0c3afcd622bf2

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22734
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22734
Committed: http://github.org/Juniper/contrail-puppet/commit/66f2c1a23d3f98eed530e8887c5ccd233cee53d8
Submitter: Zuul
Branch: master

commit 66f2c1a23d3f98eed530e8887c5ccd233cee53d8
Author: Dheeraj Gautam <email address hidden>
Date: Mon Jul 25 22:21:34 2016 -0700

SM-Mitaka: fix keystone default domain for kilo/liberty

Closes-Bug: #1597190
Closes-Bug: #1605823

missed patch2 of https://review.opencontrail.org/#/c/22243/

Change-Id: Ibd82d9e73051b3a2a13fbc06feb0c3afcd622bf2