SM provision of a ha cluster hangs at u'openstack' due to heat encryption key related issue

[wenqing liang]

R3.1.0.0 liberty SM

SM provision of a ha cluster hangs at u'openstack' due to heat encryption key related issue per Dheeraj:

"The setup had 0E59160040634493 (auto-generated) as key. this was failing as puppet was not treating as string. instead it was treated as hex number and failed."

Aug 6 22:02:46 a5d01e08-2 puppet-master[26526]: (Scope(Class[Nova::Api])) The auth_host parameter is deprecated. Please use auth_uri and identity_uri instead.
Aug 6 22:02:46 a5d01e08-2 puppet-master[26526]: (Scope(Class[Nova::Api])) The auth_port parameter is deprecated. Please use auth_uri and identity_uri instead.
Aug 6 22:02:46 a5d01e08-2 puppet-master[26526]: (Scope(Class[Nova::Api])) The auth_protocol parameter is deprecated. Please use auth_uri and identity_uri instead.
Aug 6 22:02:46 a5d01e08-2 puppet-master[26526]: (Scope(Class[Nova::Vncproxy::Common])) Could not look up qualified variable '::nova::compute::vncproxy_host'; class ::nova::compute has not been evaluated
Aug 6 22:02:46 a5d01e08-2 puppet-master[26526]: (Scope(Class[Nova::Vncproxy::Common])) Could not look up qualified variable '::nova::compute::vncproxy_protocol'; class ::nova::compute has not been evaluated
Aug 6 22:02:46 a5d01e08-2 puppet-master[26526]: (Scope(Class[Nova::Vncproxy::Common])) Could not look up qualified variable '::nova::compute::vncproxy_port'; class ::nova::compute has not been evaluated
Aug 6 22:02:46 a5d01e08-2 puppet-master[26526]: (Scope(Class[Nova::Vncproxy::Common])) Could not look up qualified variable '::nova::compute::vncproxy_path'; class ::nova::compute has not been evaluated
Aug 6 22:02:46 a5d01e08-2 puppet-master[26526]: (Scope(Class[Heat])) The keystone_host parameter is deprecated. Please use auth_uri and identity_uri instead.
Aug 6 22:02:46 a5d01e08-2 puppet-master[26526]: (Scope(Class[Heat])) The keystone_port parameter is deprecated. Please use auth_uri and identity_uri instead.
Aug 6 22:02:46 a5d01e08-2 puppet-master[26526]: (Scope(Class[Heat])) The keystone_protocol parameter is deprecated. Please use auth_uri and identity_uri instead.
Aug 6 22:02:46 a5d01e08-2 puppet-master[26526]: size(): Requires either string, array or hash to work with at /etc/puppet/environments/contrail_r31_liberty17_tgz/modules/heat/manifests/engine.pp:104 on node a5d01e03.englab.juniper.net
Aug 6 22:02:47 a5d01e08-2 dhcpd: DHCPDISCOVER from 00:e0:81:c3:88:fc via eth0: network 10.87.128.0/19: no free leases
Aug 6 22:02:47 a5d01e08-2 dhcpd: DHCPDISCOVER from 00:25:90:7e:10:ac via eth0: network 10.87.128.0/19: no free leases
Aug 6 22:02:41 a5d01e08-2 puppet-master[27929]: message repeated 2 times: [ size(): Requires either string, array or hash to work with at /etc/puppet/environments/contrail_r31_liberty17_tgz/modules/heat/manifests/engine.pp:104 on node a5d01e04.englab.juniper.net]
Aug 6 22:02:48 a5d01e08-2 puppet-master[27929]: (Scope(Class[Memcached])) type() DEPRECATED: This function will cease to function on Puppet 4; please use type3x() before upgrading to puppet 4 for backwards-compatibility, or migrate to the new parser's typing system.

Aug 6 10:32:52 a5d01e03 kernel: [ 4630.208509] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-vhost instead.
Aug 6 10:32:56 a5d01e03 puppet-agent[11786]: Could not retrieve catalog from remote server: Error 400 on SERVER: size(): Requires either string, array or hash to work with at /etc/puppet/environments/contrail_r31_liberty17_tgz/modules/heat/manifests/engine.pp:104 on node a5d01e03.englab.juniper.net
Aug 6 10:32:56 a5d01e03 puppet-agent[11786]: Not using cache on failed catalog
Aug 6 10:32:56 a5d01e03 puppet-agent[11786]: Could not retrieve catalog; skipping run
Aug 6 10:32:59 a5d01e03 puppet-agent[12502]: Local environment: "production" doesn't match server specified node environment "contrail_r31_liberty17_tgz", switching agent to "contrail_r31_liberty17_tgz".
Aug 6 10:33:02 a5d01e03 kernel: [ 4640.248749] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-vhost instead.
Aug 6 10:33:02 a5d01e03 kernel: [ 4640.256304] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-vhost instead.
Aug 6 10:33:06 a5d01e03 puppet-agent[12502]: Could not retrieve catalog from remote server: Error 400 on SERVER: size(): Requires either string, array or hash to work with at /etc/puppet/environments/contrail_r31_liberty17_tgz/modules/heat/manifests/engine.pp:104 on node a5d01e03.englab.juniper.net
Aug 6 10:33:06 a5d01e03 puppet-agent[12502]: Not using cache on failed catalog
Aug 6 10:33:06 a5d01e03 puppet-agent[12502]: Could not retrieve catalog; skipping run

[Abhay Joshi (abhayj)]

Wenqing,

This is mostly a config issue on your setup. Dheeraj will confirm.

[Dheeraj Gautam (dgautam)]

ISSUE:
======
If admin has not provided a heat_encryption key, Server Manager generates a alphanumeric key of 16 characters length. At one such instance, heat::engine is reporting wrong type for heat_encryption_key and puppet function size() fails for that.

ROOT CAUSE:
===========
Few of the auto-generated value may start with "0" (zero), this value is treat by size() as number instead of string and it fails the check for size(),

Issue is not observed when admin has configured a value that starts with alphabet or Server-Manager generated a value that starts with alphabet

E.g "0E59160040634493" fails but "E059160040634493" doesn't.

WORK-AROUND/FIX:
================
1. Configure cluster with heat_encryption_key that starts with alphabet.

[Nagabhushana R (bhushana)]

tgz is not being released as part of 3.1

[Abhay Joshi (abhayj)]

Though old title said tgz provisioning, the issue described is common to debian too. Hence reopening the bug and marking for release note. Also modified the bug description to remove tgz.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25685
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25686
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25687
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25685
Committed: http://github.org/Juniper/contrail-server-manager/commit/137c0d4e4e76acb5719f68990f284858668873bf
Submitter: Zuul
Branch: R3.1

commit 137c0d4e4e76acb5719f68990f284858668873bf
Author: nitishkrishna <email address hidden>
Date: Thu Nov 3 18:04:44 2016 -0700

Closes-Bug: #1610813 - Heat Encryption Key (generated or not) shouldn't contain 'E'

The 'E' is read by Puppet size function as a float with Exponential component
The char E is now replaced with D so that it isn't mistaken for a float number

Change-Id: I5e27d6be6b8c06cef4ca0f36da79991ec314cf47

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25687
Committed: http://github.org/Juniper/contrail-server-manager/commit/00e0e7b2548ec1f83c5898ddf30381b6d39be61f
Submitter: Zuul
Branch: master

commit 00e0e7b2548ec1f83c5898ddf30381b6d39be61f
Author: nitishkrishna <email address hidden>
Date: Thu Nov 3 18:04:44 2016 -0700

Closes-Bug: #1610813 - Heat Encryption Key (generated or not) shouldn't contain 'E'

The 'E' is read by Puppet size function as a float with Exponential component
The char E is now replaced with D so that it isn't mistaken for a float number

Change-Id: I5e27d6be6b8c06cef4ca0f36da79991ec314cf47

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25686
Committed: http://github.org/Juniper/contrail-server-manager/commit/2502d12b8336d459bd8df2085521795fa7cb8ff1
Submitter: Zuul
Branch: R3.2

commit 2502d12b8336d459bd8df2085521795fa7cb8ff1
Author: nitishkrishna <email address hidden>
Date: Thu Nov 3 18:04:44 2016 -0700

Closes-Bug: #1610813 - Heat Encryption Key (generated or not) shouldn't contain 'E'

The 'E' is read by Puppet size function as a float with Exponential component
The char E is now replaced with D so that it isn't mistaken for a float number

Change-Id: I5e27d6be6b8c06cef4ca0f36da79991ec314cf47