sudo config_controller failed with error"Evaluation Error: Unknown variable: '::platform_res_mem'. at /usr/share/puppet/modules/platform/manifests/params.pp:37:23"

[Changjin Wang]

Title
-----
sudo config_controller failed with error"Evaluation Error: Unknown variable: '::platform_res_mem'. at /usr/share/puppet/modules/platform/manifests/params.pp:37:23"

Brief Description
-----------------
Install Controller0 in KVM Guest VM
first time "sudo config_controller"

Severity
--------
Critical: System/Feature is not usable after the defect

Steps to Reproduce
------------------
virsh console stx-ctrl-0
sudo config_controller

Expected Behavior
------------------
config_controller success

Actual Behavior
----------------
Error: 2019-01-15 08:30:37 +0000 Evaluation Error: Unknown variable: '::platform_res_mem'. at /usr/share/puppet/modules/platform/manifests/params.pp:37:23 on node localhost

Reproducibility
---------------
State if the issue is 100% reproducible

System Configuration
--------------------
Multi-node system

Branch/Pull Time/Commit
-----------------------
http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_green/outputs/

Timestamp/Logs
--------------
/var/log/puppet/latest/puppet.log

2019-01-15T08:30:38.768 Debug: 2019-01-15 08:30:37 +0000 Performing a hiera indirector lookup of platform::params::security_feature with options {:variables=>Scope(Class[Platform::Params]), :merge=>#<Puppet::Pops::FirstFoundStrategy:0x00000005d58608 @options_t=#<Puppet::Pops::Types::PStructType:0x00000005d5d9c8 @elements=[#<Puppet::Pops::Types::PStructElement:0x00000005d5d9f0 @key_type=#<Puppet::Pops::Types::POptionalType:0x00000005d5da18 @type=#<Puppet::Pops::Types::PStringType:0x00000005d5da68 @size_type=nil, @values=["strategy"]>>, @value_type=#<Puppet::Pops::Types::POptionalType:0x00000005d5db58 @type=#<Puppet::Pops::Types::PPatternType:0x00000005d5dba8 @patterns=[#<Puppet::Pops::Types::PRegexpType:0x00000005d5dcc0 @pattern="first", @regexp=/first/>]>>>]>, @options={}>}
2019-01-15T08:30:38.773 Debug: 2019-01-15 08:30:37 +0000 hiera(): Looking up platform::params::security_feature in YAML backend
2019-01-15T08:30:38.777 Debug: 2019-01-15 08:30:37 +0000 hiera(): Looking for data source runtime
2019-01-15T08:30:38.781 Debug: 2019-01-15 08:30:37 +0000 hiera(): Cannot find datafile /tmp/puppet/hieradata/runtime.yaml, skipping
2019-01-15T08:30:38.786 Debug: 2019-01-15 08:30:37 +0000 hiera(): Looking for data source host
2019-01-15T08:30:38.790 Debug: 2019-01-15 08:30:37 +0000 hiera(): Cannot find datafile /tmp/puppet/hieradata/host.yaml, skipping
2019-01-15T08:30:38.795 Debug: 2019-01-15 08:30:37 +0000 hiera(): Looking for data source secure_system
2019-01-15T08:30:38.800 Debug: 2019-01-15 08:30:37 +0000 hiera(): Cannot find datafile /tmp/puppet/hieradata/secure_system.yaml, skipping
2019-01-15T08:30:38.804 Debug: 2019-01-15 08:30:37 +0000 hiera(): Looking for data source system
2019-01-15T08:30:38.808 Debug: 2019-01-15 08:30:37 +0000 hiera(): Cannot find datafile /tmp/puppet/hieradata/system.yaml, skipping
2019-01-15T08:30:38.813 Debug: 2019-01-15 08:30:37 +0000 hiera(): Looking for data source secure_static
2019-01-15T08:30:38.817 Debug: 2019-01-15 08:30:37 +0000 hiera(): Looking for data source static
2019-01-15T08:30:38.822 Debug: 2019-01-15 08:30:37 +0000 hiera(): Looking for data source personality
2019-01-15T08:30:38.826 Debug: 2019-01-15 08:30:37 +0000 hiera(): Looking for data source global
2019-01-15T08:30:38.831 Debug: 2019-01-15 08:30:37 +0000 hiera(): Found platform::params::security_feature in global
2019-01-15T08:30:38.835 Error: 2019-01-15 08:30:37 +0000 Evaluation Error: Unknown variable: '::platform_res_mem'. at /usr/share/puppet/modules/platform/manifests/params.pp:37:23 on node localhost
2019-01-15T08:30:38.839 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:39:in `optionally_fail'
2019-01-15T08:30:38.844 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:99:in `get_variable_value'
2019-01-15T08:30:38.849 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:1041:in `eval_VariableExpression'
2019-01-15T08:30:38.854 /usr/share/ruby/vendor_ruby/puppet/pops/visitor.rb:69:in `visit_this_1'
2019-01-15T08:30:38.858 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
2019-01-15T08:30:38.862 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:376:in `eval_ArithmeticExpression'
2019-01-15T08:30:38.866 /usr/share/ruby/vendor_ruby/puppet/pops/visitor.rb:69:in `visit_this_1'
2019-01-15T08:30:38.871 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
2019-01-15T08:30:38.875 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:359:in `eval_AssignmentExpression'
2019-01-15T08:30:38.879 /usr/share/ruby/vendor_ruby/puppet/pops/visitor.rb:69:in `visit_this_1'
2019-01-15T08:30:38.884 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
2019-01-15T08:30:38.888 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:638:in `block in eval_BlockExpression'
2019-01-15T08:30:38.893 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:638:in `each'
2019-01-15T08:30:38.897 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:638:in `reduce'
2019-01-15T08:30:38.902 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:638:in `eval_BlockExpression'
2019-01-15T08:30:38.906 /usr/share/ruby/vendor_ruby/puppet/pops/visitor.rb:69:in `visit_this_1'
2019-01-15T08:30:38.911 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
2019-01-15T08:30:38.915 /usr/share/ruby/vendor_ruby/puppet/pops/parser/evaluating_parser.rb:63:in `evaluate'
2019-01-15T08:30:38.920 /usr/share/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:32:in `evaluate'
2019-01-15T08:30:38.924 /usr/share/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:72:in `block (2 levels) in evaluate'
2019-01-15T08:30:38.928 /usr/share/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:71:in `catch'
2019-01-15T08:30:38.932 /usr/share/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:71:in `block in evaluate'
2019-01-15T08:30:38.937 /usr/share/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:70:in `catch'
2019-01-15T08:30:38.941 /usr/share/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:70:in `evaluate'
2019-01-15T08:30:38.945 /usr/share/ruby/vendor_ruby/puppet/parser/ast.rb:31:in `safeevaluate'
2019-01-15T08:30:38.950 /usr/share/ruby/vendor_ruby/puppet/resource/type.rb:184:in `evaluate_code'
2019-01-15T08:30:38.954 /usr/share/ruby/vendor_ruby/puppet/parser/resource.rb:81:in `block in evaluate'
2019-01-15T08:30:38.959 /usr/share/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
2019-01-15T08:30:38.963 /usr/share/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
2019-01-15T08:30:38.967 /usr/share/ruby/vendor_ruby/puppet/parser/resource.rb:73:in `evaluate'
2019-01-15T08:30:38.972 /usr/share/ruby/vendor_ruby/puppet/parser/compiler.rb:395:in `each'
2019-01-15T08:30:38.977 /usr/share/ruby/vendor_ruby/puppet/parser/compiler.rb:395:in `evaluate_classes'
2019-01-15T08:30:38.982 /usr/share/ruby/vendor_ruby/puppet/pops/loader/../../../puppet/functions/include.rb:16:in `include'
2019-01-15T08:30:38.987 /usr/share/ruby/vendor_ruby/puppet/pops/functions/dispatch.rb:47:in `invoke'
2019-01-15T08:30:38.992 /usr/share/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:36:in `block in dispatch'
2019-01-15T08:30:38.996 /usr/share/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:35:in `catch'
2019-01-15T08:30:39.001 /usr/share/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:35:in `dispatch'
2019-01-15T08:30:39.006 /usr/share/ruby/vendor_ruby/puppet/pops/functions/function.rb:46:in `block in call'
2019-01-15T08:30:39.010 /usr/share/ruby/vendor_ruby/puppet/pops/functions/function.rb:45:in `catch'
2019-01-15T08:30:39.014 /usr/share/ruby/vendor_ruby/puppet/pops/functions/function.rb:45:in `call'
2019-01-15T08:30:39.019 /usr/share/puppet/modules/platform/manifests/config.pp:330:in `block in call_function'
2019-01-15T08:30:39.024 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:284:in `eval'
2019-01-15T08:30:39.029 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:284:in `block in call_function'
2019-01-15T08:30:39.033 /usr/share/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
2019-01-15T08:30:39.037 /usr/share/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
2019-01-15T08:30:39.041 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:282:in `call_function'
2019-01-15T08:30:39.047 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:941:in `call_function_with_block'
2019-01-15T08:30:39.051 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:910:in `eval_CallNamedFunctionExpression'
2019-01-15T08:30:39.055 /usr/share/ruby/vendor_ruby/puppet/pops/visitor.rb:69:in `visit_this_1'
2019-01-15T08:30:39.060 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
2019-01-15T08:30:39.064 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:638:in `block in eval_BlockExpression'
2019-01-15T08:30:39.069 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:638:in `each'
2019-01-15T08:30:39.073 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:638:in `reduce'
2019-01-15T08:30:39.079 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:638:in `eval_BlockExpression'
2019-01-15T08:30:39.083 /usr/share/ruby/vendor_ruby/puppet/pops/visitor.rb:69:in `visit_this_1'
2019-01-15T08:30:39.089 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
2019-01-15T08:30:39.094 /usr/share/ruby/vendor_ruby/puppet/pops/parser/evaluating_parser.rb:63:in `evaluate'
2019-01-15T08:30:39.098 /usr/share/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:32:in `evaluate'
2019-01-15T08:30:39.103 /usr/share/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:72:in `block (2 levels) in evaluate'
2019-01-15T08:30:39.107 /usr/share/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:71:in `catch'
2019-01-15T08:30:39.112 /usr/share/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:71:in `block in evaluate'
2019-01-15T08:30:39.116 /usr/share/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:70:in `catch'
2019-01-15T08:30:39.121 /usr/share/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:70:in `evaluate'
2019-01-15T08:30:39.126 /usr/share/ruby/vendor_ruby/puppet/parser/ast.rb:31:in `safeevaluate'
2019-01-15T08:30:39.132 /usr/share/ruby/vendor_ruby/puppet/resource/type.rb:184:in `evaluate_code'
2019-01-15T08:30:39.136 /usr/share/ruby/vendor_ruby/puppet/parser/resource.rb:81:in `block in evaluate'
2019-01-15T08:30:39.141 /usr/share/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
2019-01-15T08:30:39.146 /usr/share/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
2019-01-15T08:30:39.149 /usr/share/ruby/vendor_ruby/puppet/parser/resource.rb:73:in `evaluate'
2019-01-15T08:30:39.154 /usr/share/ruby/vendor_ruby/puppet/parser/compiler.rb:395:in `each'
2019-01-15T08:30:39.159 /usr/share/ruby/vendor_ruby/puppet/parser/compiler.rb:395:in `evaluate_classes'
2019-01-15T08:30:39.163 /usr/share/ruby/vendor_ruby/puppet/pops/loader/../../../puppet/functions/include.rb:16:in `include'
2019-01-15T08:30:39.168 /usr/share/ruby/vendor_ruby/puppet/pops/functions/dispatch.rb:47:in `invoke'
2019-01-15T08:30:39.173 /usr/share/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:36:in `block in dispatch'
2019-01-15T08:30:39.177 /usr/share/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:35:in `catch'
2019-01-15T08:30:39.182 /usr/share/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:35:in `dispatch'
2019-01-15T08:30:39.187 /usr/share/ruby/vendor_ruby/puppet/pops/functions/function.rb:46:in `block in call'
2019-01-15T08:30:39.191 /usr/share/ruby/vendor_ruby/puppet/pops/functions/function.rb:45:in `catch'
2019-01-15T08:30:39.196 /usr/share/ruby/vendor_ruby/puppet/pops/functions/function.rb:45:in `call'
2019-01-15T08:30:39.201 /etc/puppet/manifests/bootstrap.pp:10:in `block in call_function'
2019-01-15T08:30:39.205 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:284:in `eval'
2019-01-15T08:30:39.209 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:284:in `block in call_function'
2019-01-15T08:30:39.214 /usr/share/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
2019-01-15T08:30:39.218 /usr/share/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
2019-01-15T08:30:39.223 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:282:in `call_function'
2019-01-15T08:30:39.228 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:941:in `call_function_with_block'
2019-01-15T08:30:39.232 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:910:in `eval_CallNamedFunctionExpression'
2019-01-15T08:30:39.236 /usr/share/ruby/vendor_ruby/puppet/pops/visitor.rb:48:in `block in visit_this'
2019-01-15T08:30:39.241 /usr/share/ruby/vendor_ruby/puppet/pops/visitor.rb:42:in `each'
2019-01-15T08:30:39.246 /usr/share/ruby/vendor_ruby/puppet/pops/visitor.rb:42:in `visit_this'
2019-01-15T08:30:39.250 /usr/share/ruby/vendor_ruby/puppet/pops/visitor.rb:71:in `visit_this_1'
2019-01-15T08:30:39.255 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
2019-01-15T08:30:39.259 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:638:in `block in eval_BlockExpression'
2019-01-15T08:30:39.264 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:638:in `each'
2019-01-15T08:30:39.269 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:638:in `reduce'
2019-01-15T08:30:39.274 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:638:in `eval_BlockExpression'
2019-01-15T08:30:39.277 /usr/share/ruby/vendor_ruby/puppet/pops/visitor.rb:48:in `block in visit_this'
2019-01-15T08:30:39.282 /usr/share/ruby/vendor_ruby/puppet/pops/visitor.rb:42:in `each'
2019-01-15T08:30:39.287 /usr/share/ruby/vendor_ruby/puppet/pops/visitor.rb:42:in `visit_this'
2019-01-15T08:30:39.291 /usr/share/ruby/vendor_ruby/puppet/pops/visitor.rb:71:in `visit_this_1'
2019-01-15T08:30:39.296 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
2019-01-15T08:30:39.301 /etc/puppet/manifests/bootstrap.pp:in `stack'
2019-01-15T08:30:39.305 /usr/share/ruby/vendor_ruby/puppet/pops/puppet_stack.rb:30:in `eval'
2019-01-15T08:30:39.310 /usr/share/ruby/vendor_ruby/puppet/pops/puppet_stack.rb:30:in `stack'
2019-01-15T08:30:39.314 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:715:in `eval_Program'
2019-01-15T08:30:39.319 /usr/share/ruby/vendor_ruby/puppet/pops/visitor.rb:48:in `block in visit_this'
2019-01-15T08:30:39.323 /usr/share/ruby/vendor_ruby/puppet/pops/visitor.rb:42:in `each'
2019-01-15T08:30:39.328 /usr/share/ruby/vendor_ruby/puppet/pops/visitor.rb:42:in `visit_this'
2019-01-15T08:30:39.332 /usr/share/ruby/vendor_ruby/puppet/pops/visitor.rb:71:in `visit_this_1'
2019-01-15T08:30:39.337 /usr/share/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
2019-01-15T08:30:39.342 /usr/share/ruby/vendor_ruby/puppet/pops/parser/evaluating_parser.rb:63:in `evaluate'
2019-01-15T08:30:39.345 /usr/share/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:132:in `evaluate'
2019-01-15T08:30:39.349 /usr/share/ruby/vendor_ruby/puppet/parser/ast.rb:31:in `safeevaluate'
2019-01-15T08:30:39.354 /usr/share/ruby/vendor_ruby/puppet/resource/type.rb:184:in `evaluate_code'
2019-01-15T08:30:39.358 /usr/share/ruby/vendor_ruby/puppet/parser/resource.rb:81:in `block in evaluate'
2019-01-15T08:30:39.363 /usr/share/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
2019-01-15T08:30:39.368 /usr/share/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
2019-01-15T08:30:39.372 /usr/share/ruby/vendor_ruby/puppet/parser/resource.rb:73:in `evaluate'
2019-01-15T08:30:39.376 /usr/share/ruby/vendor_ruby/puppet/parser/compiler.rb:639:in `evaluate_main'
2019-01-15T08:30:39.381 /usr/share/ruby/vendor_ruby/puppet/parser/compiler.rb:174:in `block (2 levels) in compile'
2019-01-15T08:30:39.386 /usr/share/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
2019-01-15T08:30:39.390 /usr/share/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
2019-01-15T08:30:39.394 /usr/share/ruby/vendor_ruby/puppet/parser/compiler.rb:174:in `block in compile'
2019-01-15T08:30:39.399 /usr/share/ruby/vendor_ruby/puppet/context.rb:65:in `override'
2019-01-15T08:30:39.405 /usr/share/ruby/vendor_ruby/puppet.rb:241:in `override'
2019-01-15T08:30:39.410 /usr/share/ruby/vendor_ruby/puppet/parser/compiler.rb:162:in `compile'
2019-01-15T08:30:39.415 /usr/share/ruby/vendor_ruby/puppet/parser/compiler.rb:33:in `compile'
2019-01-15T08:30:39.420 /usr/share/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:266:in `block (2 levels) in compile'
2019-01-15T08:30:39.425 /usr/share/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
2019-01-15T08:30:39.430 /usr/share/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
2019-01-15T08:30:39.434 /usr/share/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:264:in `block in compile'
2019-01-15T08:30:39.439 /usr/share/ruby/vendor_ruby/puppet/util.rb:223:in `block in benchmark'
2019-01-15T08:30:39.443 /usr/share/ruby/benchmark.rb:296:in `realtime'
2019-01-15T08:30:39.448 /usr/share/ruby/vendor_ruby/puppet/util.rb:222:in `benchmark'
2019-01-15T08:30:39.453 /usr/share/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:262:in `compile'
2019-01-15T08:30:39.458 /usr/share/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:53:in `find'
2019-01-15T08:30:39.462 /usr/share/ruby/vendor_ruby/puppet/indirector/indirection.rb:194:in `find'
2019-01-15T08:30:39.467 /usr/share/ruby/vendor_ruby/puppet/application/apply.rb:256:in `block in main'
2019-01-15T08:30:39.472 /usr/share/ruby/vendor_ruby/puppet/context.rb:65:in `override'
2019-01-15T08:30:39.476 /usr/share/ruby/vendor_ruby/puppet.rb:241:in `override'
2019-01-15T08:30:39.480 /usr/share/ruby/vendor_ruby/puppet/application/apply.rb:225:in `main'
2019-01-15T08:30:39.485 /usr/share/ruby/vendor_ruby/puppet/application/apply.rb:170:in `run_command'
2019-01-15T08:30:39.490 /usr/share/ruby/vendor_ruby/puppet/application.rb:344:in `block in run'
2019-01-15T08:30:39.494 /usr/share/ruby/vendor_ruby/puppet/util.rb:540:in `exit_on_fail'
2019-01-15T08:30:39.499 /usr/share/ruby/vendor_ruby/puppet/application.rb:344:in `run'
2019-01-15T08:30:39.503 /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:132:in `run'
2019-01-15T08:30:39.508 /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:72:in `execute'
2019-01-15T08:30:39.513 /usr/bin/puppet:5:in `<main>'

[Ghada Khalil (gkhalil)]

Needs further investigation as this issue is not reported in the sanity run on the CENGN build

[yong hu (yhu6)]

I couldn't reproduce this failure on recent build. As follow-up, I am contacting the submitter for more information.

[Changjin Wang (wang-changjin)]

1. Where was your boot image from? Your own built or downloading somewhere? (if so, pls share the URL.)

//ISO download url http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190115T060000Z/outputs/iso/bootimage.iso

2. When you were installing this Controller-0, what config did you select? There are a few options:

                Standard Controller Configuration, All-in-One Controller Configuration, or All-in-One Controller (Low latency) Configuration?
//I try both Standard Controller Configuration, All-in-One Controller Configuration,the same error

3. Did you choose STANDARD Security Boot Profile or EXTENDED Security Boot Profile?
// STANDARD Security Boot Profile

4. Could you upload the complete puppet.log from /var/log/puppet/latest/?
// see log.txt

[yong hu (yhu6)]

I installed this image on my QEMU-KVM and it worked. (See the screenshot as attached).

[Changjin Wang (wang-changjin)]

Thie is my VM install shell,can you find any problem?

for i in {1..4}; do
virsh net-define /dev/stdin <<EOF
<network>
  <name>${BRIDGE_INTERFACE}$i</name>
  <bridge name="${BRIDGE_INTERFACE}$i" />
  <forward mode="bridge"/>
</network>
EOF
virsh net-autostart ${BRIDGE_INTERFACE}$i
virsh net-start ${BRIDGE_INTERFACE}$i
done
sudo qemu-img create -f qcow2 /var/lib/libvirt/images/stx-disk0.qcow2 600G
sudo qemu-img create -f qcow2 /var/lib/libvirt/images/stx-disk1.qcow2 200G
sudo qemu-img create -f qcow2 /var/lib/libvirt/images/stx-disk2.qcow2 200G
sudo chown libvirt-qemu:kvm /var/lib/libvirt/images/stx-disk0.qcow2
sudo chown libvirt-qemu:kvm /var/lib/libvirt/images/stx-disk1.qcow2
sudo chown libvirt-qemu:kvm /var/lib/libvirt/images/stx-disk2.qcow2

sudo virt-install --connect qemu:///system --name stx-ctrl-0 \
  --ram 32768 --vcpus=16,maxvcpus=16,sockets=2,cores=8 \
  --network network=${BRIDGE_INTERFACE}1,model=e1000 --network network=${BRIDGE_INTERFACE}2,model=e1000 \
  --network network=${BRIDGE_INTERFACE}3,model=virtio --network network=${BRIDGE_INTERFACE}4,model=virtio \
  --disk path=/var/lib/libvirt/images/stx-disk0.qcow2,format=qcow2,device=disk,bus=sata \
  --disk path=/var/lib/libvirt/images/stx-disk1.qcow2,format=qcow2,device=disk,bus=sata \
  --disk path=/var/lib/libvirt/images/stx-disk2.qcow2,format=qcow2,device=disk,bus=sata \
  --cdrom /var/lib/libvirt/images/bootimage.iso \
  --graphics=vnc,listen=0.0.0.0 --hvm --cpu=IvyBridge,+vmx \
  --os-type linux --os-variant=rhel7

[yong hu (yhu6)]

Very likely the root cause was memory "KVM Guest VM" was too small.
The analysis is following: This variable '::platform_res_mem was supposedly to assigned in "/usr/share/puppet/modules/platform/lib/facter/platform_res_mem.rb" by "memtop". However, due to the lack of available+anonymous memory, this variable was not having a valid value. And consequently it led to the error message as shown in puppet.log, when it was being used by "params.pp", during the 2nd step of "config_controller" (02/08: Applying bootstrap manifest).

In my environment, VM has 12GB memory for controller-0, and it seems fine.
So strongly suggest the submitter to increase the size of RAM and try the installation again.

In addition, 16GB memory for controller is preferable. As well, we need to secure the hard disk size, like, > 250GB.

BTW: In near future, this kind of HW requirements need be increased further, thanks to the new design of containerization systems.

[Changjin Wang (wang-changjin)]

But I set the VM Mem 32GB(32768),is it too small?

[yong hu (yhu6)]

it seems "--ram" is deprecated, so have a try with --memory

--memory OPTIONS
           Memory to allocate for the guest, in MiB. Sub options are available, like 'maxmemory' and 'hugepages'. This deprecates the -r/--ram option.

As well, in your Linux host, to dump the VM config xml, by this cmd:
$ sudo virsh dumpxml <your_controller_0_VM_instance or ID>

it shows the actual VM configs.
See mime in the attachment.

[yong hu (yhu6)]

Or, on host to run this command to dump memory data of your VM.

$ virsh dommemstat <vm_id>

[Changjin Wang (wang-changjin)]

ubuntu@kvm-ubuntu:~$ virsh list
 Id Name State
----------------------------------------------------
 1 nsx-manager running
 2 nsx-controller running
 31 devstack-ubuntu running
 33 stx-ctrl-0 running

ubuntu@kvm-ubuntu:~$ virsh dommemstat 33
actual 33554432
swap_in 0
swap_out 0
major_fault 207
minor_fault 137564
unused 32469476
available 32779824
usable 32332180
last_update 1548309468
rss 2894240

[Changjin Wang (wang-changjin)]

ubuntu@kvm-ubuntu:~$ sudo virsh dumpxml 33
<domain type='kvm' id='33'>
  <name>stx-ctrl-0</name>
  <uuid>a9896905-1e84-434b-b0d1-40babbe9b871</uuid>
  <memory unit='KiB'>33554432</memory>
  <currentMemory unit='KiB'>33554432</currentMemory>
  <vcpu placement='static'>16</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-bionic'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>IvyBridge</model>
    <topology sockets='2' cores='8' threads='1'/>
    <feature policy='force' name='vmx'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='xsaveopt'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/stx-disk0.qcow2'/>
      <backingStore/>
      <target dev='sda' bus='sata'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/stx-disk1.qcow2'/>
      <backingStore/>
      <target dev='sdb' bus='sata'/>
      <alias name='sata0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/stx-disk2.qcow2'/>
      <backingStore/>
      <target dev='sdc' bus='sata'/>
      <alias name='sata0-0-2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <disk type='file' device='cdrom'>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' func...

[yong hu (yhu6)]

Have a try to run this cmd before running "config_controller"
$ sudo memtop | awk 'FNR == 3 {a=$13+$14} END {print a}'

As well, I noticed in your xml: <topology sockets='2' cores='8' threads='1'/>
which is different from mine: <topology sockets='1' cores='4' threads='1'/>

You can have a try by limiting CPU sockets='1'.

This might be related to QEMU-KVM supports to NUMA, and I need more investigation on it.

[Changjin Wang (wang-changjin)]

when set sockets=1,it can work
...
Applying configuration (this will take several minutes):

01/08: Creating bootstrap configuration ... DONE
02/08: Applying bootstrap manifest ... DONE
03/08: Persisting local configuration ... DONE
04/08: Populating initial system inventory ...

[yong hu (yhu6)]

The cause is: when defining VM with multiple CPU sockets, we have to specifically set "numa" properties as follow in <cpu> </cpu> session (see my xml for a complete controller config file)

    <numa>
       <cell id='0' cpus='0-2' memory='8' unit='GiB'/>
       <cell id='1' cpus='4-6' memory='8' unit='GiB'/>
    </numa>

otherwise, NUMA node won't be correctly handled by Linux. In such a case, there will be only one node indicated in "/sys/devices/system/node/node%d". As well "lscpu" won't tell the correct NUMA node(s) info.

And missing correct node%d results in a failure in "memtop", which is used to learn available memory for variable "platform_res_mem".

I made a test with the correct QEMU-KVM config, and STX worked well with 2 sockets for VM.

[Cindy Xie (xxie1)]

shall we set this issue as "invalid"?

[yong hu (yhu6)]

yes, I think so.
It has nothing to do with StarlingX itself.
Instead it is about how to make a VM with an appropriate NUMA config.