Installed openstack Ocata with fuel 11, installation finished. After a few hours of finishing the install, I've added more compute nodes, that succeeded.
After a few days I wanted to add even more compute nodes, but the instalation fails with the following message:
Error
All nodes are finished. Failed tasks: Task[hosts/1], Task[hosts/3], Task[hosts/2] Stopping the deployment process!
In the puppet.log on the nodes, found this:
2017-07-13 14:49:30 +0000 Puppet (err): Duplicate declaration: Host[node-13.xx.com] is already declared; cannot redeclare at /etc/puppet/modules/osnailyfacter/manifests/hosts/hosts.pp:25 on node node-1.xx.com
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:500:in `fail_on_duplicate_type_and_title'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:86:in `add_one_resource'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:72:in `block in add_resource'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:71:in `each'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:71:in `add_resource'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:88:in `add_resource'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:59:in `block (3 levels) in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:64:in `exceptwrap'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:43:in `block (2 levels) in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:42:in `collect'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:42:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:25:in `collect'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:25:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/functions/create_resources.rb:73:in `block in <top (required)>'
/usr/lib/ruby/vendor_ruby/puppet/parser/functions.rb:164:in `block (2 levels) in newfunction'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/parser/functions.rb:157:in `block in newfunction'
/etc/puppet/modules/stdlib/lib/puppet/parser/functions/ensure_resource.rb:43:in `block (2 levels) in <top (required)>'
/etc/puppet/modules/stdlib/lib/puppet/parser/functions/ensure_resource.rb:36:in `each'
/etc/puppet/modules/stdlib/lib/puppet/parser/functions/ensure_resource.rb:36:in `block in <top (required)>'
/usr/lib/ruby/vendor_ruby/puppet/parser/functions.rb:164:in `block (2 levels) in newfunction'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/parser/functions.rb:157:in `block in newfunction'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/function.rb:42:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:11:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:10:in `each'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:10:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/ifstatement.rb:25:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:11:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:10:in `each'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:10:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/type.rb:129:in `evaluate_code'
/usr/lib/ruby/vendor_ruby/puppet/parser/resource.rb:81:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/parser/resource.rb:77:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:236:in `each'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:236:in `evaluate_classes'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:60:in `block (3 levels) in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:64:in `exceptwrap'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:43:in `block (2 levels) in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:42:in `collect'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:42:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/branch.rb:12:in `block in each'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/branch.rb:11:in `each'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/branch.rb:11:in `each'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:25:in `collect'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:25:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:11:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:10:in `each'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:10:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/type.rb:129:in `evaluate_code'
/usr/lib/ruby/vendor_ruby/puppet/parser/resource.rb:81:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/parser/resource.rb:77:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:426:in `evaluate_main'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:135:in `block (2 levels) in compile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:135:in `block in compile'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:121:in `compile'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:34:in `compile'
/usr/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:95:in `block (2 levels) in compile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:93:in `block in compile'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/lib/ruby/2.3.0/benchmark.rb:308:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:92:in `compile'
/usr/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:52:in `find'
/usr/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:201:in `find'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:222:in `block in main'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
/usr/bin/puppet:8:in `<main>'
Detailed bug description:
Installed openstack Ocata with fuel 11, installation finished. After a few hours of finishing the install, I've added more compute nodes, that succeeded.
After a few days I wanted to add even more compute nodes, but the instalation fails with the following message:
Error
All nodes are finished. Failed tasks: Task[hosts/1], Task[hosts/3], Task[hosts/2] Stopping the deployment process!
In the puppet.log on the nodes, found this:
2017-07-13 14:49:30 +0000 Puppet (err): Duplicate declaration: Host[node- 13.xx.com] is already declared; cannot redeclare at /etc/puppet/ modules/ osnailyfacter/ manifests/ hosts/hosts. pp:25 on node node-1.xx.com ruby/vendor_ ruby/puppet/ resource/ catalog. rb:500: in `fail_on_ duplicate_ type_and_ title' ruby/vendor_ ruby/puppet/ resource/ catalog. rb:86:in `add_one_resource' ruby/vendor_ ruby/puppet/ resource/ catalog. rb:72:in `block in add_resource' ruby/vendor_ ruby/puppet/ resource/ catalog. rb:71:in `each' ruby/vendor_ ruby/puppet/ resource/ catalog. rb:71:in `add_resource' ruby/vendor_ ruby/puppet/ parser/ compiler. rb:88:in `add_resource' ruby/vendor_ ruby/puppet/ parser/ ast/resource. rb:59:in `block (3 levels) in evaluate' ruby/vendor_ ruby/puppet/ util/errors. rb:64:in `exceptwrap' ruby/vendor_ ruby/puppet/ parser/ ast/resource. rb:43:in `block (2 levels) in evaluate' ruby/vendor_ ruby/puppet/ parser/ ast/resource. rb:42:in `collect' ruby/vendor_ ruby/puppet/ parser/ ast/resource. rb:42:in `block in evaluate' ruby/vendor_ ruby/puppet/ parser/ ast/resource. rb:25:in `collect' ruby/vendor_ ruby/puppet/ parser/ ast/resource. rb:25:in `evaluate' ruby/vendor_ ruby/puppet/ parser/ ast.rb: 61:in `safeevaluate' ruby/vendor_ ruby/puppet/ parser/ functions/ create_ resources. rb:73:in `block in <top (required)>' ruby/vendor_ ruby/puppet/ parser/ functions. rb:164: in `block (2 levels) in newfunction' ruby/vendor_ ruby/puppet/ util/profiler/ around_ profiler. rb:58:in `profile' ruby/vendor_ ruby/puppet/ util/profiler. rb:51:in `profile' ruby/vendor_ ruby/puppet/ parser/ functions. rb:157: in `block in newfunction' modules/ stdlib/ lib/puppet/ parser/ functions/ ensure_ resource. rb:43:in `block (2 levels) in <top (required)>' modules/ stdlib/ lib/puppet/ parser/ functions/ ensure_ resource. rb:36:in `each' modules/ stdlib/ lib/puppet/ parser/ functions/ ensure_ resource. rb:36:in `block in <top (required)>' ruby/vendor_ ruby/puppet/ parser/ functions. rb:164: in `block (2 levels) in newfunction' ruby/vendor_ ruby/puppet/ util/profiler/ around_ profiler. rb:58:in `profile' ruby/vendor_ ruby/puppet/ util/profiler. rb:51:in `profile' ruby/vendor_ ruby/puppet/ parser/ functions. rb:157: in `block in newfunction' ruby/vendor_ ruby/puppet/ parser/ ast/function. rb:42:in `evaluate' ruby/vendor_ ruby/puppet/ parser/ ast.rb: 61:in `safeevaluate' ruby/vendor_ ruby/puppet/ parser/ ast/block_ expression. rb:11:in `block in evaluate' ruby/vendor_ ruby/puppet/ parser/ ast/block_ expression. rb:10:in `each' ruby/vendor_ ruby/puppet/ parser/ ast/block_ expression. rb:10:in `evaluate' ruby/vendor_ ruby/puppet/ parser/ ast.rb: 61:in `safeevaluate' ruby/vendor_ ruby/puppet/ parser/ ast/ifstatement .rb:25: in `evaluate' ruby/vendor_ ruby/puppet/ parser/ ast.rb: 61:in `safeevaluate' ruby/vendor_ ruby/puppet/ parser/ ast/block_ expression. rb:11:in `block in evaluate' ruby/vendor_ ruby/puppet/ parser/ ast/block_ expression. rb:10:in `each' ruby/vendor_ ruby/puppet/ parser/ ast/block_ expression. rb:10:in `evaluate' ruby/vendor_ ruby/puppet/ parser/ ast.rb: 61:in `safeevaluate' ruby/vendor_ ruby/puppet/ resource/ type.rb: 129:in `evaluate_code' ruby/vendor_ ruby/puppet/ parser/ resource. rb:81:in `block in evaluate' ruby/vendor_ ruby/puppet/ util/profiler/ around_ profiler. rb:58:in `profile' ruby/vendor_ ruby/puppet/ util/profiler. rb:51:in `profile' ruby/vendor_ ruby/puppet/ parser/ resource. rb:77:in `evaluate' ruby/vendor_ ruby/puppet/ parser/ compiler. rb:236: in `each' ruby/vendor_ ruby/puppet/ parser/ compiler. rb:236: in `evaluate_classes' ruby/vendor_ ruby/puppet/ parser/ ast/resource. rb:60:in `block (3 levels) in evaluate' ruby/vendor_ ruby/puppet/ util/errors. rb:64:in `exceptwrap' ruby/vendor_ ruby/puppet/ parser/ ast/resource. rb:43:in `block (2 levels) in evaluate' ruby/vendor_ ruby/puppet/ parser/ ast/resource. rb:42:in `collect' ruby/vendor_ ruby/puppet/ parser/ ast/resource. rb:42:in `block in evaluate' ruby/vendor_ ruby/puppet/ parser/ ast/branch. rb:12:in `block in each' ruby/vendor_ ruby/puppet/ parser/ ast/branch. rb:11:in `each' ruby/vendor_ ruby/puppet/ parser/ ast/branch. rb:11:in `each' ruby/vendor_ ruby/puppet/ parser/ ast/resource. rb:25:in `collect' ruby/vendor_ ruby/puppet/ parser/ ast/resource. rb:25:in `evaluate' ruby/vendor_ ruby/puppet/ parser/ ast.rb: 61:in `safeevaluate' ruby/vendor_ ruby/puppet/ parser/ ast/block_ expression. rb:11:in `block in evaluate' ruby/vendor_ ruby/puppet/ parser/ ast/block_ expression. rb:10:in `each' ruby/vendor_ ruby/puppet/ parser/ ast/block_ expression. rb:10:in `evaluate' ruby/vendor_ ruby/puppet/ parser/ ast.rb: 61:in `safeevaluate' ruby/vendor_ ruby/puppet/ resource/ type.rb: 129:in `evaluate_code' ruby/vendor_ ruby/puppet/ parser/ resource. rb:81:in `block in evaluate' ruby/vendor_ ruby/puppet/ util/profiler/ around_ profiler. rb:58:in `profile' ruby/vendor_ ruby/puppet/ util/profiler. rb:51:in `profile' ruby/vendor_ ruby/puppet/ parser/ resource. rb:77:in `evaluate' ruby/vendor_ ruby/puppet/ parser/ compiler. rb:426: in `evaluate_main' ruby/vendor_ ruby/puppet/ parser/ compiler. rb:135: in `block (2 levels) in compile' ruby/vendor_ ruby/puppet/ util/profiler/ around_ profiler. rb:58:in `profile' ruby/vendor_ ruby/puppet/ util/profiler. rb:51:in `profile' ruby/vendor_ ruby/puppet/ parser/ compiler. rb:135: in `block in compile' ruby/vendor_ ruby/puppet/ context. rb:64:in `override' ruby/vendor_ ruby/puppet. rb:246: in `override' ruby/vendor_ ruby/puppet/ parser/ compiler. rb:121: in `compile' ruby/vendor_ ruby/puppet/ parser/ compiler. rb:34:in `compile' ruby/vendor_ ruby/puppet/ indirector/ catalog/ compiler. rb:95:in `block (2 levels) in compile' ruby/vendor_ ruby/puppet/ util/profiler/ around_ profiler. rb:58:in `profile' ruby/vendor_ ruby/puppet/ util/profiler. rb:51:in `profile' ruby/vendor_ ruby/puppet/ indirector/ catalog/ compiler. rb:93:in `block in compile' ruby/vendor_ ruby/puppet/ util.rb: 161:in `block in benchmark' ruby/2. 3.0/benchmark. rb:308: in `realtime' ruby/vendor_ ruby/puppet/ util.rb: 160:in `benchmark' ruby/vendor_ ruby/puppet/ indirector/ catalog/ compiler. rb:92:in `compile' ruby/vendor_ ruby/puppet/ indirector/ catalog/ compiler. rb:52:in `find' ruby/vendor_ ruby/puppet/ indirector/ indirection. rb:201: in `find' ruby/vendor_ ruby/puppet/ application/ apply.rb: 222:in `block in main' ruby/vendor_ ruby/puppet/ context. rb:64:in `override' ruby/vendor_ ruby/puppet. rb:246: in `override' ruby/vendor_ ruby/puppet/ application/ apply.rb: 198:in `main' ruby/vendor_ ruby/puppet/ application/ apply.rb: 159:in `run_command' ruby/vendor_ ruby/puppet/ application. rb:381: in `block (2 levels) in run' ruby/vendor_ ruby/puppet/ application. rb:507: in `plugin_hook' ruby/vendor_ ruby/puppet/ application. rb:381: in `block in run' ruby/vendor_ ruby/puppet/ util.rb: 496:in `exit_on_fail' ruby/vendor_ ruby/puppet/ application. rb:381: in `run' ruby/vendor_ ruby/puppet/ util/command_ line.rb: 146:in `run' ruby/vendor_ ruby/puppet/ util/command_ line.rb: 92:in `execute' puppet: 8:in `<main>'
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/etc/puppet/
/etc/puppet/
/etc/puppet/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/bin/
Checking the source code, found this file:
root@node-1:~# cat /etc/hiera/ deleted_ nodes.yaml
deleted_nodes:
- node-13.xx.com
- node-31.xx.com
- node-30.xx.com
- node-11.xx.com
- node-35.xx.com
- node-17.xx.com
- node-16.xx.com
- node-10.xx.com
According to this file, these nodes are failed but that's not true because they're online and running right now, fuel shows them as online.
Impact:
Currently I can't add or remove any nodes from my environment because it fails on this error.
Version: fuel 11 ocata
Thank you for looking into it in advance,
Zoltan