Fuel for OpenStack

adding nodes after the deplpoyment finished fails, because some active hosts are already declared (or appears to be deleted)

Bug #1704157 reported by Zoltan Langi on 2017-07-13

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Incomplete	Undecided	MOS Maintenance	Fuel for OpenStack 11.x-updates

Bug Description

Detailed bug description:

Installed openstack Ocata with fuel 11, installation finished. After a few hours of finishing the install, I've added more compute nodes, that succeeded.
After a few days I wanted to add even more compute nodes, but the instalation fails with the following message:

Error
All nodes are finished. Failed tasks: Task[hosts/1], Task[hosts/3], Task[hosts/2] Stopping the deployment process!

In the puppet.log on the nodes, found this:

2017-07-13 14:49:30 +0000 Puppet (err): Duplicate declaration: Host[node-13.xx.com] is already declared; cannot redeclare at /etc/puppet/modules/osnailyfacter/manifests/hosts/hosts.pp:25 on node node-1.xx.com
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:500:in `fail_on_duplicate_type_and_title'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:86:in `add_one_resource'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:72:in `block in add_resource'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:71:in `each'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:71:in `add_resource'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:88:in `add_resource'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:59:in `block (3 levels) in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:64:in `exceptwrap'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:43:in `block (2 levels) in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:42:in `collect'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:42:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:25:in `collect'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:25:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/functions/create_resources.rb:73:in `block in <top (required)>'
/usr/lib/ruby/vendor_ruby/puppet/parser/functions.rb:164:in `block (2 levels) in newfunction'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/parser/functions.rb:157:in `block in newfunction'
/etc/puppet/modules/stdlib/lib/puppet/parser/functions/ensure_resource.rb:43:in `block (2 levels) in <top (required)>'
/etc/puppet/modules/stdlib/lib/puppet/parser/functions/ensure_resource.rb:36:in `each'
/etc/puppet/modules/stdlib/lib/puppet/parser/functions/ensure_resource.rb:36:in `block in <top (required)>'
/usr/lib/ruby/vendor_ruby/puppet/parser/functions.rb:164:in `block (2 levels) in newfunction'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/parser/functions.rb:157:in `block in newfunction'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/function.rb:42:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:11:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:10:in `each'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:10:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/ifstatement.rb:25:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:11:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:10:in `each'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:10:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/type.rb:129:in `evaluate_code'
/usr/lib/ruby/vendor_ruby/puppet/parser/resource.rb:81:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/parser/resource.rb:77:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:236:in `each'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:236:in `evaluate_classes'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:60:in `block (3 levels) in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:64:in `exceptwrap'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:43:in `block (2 levels) in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:42:in `collect'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:42:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/branch.rb:12:in `block in each'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/branch.rb:11:in `each'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/branch.rb:11:in `each'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:25:in `collect'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:25:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:11:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:10:in `each'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:10:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/type.rb:129:in `evaluate_code'
/usr/lib/ruby/vendor_ruby/puppet/parser/resource.rb:81:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/parser/resource.rb:77:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:426:in `evaluate_main'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:135:in `block (2 levels) in compile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:135:in `block in compile'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:121:in `compile'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:34:in `compile'
/usr/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:95:in `block (2 levels) in compile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:93:in `block in compile'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/lib/ruby/2.3.0/benchmark.rb:308:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:92:in `compile'
/usr/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:52:in `find'
/usr/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:201:in `find'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:222:in `block in main'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
/usr/bin/puppet:8:in `<main>'

Checking the source code, found this file:

root@node-1:~# cat /etc/hiera/deleted_nodes.yaml
deleted_nodes:
- node-13.xx.com
- node-31.xx.com
- node-30.xx.com
- node-11.xx.com
- node-35.xx.com
- node-17.xx.com
- node-16.xx.com
- node-10.xx.com

According to this file, these nodes are deleted but that's not true because they're online and running, serving right now, fuel node list shows them as online.

Impact:
Currently I can't add or remove any nodes from my environment or do any changed like password, certificate change because it fails on this error.

Version: fuel 11 ocata

Thank you for looking into it in advance,
Zoltan

See original description

Zoltan Langi (zlangi) on 2017-07-13

description:

updated

Zoltan Langi (zlangi) on 2017-07-14

summary:	- adding compute nodes after deplpoyment finished failes because some - hosts are already declared + adding nodes after deplpoyment finished fails because some hosts are + already declared (or appears to be deleted)
summary:	- adding nodes after deplpoyment finished fails because some hosts are - already declared (or appears to be deleted) + adding nodes after the deplpoyment finished fails, because some active + hosts are already declared (or appears to be deleted)
description:	updated

Revision history for this message

Oleksiy Molchanov (omolchanov) wrote on 2017-11-20:

Please provide diagnostic snapshot.

Changed in fuel:
status:	New → Incomplete
milestone:	none → 11.x-updates
assignee:	nobody → MOS Maintenance (mos-maintenance)

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.