Fuel 10 is giving new node an old hostname

[James Okken]

hi all

I am using Fuel 10. I have a known good environment, I've used fuel to deploy the controllers, etc and 10 compute nodes so far. I have instances running on those nodes..

when attempting to deploy a 11th compute node I am running into issues. Way back, before the 10 compute nodes, there was a previous attempt to deploy this node, this attempt was unsuccessful because of a bad hard drive. Now that this is fixed I am deploying again.

Back then Fuel refered to it as hostname node-20. Now fuel is using hostname node-31. Everything I see on the surface, including the fuel2 CLI, shows this node as node-31. Yet, after Fuel provisions the Ubuntu OS and the host reboots cloud-init (i believe) is setting the old node-20 hostname, and using many settings from node-20 (ie: RSA SSH keys).

I believe this is causing Fuel not to be able to communicate with the node and the deployment times out with #<runtimeerror: could not find any hosts in discover data provided>

[fuel2 node show 31
| id | 31 |
| name | Untitled (68:58) |
| status | ready |
| os_platform | ubuntu |
| roles | [u'compute'] |
| kernel_params | None |
| pending_roles | [] |
| hostname | node-31 |
| fqdn | node-31.ourdomain.com |
| platform_name | ProLiant BL460c Gen9 |

on this broken node-31 I find these files/values in /var/lib/cloud

root@node-20:/var/lib/cloud# grep -R node-20 *

Binary file instance/obj.pkl matches
instance/user-data.txt:hostname: node-20.ourdomain.com
instance/user-data.txt:fqdn: node-20.ourdomain.com
instance/user-data.txt.i:hostname: node-20.ourdomain.com
instance/user-data.txt.i:fqdn: node-20.ourdomain.com
instance/cloud-config.txt:fqdn: node-20.ourdomain.com
instance/cloud-config.txt:hostname: node-20.ourdomain.com
Binary file instances/nocloud/obj.pkl matches
instances/nocloud/user-data.txt:hostname: node-20.ourdomain.com
instances/nocloud/user-data.txt:fqdn: node-20.ourdomain.com
instances/nocloud/user-data.txt.i:hostname: node-20.ourdomain.com
instances/nocloud/user-data.txt.i:fqdn: node-20.ourdomain.com
instances/nocloud/cloud-config.txt:fqdn: node-20.ourdomain.com
instances/nocloud/cloud-config.txt:hostname: node-20.ourdomain.com

please let me know what in Fuel 10 is setting these values? is it cloud-init? And how can I intervene and avoid it being set incorrectly?

If I try to delete node-20 I get an error showing it doesn't exist from fuel/2's perspective:
fuel node --node-id 20 --delete-from-db --force
404 Client Error: Not Found for url: http://10.20.20.1:8000/api/v1/nodes/?ids=20 (NodeCollection not found)
fuel2 node undiscover -n 20 -f
404 Client Error: Not Found for url: http://10.20.20.1:8000/api/v1/nodes/20/ (Node not found)

I guess this has something to do with this file or one similar:
/usr/share/fuel-agent/cloud-init-templates/cloud_config_ubuntu.jinja2
in that file I see
hostname: {{ common.hostname }}
fqdn: {{ common.fqdn }}

I assume the node's UUID or MAC address is already in some Fuel database associated with node-20, and these files are being populated incorrectly because of that.

I've tried many different things to get passed this issue
as someone suggested, I renamed the hostname to something completely different using the fuel2 command
"fuel2 node update -H blade13 31"
Hoping if I went away from the node-* naming convention it would clear this up, but it didn't

If we do have a bug here I am happy to test a fix!
how/where can I intervene and avoid this hostname being set incorrectly?
thanks!

Errors from the Fuel Astute log:
2017-09-25 21:06:28 ERROR [1565] Error running provisioning: #<RuntimeError: Could not find any hosts in discovery data provided> , trace: ["/usr/share/gems/gems/astute-10.0.0/lib/astute/mclient.rb:178:in `rescue in initialize_mclient'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/mclient.rb:161:in `initialize_mclient'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/mclient.rb:51:in `initialize'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:421:in `new'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:421:in `run_shell_without_check'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:449:in `update_node_status'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:313:in `reboot_hook'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:38:in `block in process'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:26:in `each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_
 hooks.rb:26:in `process'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/image_provision.rb:117:in `reboot'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:273:in `soft_reboot'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:240:in `provision_piece'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:126:in `block (3 levels) in provision_and_watch_progress'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:309:in `call'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:309:in `sleep_not_greater_than'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:120:in `block (2 levels) in provision_and_watch_progress'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:119:in `loop'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:119:in `block in provision_and_watch_progress'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:118:in `catch'", "/usr/share/gems/gems/astute-10.0.0/
 lib/astute/provision.rb:118:in `provision_and_watch_progress'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:52:in `provision'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/orchestrator.rb:109:in `provision'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/dispatcher.rb:46:in `provision'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/dispatcher.rb:37:in `image_provision'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:172:in `dispatch_message'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:131:in `block in dispatch'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task_queue.rb:64:in `call'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task_queue.rb:64:in `block in each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task_queue.rb:56:in `each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task_queue.rb:56:in `each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server
 /server.rb:128:in `each_with_index'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:128:in `dispatch'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:106:in `block in perform_main_job'"]
2017-09-25 21:06:26 ERROR [1565] Error occured while provisioning: #<Astute::MClientError: #<RuntimeError: Could not find any hosts in discovery data provided> >
2017-09-25 21:06:26 ERROR [1565] No more retries for MCollective client instantiation after exception: ["/usr/share/gems/gems/mcollective-client-2.8.4/lib/mcollective/rpc/client.rb:507:in `discover'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/mclient.rb:167:in `initialize_mclient'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/mclient.rb:51:in `initialize'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:421:in `new'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:421:in `run_shell_without_check'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:449:in `update_node_status'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:313:in `reboot_hook'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:38:in `block in process'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:26:in `each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:26:in `process'", "/usr/shar
 e/gems/gems/astute-10.0.0/lib/astute/image_provision.rb:117:in `reboot'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:273:in `soft_reboot'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:240:in `provision_piece'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:126:in `block (3 levels) in provision_and_watch_progress'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:309:in `call'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:309:in `sleep_not_greater_than'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:120:in `block (2 levels) in provision_and_watch_progress'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:119:in `loop'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:119:in `block in provision_and_watch_progress'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:118:in `catch'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:118:in `provi sion_and_watch_progress'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:52:in `provision'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/orchestrator.rb:109:in `provision'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/dispatcher.rb:46:in `provision'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/dispatcher.rb:37:in `image_provision'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:172:in `dispatch_message'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:131:in `block in dispatch'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task_queue.rb:64:in `call'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task_queue.rb:64:in `block in each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task_queue.rb:56:in `each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task_queue.rb:56:in `each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:128:in `each_with_index'",
  "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:128:in `dispatch'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:106:in `block in perform_main_job'"]
2017-09-25 21:06:21 ERROR [1565] Retrying RPC client instantiation after exception: #<RuntimeError: Could not find any hosts in discovery data provided>
2017-09-25 21:06:16 ERROR [1565] Retrying RPC client instantiation after exception: #<RuntimeError: Could not find any hosts in discovery data provided>
Version: 10.0

[James Okken (jokken)]

is there anyone looking at this question who develops in this area of Fuel, which deals with cloud-init on newly provisioned nodes?

[James Okken (jokken)]

I have gotten around this issue. I am not sure if there was anything found by developers in the community but I see this issue now has a milestone set.

I'm not sure how I got around this issue, but my theory is something I noticed quite by accident.

The servers I was having these troubles on use 80GB hard drives, but they also have a flash drive in them, for small OS deployments. I stumbled across a /dev/sda6 partition on these flash drives.

On this partition I found 2 files: meta-data and user-data. In those files was the old node name I mentioned in the original post. This partition must have been detected, and these stale files were used by fuel-agent when the newly provisioned node first booted, even though they were booting from the 80GB drive which had its own /dev/sda6 partition.

I suspect the stale/incorrect /dev/sda6 is probably is from Fuel 8 when we tried to deploy on some of these flash drives...

once I deleted the stale/incorrect /dev/sda6 then theprovisioning and deployment went perfectly

thanks