Fuel deployment failed to execute hook "puppet" using VirtualBox script

[Nate Potter]

After successfully installing fuel using the steps for installing on VirtualBox here https://docs.mirantis.com/openstack/fuel/fuel-6.1/virtualbox.html, deployment fails saying 'failed to execute hook: puppet', and in the logs the error occurs due to a failure to communicate with the default ntp servers.

The workaround that I used to make this work was to set up fuel master as the ntp server all of the slaves point to - after doing this the deployment completed successfully. It seems like the nodes are unable to communicate with external servers after being set up with the given VirtualBox scripts.

[Roman Prykhodchenko (romcheg)]

Nathaniel, thank you for this report. Could you please attach a snapshot of that environment or at least the logs you are referring to in the bug description?

[Nate Potter (ntpttr)]

Sure, here's the error message and corresponding error in the Astute logs: http://pastebin.com/qkin4tfz.

[Mike Scherbakov (mihgen)]

Nathaniel,
according to the log, there was an issue with puppet run. We need puppet logs as well. Can you please provide full diagnostic snapshot of the environment (you can generate it in Fuel UI on Support tab).

[Nate Potter (ntpttr)]

Yes, here is the snapshot.

[Alex Schultz (alex-schultz)]

If it's a failure of the ntp-check task, it's because the controllers do not have access to the internet to query the provided (internet based) ntp servers.

ERROR: Unable to communicate with at least one of NTP server, checked the following host(s): ["0.pool.ntp.org", "1.pool.ntp.org", "2.pool.ntp.org"] on node node-1.domain.tld

Can you verify that the servers being deployed to have access to the internet and can reach these hosts?

[Nate Potter (ntpttr)]

The servers being deployed do have access to the internet and all of them are able to successfully ping the NTP servers.

[Alex Schultz (alex-schultz)]

Can they query ntp? Ping being ICMP and ntp being UDP sometimes firewall rules get in the way. You can try and manually run the ntp-check task to see if the problem persists or it might contain a better error message.

[Nate Potter (ntpttr)]

ntpstat returns 'synchronized to NTP server (10.20.0.2) at stratum 12 \ time correct to within61 ms \ polling server every 128 s
ntpq -pn returns two servers, 10.20.0.2 and fuel.domain.tld.
Is there another ntp-check task you wanted me to run? Sorry I'm not sure what you had in mind.

[Alex Schultz (alex-schultz)]

Hey Nathaniel,

What server did you run that on? You should run it on your controller node. Based on the snapshot you provided, the ntp servers configured for the environment were "0.pool.ntp.org", "1.pool.ntp.org", "2.pool.ntp.org" and the failure occurred on node-1. You can manually rerun the failed task to trouble shoot on the node by running:

  puppet apply --modulepath=/etc/puppet/modules/ /etc/puppet/modules/osnailyfacter/modular/ntp/ntp-check.pp

If it's successful you should see something like http://paste.openstack.org/show/435374/

[Andrey Maximov (maximov)]

Moving to 8.0, we don't have time to fix it before HCF.

[Nate Potter (ntpttr)]

Hi Alex, I ran that command on my controller node after deployment failed and it returned "ERROR: Unable to communicate with at least one of NTP server, checked the following host(s): ["0.pool.ntp.org", "1.pool.ntp.org", "2.pool.ntp.org"] on node node-1.domain.tld."

[Alex Schultz (alex-schultz)]

Hey Nathaniel,

So that's the check that is blocking the deployment. Do you know if you have a firewall in place that might be blocking NTP or UDP traffic? Alternatively if you have a set of internal NTP servers for your organization, you can update the environment's settings to use those NTP servers instead of the [0-2].pool.ntp.org servers. The other option is to configure the fuel master as an ntp server and configure your environments to leverage it for NTP. The issue is that time syncing is very important for the cluster activities so we must have a time source that can be used by all of the controllers to ensure their time remains consistent across all machines.

[Nate Potter (ntpttr)]

I'm not sure, I've been running this test on a laptop given to me by my company, so I'll try it on a personal machine and see if it fixes the problem. In the past I did configure the fuel master as the NTP server and it did work.

[Alex Schultz (alex-schultz)]

I've seen NTP be locked down in organizations. Using the fuel master is a good workaround when testing and access to ntp is not necessarily available.

[Matthew Mosesohn (raytrac3r)]

Nathaniel, we can't do anything to help you but except to advise you disable external NTP or find NTP servers that are reachable from your deployment environment.