Currently some providers restricts access to 123 port on root ntp servers.
If master node is placed inside such environment - then deployment will fail with error:
2015-03-23T10:45:20 err: [517] Error running RPC method granular_deploy: Failed to execute hook .
---
priority: 300
fail_on_error: true
type: shell
uids:
- '3'
- '4'
parameters:
retries: 10
cmd: ntpdate -u $(egrep '^server' /etc/ntp.conf | egrep -v '127\.127\.[0-9]+\.[0-9]+'
| sed '/^#/d' | awk '{print $2}')
timeout: 180
interval: 1
, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/nailgun_hooks.rb:54:in `block in process'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/nailgun_hooks.rb:26:in `each'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/nailgun_hooks.rb:26:in `process'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deployment_engine/granular_deployment.rb:201:in `pre_deployment_actions'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deployment_engine.rb:32:in `deploy'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/orchestrator.rb:131:in `deploy_cluster'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/orchestrator.rb:56:in `granular_deploy'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/dispatcher.rb:111:in `granular_deploy'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:142:in `dispatch_message'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:103:in `block in dispatch'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:64:in `call'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:56:in `each'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:56:in `each'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:101:in `each_with_index'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:101:in `dispatch'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:85:in `block in perform_main_job'"]
2015-03-23T10:45:20 info: [517] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"b6c2fcee-1b56-4f75-a90c-87158d681da7", "status"=>"error", "error"=>"Method granular_d
eploy. Failed to execute hook .\n\n---\npriority: 300\nfail_on_error: true\ntype: shell\nuids:\n- '3'\n- '4'\nparameters:\n retries: 10\n cmd: ntpdate -u $(egrep '^server' /etc/ntp.conf | egre
p -v '127\\.127\\.[0-9]+\\.[0-9]+'\n | sed '/^#/d' | awk '{print $2}')\n timeout: 180\n interval: 1\n.\nInspect Astute logs for the details"}}
As you can see - master node can't reach root servers:
ntpq> peers
remote refid st t when poll reach delay offset jitter
==============================================================================
kahuna.ruselabs .INIT. 16 u - 256 0 0.000 0.000 0.000
422224.s.dediku .INIT. 16 u - 256 0 0.000 0.000 0.000
ponderosa.piney .INIT. 16 u - 256 0 0.000 0.000 0.000
ntpq> as
ind assid status conf reach auth condition last_event cnt
===========================================================
1 15020 8011 yes no none reject mobilize 1
2 15021 8011 yes no none reject mobilize 1
3 15022 8011 yes no none reject mobilize 1
and if command ntpdate -u 'fuel_ip' executed - it produces error:
23 Mar 12:32:52 ntpdate[17115]: no server suitable for synchronization found
This can be fixed by adding settings to fuel master node - that will instruct ntpd propagate itself as synced server:
--- ntp.conf.orig 2015-03-23 13:29:32.847968972 +0000
+++ ntp.conf 2015-03-23 13:13:33.706984063 +0000
@@ -16,6 +16,8 @@
server 0.pool.ntp.org iburst
server 1.pool.ntp.org iburst
server 2.pool.ntp.org iburst
+server 127.127.1.0
+fudge 127.127.1.0 stratum 10
# Driftfile.
driftfile /var/lib/ntp/drift
After this settings added, previous command start working fine:
ntpdate -u 10.20.0.2
23 Mar 13:14:01 ntpdate[21021]: adjust time server 10.20.0.2 offset -0.000010 sec
And deployment continues...
cat /etc/fuel/version.yaml
VERSION:
feature_groups:
- mirantis
production: "docker"
release: "6.1"
api: "1.0"
build_number: "216"
build_id: "2015-03-22_22-54-44"
nailgun_sha: "51974b50c3961be3ed0fdc7859570db2eeb83e9c"
python-fuelclient_sha: "b223dcaf5fdad2f714cd245958fefe03995d6207"
astute_sha: "4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9"
fuellib_sha: "a636c680e3c7d8cc66ed3e03645f38250beb8970"
ostf_sha: "b4d284e9364e30bf5162975c2ba0be6ca0f14ebd"
fuelmain_sha: "f52e4442df55a2b62637a2cf4038a24ba6f37b6f"
There is actually nothing that we can do. If we will add local undisciplined clock to our master node, it will lead to some unpredictable errors too (and it was - as you can see, we had this code just a month or two ago). Case when we need upstream servers available and should not add local clock to master node is just a less evil.