Two nodes: 3 and 4 after reboot has less time for /proc/1 that before:
* 3 -> before 1433945756, after 1433945508
* 4 -> before 1433945849, after 1433945507
But for node 1 and 2 all works fine. Before deploy we run sync time hooks.
We have 2 ways:
* change Astute reboot logic to mark node as rebooted if it return different time after reboot command for /proc/1, not especially more;
* try to find and fix source of problem why time is rollback to early after reboot.
Problem in time on nodes.
Two nodes: 3 and 4 after reboot has less time for /proc/1 that before:
* 3 -> before 1433945756, after 1433945508
* 4 -> before 1433945849, after 1433945507
But for node 1 and 2 all works fine. Before deploy we run sync time hooks.
We have 2 ways:
* change Astute reboot logic to mark node as rebooted if it return different time after reboot command for /proc/1, not especially more;
* try to find and fix source of problem why time is rollback to early after reboot.
Sync time hooks details:
2015-06-10T14:10:11 info: [568] Run hook --- .[0-9]+ \.[0-9] +'
priority: 300
fail_on_error: true
type: shell
uids:
- '1'
- '3'
- '2'
- '4'
parameters:
retries: 10
cmd: ntpdate -u $(egrep '^server' /etc/ntp.conf | egrep -v '127\.127\
| sed '/^#/d' | awk '{print $2}' | head -1)
timeout: 180
interval: 30
2015-06-10T14:10:17 debug: [568] 43dfd2b7- bae1-40ee- b24d-02da19378f 1f: MC agent 'execute_ shell_command' , method 'execute', results: {:sender=>"1", :statuscode=>0, :statusmsg=>"OK", :data=> {:stdout= >"10 Jun 14:10:17 ntpdate[8044]: adjust time server 10.20.0.2 offset -0.000414 sec\n", :stderr=>"", :exit_code=>0}} bae1-40ee- b24d-02da19378f 1f: MC agent 'execute_ shell_command' , method 'execute', results: {:sender=>"3", :statuscode=>0, :statusmsg=>"OK", :data=> {:stdout= >"10 Jun 14:10:17 ntpdate[7904]: adjust time server 10.20.0.2 offset 0.000974 sec\n", :stderr=>"", :exit_code=>0}} bae1-40ee- b24d-02da19378f 1f: MC agent 'execute_ shell_command' , method 'execute', results: {:sender=>"2", :statuscode=>0, :statusmsg=>"OK", :data=> {:stdout= >"10 Jun 14:10:17 ntpdate[7902]: adjust time server 10.20.0.2 offset -0.000861 sec\n", :stderr=>"", :exit_code=>0}} bae1-40ee- b24d-02da19378f 1f: MC agent 'execute_ shell_command' , method 'execute', results: {:sender=>"4", :statuscode=>0, :statusmsg=>"OK", :data=> {:stdout= >"10 Jun 14:10:17 ntpdate[7867]: adjust time server 10.20.0.2 offset -0.000690 sec\n", :stderr=>"", :exit_code=>0}}
2015-06-10T14:10:17 debug: [568] 43dfd2b7-
2015-06-10T14:10:17 debug: [568] 43dfd2b7-
2015-06-10T14:10:17 debug: [568] 43dfd2b7-