Comment 2 for bug 1463868

Revision history for this message
Vladimir Sharshov (vsharshov) wrote :

Problem in time on nodes.

Two nodes: 3 and 4 after reboot has less time for /proc/1 that before:

* 3 -> before 1433945756, after 1433945508
* 4 -> before 1433945849, after 1433945507

But for node 1 and 2 all works fine. Before deploy we run sync time hooks.

We have 2 ways:

* change Astute reboot logic to mark node as rebooted if it return different time after reboot command for /proc/1, not especially more;
* try to find and fix source of problem why time is rollback to early after reboot.

Sync time hooks details:

2015-06-10T14:10:11 info: [568] Run hook ---
priority: 300
fail_on_error: true
type: shell
uids:
- '1'
- '3'
- '2'
- '4'
parameters:
  retries: 10
  cmd: ntpdate -u $(egrep '^server' /etc/ntp.conf | egrep -v '127\.127\.[0-9]+\.[0-9]+'
    | sed '/^#/d' | awk '{print $2}' | head -1)
  timeout: 180
  interval: 30

2015-06-10T14:10:17 debug: [568] 43dfd2b7-bae1-40ee-b24d-02da19378f1f: MC agent 'execute_shell_command', method 'execute', results: {:sender=>"1", :statuscode=>0, :statusmsg=>"OK", :data=>{:stdout=>"10 Jun 14:10:17 ntpdate[8044]: adjust time server 10.20.0.2 offset -0.000414 sec\n", :stderr=>"", :exit_code=>0}}
2015-06-10T14:10:17 debug: [568] 43dfd2b7-bae1-40ee-b24d-02da19378f1f: MC agent 'execute_shell_command', method 'execute', results: {:sender=>"3", :statuscode=>0, :statusmsg=>"OK", :data=>{:stdout=>"10 Jun 14:10:17 ntpdate[7904]: adjust time server 10.20.0.2 offset 0.000974 sec\n", :stderr=>"", :exit_code=>0}}
2015-06-10T14:10:17 debug: [568] 43dfd2b7-bae1-40ee-b24d-02da19378f1f: MC agent 'execute_shell_command', method 'execute', results: {:sender=>"2", :statuscode=>0, :statusmsg=>"OK", :data=>{:stdout=>"10 Jun 14:10:17 ntpdate[7902]: adjust time server 10.20.0.2 offset -0.000861 sec\n", :stderr=>"", :exit_code=>0}}
2015-06-10T14:10:17 debug: [568] 43dfd2b7-bae1-40ee-b24d-02da19378f1f: MC agent 'execute_shell_command', method 'execute', results: {:sender=>"4", :statuscode=>0, :statusmsg=>"OK", :data=>{:stdout=>"10 Jun 14:10:17 ntpdate[7867]: adjust time server 10.20.0.2 offset -0.000690 sec\n", :stderr=>"", :exit_code=>0}}