[fuel-bootstrap] Time syncing freezes for ubuntu bootstrapped nodes

[Dmitry Kalashnik]

Steps to reproduce:
1. Run build_default_bootstrap test.

-or-
1. Build ubuntu bootstrap
2. Activate it
3. Bootstrap nodes
4. Try to sync time

Expected result:
Test passed successfully

Actual result:
Test freezes on time-sync

Logs:
2015-12-03 12:45:00,902 - INFO environment.py:314 -- We have snapshot with such name: ready
2015-12-03 12:45:00,902 - INFO environment.py:316 -- Reverting the snapshot 'ready' ....
2015-12-03 12:45:07,396 - INFO environment.py:319 -- Resuming the snapshot 'ready' ....
2015-12-03 12:45:07,799 - INFO environment.py:357 -- Admin node login name: 'root' , password: 'r00tme'
2015-12-03 12:45:17,003 - INFO environment.py:489 -- Synchronizing time on Fuel admin node
2015-12-03 12:45:35,023 - INFO ntp.py:97 -- Time on 'admin' = Thu Dec 3 12:45:35 UTC 2015
2015-12-03 12:45:35,102 - INFO nailgun_client.py:28 -- Initiate Nailgun client with url http://10.109.15.2:8000
2015-12-03 12:45:35,102 - INFO http.py:28 -- Initiate HTTPClient with url http://10.109.15.2:8000
2015-12-03 12:45:35,108 - WARNING http.py:95 -- Authorization failure: Authentication required
2015-12-03 12:45:35,109 - INFO http.py:38 -- Initialize keystoneclient with url http://10.109.15.2:5000/v2.0
2015-12-03 12:59:35,538 - INFO environment.py:130 -- Bootstrapping node: slave-01
2015-12-03 12:59:41,542 - INFO environment.py:130 -- Bootstrapping node: slave-02
2015-12-03 12:59:47,508 - INFO environment.py:130 -- Bootstrapping node: slave-03
2015-12-03 13:00:40,494 - INFO environment.py:489 -- Synchronizing time on Fuel admin node
2015-12-03 13:00:59,068 - INFO ntp.py:97 -- Time on 'admin' = Thu Dec 3 13:00:59 UTC 2015
2015-12-03 13:00:59,132 - INFO environment.py:501 -- Synchronizing time on other active nodes

[Aleksey Zvyagintsev (azvyagintsev)]

More info :
###
root@bootstrap:/etc# /etc/init.d/ntp status
 * NTP server is not running
root@bootstrap:/etc# /etc/init.d/ntp start
 * Starting NTP server ntpd
   ...done.
root@bootstrap:/etc# /etc/init.d/ntp status
 * NTP server is not running
###
 start-stop-daemon --start --quiet --oknodo --pidfile /var/run/ntpd.pid --startas /usr/sbin/ntpd -- -d -p /var/run/ntpd.pid -g -u 102:105
ntpd 4.2.6p5@1.2349-o Fri Oct 23 16:44:56 UTC 2015 (1)
 3 Dec 13:26:23 ntpd[6469]: proto: precision = 0.113 usec
event at 0 0.0.0.0 c01d 0d kern kernel time sync enabled
Finished Parsing!!
 3 Dec 13:26:23 ntpd[6469]: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16
 3 Dec 13:26:23 ntpd[6469]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
 3 Dec 13:26:23 ntpd[6469]: Listen and drop on 1 v6wildcard :: UDP 123
 3 Dec 13:26:23 ntpd[6469]: Listen normally on 2 lo 127.0.0.1 UDP 123
restrict: op 1 addr 127.0.0.1 mask 255.255.255.255 mflags 00003000 flags 00000001
 3 Dec 13:26:23 ntpd[6469]: Listen normally on 3 eth0 10.109.0.12 UDP 123
restrict: op 1 addr 10.109.0.12 mask 255.255.255.255 mflags 00003000 flags 00000001
 3 Dec 13:26:23 ntpd[6469]: Listen normally on 4 lo ::1 UDP 123
restrict: op 1 addr ::1 mask ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mflags 00003000 flags 00000001
 3 Dec 13:26:23 ntpd[6469]: Listen normally on 5 eth4 fe80::66b5:9fff:fecd:3a UDP 123
restrict: op 1 addr fe80::66b5:9fff:fecd:3a mask ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mflags 00003000 flags 00000001
 3 Dec 13:26:23 ntpd[6469]: Listen normally on 6 eth2 fe80::6650:3aff:fe7e:933e UDP 123
restrict: op 1 addr fe80::6650:3aff:fe7e:933e mask ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mflags 00003000 flags 00000001
 3 Dec 13:26:23 ntpd[6469]: Listen normally on 7 eth3 fe80::6624:f9ff:fec2:dab UDP 123
restrict: op 1 addr fe80::6624:f9ff:fec2:dab mask ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mflags 00003000 flags 00000001
 3 Dec 13:26:23 ntpd[6469]: Listen normally on 8 eth0 fe80::6616:63ff:fe3f:71cf UDP 123
restrict: op 1 addr fe80::6616:63ff:fe3f:71cf mask ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mflags 00003000 flags 00000001
 3 Dec 13:26:23 ntpd[6469]: Listen normally on 9 eth1 fe80::6631:fbff:fe4d:a85 UDP 123
restrict: op 1 addr fe80::6631:fbff:fe4d:a85 mask ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mflags 00003000 flags 00000001
 3 Dec 13:26:23 ntpd[6469]: peers refreshed
 3 Dec 13:26:23 ntpd[6469]: Listening on routing socket on fd #26 for interface updates
restrict: op 1 addr 0.0.0.0 mask 0.0.0.0 mflags 00000000 flags 000005d0
restrict: op 1 addr :: mask :: mflags 00000000 flags 000005d0
restrict: op 1 addr 127.0.0.1 mask 255.255.255.255 mflags 00000000 flags 00000000
restrict: op 1 addr ::1 mask ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mflags 00000000 flags 00000000
key_expire: at 0 associd 45312
peer_clear: at 0 next 1 associd 45312 refid INIT
event at 0 213.151.89.43 8011 81 mobilize assoc 45312
newpeer: 10.109.0.12->213.151.89.43 mode 3 vers 4 poll 6 10 flags 0x1 0x1 ttl 0 key 00000000
key_expire: at 0 associd 45313
peer_clear: at 0 next 2 associd 45313 refid INIT
event at 0 46.28.111.54 8011 81 mobilize assoc 45313
newpeer: 10.109.0.12->46.28.111.54 mode 3 vers 4 poll 6 ...

[Aleksey Zvyagintsev (azvyagintsev)]

Files simply Rsync's into root
https://github.com/openstack/fuel-agent/blob/master/fuel_agent/utils/build.py#L782

[Aleksey Zvyagintsev (azvyagintsev)]

How-to build ubuntu-bootstrap

1) fuel-bootstrap build --activate
2) reboot discover nodes

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/252922
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=6356e90686efcc0f5aa1c6bfc108688975ce9650
Submitter: Jenkins
Branch: master

commit 6356e90686efcc0f5aa1c6bfc108688975ce9650
Author: alexz <email address hidden>
Date: Thu Dec 3 14:24:03 2015 +0200

    Fix trusty based slave bootstrap

    * Sync fix-configs-on-startup with fuel-main/bootstrap
    * Due wrong permission for trusty/etc\usr folders, after
    rsync's files into chroot system - ntp fails to run due Access Denied
    error
    * fix ubuntu cloud-init hook for ntp.drift
    * nailgun-agent runs only from cron
    * fix file permission for trusty bootstrap
    * add 'psmisc' for fuel-agent requirements
      Centos7 don't install it by default, otherwise mcollective
      container - which used for IBP

    Change-Id: Ifecbc4220df8ae77f7373d4b342aa1e72deb9059
    Closes-Bug: #1522406

[Dmitry Kalashnik (dkalashnik)]

Verified on ISO 328