Time desync on slaves leads to Cinder errors

[Artem Panchenko]

api: '1.0'
astute_sha: ed5270bf9c6c1234797e00bd7d4dd3213253a413
auth_required: true
build_id: 2015-01-29_10-46-25
build_number: '85'
feature_groups:
- mirantis
fuellib_sha: b2fbaa9ffb74fafe1f5c2c480944a78424e1ae28
fuelmain_sha: ''
nailgun_sha: 648cad955ec88afb84e599763c1d50c1164c3b17
ostf_sha: c9100263140008abfcc2704732e98fbdfd644068
production: docker
python-fuelclient_sha: cb8928ce34f5ca88c0d6cecc6331488db75362ac
release: '6.1'

Smoke tests failed (Ubuntu, NeutronGRE, CinderLVM) because OSTF was unable to create volume:

http://jenkins-product.srt.mirantis.net:8080/view/6.1/job/6.1.ubuntu.smoke_neutron/16/testReport/junit/%28root%29/deploy_neutron_gre/deploy_neutron_gre/

Environment details:

 - Ubuntu, Simple (multinode), NeutronVlan
 - 1 controller and 2 compute+cinder nodes

I reproduced the same issue with creating volume manually:

http://paste.openstack.org/show/163834/

Cinder was unable to create new volume because all cinder-volume services were down:

http://paste.openstack.org/show/163835/

I found that time on slaves is desynced:

http://paste.openstack.org/show/163836/

And NTP daemon isn't running:

http://paste.openstack.org/show/163838/

After time synchronization on all slaves (I ran "ntpdate -vu $(awk '/^server/ && $2 !~ /127.*/ {print $2}' /etc/ntp.conf)" command), issue with cinder volumes was fixed.

[Stanislaw Bogatkin (sbogatkin)]

Ubuntu HA should be vulnerable to that bug too.
It's happened because new ntp 4.2.8 package was merged into master, and it create /var/lib/ntp directory. In our ntp preseed snippet we use long line of 'command && mkdir /var/lib/ntp && command && echo "server $server burst iburst" >> /target/etc/ntp.conf && command'. As long as package ntp-dev installed before snippet applies, so /var/lib/ntp alreade exists on moment applying that part of snippet and command that adding string with 'server' doesn't applies. After that ubuntu node sync to nothing and after that time is desynced.

Workaround is manually created string "server <server name or IP here> burst iburst" in /etc/ntp.conf.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/151620

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/151620
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=43975ae37aff712d49f3862149b812c29a9f4f02
Submitter: Jenkins
Branch: master

commit 43975ae37aff712d49f3862149b812c29a9f4f02
Author: Stanislaw Bogatkin <email address hidden>
Date: Fri Jan 30 15:24:55 2015 +0300

    Change way to create /var/lib/ntp from ubuntu snippet

    mkdir call leaved here to backward compatibility with next
    versions of ntp (which may be pre-packaged and may not create
    /var/lib/ntp directory).

    Change-Id: Ieb3b045f1c8b2cb36c8677ce7901c8fc71c9e2ed
    Closes-Bug: 1416133

[Stanislaw Bogatkin (sbogatkin)]

This particular bug cannot be applied to 6.0 series due to that fact that we hadn't ntp-dev package in 6.0.