[BVT] Clock skew detected for Ceph after cluster deployment

[Vladimir Kuklin]

Ceph ha test fails because of clock skew after snapshot revert

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  api: "1.0"
  build_number: "96"
  build_id: "2015-02-03_22-55-01"
  nailgun_sha: "62dd62897850795fa35d2f359cf4f310d33f65c5"
  python-fuelclient_sha: "2ea7b3e91c1d2ff85110bf5abb161a6f4e537358"
  astute_sha: "ed5270bf9c6c1234797e00bd7d4dd3213253a413"
  fuellib_sha: "2147da0c583a7944f440ceb51236e7cb2e6610c9"
  ostf_sha: "c9100263140008abfcc2704732e98fbdfd644068"
  fuelmain_sha: ""

2015-02-04 01:26:11,412 - WARNING fuel_web_client.py:1207 -- Clock skew detected in Ceph.
2015-02-04 01:26:11,413 - INFO environment.py:552 -- Sync time on revert for admin
2015-02-04 01:26:22,517 - INFO environment.py:572 -- Master node time: ['Wed Feb 4 01:26:22 UTC 2015\n']
2015-02-04 01:26:35,505 - INFO environment.py:547 -- Node time: ['Wed Feb 4 01:26:35 UTC 2015\n']
2015-02-04 01:26:51,505 - INFO environment.py:547 -- Node time: ['Wed Feb 4 01:26:51 UTC 2015\n']
2015-02-04 01:27:07,965 - INFO environment.py:547 -- Node time: ['Wed Feb 4 01:27:07 UTC 2015\n']
2015-02-04 01:27:25,503 - INFO environment.py:547 -- Node time: ['Wed Feb 4 01:27:25 UTC 2015\n']
2015-02-04 01:27:41,508 - INFO environment.py:547 -- Node time: ['Wed Feb 4 01:27:41 UTC 2015\n']
2015-02-04 01:27:57,503 - INFO environment.py:547 -- Node time: ['Wed Feb 4 01:27:57 UTC 2015\n']

[Stanislaw Bogatkin (sbogatkin)]

Seems that it is duplicate of https://bugs.launchpad.net/fuel/+bug/1415596

[Bogdan Dobrelya (bogdando)]

The failed BVT job does not contain the patch for #1415596, indeed. Let's update the bug status once we have next BVT build done

[Stanislaw Bogatkin (sbogatkin)]

It was merged today, so it will be in next ISO.

[Artem Panchenko (apanchenko-8)]

@Stanislaw,

this issue is still reproduced on bvt:

http://jenkins-product.srt.mirantis.net:8080/job/6.1.ubuntu.bvt_2/96/console
http://jenkins-product.srt.mirantis.net:8080/job/6.1.ubuntu.bvt_2/95/console

This bug was marked as duplicate of https://bugs.launchpad.net/fuel/+bug/1415596 , but it's a separate issue because time synchronization (on master and bootstrapped slaves) after reverting from snapshot was successful:

http://paste.openstack.org/show/167406/

Then tests deployed a cluster, found that Ceph health is bad (Clock skew detected) and synced (successfully) time to fix that:

http://paste.openstack.org/show/167407/

but it didn't help (test was waiting for Ceph health recovery 6 minutes):

2015-02-05 12:06:14,979 - DEBUG __init__.py:50 -- Done: get_ceph_health with result: HEALTH_WARN clock skew detected on mon.node-5, mon.node-6

In my opinion time wasn't correctly synchronized after provisioning before Ceph was deployed. I've found these NTP errors in logs on nodes:

http://paste.openstack.org/show/167429/

[Artem Panchenko (apanchenko-8)]

Seems that patches for https://bugs.launchpad.net/fuel/+bug/1417574 bug could fix this issue as well

[Bogdan Dobrelya (bogdando)]

Yes, they should. The ISO used with failed #96 build had not contained them, so we're waiting for new build and rerun of failed BVT

[Andrey Sledzinskiy (asledzinskiy)]

verified on {

    "build_id": "2015-02-15_22-54-44",
    "ostf_sha": "f9c37d0876141e1550eb4e703a8e500cd463282f",
    "build_number": "126",
    "release_versions":

{

    "2014.2-6.1":

{

    "VERSION":

{

    "build_id": "2015-02-15_22-54-44",
    "ostf_sha": "f9c37d0876141e1550eb4e703a8e500cd463282f",
    "build_number": "126",
    "api": "1.0",
    "nailgun_sha": "1e3a40dd8a17abe1d38f42da1e0dc1a6d4572666",
    "production": "docker",
    "python-fuelclient_sha": "61431ed16fc00039a269424bdbaa410277eff609",
    "astute_sha": "1f87a9b9a47de7498b4061d15a8c7fb9435709d5",
    "feature_groups":

            [
                "mirantis"
            ],
            "release": "6.1",
            "fuelmain_sha": "2054229e275d08898b5d079a6625ffcc79ae23b8",
            "fuellib_sha": "7f8d4382abfcd4338964182ebfea1d539f963e66"
        }
    }

},
"auth_required": true,
"api": "1.0",
"nailgun_sha": "1e3a40dd8a17abe1d38f42da1e0dc1a6d4572666",
"production": "docker",
"python-fuelclient_sha": "61431ed16fc00039a269424bdbaa410277eff609",
"astute_sha": "1f87a9b9a47de7498b4061d15a8c7fb9435709d5",
"feature_groups":

    [
        "mirantis"
    ],
    "release": "6.1",
    "fuelmain_sha": "2054229e275d08898b5d079a6625ffcc79ae23b8",
    "fuellib_sha": "7f8d4382abfcd4338964182ebfea1d539f963e66"

}

[Stanislaw Bogatkin (sbogatkin)]

This bug was created due to problems with new ntp package and revert from it. We don't have such flappy settings in 6.0. Closed.