Create volume and attach it to instance test failed on Ubuntu.BVT

[Egor Kotko]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  api: "1.0"
  build_number: "97"
  build_id: "2015-02-04_10-13-35"
  nailgun_sha: "62dd62897850795fa35d2f359cf4f310d33f65c5"
  python-fuelclient_sha: "3355e188eec37b9736282683538b4aa51edc6897"
  astute_sha: "ed5270bf9c6c1234797e00bd7d4dd3213253a413"
  fuellib_sha: "df615cc9206126b1910b0dd501eb1f5533780344"
  ostf_sha: "c9100263140008abfcc2704732e98fbdfd644068"
  fuelmain_sha: ""

Ubuntu.BVT failed with error:
Create volume and attach it to instance (failure): Time limit exceeded while waiting for volume becoming 'in-use' to finish.

Test Result:
http://jenkins-product.srt.mirantis.net:8080/view/6.1/job/6.1.ubuntu.bvt_2/92/testReport/%28root%29/ceph_ha/ceph_ha/

 Scenario:
            1. Create cluster
            2. Add 3 nodes with controller and ceph OSD roles
            3. Add 1 node with ceph OSD roles
            4. Add 2 nodes with compute and ceph OSD roles
            5. Deploy the cluster
            6. Check ceph status

        Duration 90m
        Snapshot ceph_ha

[Aleksandr Didenko (adidenko)]

I've got similar bug on custom BVT with absolutely the same simptoms.:

http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/custom.ubuntu.bvt_2/393/

I've managed to debug broken env for some time and it looks like monit is the root cause here. So here's what's happenning on compute node during deployment:

1) Puppet configures and starts monit service
Wed Feb 04 11:19:52 +0000 2015 Puppet (debug): Executing '/etc/init.d/monit start'

2) Monit starts 'nova-compute' since it's not running yet:
[UTC Feb 4 11:19:53] info : 'nova-compute' start: /usr/sbin/service

2) Puppet finishes configuration for ceph and nova-compute and evalutes 'nova-compute' service:
Wed Feb 04 11:20:01 +0000 2015 /Service[nova-compute] (info): Starting to evaluate the resource
Wed Feb 04 11:20:01 +0000 2015 /Service[nova-compute] (info): Evaluated in 0.06 seconds
But it's already started by monit.

So it looks like monit starts nova-compute whichs interfere with deployment process where this service should be configured and started by puppet.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/152950

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/152950
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=3a4beabcbd20e14cea442505628ba179fdc0c510
Submitter: Jenkins
Branch: master

commit 3a4beabcbd20e14cea442505628ba179fdc0c510
Author: Aleksandr Didenko <email address hidden>
Date: Wed Feb 4 17:37:53 2015 +0200

    Revert "Enable monit for granular roles"

    Closes-bug: #1418076
    This reverts commit 8e2fc4537915778f7ab36778c2b4cfa69ed9d054.

    Change-Id: Icef28222b978ef90d5f5845acda85685e2c74d3f

[Andrey Sledzinskiy (asledzinskiy)]

verified on {

    "build_id": "2015-02-15_22-54-44",
    "ostf_sha": "f9c37d0876141e1550eb4e703a8e500cd463282f",
    "build_number": "126",
    "release_versions":

{

    "2014.2-6.1":

{

    "VERSION":

{

    "build_id": "2015-02-15_22-54-44",
    "ostf_sha": "f9c37d0876141e1550eb4e703a8e500cd463282f",
    "build_number": "126",
    "api": "1.0",
    "nailgun_sha": "1e3a40dd8a17abe1d38f42da1e0dc1a6d4572666",
    "production": "docker",
    "python-fuelclient_sha": "61431ed16fc00039a269424bdbaa410277eff609",
    "astute_sha": "1f87a9b9a47de7498b4061d15a8c7fb9435709d5",
    "feature_groups":

            [
                "mirantis"
            ],
            "release": "6.1",
            "fuelmain_sha": "2054229e275d08898b5d079a6625ffcc79ae23b8",
            "fuellib_sha": "7f8d4382abfcd4338964182ebfea1d539f963e66"
        }
    }

},
"auth_required": true,
"api": "1.0",
"nailgun_sha": "1e3a40dd8a17abe1d38f42da1e0dc1a6d4572666",
"production": "docker",
"python-fuelclient_sha": "61431ed16fc00039a269424bdbaa410277eff609",
"astute_sha": "1f87a9b9a47de7498b4061d15a8c7fb9435709d5",
"feature_groups":

    [
        "mirantis"
    ],
    "release": "6.1",
    "fuelmain_sha": "2054229e275d08898b5d079a6625ffcc79ae23b8",
    "fuellib_sha": "7f8d4382abfcd4338964182ebfea1d539f963e66"

}