sm provision stuck at post_provision of compute

[wenqing liang]

R3.1.0.0-build14 mitaka SM provision hangs at :

            "provision_role_sequence": "{'completed': [('cmbu-ceph-perf1', 'haproxy', '2016_08_03__18_28_24'), ('cmbu-ceph-perf1', 'database', '2016_08_03__18_30_01'), ('cmbu-ceph-perf1', 'openstack', '2016_08_03__18_42_38'), ('cmbu-ceph-perf1', 'config', '2016_08_03__18_48_52'), ('cmbu-ceph-perf1', 'control', '2016_08_03__18_49_40'), ('cmbu-ceph-perf1', 'collector', '2016_08_03__18_50_45'), ('cmbu-ceph-perf1', 'webui', '2016_08_03__18_51_42'), ('cmbu-ceph-perf4', 'compute', '2016_08_03__19_02_14'), ('cmbu-ceph-perf2', 'compute', '2016_08_03__19_02_31'), ('cmbu-ceph-perf3', 'compute', '2016_08_03__19_02_51')], 'steps': [[(u'cmbu-ceph-perf2', 'post_provision'), (u'cmbu-ceph-perf3', 'post_provision'), (u'cmbu-ceph-perf4', 'post_provision'), (u'cmbu-ceph-perf2', 'storage-compute'), (u'cmbu-ceph-perf3', 'storage-compute'), (u'cmbu-ceph-perf4', 'storage-compute'), (u'cmbu-ceph-perf1', 'storage-master'), (u'cmbu-ceph-perf1', 'post_provision')]]}",
            "provisioned_id": null
        }
    ]
}
root@cmbu-auto-esx1-lnx02:~/store#

Aug 3 20:00:05 cmbu-ceph-perf2 kernel: [ 3516.096235] init: ceph-rest-api main process (2168) terminated with status 1
Aug 3 20:00:05 cmbu-ceph-perf2 kernel: [ 3516.096252] init: ceph-rest-api main process ended, respawning
Aug 3 20:00:53 cmbu-ceph-perf2 kernel: [ 3564.050458] init: ceph-disk (/dev/sde1/3195) main process (3210) terminated with status 1
Aug 3 20:00:53 cmbu-ceph-perf2 kernel: [ 3564.230592] XFS (sdc1): Mounting Filesystem
Aug 3 20:00:53 cmbu-ceph-perf2 kernel: [ 3564.265322] XFS (sdc1): Ending clean mount
Aug 3 20:00:53 cmbu-ceph-perf2 kernel: [ 3564.396720] XFS (sde1): Mounting Filesystem
Aug 3 20:00:53 cmbu-ceph-perf2 kernel: [ 3564.449577] XFS (sde1): Ending clean mount
Aug 3 20:00:54 cmbu-ceph-perf2 kernel: [ 3564.927170] XFS (sdc1): Mounting Filesystem
Aug 3 20:00:54 cmbu-ceph-perf2 kernel: [ 3564.959383] XFS (sdc1): Ending clean mount
Aug 3 20:00:54 cmbu-ceph-perf2 kernel: [ 3565.112998] XFS (sde1): Mounting Filesystem
Aug 3 20:00:54 cmbu-ceph-perf2 kernel: [ 3565.166143] XFS (sde1): Ending clean mount
Aug 3 20:00:54 cmbu-ceph-perf2 kernel: [ 3565.426625] XFS (sde1): Mounting Filesystem
Aug 3 20:00:54 cmbu-ceph-perf2 kernel: [ 3565.473986] XFS (sde1): Ending clean mount
Aug 3 20:00:55 cmbu-ceph-perf2 puppet-agent[5361]: (/Stage[storage]/Contrail::Storage/Contrail::Lib::Storage_common[storage-compute]/Contrail::Lib::Storage_disk[/dev/sdf]/Ceph::Osd[/dev/sdf]/Exec[ceph-osd-prepare-/dev/sdf]/unless) Check "/bin/true # comment to satisfy puppet syntax requirements\nset -ex\nceph-disk list | grep ' */dev/sdf.*ceph data, prepared' ||\nceph-disk list | grep ' */dev/sdf.*ceph data, active' ||\nls -l /var/lib/ceph/osd/ceph-* | grep ' /dev/sdf'\n" exceeded timeout

root@cmbu-auto-esx1-lnx02:~/store# server-manager status server
{
    "server": [
        {
            "id": "cmbu-ceph-perf1",
            "ip_address": "10.87.140.197",
            "mac_address": "00:25:90:AB:9C:88",
            "status": "storage_started"
        },
        {
            "id": "cmbu-ceph-perf2",
            "ip_address": "10.87.140.198",
            "mac_address": "00:25:90:35:8A:1F",
            "status": "storage_started"
        },
        {
            "id": "cmbu-ceph-perf3",
            "ip_address": "10.87.140.199",
            "mac_address": "00:25:90:92:0E:6C",
            "status": "storage_started"
        },
        {
            "id": "cmbu-ceph-perf4",
            "ip_address": "10.87.140.200",
            "mac_address": "00:25:90:92:0D:F2",
            "status": "storage_started"
        }
    ]
}
root@cmbu-auto-esx1-lnx02:~/store#

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23191
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23231
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23191
Committed: http://github.org/Juniper/contrail-puppet/commit/5a19fc2136b61e444b9efafee9ed643b0b88f859
Submitter: Zuul
Branch: R3.1

commit 5a19fc2136b61e444b9efafee9ed643b0b88f859
Author: Dheeraj Gautam <email address hidden>
Date: Wed Aug 10 16:32:06 2016 -0700

SM-Storage: create ceph user/group and use latest package resource

Partial Bug: #1609619
Partial Bug: #1611446

With current version of ceph, uses ceph:ceph instead of root:root. This causes
ceph-mon to fail to come up and provisioning gets stuck.

ensure latest for contrail-storage package instead of present. This will
install latest version for package available.

PATCH 2:
add ceph:ceph for /var/lib/ceph and directory/files under /var/lib/ceph. This
is needed for mitaka

Change-Id: Id317d8d00d05deb2196f3f710d0af33ee3aeccc5

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23231
Committed: http://github.org/Juniper/contrail-puppet/commit/22ac391e66d5f29a2082cbbfc4c1814cb28facfd
Submitter: Zuul
Branch: R3.1

commit 22ac391e66d5f29a2082cbbfc4c1814cb28facfd
Author: Dheeraj Gautam <email address hidden>
Date: Thu Aug 11 17:10:26 2016 -0700

SM-Storage: Fix Liberty live-migration and mitaka provisioning

Partial-Bug: #1609619

1. Fix ceph:ceph for /var/lib/ceph before starting service
2. Remove rpc_backend from cinder.conf
3. remove scheduler from cinder call

Change-Id: I4c94541d0bd13ac7f6a6dd80f16099911ffb42a6

[Dheeraj Gautam (dgautam)]

With Build 23, mitaka provisioning gets completed.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23292
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23293
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23292
Committed: http://github.org/Juniper/contrail-puppet/commit/b5843bf8721ff62540ccbbfe98e74a63ccfa9105
Submitter: Zuul
Branch: master

commit b5843bf8721ff62540ccbbfe98e74a63ccfa9105
Author: Dheeraj Gautam <email address hidden>
Date: Wed Aug 10 16:32:06 2016 -0700

SM-Storage: create ceph user/group and use latest package resource

Partial Bug: #1609619
Partial Bug: #1611446

With current version of ceph, uses ceph:ceph instead of root:root. This causes
ceph-mon to fail to come up and provisioning gets stuck.

ensure latest for contrail-storage package instead of present. This will
install latest version for package available.

PATCH 2:
add ceph:ceph for /var/lib/ceph and directory/files under /var/lib/ceph. This
is needed for mitaka

Change-Id: Id317d8d00d05deb2196f3f710d0af33ee3aeccc5

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23293
Committed: http://github.org/Juniper/contrail-puppet/commit/eb433fc1ff71a50856a506cadc52b323575a99d5
Submitter: Zuul
Branch: master

commit eb433fc1ff71a50856a506cadc52b323575a99d5
Author: Dheeraj Gautam <email address hidden>
Date: Thu Aug 11 17:10:26 2016 -0700

SM-Storage: Fix Liberty live-migration and mitaka provisioning

Partial-Bug: #1609619

1. Fix ceph:ceph for /var/lib/ceph before starting service
2. Remove rpc_backend from cinder.conf
3. remove scheduler from cinder call

Change-Id: I4c94541d0bd13ac7f6a6dd80f16099911ffb42a6