Comment 3 for bug 1461398

Wenqing, Hit this issue again today on build 47, with the patch from
https://review.opencontrail.org/#/c/11442/

Below are the analysis from my side,

I think the sequence for storage is wrong.

t@cmbu-auto-esx1-lnx02:~# server-manager status server
{
    "server": [
        {
            "id": "cmbu-ceph-perf1",
            "ip_address": "10.87.140.197",
            "mac_address": "00:25:90:AB:9C:88",
            "status": "storage_started"
        },
        {
            "id": "cmbu-ceph-perf2",
            "ip_address": "10.87.140.198",
            "mac_address": "00:25:90:92:0D:54",
            "status": "storage_started"
        },
        {
            "id": "cmbu-ceph-perf3",
            "ip_address": "10.87.140.199",
            "mac_address": "00:25:90:92:0E:6C",
            "status": "storage-master_completed"
        },
        {
            "id": "cmbu-ceph-perf4",
            "ip_address": "10.87.140.200",
            "mac_address": "00:25:90:92:0D:F2",
            "status": "storage_started"
        }
    ]
}
root@cmbu-auto-esx1-lnx02:~# server-manager show server --select "id,ip_address,roles"
{
    "server": [
        {
            "id": "cmbu-ceph-perf1",
            "ip_address": "10.87.140.197",
            "roles": [
                "config",
                "openstack",
                "control",
                "collector",
                "webui",
                "database",
                "storage-master"
            ]
        },
        {
            "id": "cmbu-ceph-perf2",
            "ip_address": "10.87.140.198",
            "roles": [
                "compute",
                "storage-compute"
            ]
        },
        {
            "id": "cmbu-ceph-perf3",
            "ip_address": "10.87.140.199",
            "roles": [
                "compute",
                "storage-compute"
            ]
        },
        {
            "id": "cmbu-ceph-perf4",
            "ip_address": "10.87.140.200",
            "roles": [
                "compute",
                "storage-compute"
            ]
        }
    ]
}
===============
Not sure why we have storage-master completed when there is no storage master at "cmbu-ceph-perf4"
and only has a storage-compute

Also, Not sure if role sequence is right?

            "provision_role_sequence": "{'completed': [('cmbu-ceph-perf1', 'haproxy', '2015_06_10__11_33_28'), ('cmbu-ceph-perf1', 'database', '2015_06_10__11_35_06'),('cmbu-ceph-perf1', 'openstack', '2015_06_10__11_43_45'), ('cmbu-ceph-perf1', 'config', '2015_06_10__11_45_57'), ('cmbu-ceph-perf1', 'control', '2015_06_10__11_46_34') ('cmbu-ceph-perf1', 'collector', '2015_06_10__11_47_08'), ('cmbu-ceph-perf1', 'webui', '2015_06_10__11_47_27'), ('cmbu-ceph-perf2', 'compute', '2015_06_10__11_52_48') ('cmbu-ceph-perf3', 'compute', '2015_06_10__11_53_00'), ('cmbu-ceph-perf4', 'compute', '2015_06_10__11_53_02')], 'steps': [[(u'cmbu-ceph-perf2', 'post_provision'), (ucmbu-ceph-perf3', 'post_provision'), (u'cmbu-ceph-perf4', 'post_provision'), (u'cmbu-ceph-perf2', 'storage-compute'), (u'cmbu-ceph-perf3', 'storage-compute'), (u'cmbu-eph-perf4', 'storage-compute'), (u'cmbu-ceph-perf1', 'storage-master'), (u'cmbu-ceph-perf1', 'post_provision')]]}",

When tries provisioning on same setup without storage,
provision completed, compute was fine also.
root@cmbu-auto-esx1-lnx02:~# server-manager status server
{
    "server": [
        {
            "id": "cmbu-ceph-perf1",
            "ip_address": "10.87.140.197",
            "mac_address": "00:25:90:AB:9C:88",
            "status": "provision_completed"
        },
        {
            "id": "cmbu-ceph-perf2",
            "ip_address": "10.87.140.198",
            "mac_address": "00:25:90:92:0D:54",
            "status": "provision_completed"
        },
        {
            "id": "cmbu-ceph-perf3",
            "ip_address": "10.87.140.199",
            "mac_address": "00:25:90:92:0E:6C",
            "status": "provision_completed"
        },
        {
            "id": "cmbu-ceph-perf4",
            "ip_address": "10.87.140.200",
            "mac_address": "00:25:90:92:0D:F2",
            "status": "provision_completed"
        }
    ]
}

root@cmbu-ceph-perf1:~# nova service-list
+----+------------------+-----------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+-----------------+----------+---------+-------+----------------------------+-----------------+
| 1 | nova-conductor | cmbu-ceph-perf1 | internal | enabled | up | 2015-06-10T23:08:09.000000 | - |
| 2 | nova-consoleauth | cmbu-ceph-perf1 | internal | enabled | up | 2015-06-10T23:08:10.000000 | - |
| 3 | nova-scheduler | cmbu-ceph-perf1 | internal | enabled | up | 2015-06-10T23:08:08.000000 | - |
| 4 | nova-console | cmbu-ceph-perf1 | internal | enabled | up | 2015-06-10T23:08:06.000000 | - |
| 5 | nova-compute | cmbu-ceph-perf3 | nova | enabled | up | 2015-06-10T23:08:10.000000 | - |
| 6 | nova-compute | cmbu-ceph-perf4 | nova | enabled | up | 2015-06-10T23:08:03.000000 | - |
| 7 | nova-compute | cmbu-ceph-perf2 | nova | enabled | up | 2015-06-10T23:08:04.000000 | - |
+----+------------------+-----------------+----------+---------+-------+----------------------------+-----------------+
root@cmbu-ceph-perf1:~#