Vcenter-as-compute: VM launch not distributed across all Esxi in multiple clusters

[Sarath]

Version: 4.0.1.0-48-newton
Topology: 3node HA with multiple computes (multi-cluster Esxi) & Kvm

Kamlesh debugged the problem setup and found nova config issue.
Please find below the debugging details,

root@5a10s31:~#
root@5a10s31:~# nova availability-zone-list
+-----------------------+----------------------------------------+
| Name | Status |
+-----------------------+----------------------------------------+
| internal | available |
| |- 5a10s29 | |
| | |- nova-conductor | enabled :-) 2017-09-08T18:20:18.000000 |
| | |- nova-scheduler | enabled :-) 2017-09-08T18:20:17.000000 |
| |- 5a10s30 | |
| | |- nova-conductor | enabled :-) 2017-09-08T18:20:18.000000 |
| | |- nova-scheduler | enabled :-) 2017-09-08T18:20:16.000000 |
| |- 5a10s31 | |
| | |- nova-conductor | enabled :-) 2017-09-08T18:20:15.000000 |
| | |- nova-scheduler | enabled :-) 2017-09-08T18:20:15.000000 |
| | |- nova-consoleauth | enabled :-) 2017-09-08T18:20:21.000000 |
| nova | available |
| |- 5a10s23 | |
| | |- nova-compute | enabled :-) 2017-09-08T18:20:18.000000 |
| |- 5a10s29 | |
| | |- nova-compute | enabled :-) 2017-09-08T18:20:14.000000 |
+-----------------------+----------------------------------------+
root@5a10s31:~#

[kamlesh parmar (kparmar)]

The configuration is not right.
The vc-compute ip sepecified at the cluster json is not correct.

                                    "datacenters": {
                                        "A6s32-VC-Compute-Sanity": {
                                            "datacenter_mtu": 1500,
                                            "dv_switches": [
                                                {
                                                    "clusternames": [
                                                        "cluster-1"
                                                    ],
                                                    "dv_port_group": {
                                                        "dv_portgroup_name": "Sanity-dvportgroup-1",
                                                        "number_of_ports": "1800"
                                                    },
                                                    "dv_switch_name": "Sanity-dvswitch-1",
                                                    "vcenter_compute": "10.87.36.12"
                                                },
                                                {
                                                    "clusternames": [
                                                        "cluster-2"
                                                    ],
                                                    "dv_port_group": {
                                                        "dv_portgroup_name": "Sanity-dvportgroup-2",
                                                        "number_of_ports": "1800"
                                                    },
                                                    "dv_switch_name": "Sanity-dvswitch-2",
                                                    "vcenter_compute": "10.87.36.15"
                                                }
                                            ]

There is no server with 10.87.36.15 ip address defined.

root@5a10s26:/opt/contrail/server_manager/ansible/playbooks/contrail_4_0_1_48_vc_new/playbooks# server-manager display server
+--------------------+-------------+-------------------+
| id | ip_address | mac_address |
+--------------------+-------------+-------------------+
| 5a10s31 | 10.87.36.10 | 00:25:90:AA:31:6C |
| 5a10s30 | 10.87.36.11 | 00:25:90:AA:A9:1E |
| 5a10s29 | 10.87.36.12 | 00:25:90:AA:31:6A |
| 5a10s01b | 10.87.36.27 | C4:54:44:44:D5:C4 |
| 5a10s23 | 10.87.36.18 | 0C:C4:7A:69:83:D2 |
| contrailvm-5a10s27 | 10.87.36.19 | 00:77:56:AA:BB:01 |
| contrailvm-5a10s25 | 10.87.36.20 | 00:77:56:AA:BB:02 |
| contrailvm-5a10s31 | 10.87.36.21 | 00:77:56:AA:BB:03 |
+--------------------+-------------+-------------------+

[Sarath (nsarath)]

From: Sarathbabu Narasimhan
Sent: Friday, September 08, 2017 3:18 PM
To: Kamlesh Parmar <email address hidden>; Sachchidanand Vaidya <email address hidden>; Abhay Joshi <email address hidden>; Sandip Dey <email address hidden>; Pavana Prakash <email address hidden>; Amudha R <email address hidden>
Cc: Jeba Paulaiyan <email address hidden>; Sudheendra Rao <email address hidden>; Rudra Rugge <email address hidden>
Subject: RE: 4.0.1 vCenter Testing

Kamlesh, sure I agree this IP issue was due to swap recently. So, in case of 16-04/newton, it using kvm and only one esxi cluster looks valid.
But incase of Mitaka, with this same config file none of the Esxi clusters used for VM-launch and all the VM launched on KVM only.

As the bug opened for this, let me re-create on Mitaka and will update bug.

Thanks
*Sarath

[Sarath (nsarath)]

Verified with latest #74-mitaka, the issue not seen and please find below the sanity logs captured for
failed #50 and current scenario #74, Marking the bug state accordingly lowering priority

failed mitaka #50 (sanity-logs),
2017-08-31 14:11:15,467 - DEBUG - Check whether image by uuid 193f9d0d-60a0-4c9b-8b72-7c5bdd652295 is active
2017-08-31 14:11:18,971 - INFO - VM ([<Server: vm1>]) created on node: (5a10s23), Zone: (nova:5a10s23)
2017-08-31 14:11:19,875 - DEBUG - Check whether image by uuid 193f9d0d-60a0-4c9b-8b72-7c5bdd652295 is active
2017-08-31 14:11:22,855 - INFO - VM ([<Server: vm2>]) created on node: (5a10s23), Zone: (nova:5a10s23)
2017-08-31 14:11:23,890 - DEBUG - Check whether image by uuid 193f9d0d-60a0-4c9b-8b72-7c5bdd652295 is active
2017-08-31 14:11:26,021 - INFO - VM ([<Server: vm3>]) created on node: (5a10s23), Zone: (nova:5a10s23)
2017-08-31 14:11:27,166 - DEBUG - Check whether image by uuid 193f9d0d-60a0-4c9b-8b72-7c5bdd652295 is active
2017-08-31 14:11:29,410 - INFO - VM ([<Server: vm4>]) created on node: (5a10s23), Zone: (nova:5a10s23)

current #74-mitaka same tests with VM launch distributed across computes in round-robbin,
2017-09-11 14:32:16,728 - INFO - VM ([<Server: vm1>]) created on node: (5a10s23), Zone: (nova:5a10s23)
2017-09-11 14:32:24,230 - INFO - VM ([<Server: vm2>]) created on node: (5a10s01b), Zone: (nova:5a10s01b)
2017-09-11 14:32:26,985 - INFO - VM ([<Server: vm3>]) created on node: (5a10s29), Zone: (nova:5a10s29)
2017-09-11 14:32:29,002 - INFO - VM ([<Server: vm4>]) created on node: (5a10s23), Zone: (nova:5a10s23)

root@5a10s31:/etc/apt# nova availability-zone-list
+-----------------------+----------------------------------------+
| Name | Status |
+-----------------------+----------------------------------------+
| internal | available |
| |- 5a10s29 | |
| | |- nova-conductor | enabled :-) 2017-09-11T21:38:18.000000 |
| | |- nova-consoleauth | enabled :-) 2017-09-11T21:38:09.000000 |
| | |- nova-scheduler | enabled :-) 2017-09-11T21:38:09.000000 |
| | |- nova-console | enabled :-) 2017-09-11T21:38:09.000000 |
| |- 5a10s30 | |
| | |- nova-conductor | enabled :-) 2017-09-11T21:38:13.000000 |
| | |- nova-consoleauth | enabled :-) 2017-09-11T21:38:14.000000 |
| | |- nova-scheduler | enabled :-) 2017-09-11T21:38:14.000000 |
| | |- nova-console | enabled :-) 2017-09-11T21:38:14.000000 |
| |- 5a10s31 | |
| | |- nova-conductor | enabled :-) 2017-09-11T21:38:09.000000 |
| | |- nova-consoleauth | enabled :-) 2017-09-11T21:38:11.000000 |
| | |- nova-scheduler | enabled :-) 2017-09-11T21:38:10.000000 |
| | |- nova-console | enabled :-) 2017-09-11T21:38:11.000000 |
| nova | available |
| |- 5a10s01b | |
| | |- nova-compute | enabled :-) 2017-09-11T21:38:14.000000 |
| |- 5a10s23 | |
| | |- nova-comput...