From: Amudha R <email address hidden>
Date: Wednesday, July 11, 2018 at 1:05 AM
To: Sandip Dey <email address hidden>, Sachchidanand Vaidya <email address hidden>, Ram Yadav <email address hidden>
Cc: Sudheendra Rao <email address hidden>
Subject: Re: R5.0.1 vcenter provisioning

Please read inline. 

Thanks,
Amudha

From: Sandip Dey <email address hidden>
Date: Tuesday, July 10, 2018 at 11:36 AM
To: Amudha R <email address hidden>, Sachchidanand Vaidya <email address hidden>, Ram Yadav <email address hidden>
Cc: Sudheendra Rao <email address hidden>
Subject: Re: R5.0.1 vcenter provisioning

My setup is up with sriov/pcie/ctrl-data single interface and eam.

You can use if you want to check things

10.204.216.61 - controller
All the files are here /root/contrail-ansible-deployer

Amudha, the pcie interface came up with name eth0 this time. 
[Amudha] - Did you mean eth20?

We need to fix the interface name something consistent for all kind of configuration of the ctrl-data interface.
[Amudha] - ctrl-data interface is a vmxnet3 interface and comes up as ens*. In case of PCI and SR-IOV interfaces,
they are ixgbe interfaces and will come up as eth* interfaces.

I have a script to give ip to the ctrl-data interface.If the the name  is not same for all the compute-vms,script will fail.
[Amudha] Can your script check for ens and eth* as well? And may be use the driver type for additional check if needed?..

Also will write a script to create bond .I would assume the interfaces names to be eth20 and eth21 for bond.
[Amudha] Should work with the current vmdk.

The vmwarephysicalinterface should also ideally be same for any kind of configuration for ctrl-data.That was the case for fab.It used to be always ‘eth1’
[Amudha] I am not sure about this.. Would think it to be eth2 in case of ctrl-data interface defined.

We have to get rid of the situation where vmwarephysicalinterface=ens192 for some case and vmwarephysicalinterface=ens224 for some other.
[Amudha] Each scenario creates different interfaces, and this is hidden from user. Would not prefer changing this in 5.0.1 beta. 
We can have a discussion once all are available, and see if it really needs changes.

Regards
Sandip

From: Sandip Dey <email address hidden>
Date: Tuesday, July 10, 2018 at 7:51 PM
To: Amudha R <email address hidden>, Sachchidanand Vaidya <email address hidden>, Ram Yadav <email address hidden>
Cc: Sudheendra Rao <email address hidden>
Subject: Re: R5.0.1 vcenter provisioning

Hi Amudha

Please see inline….

Ram,

What is the behavior if we create a pg in the vm dvs from vcenter ui for which vn is not there in api server?Does the vcenter-plugin delete it since, its not there in the api server?

I wanted to delete the contrail-vm/vm dvs from vcenter and rerun the vcenter provisioning when rest of the contrail components are up and running.I see the dvs pg on vm dvs getting deleted as soon as it was created.
I am not sure if the vcenter plugin deleting it,because there is not explicit log for deletion.I see only this.

Any port group created on that vm dvs is getting deleted.

[2018-07-10 12:50:01,070] [INFO ] [Thread-3:VCenterDB@794] [Found <dpg c4k4u14_dvpg, dvs c4k4_dvs, datacenter c4_datacenter11, vCenter https://10.204.217.246/sdk>.]
[2018-07-10 12:50:06,075] [INFO ] [Thread-3:VCenterNotify@664] [+++++++++++++ Done syncing +++++++++++++++++++++]
[2018-07-10 12:50:06,075] [INFO ] [Thread-3:VCenterNotify@667] [+++++++++++++ Waiting for events +++++++++++++++++++++]
[2018-07-10 12:50:06,075] [INFO ] [Thread-3:VncDB@155] [ Checking if api-server is alive and kicking..]
[2018-07-10 12:50:06,075] [INFO ] [Thread-3:ApiConnectorImpl@217] [>> Request: GET, /projects]
[2018-07-10 12:50:06,078] [INFO ] [Thread-3:ApiConnectorImpl@239] [<< Response Status: HTTP/1.1 200 OK]
[2018-07-10 12:50:06,080] [INFO ] [Thread-3:VncDB@171] [ Api-server alive. Got the pulse..]
[2018-07-10 12:50:06,080] [INFO ] [Thread-3:VCenterNotify@642] [+++++++++++++ Start syncing  +++++++++++++++++++++]
[2018-07-10 12:50:06,080] [INFO ] [Thread-3:VncDB@813] [Start reading virtual networks from the API server ...]
[2018-07-10 12:50:06,080] [INFO ] [Thread-3:ApiConnectorImpl@217] [>> Request: GET, /virtual-networks]
[2018-07-10 12:50:06,084] [INFO ] [Thread-3:ApiConnectorImpl@239] [<< Response Status: HTTP/1.1 200 OK]
[2018-07-10 12:50:06,086] [INFO ] [Thread-3:ApiConnectorImpl@217] [>> Request: GET, /virtual-network/2abc2b5c-8da9-4227-acc8-dd8f949891d2]
[2018-07-10 12:50:06,089] [INFO ] [Thread-3:ApiConnectorImpl@239] [<< Response Status: HTTP/1.1 200 OK]
[2018-07-10 12:50:06,091] [INFO ] [Thread-3:ApiConnectorImpl@217] [>> Request: GET, /virtual-network/007f8dd7-8bc5-42ef-8036-eac1374701c9]
[2018-07-10 12:50:06,094] [INFO ] [Thread-3:ApiConnectorImpl@239] [<< Response Status: HTTP/1.1 200 OK]
[2018-07-10 12:50:06,096] [INFO ] [Thread-3:ApiConnectorImpl@217] [>> Request: GET, /virtual-network/a9ba499e-dd29-4dd9-ac53-9a283b668526]
[2018-07-10 12:50:06,101] [INFO ] [Thread-3:ApiConnectorImpl@239] [<< Response Status: HTTP/1.1 200 OK]
[2018-07-10 12:50:06,103] [INFO ] [Thread-3:ApiConnectorImpl@217] [>> Request: GET, /virtual-network/33a11739-d562-4329-9abe-946d55c1d7b0]
[2018-07-10 12:50:06,107] [INFO ] [Thread-3:ApiConnectorImpl@239] [<< Response Status: HTTP/1.1 200 OK]
[2018-07-10 12:50:06,109] [INFO ] [Thread-3:ApiConnectorImpl@217] [>> Request: GET, /virtual-network/24da76ed-9534-47bb-b396-f4c8d2134af0]
[2018-07-10 12:50:06,112] [INFO ] [Thread-3:ApiConnectorImpl@239] [<< Response Status: HTTP/1.1 200 OK]
[2018-07-10 12:50:06,114] [INFO ] [Thread-3:VncDB@846] [Done reading from the API server, found 2 Virtual Networks]
[2018-07-10 12:50:06,114] [INFO ] [Thread-3:VCenterDB@885] [Start reading virtual networks from vcenter ...]
[2018-07-10 12:50:06,134] [ERROR] [Thread-3:VCenterDB@688] [dvSwitch: c4k4_dvs Datacenter: c4_datacenter11 Private VLAN NOTconfigured]
[2018-07-10 12:50:06,134] [ERROR] [Thread-3:VirtualNetworkInfo@277] [VN <ctest-vCenter-55191957, f90106d9-fc85-3c37-a483-bc78ae8fa5a6>Cannot populate vlan, private vlan not configured on dvSwitch: c4k4_dvs Datacenter: c4_datacenter11]
[2018-07-10 12:50:06,140] [INFO ] [Thread-3:VirtualNetworkInfo@401] [Set ipPoolId to null for VN <ctest-vCenter-55191957, f90106d9-fc85-3c37-a483-bc78ae8fa5a6>]
[2018-07-10 12:50:06,140] [INFO ] [Thread-3:VCenterDB@926] [Read from vcenter VN <ctest-vCenter-55191957, f90106d9-fc85-3c37-a483-bc78ae8fa5a6>, ipPoolId null]
[2018-07-10 12:50:06,144] [ERROR] [Thread-3:VCenterDB@688] [dvSwitch: c4k4_dvs Datacenter: c4_datacenter11 Private VLAN NOTconfigured]
[2018-07-10 12:50:06,144] [ERROR] [Thread-3:VirtualNetworkInfo@277] [VN <c4k4u14_dvpg, 75acac93-6416-38b5-9c24-1b75c0adbea3>Cannot populate vlan, private vlan not configured on dvSwitch: c4k4_dvs Datacenter: c4_datacenter11]

Regards
Sandip
From: Amudha R <email address hidden>
Date: Tuesday, July 10, 2018 at 6:40 PM
To: Sandip Dey <email address hidden>, Sachchidanand Vaidya <email address hidden>, Ram Yadav <email address hidden>
Cc: Sudheendra Rao <email address hidden>
Subject: Re: R5.0.1 vcenter provisioning

Sandip,

For 2. Even with single-interface and multi-interface – the vmware_physical_interface would be different – ens192 for single-intf and ens224 for multi-intf.

We have become aware of this value and worry about it because of the issues we had with 5.0 GA build – with the issues fixed and provisioning populating the
vmware_physical_interface to correct value, this is hidden from user..

+++sandipd: I am more concern explaining things to our guys.When things are fine, no issues.But if something goes wrong, finding the correct depending on the setup will be difficult.
                       In fab , we had consistent interface for vmwarephysical interface

The PCI and SR-IOV interfaces are renamed by a script /firstboot in the ContrailVM during bootup..

+++sandipd:For pcie, the interface rename not happening looks like. 

-Amudha

From: Sandip Dey <email address hidden>
Date: Tuesday, July 10, 2018 at 7:56 AM
To: Amudha R <email address hidden>, Sachchidanand Vaidya <email address hidden>, Ram Yadav <email address hidden>
Cc: Sudheendra Rao <email address hidden>
Subject: Re: R5.0.1 vcenter provisioning

Hi Amudha

My concern is 

1.interfaces are coming up with different names for different contrail_vms
2.Again vmwarephysicalinterface was different (for some ens192 or ens224) in different agents.

Do you think theses are problems?At least #2 is a problem to me.It is really confusing.

Regards
Sandip

From: Amudha R <email address hidden>
Date: Tuesday, July 10, 2018 at 5:12 PM
To: Sandip Dey <email address hidden>, Sachchidanand Vaidya <email address hidden>, Ram Yadav <email address hidden>
Cc: Sudheendra Rao <email address hidden>
Subject: Re: R5.0.1 vcenter provisioning

PCI and SR-IOV interfaces, both should be renamed as eth20, eth21 and so on.
Like we discussed, this was to avoid confusion while finding the vmware_physical_interface.

      vrouter:
        PHYSICAL_INTERFACE: 

PHYSICAL_INTERFACE is optional and CONTROL_NODES can be specified to deduce the physical interface.
And this is populated in instances.yaml after the ContrailVM is up and the interfaces are setup,
and the users would be aware of the interfaces configured on the ContrailVM. 
vmware_physical_interface is populated in contrail-vrouter-agent.conf by provisioning and is from the user.


  bms2:
    provider: bms
    esxi_host: 10.84.24.4
    ip: 10.84.24.26
    roles:
      vrouter:
  bms3:
    provider: bms
    esxi_host: 10.84.24.7
    ip: 10.84.24.235
    roles:
      vrouter:

global_configuration:
  CONTAINER_REGISTRY: opencontrailnightly
  CONTRAIL_VERSION: 5.0-94

contrail_configuration:
  CLOUD_ORCHESTRATOR: vcenter
  CONTROLLER_NODES: 10.84.13.14
  CONTROL_NODES: 192.168.0.241

There were issues when I tried renaming the PCI and SRIOV interfaces to ens* interfaces.
I will again check the script and update here.

Please update once you provision contrail and test functionality.

Thanks,
Amudha

From: Sandip Dey <email address hidden>
Date: Tuesday, July 10, 2018 at 2:53 AM
To: Sachchidanand Vaidya <email address hidden>, Ram Yadav <email address hidden>, Amudha R <email address hidden>
Cc: Sudheendra Rao <email address hidden>
Subject: R5.0.1 vcenter provisioning

Hi Amudha

I am able to bring up the contrail-vm.Yet to provision the contrail.I see these issues

ISSUES
====== 
Problem 1:

1.SRIOV interface names : eth20,eth21
2.single ctrl-data ens192
3.pcie ens160f0,ens160f1

Should we have a consistent naming?

Problem 2
=========
1.pcie enabled vm - ens192 connected to vm-dvs
2.sriov enabled vm - ens192 connected to vm-dvs
3.single ctrl-data interface - ens224 connected to vm-dvs and ens192 connected to ctrl-data-dvs

Again inconsistency. Should not we create consistent interface connecting to vm-dvs.The above behavior will definitely confuse users while making the instance.yml

bms4:
    provider: bms
    esxi_host: 10.204.216.224
    ip: 10.204.216.181
    roles:
      vrouter:
        PHYSICAL_INTERFACE: ens192     >>>> this entry we do for the ctrl-data interface.Now it will be different for single ctrl/pcie /sriov
  bms5:
    provider: bms
    esxi_host: 10.204.216.225
    ip: 10.204.216.182
    roles:
      vrouter:
        PHYSICAL_INTERFACE: ens192

Let me know your thoughts.Will raise bugs accordingly.

Regards
Sandip