10/23/2018 07:31:05 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 07:31:05 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 07:31:05 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 07:31:05 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 07:31:05 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 07:31:05 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 07:31:05 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 07:31:05 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 07:31:05 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 07:31:05 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540279865866222 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.139:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 07:31:05 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 07:31:06 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 07:31:06 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 07:31:06 AM [cvm.controllers] [INFO]: Synchronization complete 10/23/2018 07:31:09 AM [cvm] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/23/2018 07:31:09 AM [cvm] [INFO]: Session Event: TCP Connected 10/23/2018 07:31:09 AM [cvm] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/23/2018 07:31:09 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/23/2018 07:31:09 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ 10.204.217.140:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector connection down >>, ] >> 10/23/2018 07:31:09 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/23/2018 07:31:09 AM [cvm] [INFO]: Processing event[EvTcpConnected] in state[Connect] 10/23/2018 07:31:09 AM [cvm] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 10/23/2018 07:31:09 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 07:31:09 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 07:31:39 AM [cvm] [INFO]: Processing event[EvConnectTimerExpired] in state[ClientInit] 10/23/2018 07:31:39 AM [cvm] [ERROR]: Session Event: TCP Connection Closed 10/23/2018 07:31:39 AM [cvm] [INFO]: Sandesh Client: Event[EvConnectTimerExpired] => State[ClientInit] -> State[Idle] 10/23/2018 07:31:39 AM [cvm] [INFO]: Processing event[EvTcpClose] in state[Idle] 10/23/2018 07:31:39 AM [cvm] [INFO]: Ignore event [EvTcpClose] received for old session 10/23/2018 07:31:39 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 07:31:39 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ 10.204.217.140:8086, ] status = Down description = ClientInit to Idle on EvConnectTimerExpired >>, ] description = Collector connection down >>, ] >> 10/23/2018 07:31:39 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 07:31:39 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 07:31:39 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 9090 start_time = 1540279865866222 collector_name = collector_ip = 10.204.217.140:8086 collector_list = [ 10.204.217.140:8086, 10.204.217.139:8086, 10.204.217.144:8086, ] >> sm_queue_count = 2 max_sm_queue_count = 2 >> 10/23/2018 07:31:39 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 07:31:43 AM [cvm] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/23/2018 07:31:43 AM [cvm] [INFO]: Session Event: TCP Connected 10/23/2018 07:31:43 AM [cvm] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/23/2018 07:31:43 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/23/2018 07:31:43 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ 10.204.217.139:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector connection down >>, ] >> 10/23/2018 07:31:43 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/23/2018 07:31:43 AM [cvm] [INFO]: Processing event[EvTcpConnected] in state[Connect] 10/23/2018 07:31:43 AM [cvm] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 10/23/2018 07:31:43 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 07:31:43 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 07:31:54 AM [cvm] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 10/23/2018 07:31:54 AM [cvm] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 10/23/2018 07:31:54 AM [cvm] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 10/23/2018 07:31:54 AM [cvm] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 10/23/2018 08:25:58 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 08:25:58 AM [cvm.clients] [INFO]: Read from ESXi API vcenter_tiny_vm properties: {u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '50242a1d-212e-3235-01f3-63d057df0c79', u'runtime.powerState': 'poweredOff', u'name': 'vcenter_tiny_vm', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 08:25:59 AM [cvm.database] [INFO]: Could not find VM model with uuid 50242a1d-212e-3235-01f3-63d057df0c79. 10/23/2018 08:25:59 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine 50242a1d-212e-3235-01f3-63d057df0c79 in VNC 10/23/2018 08:25:59 AM [cvm.clients] [INFO]: Virtual Machine 50242a1d-212e-3235-01f3-63d057df0c79 not found in VNC - creating 10/23/2018 08:25:59 AM [cvm.clients] [INFO]: Virtual Machine 50242a1d-212e-3235-01f3-63d057df0c79 created in VNC 10/23/2018 08:25:59 AM [cvm.services] [INFO]: Created VirtualMachineModel(uuid=50242a1d-212e-3235-01f3-63d057df0c79, name=vcenter_tiny_vm, vm_properties={u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '50242a1d-212e-3235-01f3-63d057df0c79', u'runtime.powerState': 'poweredOff', u'name': 'vcenter_tiny_vm', u'summary.runtime.host': 'vim.HostSystem:ha-host'}, interfaces=[]) 10/23/2018 08:25:59 AM [cvm.database] [INFO]: Saved Virtual Machine model for vcenter_tiny_vm 10/23/2018 08:25:59 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 08:25:59 AM [cvm.services] [INFO]: Deleting VirtualMachineModel(uuid=50242a1d-212e-3235-01f3-63d057df0c79, name=vcenter_tiny_vm, vm_properties={u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '50242a1d-212e-3235-01f3-63d057df0c79', u'runtime.powerState': 'poweredOff', u'name': 'vcenter_tiny_vm', u'summary.runtime.host': 'vim.HostSystem:ha-host'}, interfaces=[]) 10/23/2018 08:26:00 AM [cvm.services] [INFO]: VM vcenter_tiny_vm still exists on another host and can't be deleted from VNC 10/23/2018 08:41:24 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 08:41:24 AM [cvm.clients] [INFO]: Read from ESXi API ctest-pt_svm0-88664358 properties: {u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '502438b4-1f89-fcda-d123-55803bf24615', u'runtime.powerState': 'poweredOff', u'name': 'ctest-pt_svm0-88664358', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 08:41:24 AM [cvm.database] [INFO]: Could not find VM model with uuid 502438b4-1f89-fcda-d123-55803bf24615. 10/23/2018 08:41:24 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine 502438b4-1f89-fcda-d123-55803bf24615 in VNC 10/23/2018 08:41:24 AM [cvm.clients] [INFO]: Virtual Machine 502438b4-1f89-fcda-d123-55803bf24615 not found in VNC - creating 10/23/2018 08:41:24 AM [cvm.clients] [INFO]: Virtual Machine 502438b4-1f89-fcda-d123-55803bf24615 created in VNC 10/23/2018 08:41:24 AM [cvm.services] [INFO]: Created VirtualMachineModel(uuid=502438b4-1f89-fcda-d123-55803bf24615, name=ctest-pt_svm0-88664358, vm_properties={u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '502438b4-1f89-fcda-d123-55803bf24615', u'runtime.powerState': 'poweredOff', u'name': 'ctest-pt_svm0-88664358', u'summary.runtime.host': 'vim.HostSystem:ha-host'}, interfaces=[]) 10/23/2018 08:41:24 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-88664358 10/23/2018 08:41:25 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 08:41:25 AM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 08:41:25 AM [cvm.database] [INFO]: Could not find VN model with key dvportgroup-48408. 10/23/2018 08:41:25 AM [cvm.services] [INFO]: Fetching new portgroup for key: dvportgroup-48408 10/23/2018 08:41:25 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48408 name: ctest-mgmt_vCenter-78493728 10/23/2018 08:41:25 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-mgmt_vCenter-78493728 already allowed. 10/23/2018 08:41:25 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-mgmt_vCenter-78493728 10/23/2018 08:41:25 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=b1b1dcab-b03c-4bb6-b4ba-b4c316c957ff, key=dvportgroup-48408, name=ctest-mgmt_vCenter-78493728) 10/23/2018 08:41:25 AM [cvm.database] [INFO]: Could not find VN model with key dvportgroup-48410. 10/23/2018 08:41:25 AM [cvm.services] [INFO]: Fetching new portgroup for key: dvportgroup-48410 10/23/2018 08:41:25 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48410 name: ctest-right_vCenter-98633662 10/23/2018 08:41:25 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-right_vCenter-98633662 already allowed. 10/23/2018 08:41:25 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-right_vCenter-98633662 10/23/2018 08:41:25 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f5ce463-d3bc-49ae-bfcf-a8cebd94b9e4, key=dvportgroup-48410, name=ctest-right_vCenter-98633662) 10/23/2018 08:41:25 AM [cvm.database] [INFO]: Could not find VN model with key dvportgroup-48409. 10/23/2018 08:41:25 AM [cvm.services] [INFO]: Fetching new portgroup for key: dvportgroup-48409 10/23/2018 08:41:26 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48409 name: ctest-left_vCenter-67859151 10/23/2018 08:41:26 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-left_vCenter-67859151 already allowed. 10/23/2018 08:41:26 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-left_vCenter-67859151 10/23/2018 08:41:26 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=27197e14-7e12-467d-9aa8-f4fd0973f008, key=dvportgroup-48409, name=ctest-left_vCenter-67859151) 10/23/2018 08:41:26 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=633b7123-515e-31f3-9791-cb5ced964ebd, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:41:ef, port_key=112160, portgroup_key=dvportgroup-48408, vlan_id=None), ip_address=unset) 10/23/2018 08:41:26 AM [cvm.clients] [INFO]: Reading VLAN ID of port 112160 10/23/2018 08:41:26 AM [cvm.clients] [INFO]: Port: 112160 has no VLAN ID 10/23/2018 08:41:29 AM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 112160 to 1 10/23/2018 08:41:29 AM [cvm.clients] [INFO]: Successfully set VLAN ID: 1 for port: 112160 10/23/2018 08:41:29 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 633b7123-515e-31f3-9791-cb5ced964ebd in VNC 10/23/2018 08:41:29 AM [cvm.clients] [INFO]: Virtual Machine Interface 633b7123-515e-31f3-9791-cb5ced964ebd not found in VNC - creating 10/23/2018 08:41:29 AM [cvm.clients] [INFO]: Virtual Machine Interface 633b7123-515e-31f3-9791-cb5ced964ebd created in VNC 10/23/2018 08:41:29 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-mgmt_vCenter-78493728-ctest-pt_svm0-88664358 10/23/2018 08:41:29 AM [cvm.clients] [INFO]: Created Instance IP: 530dffbe-433b-3cfe-bcb8-56cb1cbaa247 10/23/2018 08:41:29 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-mgmt_vCenter-78493728-ctest-pt_svm0-88664358 10/23/2018 08:41:29 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=633b7123-515e-31f3-9791-cb5ced964ebd, display_name=vmi-ctest-mgmt_vCenter-78493728-ctest-pt_svm0-88664358, vcenter_port=VCenterPort(mac_address=00:50:56:a4:41:ef, port_key=112160, portgroup_key=dvportgroup-48408, vlan_id=1), ip_address=214.164.224.252) 10/23/2018 08:41:29 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=cbfa0400-0942-3f9e-82e0-726ea5ed1474, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:8c:b9, port_key=112542, portgroup_key=dvportgroup-48410, vlan_id=None), ip_address=unset) 10/23/2018 08:41:29 AM [cvm.clients] [INFO]: Reading VLAN ID of port 112542 10/23/2018 08:41:29 AM [cvm.clients] [INFO]: Port: 112542 has no VLAN ID 10/23/2018 08:41:32 AM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 112542 to 2 10/23/2018 08:41:32 AM [cvm.clients] [INFO]: Successfully set VLAN ID: 2 for port: 112542 10/23/2018 08:41:32 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface cbfa0400-0942-3f9e-82e0-726ea5ed1474 in VNC 10/23/2018 08:41:32 AM [cvm.clients] [INFO]: Virtual Machine Interface cbfa0400-0942-3f9e-82e0-726ea5ed1474 not found in VNC - creating 10/23/2018 08:41:32 AM [cvm.clients] [INFO]: Virtual Machine Interface cbfa0400-0942-3f9e-82e0-726ea5ed1474 created in VNC 10/23/2018 08:41:32 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-right_vCenter-98633662-ctest-pt_svm0-88664358 10/23/2018 08:41:32 AM [cvm.clients] [INFO]: Created Instance IP: bb152fbe-bece-3291-8682-7e7105ddeb90 10/23/2018 08:41:32 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-right_vCenter-98633662-ctest-pt_svm0-88664358 10/23/2018 08:41:32 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=cbfa0400-0942-3f9e-82e0-726ea5ed1474, display_name=vmi-ctest-right_vCenter-98633662-ctest-pt_svm0-88664358, vcenter_port=VCenterPort(mac_address=00:50:56:a4:8c:b9, port_key=112542, portgroup_key=dvportgroup-48410, vlan_id=2), ip_address=96.180.215.251) 10/23/2018 08:41:32 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=fcde84f9-7668-3ea5-88a3-13bfb329ecf9, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:0e:85, port_key=112289, portgroup_key=dvportgroup-48409, vlan_id=None), ip_address=unset) 10/23/2018 08:41:33 AM [cvm.clients] [INFO]: Reading VLAN ID of port 112289 10/23/2018 08:41:33 AM [cvm.clients] [INFO]: Port: 112289 has no VLAN ID 10/23/2018 08:41:36 AM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 112289 to 3 10/23/2018 08:41:36 AM [cvm.clients] [INFO]: Successfully set VLAN ID: 3 for port: 112289 10/23/2018 08:41:36 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface fcde84f9-7668-3ea5-88a3-13bfb329ecf9 in VNC 10/23/2018 08:41:36 AM [cvm.clients] [INFO]: Virtual Machine Interface fcde84f9-7668-3ea5-88a3-13bfb329ecf9 not found in VNC - creating 10/23/2018 08:41:36 AM [cvm.clients] [INFO]: Virtual Machine Interface fcde84f9-7668-3ea5-88a3-13bfb329ecf9 created in VNC 10/23/2018 08:41:36 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-left_vCenter-67859151-ctest-pt_svm0-88664358 10/23/2018 08:41:36 AM [cvm.clients] [INFO]: Created Instance IP: 65e66756-4d1b-33b4-86a8-a93c93c2b591 10/23/2018 08:41:36 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-left_vCenter-67859151-ctest-pt_svm0-88664358 10/23/2018 08:41:36 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=fcde84f9-7668-3ea5-88a3-13bfb329ecf9, display_name=vmi-ctest-left_vCenter-67859151-ctest-pt_svm0-88664358, vcenter_port=VCenterPort(mac_address=00:50:56:a4:0e:85, port_key=112289, portgroup_key=dvportgroup-48409, vlan_id=3), ip_address=190.124.167.251) 10/23/2018 08:41:36 AM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: 633b7123-515e-31f3-9791-cb5ced964ebd 10/23/2018 08:41:36 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 633b7123-515e-31f3-9791-cb5ced964ebd 10/23/2018 08:41:36 AM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': '633b7123-515e-31f3-9791-cb5ced964ebd', 'display_name': 'ctest-pt_svm0-88664358', 'rx_vlan': 1, 'vn_id': 'b1b1dcab-b03c-4bb6-b4ba-b4c316c957ff', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 1, 'port_type': 2, 'mac_address': '00:50:56:a4:41:ef', 'interface_name': '633b7123-515e-31f3-9791-cb5ced964ebd', 'ip_address': u'214.164.224.252', 'vm_uuid_str': '502438b4-1f89-fcda-d123-55803bf24615'} 10/23/2018 08:41:36 AM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: cbfa0400-0942-3f9e-82e0-726ea5ed1474 10/23/2018 08:41:36 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: cbfa0400-0942-3f9e-82e0-726ea5ed1474 10/23/2018 08:41:37 AM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': 'cbfa0400-0942-3f9e-82e0-726ea5ed1474', 'display_name': 'ctest-pt_svm0-88664358', 'rx_vlan': 2, 'vn_id': '9f5ce463-d3bc-49ae-bfcf-a8cebd94b9e4', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 2, 'port_type': 2, 'mac_address': '00:50:56:a4:8c:b9', 'interface_name': 'cbfa0400-0942-3f9e-82e0-726ea5ed1474', 'ip_address': u'96.180.215.251', 'vm_uuid_str': '502438b4-1f89-fcda-d123-55803bf24615'} 10/23/2018 08:41:37 AM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: fcde84f9-7668-3ea5-88a3-13bfb329ecf9 10/23/2018 08:41:37 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: fcde84f9-7668-3ea5-88a3-13bfb329ecf9 10/23/2018 08:41:37 AM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': 'fcde84f9-7668-3ea5-88a3-13bfb329ecf9', 'display_name': 'ctest-pt_svm0-88664358', 'rx_vlan': 3, 'vn_id': '27197e14-7e12-467d-9aa8-f4fd0973f008', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 3, 'port_type': 2, 'mac_address': '00:50:56:a4:0e:85', 'interface_name': 'fcde84f9-7668-3ea5-88a3-13bfb329ecf9', 'ip_address': u'190.124.167.251', 'vm_uuid_str': '502438b4-1f89-fcda-d123-55803bf24615'} 10/23/2018 08:41:37 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 633b7123-515e-31f3-9791-cb5ced964ebd 10/23/2018 08:41:37 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: cbfa0400-0942-3f9e-82e0-726ea5ed1474 10/23/2018 08:41:37 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: fcde84f9-7668-3ea5-88a3-13bfb329ecf9 10/23/2018 08:41:37 AM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 08:41:37 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=633b7123-515e-31f3-9791-cb5ced964ebd, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:41:ef, port_key=112160, portgroup_key=dvportgroup-48408, vlan_id=None), ip_address=unset) 10/23/2018 08:41:38 AM [cvm.clients] [INFO]: Reading VLAN ID of port 112160 10/23/2018 08:41:38 AM [cvm.clients] [INFO]: Port: 112160 VLAN ID: 1 10/23/2018 08:41:38 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 633b7123-515e-31f3-9791-cb5ced964ebd in VNC 10/23/2018 08:41:38 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface 633b7123-515e-31f3-9791-cb5ced964ebd 10/23/2018 08:41:38 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface 633b7123-515e-31f3-9791-cb5ced964ebd to ctest-mgmt_vCenter-78493728 10/23/2018 08:41:38 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-mgmt_vCenter-78493728-ctest-pt_svm0-88664358 10/23/2018 08:41:38 AM [cvm.clients] [INFO]: Created Instance IP: 530dffbe-433b-3cfe-bcb8-56cb1cbaa247 10/23/2018 08:41:38 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-mgmt_vCenter-78493728-ctest-pt_svm0-88664358 10/23/2018 08:41:38 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=633b7123-515e-31f3-9791-cb5ced964ebd, display_name=vmi-ctest-mgmt_vCenter-78493728-ctest-pt_svm0-88664358, vcenter_port=VCenterPort(mac_address=00:50:56:a4:41:ef, port_key=112160, portgroup_key=dvportgroup-48408, vlan_id=1), ip_address=214.164.224.252) 10/23/2018 08:41:38 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=cbfa0400-0942-3f9e-82e0-726ea5ed1474, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:8c:b9, port_key=112542, portgroup_key=dvportgroup-48410, vlan_id=None), ip_address=unset) 10/23/2018 08:41:38 AM [cvm.clients] [INFO]: Reading VLAN ID of port 112542 10/23/2018 08:41:38 AM [cvm.clients] [INFO]: Port: 112542 VLAN ID: 2 10/23/2018 08:41:38 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface cbfa0400-0942-3f9e-82e0-726ea5ed1474 in VNC 10/23/2018 08:41:38 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface cbfa0400-0942-3f9e-82e0-726ea5ed1474 10/23/2018 08:41:38 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface cbfa0400-0942-3f9e-82e0-726ea5ed1474 to ctest-right_vCenter-98633662 10/23/2018 08:41:38 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-right_vCenter-98633662-ctest-pt_svm0-88664358 10/23/2018 08:41:38 AM [cvm.clients] [INFO]: Created Instance IP: bb152fbe-bece-3291-8682-7e7105ddeb90 10/23/2018 08:41:38 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-right_vCenter-98633662-ctest-pt_svm0-88664358 10/23/2018 08:41:38 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=cbfa0400-0942-3f9e-82e0-726ea5ed1474, display_name=vmi-ctest-right_vCenter-98633662-ctest-pt_svm0-88664358, vcenter_port=VCenterPort(mac_address=00:50:56:a4:8c:b9, port_key=112542, portgroup_key=dvportgroup-48410, vlan_id=2), ip_address=96.180.215.251) 10/23/2018 08:41:38 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=fcde84f9-7668-3ea5-88a3-13bfb329ecf9, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:0e:85, port_key=112289, portgroup_key=dvportgroup-48409, vlan_id=None), ip_address=unset) 10/23/2018 08:41:38 AM [cvm.clients] [INFO]: Reading VLAN ID of port 112289 10/23/2018 08:41:38 AM [cvm.clients] [INFO]: Port: 112289 VLAN ID: 3 10/23/2018 08:41:39 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface fcde84f9-7668-3ea5-88a3-13bfb329ecf9 in VNC 10/23/2018 08:41:39 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface fcde84f9-7668-3ea5-88a3-13bfb329ecf9 10/23/2018 08:41:39 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface fcde84f9-7668-3ea5-88a3-13bfb329ecf9 to ctest-left_vCenter-67859151 10/23/2018 08:41:39 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-left_vCenter-67859151-ctest-pt_svm0-88664358 10/23/2018 08:41:39 AM [cvm.clients] [INFO]: Created Instance IP: 65e66756-4d1b-33b4-86a8-a93c93c2b591 10/23/2018 08:41:39 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-left_vCenter-67859151-ctest-pt_svm0-88664358 10/23/2018 08:41:39 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=fcde84f9-7668-3ea5-88a3-13bfb329ecf9, display_name=vmi-ctest-left_vCenter-67859151-ctest-pt_svm0-88664358, vcenter_port=VCenterPort(mac_address=00:50:56:a4:0e:85, port_key=112289, portgroup_key=dvportgroup-48409, vlan_id=3), ip_address=190.124.167.251) 10/23/2018 08:41:39 AM [cvm.clients] [INFO]: Read vRouter port with uuid: 633b7123-515e-31f3-9791-cb5ced964ebd, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427856:41:39.240547', u'dns-server': u'214.164.224.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:41:ef', u'rx-vlan-id': 1, u'tx-vlan-id': 1, u'gateway': u'214.164.224.254', u'system-name': u'633b7123-515e-31f3-9791-cb5ced964ebd', u'ip-address': u'214.164.224.252', u'vn-id': u'b1b1dcab-b03c-4bb6-b4ba-b4c316c957ff', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'633b7123-515e-31f3-9791-cb5ced964ebd', u'instance-id': u'502438b4-1f89-fcda-d123-55803bf24615'} 10/23/2018 08:41:39 AM [cvm.clients] [INFO]: Read vRouter port with uuid: cbfa0400-0942-3f9e-82e0-726ea5ed1474, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427856:41:39.245240', u'dns-server': u'96.180.215.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:8c:b9', u'rx-vlan-id': 2, u'tx-vlan-id': 2, u'gateway': u'96.180.215.254', u'system-name': u'cbfa0400-0942-3f9e-82e0-726ea5ed1474', u'ip-address': u'96.180.215.251', u'vn-id': u'9f5ce463-d3bc-49ae-bfcf-a8cebd94b9e4', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'cbfa0400-0942-3f9e-82e0-726ea5ed1474', u'instance-id': u'502438b4-1f89-fcda-d123-55803bf24615'} 10/23/2018 08:41:39 AM [cvm.clients] [INFO]: Read vRouter port with uuid: fcde84f9-7668-3ea5-88a3-13bfb329ecf9, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427856:41:39.249859', u'dns-server': u'190.124.167.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:0e:85', u'rx-vlan-id': 3, u'tx-vlan-id': 3, u'gateway': u'190.124.167.254', u'system-name': u'fcde84f9-7668-3ea5-88a3-13bfb329ecf9', u'ip-address': u'190.124.167.251', u'vn-id': u'27197e14-7e12-467d-9aa8-f4fd0973f008', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'fcde84f9-7668-3ea5-88a3-13bfb329ecf9', u'instance-id': u'502438b4-1f89-fcda-d123-55803bf24615'} 10/23/2018 08:41:39 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 633b7123-515e-31f3-9791-cb5ced964ebd 10/23/2018 08:41:39 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: cbfa0400-0942-3f9e-82e0-726ea5ed1474 10/23/2018 08:41:39 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: fcde84f9-7668-3ea5-88a3-13bfb329ecf9 10/23/2018 08:41:39 AM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 08:41:39 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=633b7123-515e-31f3-9791-cb5ced964ebd, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:41:ef, port_key=112160, portgroup_key=dvportgroup-48408, vlan_id=None), ip_address=unset) 10/23/2018 08:41:39 AM [cvm.clients] [INFO]: Reading VLAN ID of port 112160 10/23/2018 08:41:39 AM [cvm.clients] [INFO]: Port: 112160 VLAN ID: 1 10/23/2018 08:41:39 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 633b7123-515e-31f3-9791-cb5ced964ebd in VNC 10/23/2018 08:41:40 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface 633b7123-515e-31f3-9791-cb5ced964ebd 10/23/2018 08:41:40 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface 633b7123-515e-31f3-9791-cb5ced964ebd to ctest-mgmt_vCenter-78493728 10/23/2018 08:41:40 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-mgmt_vCenter-78493728-ctest-pt_svm0-88664358 10/23/2018 08:41:40 AM [cvm.clients] [INFO]: Created Instance IP: 530dffbe-433b-3cfe-bcb8-56cb1cbaa247 10/23/2018 08:41:40 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-mgmt_vCenter-78493728-ctest-pt_svm0-88664358 10/23/2018 08:41:40 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=633b7123-515e-31f3-9791-cb5ced964ebd, display_name=vmi-ctest-mgmt_vCenter-78493728-ctest-pt_svm0-88664358, vcenter_port=VCenterPort(mac_address=00:50:56:a4:41:ef, port_key=112160, portgroup_key=dvportgroup-48408, vlan_id=1), ip_address=214.164.224.252) 10/23/2018 08:41:40 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=cbfa0400-0942-3f9e-82e0-726ea5ed1474, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:8c:b9, port_key=112542, portgroup_key=dvportgroup-48410, vlan_id=None), ip_address=unset) 10/23/2018 08:41:40 AM [cvm.clients] [INFO]: Reading VLAN ID of port 112542 10/23/2018 08:41:40 AM [cvm.clients] [INFO]: Port: 112542 VLAN ID: 2 10/23/2018 08:41:40 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface cbfa0400-0942-3f9e-82e0-726ea5ed1474 in VNC 10/23/2018 08:41:40 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface cbfa0400-0942-3f9e-82e0-726ea5ed1474 10/23/2018 08:41:40 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface cbfa0400-0942-3f9e-82e0-726ea5ed1474 to ctest-right_vCenter-98633662 10/23/2018 08:41:40 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-right_vCenter-98633662-ctest-pt_svm0-88664358 10/23/2018 08:41:40 AM [cvm.clients] [INFO]: Created Instance IP: bb152fbe-bece-3291-8682-7e7105ddeb90 10/23/2018 08:41:40 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-right_vCenter-98633662-ctest-pt_svm0-88664358 10/23/2018 08:41:40 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=cbfa0400-0942-3f9e-82e0-726ea5ed1474, display_name=vmi-ctest-right_vCenter-98633662-ctest-pt_svm0-88664358, vcenter_port=VCenterPort(mac_address=00:50:56:a4:8c:b9, port_key=112542, portgroup_key=dvportgroup-48410, vlan_id=2), ip_address=96.180.215.251) 10/23/2018 08:41:40 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=fcde84f9-7668-3ea5-88a3-13bfb329ecf9, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:0e:85, port_key=112289, portgroup_key=dvportgroup-48409, vlan_id=None), ip_address=unset) 10/23/2018 08:41:40 AM [cvm.clients] [INFO]: Reading VLAN ID of port 112289 10/23/2018 08:41:40 AM [cvm.clients] [INFO]: Port: 112289 VLAN ID: 3 10/23/2018 08:41:40 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface fcde84f9-7668-3ea5-88a3-13bfb329ecf9 in VNC 10/23/2018 08:41:40 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface fcde84f9-7668-3ea5-88a3-13bfb329ecf9 10/23/2018 08:41:40 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface fcde84f9-7668-3ea5-88a3-13bfb329ecf9 to ctest-left_vCenter-67859151 10/23/2018 08:41:40 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-left_vCenter-67859151-ctest-pt_svm0-88664358 10/23/2018 08:41:41 AM [cvm.clients] [INFO]: Created Instance IP: 65e66756-4d1b-33b4-86a8-a93c93c2b591 10/23/2018 08:41:41 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-left_vCenter-67859151-ctest-pt_svm0-88664358 10/23/2018 08:41:41 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=fcde84f9-7668-3ea5-88a3-13bfb329ecf9, display_name=vmi-ctest-left_vCenter-67859151-ctest-pt_svm0-88664358, vcenter_port=VCenterPort(mac_address=00:50:56:a4:0e:85, port_key=112289, portgroup_key=dvportgroup-48409, vlan_id=3), ip_address=190.124.167.251) 10/23/2018 08:41:41 AM [cvm.clients] [INFO]: Read vRouter port with uuid: 633b7123-515e-31f3-9791-cb5ced964ebd, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427856:41:41.050539', u'dns-server': u'214.164.224.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:41:ef', u'rx-vlan-id': 1, u'tx-vlan-id': 1, u'gateway': u'214.164.224.254', u'system-name': u'633b7123-515e-31f3-9791-cb5ced964ebd', u'ip-address': u'214.164.224.252', u'vn-id': u'b1b1dcab-b03c-4bb6-b4ba-b4c316c957ff', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'633b7123-515e-31f3-9791-cb5ced964ebd', u'instance-id': u'502438b4-1f89-fcda-d123-55803bf24615'} 10/23/2018 08:41:41 AM [cvm.clients] [INFO]: Read vRouter port with uuid: cbfa0400-0942-3f9e-82e0-726ea5ed1474, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427856:41:41.055397', u'dns-server': u'96.180.215.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:8c:b9', u'rx-vlan-id': 2, u'tx-vlan-id': 2, u'gateway': u'96.180.215.254', u'system-name': u'cbfa0400-0942-3f9e-82e0-726ea5ed1474', u'ip-address': u'96.180.215.251', u'vn-id': u'9f5ce463-d3bc-49ae-bfcf-a8cebd94b9e4', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'cbfa0400-0942-3f9e-82e0-726ea5ed1474', u'instance-id': u'502438b4-1f89-fcda-d123-55803bf24615'} 10/23/2018 08:41:41 AM [cvm.clients] [INFO]: Read vRouter port with uuid: fcde84f9-7668-3ea5-88a3-13bfb329ecf9, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427856:41:41.059982', u'dns-server': u'190.124.167.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:0e:85', u'rx-vlan-id': 3, u'tx-vlan-id': 3, u'gateway': u'190.124.167.254', u'system-name': u'fcde84f9-7668-3ea5-88a3-13bfb329ecf9', u'ip-address': u'190.124.167.251', u'vn-id': u'27197e14-7e12-467d-9aa8-f4fd0973f008', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'fcde84f9-7668-3ea5-88a3-13bfb329ecf9', u'instance-id': u'502438b4-1f89-fcda-d123-55803bf24615'} 10/23/2018 08:41:41 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 633b7123-515e-31f3-9791-cb5ced964ebd 10/23/2018 08:41:41 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: cbfa0400-0942-3f9e-82e0-726ea5ed1474 10/23/2018 08:41:41 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: fcde84f9-7668-3ea5-88a3-13bfb329ecf9 10/23/2018 08:41:41 AM [cvm.services] [INFO]: VMware tools on VM ctest-pt_svm0-88664358 are running 10/23/2018 08:41:41 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-88664358 10/23/2018 08:41:41 AM [cvm.services] [INFO]: VM ctest-pt_svm0-88664358 was powered on 10/23/2018 08:41:41 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-88664358 10/23/2018 08:41:41 AM [cvm.clients] [INFO]: Enabled vRouter port with uuid: 633b7123-515e-31f3-9791-cb5ced964ebd 10/23/2018 08:41:41 AM [cvm.clients] [INFO]: Enabled vRouter port with uuid: fcde84f9-7668-3ea5-88a3-13bfb329ecf9 10/23/2018 08:41:42 AM [cvm.clients] [INFO]: Enabled vRouter port with uuid: cbfa0400-0942-3f9e-82e0-726ea5ed1474 10/23/2018 08:45:26 AM [cvm.services] [INFO]: VMware tools on VM ctest-pt_svm0-88664358 are not running 10/23/2018 08:45:26 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-88664358 10/23/2018 08:45:27 AM [cvm.services] [INFO]: VM ctest-pt_svm0-88664358 was powered off 10/23/2018 08:45:27 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-88664358 10/23/2018 08:45:27 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 633b7123-515e-31f3-9791-cb5ced964ebd 10/23/2018 08:45:27 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: fcde84f9-7668-3ea5-88a3-13bfb329ecf9 10/23/2018 08:45:27 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: cbfa0400-0942-3f9e-82e0-726ea5ed1474 10/23/2018 08:45:28 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 08:45:29 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 171, in main gevent.spawn(vmware_monitor.start()), File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 13, in start self._controller.handle_update(update_set) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 31, in handle_update self._update_handler.handle_update(update_set) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 43, in handle_update handler.handle_change(object_update.obj, property_change) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 54, in handle_change self._handle_change(obj, value) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 89, in _handle_change self._handle_event(value) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 216, in _handle_event self._vmi_service.remove_vmis_for_vm_model(vm_name) File "/usr/lib/python2.7/site-packages/cvm/services.py", line 177, in remove_vmis_for_vm_model full_remove = self._vcenter_api_client.can_remove_vm(vm_model.uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 574, in __call__ return self.f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 394, in _InvokeAccessor return self._stub.InvokeAccessor(self, info) File "/usr/lib/python2.7/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor options=self._pcType.RetrieveOptions(maxObjects=1)) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 580, in self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 386, in _InvokeMethod return self._stub.InvokeMethod(self, info, args) File "/usr/lib/python2.7/site-packages/pyVmomi/SoapAdapter.py", line 1370, in InvokeMethod raise obj # pylint: disable-msg=E0702 vmodl.fault.ManagedObjectNotFound: (vmodl.fault.ManagedObjectNotFound) { dynamicType = , dynamicProperty = (vmodl.DynamicProperty) [], msg = "The object 'vim.VirtualMachine:vm-48413' has already been deleted or has not been completely created", faultCause = , faultMessage = (vmodl.LocalizableMessage) [], obj = 'vim.VirtualMachine:vm-48413' } 10/23/2018 08:45:31 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 08:45:31 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 08:45:31 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 08:45:31 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 08:45:31 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 08:45:31 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 08:45:31 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 08:45:31 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 08:45:31 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 08:45:31 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540284331085715 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.140:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 08:45:31 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 08:45:31 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 08:45:31 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 08:45:31 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48408 name: ctest-mgmt_vCenter-78493728 10/23/2018 08:45:31 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-mgmt_vCenter-78493728 already allowed. 10/23/2018 08:45:31 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-mgmt_vCenter-78493728 10/23/2018 08:45:31 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=b1b1dcab-b03c-4bb6-b4ba-b4c316c957ff, key=dvportgroup-48408, name=ctest-mgmt_vCenter-78493728) 10/23/2018 08:45:31 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48409 name: ctest-left_vCenter-67859151 10/23/2018 08:45:31 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-left_vCenter-67859151 already allowed. 10/23/2018 08:45:31 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-left_vCenter-67859151 10/23/2018 08:45:31 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=27197e14-7e12-467d-9aa8-f4fd0973f008, key=dvportgroup-48409, name=ctest-left_vCenter-67859151) 10/23/2018 08:45:31 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48410 name: ctest-right_vCenter-98633662 10/23/2018 08:45:31 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-right_vCenter-98633662 already allowed. 10/23/2018 08:45:31 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-right_vCenter-98633662 10/23/2018 08:45:31 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f5ce463-d3bc-49ae-bfcf-a8cebd94b9e4, key=dvportgroup-48410, name=ctest-right_vCenter-98633662) 10/23/2018 08:45:34 AM [cvm.services] [INFO]: Deleting 2568f07c-e4b6-3bb1-b9bc-50bc99a5b6fb from VNC. 10/23/2018 08:45:34 AM [cvm.clients] [ERROR]: Could not find VMI 2568f07c-e4b6-3bb1-b9bc-50bc99a5b6fb in VNC 10/23/2018 08:45:34 AM [cvm.clients] [ERROR]: Virtual Machine Interface 2568f07c-e4b6-3bb1-b9bc-50bc99a5b6fb not found in VNC. Unable to delete 10/23/2018 08:45:35 AM [cvm.services] [INFO]: Deleting e6959455-bd45-3eb4-afae-85aa2aecdf2e from VNC. 10/23/2018 08:45:35 AM [cvm.clients] [ERROR]: Could not find VMI e6959455-bd45-3eb4-afae-85aa2aecdf2e in VNC 10/23/2018 08:45:35 AM [cvm.clients] [ERROR]: Virtual Machine Interface e6959455-bd45-3eb4-afae-85aa2aecdf2e not found in VNC. Unable to delete 10/23/2018 08:45:35 AM [cvm] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/23/2018 08:45:35 AM [cvm] [INFO]: Session Event: TCP Connected 10/23/2018 08:45:35 AM [cvm] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/23/2018 08:45:35 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/23/2018 08:45:35 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ 10.204.217.139:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector connection down >>, ] >> 10/23/2018 08:45:35 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/23/2018 08:45:35 AM [cvm] [INFO]: Processing event[EvTcpConnected] in state[Connect] 10/23/2018 08:45:35 AM [cvm] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 10/23/2018 08:45:35 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 08:45:35 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 08:45:35 AM [cvm] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 10/23/2018 08:45:35 AM [cvm] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 10/23/2018 08:45:35 AM [cvm] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 10/23/2018 08:45:35 AM [cvm] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 10/23/2018 08:45:35 AM [cvm.services] [INFO]: Deleting 2619b78b-c6a2-3f61-8238-3286c85a49a2 from VNC. 10/23/2018 08:45:35 AM [cvm.clients] [ERROR]: Could not find VMI 2619b78b-c6a2-3f61-8238-3286c85a49a2 in VNC 10/23/2018 08:45:35 AM [cvm.clients] [ERROR]: Virtual Machine Interface 2619b78b-c6a2-3f61-8238-3286c85a49a2 not found in VNC. Unable to delete 10/23/2018 08:45:35 AM [cvm.database] [INFO]: Could not find VM model with uuid 502439ff-e7eb-3203-0595-f798b0fab1de. 10/23/2018 08:45:36 AM [cvm.database] [INFO]: Could not find VM model with uuid 502452f9-0d96-8884-3822-f2925fb51074. 10/23/2018 08:45:37 AM [cvm.database] [INFO]: Could not find VM model with uuid 50242a1d-212e-3235-01f3-63d057df0c79. 10/23/2018 08:45:38 AM [cvm.database] [INFO]: Could not find VM model with uuid 5024ec42-d8a7-8155-21cd-79376e6b43f1. 10/23/2018 08:45:39 AM [cvm.database] [INFO]: Could not find VM model with uuid 50265637-8829-fc50-89db-d80f49f1904a. 10/23/2018 08:45:39 AM [cvm.database] [INFO]: Could not find VM model with uuid 502438b4-1f89-fcda-d123-55803bf24615. 10/23/2018 08:45:40 AM [cvm.services] [INFO]: Deleting 502438b4-1f89-fcda-d123-55803bf24615 from VNC 10/23/2018 08:45:40 AM [cvm.clients] [INFO]: Attempting to delete Virtual Machine 502438b4-1f89-fcda-d123-55803bf24615 from VNC... 10/23/2018 08:45:40 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 25, in sync self._vm_service.delete_unused_vms_in_vnc() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 280, in delete_unused_vms_in_vnc self._vnc_api_client.delete_vm(vnc_vm.uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 320, in delete_vm vm = self.read_vm(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 350, in read_vm return self.vnc_lib.virtual_machine_read(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 670, in _object_read response = self._request_server(OP_GET, uri, query_params) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1085, in _request % (op, url, data, content)) NoIdError: Unknown id: Error: oper 2 url /virtual-machine/502438b4-1f89-fcda-d123-55803bf24615 body {'exclude_back_refs': True, 'exclude_children': True} response Unknown id: 502438b4-1f89-fcda-d123-55803bf24615 10/23/2018 08:45:42 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 08:45:42 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 08:45:42 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 08:45:42 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 08:45:42 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 08:45:42 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 08:45:42 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 08:45:42 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 08:45:42 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 08:45:42 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540284342616423 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.140:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 08:45:42 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 08:45:42 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 08:45:43 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 08:45:43 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48408 name: ctest-mgmt_vCenter-78493728 10/23/2018 08:45:43 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-mgmt_vCenter-78493728 already allowed. 10/23/2018 08:45:43 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-mgmt_vCenter-78493728 10/23/2018 08:45:43 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=b1b1dcab-b03c-4bb6-b4ba-b4c316c957ff, key=dvportgroup-48408, name=ctest-mgmt_vCenter-78493728) 10/23/2018 08:45:43 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48409 name: ctest-left_vCenter-67859151 10/23/2018 08:45:43 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-left_vCenter-67859151 already allowed. 10/23/2018 08:45:43 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-left_vCenter-67859151 10/23/2018 08:45:43 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=27197e14-7e12-467d-9aa8-f4fd0973f008, key=dvportgroup-48409, name=ctest-left_vCenter-67859151) 10/23/2018 08:45:43 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48410 name: ctest-right_vCenter-98633662 10/23/2018 08:45:43 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-right_vCenter-98633662 already allowed. 10/23/2018 08:45:43 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-right_vCenter-98633662 10/23/2018 08:45:43 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f5ce463-d3bc-49ae-bfcf-a8cebd94b9e4, key=dvportgroup-48410, name=ctest-right_vCenter-98633662) 10/23/2018 08:45:45 AM [cvm.database] [INFO]: Could not find VM model with uuid 50265637-8829-fc50-89db-d80f49f1904a. 10/23/2018 08:45:46 AM [cvm.database] [INFO]: Could not find VM model with uuid 502439ff-e7eb-3203-0595-f798b0fab1de. 10/23/2018 08:45:46 AM [cvm] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/23/2018 08:45:46 AM [cvm] [INFO]: Session Event: TCP Connected 10/23/2018 08:45:46 AM [cvm] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/23/2018 08:45:46 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/23/2018 08:45:46 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ 10.204.217.144:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector connection down >>, ] >> 10/23/2018 08:45:46 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/23/2018 08:45:46 AM [cvm] [INFO]: Processing event[EvTcpConnected] in state[Connect] 10/23/2018 08:45:46 AM [cvm] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 10/23/2018 08:45:46 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 08:45:46 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 08:45:46 AM [cvm] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 10/23/2018 08:45:46 AM [cvm] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 10/23/2018 08:45:46 AM [cvm] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 10/23/2018 08:45:46 AM [cvm] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 10/23/2018 08:45:46 AM [cvm.database] [INFO]: Could not find VM model with uuid 502452f9-0d96-8884-3822-f2925fb51074. 10/23/2018 08:45:47 AM [cvm.database] [INFO]: Could not find VM model with uuid 50242a1d-212e-3235-01f3-63d057df0c79. 10/23/2018 08:45:48 AM [cvm.database] [INFO]: Could not find VM model with uuid 5024ec42-d8a7-8155-21cd-79376e6b43f1. 10/23/2018 08:45:49 AM [cvm.services] [INFO]: Deleting 5024ec42-d8a7-8155-21cd-79376e6b43f1 from VNC 10/23/2018 08:45:49 AM [cvm.clients] [INFO]: Attempting to delete Virtual Machine 5024ec42-d8a7-8155-21cd-79376e6b43f1 from VNC... 10/23/2018 08:45:49 AM [cvm.clients] [INFO]: Virtual Machine Interface 515e13ae-12de-36e8-8063-98b60e74c2e3 removed from VNC 10/23/2018 08:45:49 AM [cvm.clients] [INFO]: Virtual Machine 5024ec42-d8a7-8155-21cd-79376e6b43f1 removed from VNC 10/23/2018 08:45:49 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 8ade3b7a-7271-3a15-9365-6a6a9daaf0fd 10/23/2018 08:45:49 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 515e13ae-12de-36e8-8063-98b60e74c2e3 10/23/2018 08:45:49 AM [cvm.controllers] [INFO]: Synchronization complete 10/23/2018 08:53:29 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 08:53:30 AM [cvm.clients] [INFO]: Read from ESXi API ctest-pt_svm0-84512993 properties: {u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '502410a0-9417-45c7-aec3-6387b409e4c5', u'runtime.powerState': 'poweredOff', u'name': 'ctest-pt_svm0-84512993', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 08:53:30 AM [cvm.database] [INFO]: Could not find VM model with uuid 502410a0-9417-45c7-aec3-6387b409e4c5. 10/23/2018 08:53:30 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine 502410a0-9417-45c7-aec3-6387b409e4c5 in VNC 10/23/2018 08:53:30 AM [cvm.clients] [INFO]: Virtual Machine 502410a0-9417-45c7-aec3-6387b409e4c5 not found in VNC - creating 10/23/2018 08:53:30 AM [cvm.clients] [INFO]: Virtual Machine 502410a0-9417-45c7-aec3-6387b409e4c5 created in VNC 10/23/2018 08:53:30 AM [cvm.services] [INFO]: Created VirtualMachineModel(uuid=502410a0-9417-45c7-aec3-6387b409e4c5, name=ctest-pt_svm0-84512993, vm_properties={u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '502410a0-9417-45c7-aec3-6387b409e4c5', u'runtime.powerState': 'poweredOff', u'name': 'ctest-pt_svm0-84512993', u'summary.runtime.host': 'vim.HostSystem:ha-host'}, interfaces=[]) 10/23/2018 08:53:30 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-84512993 10/23/2018 08:53:30 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 08:53:30 AM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 08:53:30 AM [cvm.database] [INFO]: Could not find VN model with key dvportgroup-48417. 10/23/2018 08:53:30 AM [cvm.services] [INFO]: Fetching new portgroup for key: dvportgroup-48417 10/23/2018 08:53:31 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48417 name: ctest-right_vCenter-87273386 10/23/2018 08:53:31 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-right_vCenter-87273386 already allowed. 10/23/2018 08:53:31 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-right_vCenter-87273386 10/23/2018 08:53:31 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=8671e8ee-9c98-473c-b320-80f7de2d4bb0, key=dvportgroup-48417, name=ctest-right_vCenter-87273386) 10/23/2018 08:53:31 AM [cvm.database] [INFO]: Could not find VN model with key dvportgroup-48415. 10/23/2018 08:53:31 AM [cvm.services] [INFO]: Fetching new portgroup for key: dvportgroup-48415 10/23/2018 08:53:31 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48415 name: ctest-mgmt_vCenter-88906235 10/23/2018 08:53:31 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-mgmt_vCenter-88906235 already allowed. 10/23/2018 08:53:31 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-mgmt_vCenter-88906235 10/23/2018 08:53:31 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=e4260a2b-2a9d-450e-993e-e4687314b554, key=dvportgroup-48415, name=ctest-mgmt_vCenter-88906235) 10/23/2018 08:53:31 AM [cvm.database] [INFO]: Could not find VN model with key dvportgroup-48416. 10/23/2018 08:53:31 AM [cvm.services] [INFO]: Fetching new portgroup for key: dvportgroup-48416 10/23/2018 08:53:31 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48416 name: ctest-left_vCenter-66464404 10/23/2018 08:53:31 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-left_vCenter-66464404 already allowed. 10/23/2018 08:53:31 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-left_vCenter-66464404 10/23/2018 08:53:31 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=103f6c38-fdcd-4f50-90e6-8802e56cf625, key=dvportgroup-48416, name=ctest-left_vCenter-66464404) 10/23/2018 08:53:31 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=d5996227-b663-344d-8532-80c4cc0f1d40, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:5d:07, port_key=112926, portgroup_key=dvportgroup-48417, vlan_id=None), ip_address=unset) 10/23/2018 08:53:31 AM [cvm.clients] [INFO]: Reading VLAN ID of port 112926 10/23/2018 08:53:31 AM [cvm.clients] [INFO]: Port: 112926 has no VLAN ID 10/23/2018 08:53:34 AM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 112926 to 1 10/23/2018 08:53:34 AM [cvm.clients] [INFO]: Successfully set VLAN ID: 1 for port: 112926 10/23/2018 08:53:34 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface d5996227-b663-344d-8532-80c4cc0f1d40 in VNC 10/23/2018 08:53:34 AM [cvm.clients] [INFO]: Virtual Machine Interface d5996227-b663-344d-8532-80c4cc0f1d40 not found in VNC - creating 10/23/2018 08:53:34 AM [cvm.clients] [INFO]: Virtual Machine Interface d5996227-b663-344d-8532-80c4cc0f1d40 created in VNC 10/23/2018 08:53:34 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-right_vCenter-87273386-ctest-pt_svm0-84512993 10/23/2018 08:53:34 AM [cvm.clients] [INFO]: Created Instance IP: efc1faac-fbb4-3d28-95bd-bf11724a41e3 10/23/2018 08:53:35 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-right_vCenter-87273386-ctest-pt_svm0-84512993 10/23/2018 08:53:35 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=d5996227-b663-344d-8532-80c4cc0f1d40, display_name=vmi-ctest-right_vCenter-87273386-ctest-pt_svm0-84512993, vcenter_port=VCenterPort(mac_address=00:50:56:a4:5d:07, port_key=112926, portgroup_key=dvportgroup-48417, vlan_id=1), ip_address=4.239.41.251) 10/23/2018 08:53:35 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=544abab5-4537-3a6d-a994-3b15cd6b93d3, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:6b:05, port_key=112671, portgroup_key=dvportgroup-48415, vlan_id=None), ip_address=unset) 10/23/2018 08:53:35 AM [cvm.clients] [INFO]: Reading VLAN ID of port 112671 10/23/2018 08:53:35 AM [cvm.clients] [INFO]: Port: 112671 has no VLAN ID 10/23/2018 08:53:38 AM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 112671 to 2 10/23/2018 08:53:38 AM [cvm.clients] [INFO]: Successfully set VLAN ID: 2 for port: 112671 10/23/2018 08:53:38 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 544abab5-4537-3a6d-a994-3b15cd6b93d3 in VNC 10/23/2018 08:53:38 AM [cvm.clients] [INFO]: Virtual Machine Interface 544abab5-4537-3a6d-a994-3b15cd6b93d3 not found in VNC - creating 10/23/2018 08:53:38 AM [cvm.clients] [INFO]: Virtual Machine Interface 544abab5-4537-3a6d-a994-3b15cd6b93d3 created in VNC 10/23/2018 08:53:38 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-mgmt_vCenter-88906235-ctest-pt_svm0-84512993 10/23/2018 08:53:38 AM [cvm.clients] [INFO]: Created Instance IP: b306a4c7-4872-3ea6-bf77-6f22c6dd3fd6 10/23/2018 08:53:38 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-mgmt_vCenter-88906235-ctest-pt_svm0-84512993 10/23/2018 08:53:38 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=544abab5-4537-3a6d-a994-3b15cd6b93d3, display_name=vmi-ctest-mgmt_vCenter-88906235-ctest-pt_svm0-84512993, vcenter_port=VCenterPort(mac_address=00:50:56:a4:6b:05, port_key=112671, portgroup_key=dvportgroup-48415, vlan_id=2), ip_address=34.175.196.252) 10/23/2018 08:53:38 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=7ef8c679-eae2-323c-9cd2-110d1f857efc, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:cd:58, port_key=112798, portgroup_key=dvportgroup-48416, vlan_id=None), ip_address=unset) 10/23/2018 08:53:38 AM [cvm.clients] [INFO]: Reading VLAN ID of port 112798 10/23/2018 08:53:38 AM [cvm.clients] [INFO]: Port: 112798 has no VLAN ID 10/23/2018 08:53:41 AM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 112798 to 3 10/23/2018 08:53:41 AM [cvm.clients] [INFO]: Successfully set VLAN ID: 3 for port: 112798 10/23/2018 08:53:41 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 7ef8c679-eae2-323c-9cd2-110d1f857efc in VNC 10/23/2018 08:53:41 AM [cvm.clients] [INFO]: Virtual Machine Interface 7ef8c679-eae2-323c-9cd2-110d1f857efc not found in VNC - creating 10/23/2018 08:53:41 AM [cvm.clients] [INFO]: Virtual Machine Interface 7ef8c679-eae2-323c-9cd2-110d1f857efc created in VNC 10/23/2018 08:53:41 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-left_vCenter-66464404-ctest-pt_svm0-84512993 10/23/2018 08:53:41 AM [cvm.clients] [INFO]: Created Instance IP: f4b30bb0-ff4f-3e01-b945-d4122bd84bf8 10/23/2018 08:53:41 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-left_vCenter-66464404-ctest-pt_svm0-84512993 10/23/2018 08:53:41 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=7ef8c679-eae2-323c-9cd2-110d1f857efc, display_name=vmi-ctest-left_vCenter-66464404-ctest-pt_svm0-84512993, vcenter_port=VCenterPort(mac_address=00:50:56:a4:cd:58, port_key=112798, portgroup_key=dvportgroup-48416, vlan_id=3), ip_address=149.191.235.251) 10/23/2018 08:53:41 AM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: d5996227-b663-344d-8532-80c4cc0f1d40 10/23/2018 08:53:42 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: d5996227-b663-344d-8532-80c4cc0f1d40 10/23/2018 08:53:42 AM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': 'd5996227-b663-344d-8532-80c4cc0f1d40', 'display_name': 'ctest-pt_svm0-84512993', 'rx_vlan': 1, 'vn_id': '8671e8ee-9c98-473c-b320-80f7de2d4bb0', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 1, 'port_type': 2, 'mac_address': '00:50:56:a4:5d:07', 'interface_name': 'd5996227-b663-344d-8532-80c4cc0f1d40', 'ip_address': u'4.239.41.251', 'vm_uuid_str': '502410a0-9417-45c7-aec3-6387b409e4c5'} 10/23/2018 08:53:42 AM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: 544abab5-4537-3a6d-a994-3b15cd6b93d3 10/23/2018 08:53:42 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 544abab5-4537-3a6d-a994-3b15cd6b93d3 10/23/2018 08:53:42 AM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': '544abab5-4537-3a6d-a994-3b15cd6b93d3', 'display_name': 'ctest-pt_svm0-84512993', 'rx_vlan': 2, 'vn_id': 'e4260a2b-2a9d-450e-993e-e4687314b554', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 2, 'port_type': 2, 'mac_address': '00:50:56:a4:6b:05', 'interface_name': '544abab5-4537-3a6d-a994-3b15cd6b93d3', 'ip_address': u'34.175.196.252', 'vm_uuid_str': '502410a0-9417-45c7-aec3-6387b409e4c5'} 10/23/2018 08:53:42 AM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: 7ef8c679-eae2-323c-9cd2-110d1f857efc 10/23/2018 08:53:42 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 7ef8c679-eae2-323c-9cd2-110d1f857efc 10/23/2018 08:53:42 AM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': '7ef8c679-eae2-323c-9cd2-110d1f857efc', 'display_name': 'ctest-pt_svm0-84512993', 'rx_vlan': 3, 'vn_id': '103f6c38-fdcd-4f50-90e6-8802e56cf625', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 3, 'port_type': 2, 'mac_address': '00:50:56:a4:cd:58', 'interface_name': '7ef8c679-eae2-323c-9cd2-110d1f857efc', 'ip_address': u'149.191.235.251', 'vm_uuid_str': '502410a0-9417-45c7-aec3-6387b409e4c5'} 10/23/2018 08:53:43 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: d5996227-b663-344d-8532-80c4cc0f1d40 10/23/2018 08:53:43 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 544abab5-4537-3a6d-a994-3b15cd6b93d3 10/23/2018 08:53:43 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 7ef8c679-eae2-323c-9cd2-110d1f857efc 10/23/2018 08:53:43 AM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 08:53:43 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=d5996227-b663-344d-8532-80c4cc0f1d40, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:5d:07, port_key=112926, portgroup_key=dvportgroup-48417, vlan_id=None), ip_address=unset) 10/23/2018 08:53:43 AM [cvm.clients] [INFO]: Reading VLAN ID of port 112926 10/23/2018 08:53:43 AM [cvm.clients] [INFO]: Port: 112926 VLAN ID: 1 10/23/2018 08:53:43 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface d5996227-b663-344d-8532-80c4cc0f1d40 in VNC 10/23/2018 08:53:43 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface d5996227-b663-344d-8532-80c4cc0f1d40 10/23/2018 08:53:43 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface d5996227-b663-344d-8532-80c4cc0f1d40 to ctest-right_vCenter-87273386 10/23/2018 08:53:43 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-right_vCenter-87273386-ctest-pt_svm0-84512993 10/23/2018 08:53:43 AM [cvm.clients] [INFO]: Created Instance IP: efc1faac-fbb4-3d28-95bd-bf11724a41e3 10/23/2018 08:53:43 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-right_vCenter-87273386-ctest-pt_svm0-84512993 10/23/2018 08:53:43 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=d5996227-b663-344d-8532-80c4cc0f1d40, display_name=vmi-ctest-right_vCenter-87273386-ctest-pt_svm0-84512993, vcenter_port=VCenterPort(mac_address=00:50:56:a4:5d:07, port_key=112926, portgroup_key=dvportgroup-48417, vlan_id=1), ip_address=4.239.41.251) 10/23/2018 08:53:43 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=544abab5-4537-3a6d-a994-3b15cd6b93d3, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:6b:05, port_key=112671, portgroup_key=dvportgroup-48415, vlan_id=None), ip_address=unset) 10/23/2018 08:53:44 AM [cvm.clients] [INFO]: Reading VLAN ID of port 112671 10/23/2018 08:53:44 AM [cvm.clients] [INFO]: Port: 112671 VLAN ID: 2 10/23/2018 08:53:44 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 544abab5-4537-3a6d-a994-3b15cd6b93d3 in VNC 10/23/2018 08:53:44 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface 544abab5-4537-3a6d-a994-3b15cd6b93d3 10/23/2018 08:53:44 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface 544abab5-4537-3a6d-a994-3b15cd6b93d3 to ctest-mgmt_vCenter-88906235 10/23/2018 08:53:44 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-mgmt_vCenter-88906235-ctest-pt_svm0-84512993 10/23/2018 08:53:44 AM [cvm.clients] [INFO]: Created Instance IP: b306a4c7-4872-3ea6-bf77-6f22c6dd3fd6 10/23/2018 08:53:44 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-mgmt_vCenter-88906235-ctest-pt_svm0-84512993 10/23/2018 08:53:44 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=544abab5-4537-3a6d-a994-3b15cd6b93d3, display_name=vmi-ctest-mgmt_vCenter-88906235-ctest-pt_svm0-84512993, vcenter_port=VCenterPort(mac_address=00:50:56:a4:6b:05, port_key=112671, portgroup_key=dvportgroup-48415, vlan_id=2), ip_address=34.175.196.252) 10/23/2018 08:53:44 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=7ef8c679-eae2-323c-9cd2-110d1f857efc, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:cd:58, port_key=112798, portgroup_key=dvportgroup-48416, vlan_id=None), ip_address=unset) 10/23/2018 08:53:44 AM [cvm.clients] [INFO]: Reading VLAN ID of port 112798 10/23/2018 08:53:44 AM [cvm.clients] [INFO]: Port: 112798 VLAN ID: 3 10/23/2018 08:53:44 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 7ef8c679-eae2-323c-9cd2-110d1f857efc in VNC 10/23/2018 08:53:44 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface 7ef8c679-eae2-323c-9cd2-110d1f857efc 10/23/2018 08:53:44 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface 7ef8c679-eae2-323c-9cd2-110d1f857efc to ctest-left_vCenter-66464404 10/23/2018 08:53:44 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-left_vCenter-66464404-ctest-pt_svm0-84512993 10/23/2018 08:53:44 AM [cvm.clients] [INFO]: Created Instance IP: f4b30bb0-ff4f-3e01-b945-d4122bd84bf8 10/23/2018 08:53:44 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-left_vCenter-66464404-ctest-pt_svm0-84512993 10/23/2018 08:53:44 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=7ef8c679-eae2-323c-9cd2-110d1f857efc, display_name=vmi-ctest-left_vCenter-66464404-ctest-pt_svm0-84512993, vcenter_port=VCenterPort(mac_address=00:50:56:a4:cd:58, port_key=112798, portgroup_key=dvportgroup-48416, vlan_id=3), ip_address=149.191.235.251) 10/23/2018 08:53:44 AM [cvm.clients] [INFO]: Read vRouter port with uuid: d5996227-b663-344d-8532-80c4cc0f1d40, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427856:53:44.668397', u'dns-server': u'4.239.41.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:5d:07', u'rx-vlan-id': 1, u'tx-vlan-id': 1, u'gateway': u'4.239.41.254', u'system-name': u'd5996227-b663-344d-8532-80c4cc0f1d40', u'ip-address': u'4.239.41.251', u'vn-id': u'8671e8ee-9c98-473c-b320-80f7de2d4bb0', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'd5996227-b663-344d-8532-80c4cc0f1d40', u'instance-id': u'502410a0-9417-45c7-aec3-6387b409e4c5'} 10/23/2018 08:53:44 AM [cvm.clients] [INFO]: Read vRouter port with uuid: 544abab5-4537-3a6d-a994-3b15cd6b93d3, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427856:53:44.673023', u'dns-server': u'34.175.196.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:6b:05', u'rx-vlan-id': 2, u'tx-vlan-id': 2, u'gateway': u'34.175.196.254', u'system-name': u'544abab5-4537-3a6d-a994-3b15cd6b93d3', u'ip-address': u'34.175.196.252', u'vn-id': u'e4260a2b-2a9d-450e-993e-e4687314b554', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'544abab5-4537-3a6d-a994-3b15cd6b93d3', u'instance-id': u'502410a0-9417-45c7-aec3-6387b409e4c5'} 10/23/2018 08:53:44 AM [cvm.clients] [INFO]: Read vRouter port with uuid: 7ef8c679-eae2-323c-9cd2-110d1f857efc, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427856:53:44.677601', u'dns-server': u'149.191.235.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:cd:58', u'rx-vlan-id': 3, u'tx-vlan-id': 3, u'gateway': u'149.191.235.254', u'system-name': u'7ef8c679-eae2-323c-9cd2-110d1f857efc', u'ip-address': u'149.191.235.251', u'vn-id': u'103f6c38-fdcd-4f50-90e6-8802e56cf625', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'7ef8c679-eae2-323c-9cd2-110d1f857efc', u'instance-id': u'502410a0-9417-45c7-aec3-6387b409e4c5'} 10/23/2018 08:53:44 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: d5996227-b663-344d-8532-80c4cc0f1d40 10/23/2018 08:53:44 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 544abab5-4537-3a6d-a994-3b15cd6b93d3 10/23/2018 08:53:45 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 7ef8c679-eae2-323c-9cd2-110d1f857efc 10/23/2018 08:53:45 AM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 08:53:45 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=d5996227-b663-344d-8532-80c4cc0f1d40, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:5d:07, port_key=112926, portgroup_key=dvportgroup-48417, vlan_id=None), ip_address=unset) 10/23/2018 08:53:45 AM [cvm.clients] [INFO]: Reading VLAN ID of port 112926 10/23/2018 08:53:45 AM [cvm.clients] [INFO]: Port: 112926 VLAN ID: 1 10/23/2018 08:53:45 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface d5996227-b663-344d-8532-80c4cc0f1d40 in VNC 10/23/2018 08:53:45 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface d5996227-b663-344d-8532-80c4cc0f1d40 10/23/2018 08:53:45 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface d5996227-b663-344d-8532-80c4cc0f1d40 to ctest-right_vCenter-87273386 10/23/2018 08:53:45 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-right_vCenter-87273386-ctest-pt_svm0-84512993 10/23/2018 08:53:45 AM [cvm.clients] [INFO]: Created Instance IP: efc1faac-fbb4-3d28-95bd-bf11724a41e3 10/23/2018 08:53:45 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-right_vCenter-87273386-ctest-pt_svm0-84512993 10/23/2018 08:53:45 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=d5996227-b663-344d-8532-80c4cc0f1d40, display_name=vmi-ctest-right_vCenter-87273386-ctest-pt_svm0-84512993, vcenter_port=VCenterPort(mac_address=00:50:56:a4:5d:07, port_key=112926, portgroup_key=dvportgroup-48417, vlan_id=1), ip_address=4.239.41.251) 10/23/2018 08:53:45 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=544abab5-4537-3a6d-a994-3b15cd6b93d3, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:6b:05, port_key=112671, portgroup_key=dvportgroup-48415, vlan_id=None), ip_address=unset) 10/23/2018 08:53:45 AM [cvm.clients] [INFO]: Reading VLAN ID of port 112671 10/23/2018 08:53:45 AM [cvm.clients] [INFO]: Port: 112671 VLAN ID: 2 10/23/2018 08:53:45 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 544abab5-4537-3a6d-a994-3b15cd6b93d3 in VNC 10/23/2018 08:53:45 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface 544abab5-4537-3a6d-a994-3b15cd6b93d3 10/23/2018 08:53:45 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface 544abab5-4537-3a6d-a994-3b15cd6b93d3 to ctest-mgmt_vCenter-88906235 10/23/2018 08:53:46 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-mgmt_vCenter-88906235-ctest-pt_svm0-84512993 10/23/2018 08:53:46 AM [cvm.clients] [INFO]: Created Instance IP: b306a4c7-4872-3ea6-bf77-6f22c6dd3fd6 10/23/2018 08:53:46 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-mgmt_vCenter-88906235-ctest-pt_svm0-84512993 10/23/2018 08:53:46 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=544abab5-4537-3a6d-a994-3b15cd6b93d3, display_name=vmi-ctest-mgmt_vCenter-88906235-ctest-pt_svm0-84512993, vcenter_port=VCenterPort(mac_address=00:50:56:a4:6b:05, port_key=112671, portgroup_key=dvportgroup-48415, vlan_id=2), ip_address=34.175.196.252) 10/23/2018 08:53:46 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=7ef8c679-eae2-323c-9cd2-110d1f857efc, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:cd:58, port_key=112798, portgroup_key=dvportgroup-48416, vlan_id=None), ip_address=unset) 10/23/2018 08:53:46 AM [cvm.clients] [INFO]: Reading VLAN ID of port 112798 10/23/2018 08:53:46 AM [cvm.clients] [INFO]: Port: 112798 VLAN ID: 3 10/23/2018 08:53:46 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 7ef8c679-eae2-323c-9cd2-110d1f857efc in VNC 10/23/2018 08:53:46 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface 7ef8c679-eae2-323c-9cd2-110d1f857efc 10/23/2018 08:53:46 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface 7ef8c679-eae2-323c-9cd2-110d1f857efc to ctest-left_vCenter-66464404 10/23/2018 08:53:46 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-left_vCenter-66464404-ctest-pt_svm0-84512993 10/23/2018 08:53:46 AM [cvm.clients] [INFO]: Created Instance IP: f4b30bb0-ff4f-3e01-b945-d4122bd84bf8 10/23/2018 08:53:46 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-left_vCenter-66464404-ctest-pt_svm0-84512993 10/23/2018 08:53:46 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=7ef8c679-eae2-323c-9cd2-110d1f857efc, display_name=vmi-ctest-left_vCenter-66464404-ctest-pt_svm0-84512993, vcenter_port=VCenterPort(mac_address=00:50:56:a4:cd:58, port_key=112798, portgroup_key=dvportgroup-48416, vlan_id=3), ip_address=149.191.235.251) 10/23/2018 08:53:46 AM [cvm.clients] [INFO]: Read vRouter port with uuid: d5996227-b663-344d-8532-80c4cc0f1d40, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427856:53:46.523198', u'dns-server': u'4.239.41.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:5d:07', u'rx-vlan-id': 1, u'tx-vlan-id': 1, u'gateway': u'4.239.41.254', u'system-name': u'd5996227-b663-344d-8532-80c4cc0f1d40', u'ip-address': u'4.239.41.251', u'vn-id': u'8671e8ee-9c98-473c-b320-80f7de2d4bb0', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'd5996227-b663-344d-8532-80c4cc0f1d40', u'instance-id': u'502410a0-9417-45c7-aec3-6387b409e4c5'} 10/23/2018 08:53:46 AM [cvm.clients] [INFO]: Read vRouter port with uuid: 544abab5-4537-3a6d-a994-3b15cd6b93d3, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427856:53:46.529184', u'dns-server': u'34.175.196.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:6b:05', u'rx-vlan-id': 2, u'tx-vlan-id': 2, u'gateway': u'34.175.196.254', u'system-name': u'544abab5-4537-3a6d-a994-3b15cd6b93d3', u'ip-address': u'34.175.196.252', u'vn-id': u'e4260a2b-2a9d-450e-993e-e4687314b554', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'544abab5-4537-3a6d-a994-3b15cd6b93d3', u'instance-id': u'502410a0-9417-45c7-aec3-6387b409e4c5'} 10/23/2018 08:53:46 AM [cvm.clients] [INFO]: Read vRouter port with uuid: 7ef8c679-eae2-323c-9cd2-110d1f857efc, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427856:53:46.533824', u'dns-server': u'149.191.235.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:cd:58', u'rx-vlan-id': 3, u'tx-vlan-id': 3, u'gateway': u'149.191.235.254', u'system-name': u'7ef8c679-eae2-323c-9cd2-110d1f857efc', u'ip-address': u'149.191.235.251', u'vn-id': u'103f6c38-fdcd-4f50-90e6-8802e56cf625', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'7ef8c679-eae2-323c-9cd2-110d1f857efc', u'instance-id': u'502410a0-9417-45c7-aec3-6387b409e4c5'} 10/23/2018 08:53:46 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: d5996227-b663-344d-8532-80c4cc0f1d40 10/23/2018 08:53:46 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 544abab5-4537-3a6d-a994-3b15cd6b93d3 10/23/2018 08:53:47 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 7ef8c679-eae2-323c-9cd2-110d1f857efc 10/23/2018 08:53:47 AM [cvm.services] [INFO]: VMware tools on VM ctest-pt_svm0-84512993 are running 10/23/2018 08:53:47 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-84512993 10/23/2018 08:53:47 AM [cvm.services] [INFO]: VM ctest-pt_svm0-84512993 was powered on 10/23/2018 08:53:47 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-84512993 10/23/2018 08:53:47 AM [cvm.clients] [INFO]: Enabled vRouter port with uuid: 544abab5-4537-3a6d-a994-3b15cd6b93d3 10/23/2018 08:53:47 AM [cvm.clients] [INFO]: Enabled vRouter port with uuid: 7ef8c679-eae2-323c-9cd2-110d1f857efc 10/23/2018 08:53:47 AM [cvm.clients] [INFO]: Enabled vRouter port with uuid: d5996227-b663-344d-8532-80c4cc0f1d40 10/23/2018 08:56:21 AM [cvm.services] [INFO]: VMware tools on VM ctest-pt_svm0-84512993 are not running 10/23/2018 08:56:21 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-84512993 10/23/2018 08:56:21 AM [cvm.services] [INFO]: VM ctest-pt_svm0-84512993 was powered off 10/23/2018 08:56:21 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-84512993 10/23/2018 08:56:21 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 544abab5-4537-3a6d-a994-3b15cd6b93d3 10/23/2018 08:56:21 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 7ef8c679-eae2-323c-9cd2-110d1f857efc 10/23/2018 08:56:21 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: d5996227-b663-344d-8532-80c4cc0f1d40 10/23/2018 08:56:22 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 08:56:23 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 171, in main gevent.spawn(vmware_monitor.start()), File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 13, in start self._controller.handle_update(update_set) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 31, in handle_update self._update_handler.handle_update(update_set) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 43, in handle_update handler.handle_change(object_update.obj, property_change) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 54, in handle_change self._handle_change(obj, value) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 89, in _handle_change self._handle_event(value) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 216, in _handle_event self._vmi_service.remove_vmis_for_vm_model(vm_name) File "/usr/lib/python2.7/site-packages/cvm/services.py", line 177, in remove_vmis_for_vm_model full_remove = self._vcenter_api_client.can_remove_vm(vm_model.uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 574, in __call__ return self.f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 394, in _InvokeAccessor return self._stub.InvokeAccessor(self, info) File "/usr/lib/python2.7/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor options=self._pcType.RetrieveOptions(maxObjects=1)) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 580, in self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 386, in _InvokeMethod return self._stub.InvokeMethod(self, info, args) File "/usr/lib/python2.7/site-packages/pyVmomi/SoapAdapter.py", line 1370, in InvokeMethod raise obj # pylint: disable-msg=E0702 vmodl.fault.ManagedObjectNotFound: (vmodl.fault.ManagedObjectNotFound) { dynamicType = , dynamicProperty = (vmodl.DynamicProperty) [], msg = "The object 'vim.VirtualMachine:vm-48420' has already been deleted or has not been completely created", faultCause = , faultMessage = (vmodl.LocalizableMessage) [], obj = 'vim.VirtualMachine:vm-48420' } 10/23/2018 08:56:25 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 08:56:25 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 08:56:25 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 08:56:25 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 08:56:25 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 08:56:25 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 08:56:25 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 08:56:25 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 08:56:25 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 08:56:25 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540284985008181 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.140:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 08:56:25 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 08:56:25 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 08:56:25 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 08:56:25 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48415 name: ctest-mgmt_vCenter-88906235 10/23/2018 08:56:25 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-mgmt_vCenter-88906235 already allowed. 10/23/2018 08:56:25 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-mgmt_vCenter-88906235 10/23/2018 08:56:25 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=e4260a2b-2a9d-450e-993e-e4687314b554, key=dvportgroup-48415, name=ctest-mgmt_vCenter-88906235) 10/23/2018 08:56:25 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48416 name: ctest-left_vCenter-66464404 10/23/2018 08:56:25 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-left_vCenter-66464404 already allowed. 10/23/2018 08:56:25 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-left_vCenter-66464404 10/23/2018 08:56:25 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=103f6c38-fdcd-4f50-90e6-8802e56cf625, key=dvportgroup-48416, name=ctest-left_vCenter-66464404) 10/23/2018 08:56:25 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48417 name: ctest-right_vCenter-87273386 10/23/2018 08:56:25 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-right_vCenter-87273386 already allowed. 10/23/2018 08:56:25 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-right_vCenter-87273386 10/23/2018 08:56:25 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=8671e8ee-9c98-473c-b320-80f7de2d4bb0, key=dvportgroup-48417, name=ctest-right_vCenter-87273386) 10/23/2018 08:56:28 AM [cvm.services] [INFO]: Deleting d5996227-b663-344d-8532-80c4cc0f1d40 from VNC. 10/23/2018 08:56:28 AM [cvm.clients] [INFO]: Virtual Machine Interface d5996227-b663-344d-8532-80c4cc0f1d40 removed from VNC 10/23/2018 08:56:29 AM [cvm] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/23/2018 08:56:29 AM [cvm] [INFO]: Session Event: TCP Connected 10/23/2018 08:56:29 AM [cvm] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/23/2018 08:56:29 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/23/2018 08:56:29 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ 10.204.217.139:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector connection down >>, ] >> 10/23/2018 08:56:29 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/23/2018 08:56:29 AM [cvm] [INFO]: Processing event[EvTcpConnected] in state[Connect] 10/23/2018 08:56:29 AM [cvm] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 10/23/2018 08:56:29 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 08:56:29 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 08:56:29 AM [cvm] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 10/23/2018 08:56:29 AM [cvm] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 10/23/2018 08:56:29 AM [cvm] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 10/23/2018 08:56:29 AM [cvm] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 10/23/2018 08:56:29 AM [cvm.services] [INFO]: Deleting 544abab5-4537-3a6d-a994-3b15cd6b93d3 from VNC. 10/23/2018 08:56:29 AM [cvm.clients] [ERROR]: Instance IP not found: b306a4c7-4872-3ea6-bf77-6f22c6dd3fd6 10/23/2018 08:56:29 AM [cvm.clients] [INFO]: Virtual Machine Interface 544abab5-4537-3a6d-a994-3b15cd6b93d3 removed from VNC 10/23/2018 08:56:29 AM [cvm.services] [INFO]: Deleting 7ef8c679-eae2-323c-9cd2-110d1f857efc from VNC. 10/23/2018 08:56:30 AM [cvm.clients] [INFO]: Virtual Machine Interface 7ef8c679-eae2-323c-9cd2-110d1f857efc removed from VNC 10/23/2018 08:56:30 AM [cvm.services] [INFO]: Deleting a9ccb92c-4729-35ac-b5f0-82c6cb4b69a3 from VNC. 10/23/2018 08:56:30 AM [cvm.clients] [INFO]: Virtual Machine Interface a9ccb92c-4729-35ac-b5f0-82c6cb4b69a3 removed from VNC 10/23/2018 08:56:31 AM [cvm.services] [INFO]: Deleting 799f2161-67d9-3fee-ad46-77009e617ca6 from VNC. 10/23/2018 08:56:31 AM [cvm.clients] [INFO]: Virtual Machine Interface 799f2161-67d9-3fee-ad46-77009e617ca6 removed from VNC 10/23/2018 08:56:32 AM [cvm.services] [INFO]: Deleting 59b171cd-a3dd-3c30-83e5-b55b0fc3e356 from VNC. 10/23/2018 08:56:32 AM [cvm.clients] [INFO]: Virtual Machine Interface 59b171cd-a3dd-3c30-83e5-b55b0fc3e356 removed from VNC 10/23/2018 08:56:32 AM [cvm.database] [INFO]: Could not find VM model with uuid 502439ff-e7eb-3203-0595-f798b0fab1de. 10/23/2018 08:56:33 AM [cvm.database] [INFO]: Could not find VM model with uuid 5024ffbc-d297-1a97-301c-5f0fd226f6b7. 10/23/2018 08:56:33 AM [cvm.database] [INFO]: Could not find VM model with uuid 50242a1d-212e-3235-01f3-63d057df0c79. 10/23/2018 08:56:34 AM [cvm.database] [INFO]: Could not find VM model with uuid 50265637-8829-fc50-89db-d80f49f1904a. 10/23/2018 08:56:35 AM [cvm.database] [INFO]: Could not find VM model with uuid 502410a0-9417-45c7-aec3-6387b409e4c5. 10/23/2018 08:56:36 AM [cvm.services] [INFO]: Deleting 502410a0-9417-45c7-aec3-6387b409e4c5 from VNC 10/23/2018 08:56:36 AM [cvm.clients] [INFO]: Attempting to delete Virtual Machine 502410a0-9417-45c7-aec3-6387b409e4c5 from VNC... 10/23/2018 08:56:36 AM [cvm.clients] [INFO]: Virtual Machine 502410a0-9417-45c7-aec3-6387b409e4c5 removed from VNC 10/23/2018 08:56:36 AM [cvm.database] [INFO]: Could not find VM model with uuid 502417b2-785f-4f24-7e5e-c63ef025bfe8. 10/23/2018 08:56:37 AM [cvm.database] [INFO]: Could not find VM model with uuid 5024b44b-b722-2e8b-9100-588b1f2af5d9. 10/23/2018 08:56:37 AM [cvm.services] [INFO]: Deleting 5024b44b-b722-2e8b-9100-588b1f2af5d9 from VNC 10/23/2018 08:56:37 AM [cvm.clients] [INFO]: Attempting to delete Virtual Machine 5024b44b-b722-2e8b-9100-588b1f2af5d9 from VNC... 10/23/2018 08:56:37 AM [cvm.clients] [INFO]: Virtual Machine 5024b44b-b722-2e8b-9100-588b1f2af5d9 removed from VNC 10/23/2018 08:56:37 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: f5bd1f17-da86-31eb-b600-7f93df3e1ec2 10/23/2018 08:56:38 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 36dadc64-a045-32b0-b4ed-6856be693f41 10/23/2018 08:56:38 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: d5996227-b663-344d-8532-80c4cc0f1d40 10/23/2018 08:56:38 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 544abab5-4537-3a6d-a994-3b15cd6b93d3 10/23/2018 08:56:38 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 7ef8c679-eae2-323c-9cd2-110d1f857efc 10/23/2018 08:56:38 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: a9ccb92c-4729-35ac-b5f0-82c6cb4b69a3 10/23/2018 08:56:38 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 799f2161-67d9-3fee-ad46-77009e617ca6 10/23/2018 08:56:38 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 59b171cd-a3dd-3c30-83e5-b55b0fc3e356 10/23/2018 08:56:38 AM [cvm.controllers] [INFO]: Synchronization complete 10/23/2018 09:04:18 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 09:04:18 AM [cvm.clients] [INFO]: Read from ESXi API ctest-pt_svm0-41837086 properties: {u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '5024b109-8611-b3bc-f302-d1680ff800c3', u'runtime.powerState': 'poweredOff', u'name': 'ctest-pt_svm0-41837086', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 09:04:18 AM [cvm.database] [INFO]: Could not find VM model with uuid 5024b109-8611-b3bc-f302-d1680ff800c3. 10/23/2018 09:04:18 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine 5024b109-8611-b3bc-f302-d1680ff800c3 in VNC 10/23/2018 09:04:18 AM [cvm.clients] [INFO]: Virtual Machine 5024b109-8611-b3bc-f302-d1680ff800c3 not found in VNC - creating 10/23/2018 09:04:19 AM [cvm.clients] [INFO]: Virtual Machine 5024b109-8611-b3bc-f302-d1680ff800c3 created in VNC 10/23/2018 09:04:19 AM [cvm.services] [INFO]: Created VirtualMachineModel(uuid=5024b109-8611-b3bc-f302-d1680ff800c3, name=ctest-pt_svm0-41837086, vm_properties={u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '5024b109-8611-b3bc-f302-d1680ff800c3', u'runtime.powerState': 'poweredOff', u'name': 'ctest-pt_svm0-41837086', u'summary.runtime.host': 'vim.HostSystem:ha-host'}, interfaces=[]) 10/23/2018 09:04:19 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-41837086 10/23/2018 09:04:19 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 09:04:19 AM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 09:04:19 AM [cvm.database] [INFO]: Could not find VN model with key dvportgroup-48422. 10/23/2018 09:04:19 AM [cvm.services] [INFO]: Fetching new portgroup for key: dvportgroup-48422 10/23/2018 09:04:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48422 name: ctest-mgmt_vCenter-98202655 10/23/2018 09:04:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-mgmt_vCenter-98202655 already allowed. 10/23/2018 09:04:19 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-mgmt_vCenter-98202655 10/23/2018 09:04:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=5bec593d-6eac-4feb-8f21-a68cd8869c69, key=dvportgroup-48422, name=ctest-mgmt_vCenter-98202655) 10/23/2018 09:04:19 AM [cvm.database] [INFO]: Could not find VN model with key dvportgroup-48423. 10/23/2018 09:04:19 AM [cvm.services] [INFO]: Fetching new portgroup for key: dvportgroup-48423 10/23/2018 09:04:20 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48423 name: ctest-left_vCenter-81492784 10/23/2018 09:04:20 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-left_vCenter-81492784 already allowed. 10/23/2018 09:04:20 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-left_vCenter-81492784 10/23/2018 09:04:20 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=825d5d15-825a-4725-be91-85aa6777c99a, key=dvportgroup-48423, name=ctest-left_vCenter-81492784) 10/23/2018 09:04:20 AM [cvm.database] [INFO]: Could not find VN model with key dvportgroup-48424. 10/23/2018 09:04:20 AM [cvm.services] [INFO]: Fetching new portgroup for key: dvportgroup-48424 10/23/2018 09:04:20 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48424 name: ctest-right_vCenter-36892077 10/23/2018 09:04:20 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-right_vCenter-36892077 already allowed. 10/23/2018 09:04:20 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-right_vCenter-36892077 10/23/2018 09:04:20 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=dc3ff0cc-cf6c-4e92-8f67-ae5d79f85ca8, key=dvportgroup-48424, name=ctest-right_vCenter-36892077) 10/23/2018 09:04:20 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=ffa55db6-f45d-3490-b5d5-512eef35b9d5, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:1c:e2, port_key=113055, portgroup_key=dvportgroup-48422, vlan_id=None), ip_address=unset) 10/23/2018 09:04:20 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113055 10/23/2018 09:04:20 AM [cvm.clients] [INFO]: Port: 113055 has no VLAN ID 10/23/2018 09:04:23 AM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 113055 to 1 10/23/2018 09:04:23 AM [cvm.clients] [INFO]: Successfully set VLAN ID: 1 for port: 113055 10/23/2018 09:04:23 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface ffa55db6-f45d-3490-b5d5-512eef35b9d5 in VNC 10/23/2018 09:04:23 AM [cvm.clients] [INFO]: Virtual Machine Interface ffa55db6-f45d-3490-b5d5-512eef35b9d5 not found in VNC - creating 10/23/2018 09:04:23 AM [cvm.clients] [INFO]: Virtual Machine Interface ffa55db6-f45d-3490-b5d5-512eef35b9d5 created in VNC 10/23/2018 09:04:23 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-mgmt_vCenter-98202655-ctest-pt_svm0-41837086 10/23/2018 09:04:23 AM [cvm.clients] [INFO]: Created Instance IP: 2f67f0bc-ce12-3661-a577-2b6c6fc49638 10/23/2018 09:04:23 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-mgmt_vCenter-98202655-ctest-pt_svm0-41837086 10/23/2018 09:04:23 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=ffa55db6-f45d-3490-b5d5-512eef35b9d5, display_name=vmi-ctest-mgmt_vCenter-98202655-ctest-pt_svm0-41837086, vcenter_port=VCenterPort(mac_address=00:50:56:a4:1c:e2, port_key=113055, portgroup_key=dvportgroup-48422, vlan_id=1), ip_address=28.96.129.252) 10/23/2018 09:04:23 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=636c8df6-1398-30d2-9269-2014e3e6fd03, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:43:d9, port_key=113182, portgroup_key=dvportgroup-48423, vlan_id=None), ip_address=unset) 10/23/2018 09:04:24 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113182 10/23/2018 09:04:24 AM [cvm.clients] [INFO]: Port: 113182 has no VLAN ID 10/23/2018 09:04:27 AM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 113182 to 2 10/23/2018 09:04:27 AM [cvm.clients] [INFO]: Successfully set VLAN ID: 2 for port: 113182 10/23/2018 09:04:27 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 636c8df6-1398-30d2-9269-2014e3e6fd03 in VNC 10/23/2018 09:04:27 AM [cvm.clients] [INFO]: Virtual Machine Interface 636c8df6-1398-30d2-9269-2014e3e6fd03 not found in VNC - creating 10/23/2018 09:04:27 AM [cvm.clients] [INFO]: Virtual Machine Interface 636c8df6-1398-30d2-9269-2014e3e6fd03 created in VNC 10/23/2018 09:04:27 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-left_vCenter-81492784-ctest-pt_svm0-41837086 10/23/2018 09:04:27 AM [cvm.clients] [INFO]: Created Instance IP: 29aa5783-c6db-33a0-a375-1a1f36eb921d 10/23/2018 09:04:27 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-left_vCenter-81492784-ctest-pt_svm0-41837086 10/23/2018 09:04:27 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=636c8df6-1398-30d2-9269-2014e3e6fd03, display_name=vmi-ctest-left_vCenter-81492784-ctest-pt_svm0-41837086, vcenter_port=VCenterPort(mac_address=00:50:56:a4:43:d9, port_key=113182, portgroup_key=dvportgroup-48423, vlan_id=2), ip_address=59.206.118.251) 10/23/2018 09:04:27 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:53:c4, port_key=113310, portgroup_key=dvportgroup-48424, vlan_id=None), ip_address=unset) 10/23/2018 09:04:27 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113310 10/23/2018 09:04:27 AM [cvm.clients] [INFO]: Port: 113310 has no VLAN ID 10/23/2018 09:04:30 AM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 113310 to 3 10/23/2018 09:04:30 AM [cvm.clients] [INFO]: Successfully set VLAN ID: 3 for port: 113310 10/23/2018 09:04:30 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3 in VNC 10/23/2018 09:04:30 AM [cvm.clients] [INFO]: Virtual Machine Interface d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3 not found in VNC - creating 10/23/2018 09:04:30 AM [cvm.clients] [INFO]: Virtual Machine Interface d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3 created in VNC 10/23/2018 09:04:30 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-right_vCenter-36892077-ctest-pt_svm0-41837086 10/23/2018 09:04:30 AM [cvm.clients] [INFO]: Created Instance IP: b011dd8a-91e3-3d48-8b84-c4d698843e97 10/23/2018 09:04:30 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-right_vCenter-36892077-ctest-pt_svm0-41837086 10/23/2018 09:04:30 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3, display_name=vmi-ctest-right_vCenter-36892077-ctest-pt_svm0-41837086, vcenter_port=VCenterPort(mac_address=00:50:56:a4:53:c4, port_key=113310, portgroup_key=dvportgroup-48424, vlan_id=3), ip_address=25.142.22.251) 10/23/2018 09:04:30 AM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: ffa55db6-f45d-3490-b5d5-512eef35b9d5 10/23/2018 09:04:30 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: ffa55db6-f45d-3490-b5d5-512eef35b9d5 10/23/2018 09:04:31 AM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': 'ffa55db6-f45d-3490-b5d5-512eef35b9d5', 'display_name': 'ctest-pt_svm0-41837086', 'rx_vlan': 1, 'vn_id': '5bec593d-6eac-4feb-8f21-a68cd8869c69', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 1, 'port_type': 2, 'mac_address': '00:50:56:a4:1c:e2', 'interface_name': 'ffa55db6-f45d-3490-b5d5-512eef35b9d5', 'ip_address': u'28.96.129.252', 'vm_uuid_str': '5024b109-8611-b3bc-f302-d1680ff800c3'} 10/23/2018 09:04:31 AM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: 636c8df6-1398-30d2-9269-2014e3e6fd03 10/23/2018 09:04:31 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 636c8df6-1398-30d2-9269-2014e3e6fd03 10/23/2018 09:04:31 AM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': '636c8df6-1398-30d2-9269-2014e3e6fd03', 'display_name': 'ctest-pt_svm0-41837086', 'rx_vlan': 2, 'vn_id': '825d5d15-825a-4725-be91-85aa6777c99a', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 2, 'port_type': 2, 'mac_address': '00:50:56:a4:43:d9', 'interface_name': '636c8df6-1398-30d2-9269-2014e3e6fd03', 'ip_address': u'59.206.118.251', 'vm_uuid_str': '5024b109-8611-b3bc-f302-d1680ff800c3'} 10/23/2018 09:04:31 AM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3 10/23/2018 09:04:31 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3 10/23/2018 09:04:31 AM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': 'd0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3', 'display_name': 'ctest-pt_svm0-41837086', 'rx_vlan': 3, 'vn_id': 'dc3ff0cc-cf6c-4e92-8f67-ae5d79f85ca8', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 3, 'port_type': 2, 'mac_address': '00:50:56:a4:53:c4', 'interface_name': 'd0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3', 'ip_address': u'25.142.22.251', 'vm_uuid_str': '5024b109-8611-b3bc-f302-d1680ff800c3'} 10/23/2018 09:04:31 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: ffa55db6-f45d-3490-b5d5-512eef35b9d5 10/23/2018 09:04:32 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 636c8df6-1398-30d2-9269-2014e3e6fd03 10/23/2018 09:04:32 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3 10/23/2018 09:04:32 AM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 09:04:32 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=ffa55db6-f45d-3490-b5d5-512eef35b9d5, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:1c:e2, port_key=113055, portgroup_key=dvportgroup-48422, vlan_id=None), ip_address=unset) 10/23/2018 09:04:32 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113055 10/23/2018 09:04:32 AM [cvm.clients] [INFO]: Port: 113055 VLAN ID: 1 10/23/2018 09:04:32 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface ffa55db6-f45d-3490-b5d5-512eef35b9d5 in VNC 10/23/2018 09:04:32 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface ffa55db6-f45d-3490-b5d5-512eef35b9d5 10/23/2018 09:04:32 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface ffa55db6-f45d-3490-b5d5-512eef35b9d5 to ctest-mgmt_vCenter-98202655 10/23/2018 09:04:32 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-mgmt_vCenter-98202655-ctest-pt_svm0-41837086 10/23/2018 09:04:32 AM [cvm.clients] [INFO]: Created Instance IP: 2f67f0bc-ce12-3661-a577-2b6c6fc49638 10/23/2018 09:04:32 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-mgmt_vCenter-98202655-ctest-pt_svm0-41837086 10/23/2018 09:04:32 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=ffa55db6-f45d-3490-b5d5-512eef35b9d5, display_name=vmi-ctest-mgmt_vCenter-98202655-ctest-pt_svm0-41837086, vcenter_port=VCenterPort(mac_address=00:50:56:a4:1c:e2, port_key=113055, portgroup_key=dvportgroup-48422, vlan_id=1), ip_address=28.96.129.252) 10/23/2018 09:04:32 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=636c8df6-1398-30d2-9269-2014e3e6fd03, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:43:d9, port_key=113182, portgroup_key=dvportgroup-48423, vlan_id=None), ip_address=unset) 10/23/2018 09:04:32 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113182 10/23/2018 09:04:32 AM [cvm.clients] [INFO]: Port: 113182 VLAN ID: 2 10/23/2018 09:04:32 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 636c8df6-1398-30d2-9269-2014e3e6fd03 in VNC 10/23/2018 09:04:32 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface 636c8df6-1398-30d2-9269-2014e3e6fd03 10/23/2018 09:04:32 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface 636c8df6-1398-30d2-9269-2014e3e6fd03 to ctest-left_vCenter-81492784 10/23/2018 09:04:33 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-left_vCenter-81492784-ctest-pt_svm0-41837086 10/23/2018 09:04:33 AM [cvm.clients] [INFO]: Created Instance IP: 29aa5783-c6db-33a0-a375-1a1f36eb921d 10/23/2018 09:04:33 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-left_vCenter-81492784-ctest-pt_svm0-41837086 10/23/2018 09:04:33 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=636c8df6-1398-30d2-9269-2014e3e6fd03, display_name=vmi-ctest-left_vCenter-81492784-ctest-pt_svm0-41837086, vcenter_port=VCenterPort(mac_address=00:50:56:a4:43:d9, port_key=113182, portgroup_key=dvportgroup-48423, vlan_id=2), ip_address=59.206.118.251) 10/23/2018 09:04:33 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:53:c4, port_key=113310, portgroup_key=dvportgroup-48424, vlan_id=None), ip_address=unset) 10/23/2018 09:04:33 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113310 10/23/2018 09:04:33 AM [cvm.clients] [INFO]: Port: 113310 VLAN ID: 3 10/23/2018 09:04:33 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3 in VNC 10/23/2018 09:04:33 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3 10/23/2018 09:04:33 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3 to ctest-right_vCenter-36892077 10/23/2018 09:04:33 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-right_vCenter-36892077-ctest-pt_svm0-41837086 10/23/2018 09:04:33 AM [cvm.clients] [INFO]: Created Instance IP: b011dd8a-91e3-3d48-8b84-c4d698843e97 10/23/2018 09:04:33 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-right_vCenter-36892077-ctest-pt_svm0-41837086 10/23/2018 09:04:33 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3, display_name=vmi-ctest-right_vCenter-36892077-ctest-pt_svm0-41837086, vcenter_port=VCenterPort(mac_address=00:50:56:a4:53:c4, port_key=113310, portgroup_key=dvportgroup-48424, vlan_id=3), ip_address=25.142.22.251) 10/23/2018 09:04:33 AM [cvm.clients] [INFO]: Read vRouter port with uuid: ffa55db6-f45d-3490-b5d5-512eef35b9d5, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:04:33.488525', u'dns-server': u'28.96.129.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:1c:e2', u'rx-vlan-id': 1, u'tx-vlan-id': 1, u'gateway': u'28.96.129.254', u'system-name': u'ffa55db6-f45d-3490-b5d5-512eef35b9d5', u'ip-address': u'28.96.129.252', u'vn-id': u'5bec593d-6eac-4feb-8f21-a68cd8869c69', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'ffa55db6-f45d-3490-b5d5-512eef35b9d5', u'instance-id': u'5024b109-8611-b3bc-f302-d1680ff800c3'} 10/23/2018 09:04:33 AM [cvm.clients] [INFO]: Read vRouter port with uuid: 636c8df6-1398-30d2-9269-2014e3e6fd03, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:04:33.493109', u'dns-server': u'59.206.118.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:43:d9', u'rx-vlan-id': 2, u'tx-vlan-id': 2, u'gateway': u'59.206.118.254', u'system-name': u'636c8df6-1398-30d2-9269-2014e3e6fd03', u'ip-address': u'59.206.118.251', u'vn-id': u'825d5d15-825a-4725-be91-85aa6777c99a', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'636c8df6-1398-30d2-9269-2014e3e6fd03', u'instance-id': u'5024b109-8611-b3bc-f302-d1680ff800c3'} 10/23/2018 09:04:33 AM [cvm.clients] [INFO]: Read vRouter port with uuid: d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:04:33.497761', u'dns-server': u'25.142.22.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:53:c4', u'rx-vlan-id': 3, u'tx-vlan-id': 3, u'gateway': u'25.142.22.254', u'system-name': u'd0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3', u'ip-address': u'25.142.22.251', u'vn-id': u'dc3ff0cc-cf6c-4e92-8f67-ae5d79f85ca8', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'd0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3', u'instance-id': u'5024b109-8611-b3bc-f302-d1680ff800c3'} 10/23/2018 09:04:33 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: ffa55db6-f45d-3490-b5d5-512eef35b9d5 10/23/2018 09:04:33 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 636c8df6-1398-30d2-9269-2014e3e6fd03 10/23/2018 09:04:33 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3 10/23/2018 09:04:33 AM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 09:04:34 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=ffa55db6-f45d-3490-b5d5-512eef35b9d5, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:1c:e2, port_key=113055, portgroup_key=dvportgroup-48422, vlan_id=None), ip_address=unset) 10/23/2018 09:04:34 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113055 10/23/2018 09:04:34 AM [cvm.clients] [INFO]: Port: 113055 VLAN ID: 1 10/23/2018 09:04:34 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface ffa55db6-f45d-3490-b5d5-512eef35b9d5 in VNC 10/23/2018 09:04:34 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface ffa55db6-f45d-3490-b5d5-512eef35b9d5 10/23/2018 09:04:34 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface ffa55db6-f45d-3490-b5d5-512eef35b9d5 to ctest-mgmt_vCenter-98202655 10/23/2018 09:04:34 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-mgmt_vCenter-98202655-ctest-pt_svm0-41837086 10/23/2018 09:04:34 AM [cvm.clients] [INFO]: Created Instance IP: 2f67f0bc-ce12-3661-a577-2b6c6fc49638 10/23/2018 09:04:34 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-mgmt_vCenter-98202655-ctest-pt_svm0-41837086 10/23/2018 09:04:34 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=ffa55db6-f45d-3490-b5d5-512eef35b9d5, display_name=vmi-ctest-mgmt_vCenter-98202655-ctest-pt_svm0-41837086, vcenter_port=VCenterPort(mac_address=00:50:56:a4:1c:e2, port_key=113055, portgroup_key=dvportgroup-48422, vlan_id=1), ip_address=28.96.129.252) 10/23/2018 09:04:34 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=636c8df6-1398-30d2-9269-2014e3e6fd03, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:43:d9, port_key=113182, portgroup_key=dvportgroup-48423, vlan_id=None), ip_address=unset) 10/23/2018 09:04:34 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113182 10/23/2018 09:04:34 AM [cvm.clients] [INFO]: Port: 113182 VLAN ID: 2 10/23/2018 09:04:34 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 636c8df6-1398-30d2-9269-2014e3e6fd03 in VNC 10/23/2018 09:04:34 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface 636c8df6-1398-30d2-9269-2014e3e6fd03 10/23/2018 09:04:34 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface 636c8df6-1398-30d2-9269-2014e3e6fd03 to ctest-left_vCenter-81492784 10/23/2018 09:04:34 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-left_vCenter-81492784-ctest-pt_svm0-41837086 10/23/2018 09:04:34 AM [cvm.clients] [INFO]: Created Instance IP: 29aa5783-c6db-33a0-a375-1a1f36eb921d 10/23/2018 09:04:34 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-left_vCenter-81492784-ctest-pt_svm0-41837086 10/23/2018 09:04:34 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=636c8df6-1398-30d2-9269-2014e3e6fd03, display_name=vmi-ctest-left_vCenter-81492784-ctest-pt_svm0-41837086, vcenter_port=VCenterPort(mac_address=00:50:56:a4:43:d9, port_key=113182, portgroup_key=dvportgroup-48423, vlan_id=2), ip_address=59.206.118.251) 10/23/2018 09:04:34 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:53:c4, port_key=113310, portgroup_key=dvportgroup-48424, vlan_id=None), ip_address=unset) 10/23/2018 09:04:35 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113310 10/23/2018 09:04:35 AM [cvm.clients] [INFO]: Port: 113310 VLAN ID: 3 10/23/2018 09:04:35 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3 in VNC 10/23/2018 09:04:35 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3 10/23/2018 09:04:35 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3 to ctest-right_vCenter-36892077 10/23/2018 09:04:35 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-right_vCenter-36892077-ctest-pt_svm0-41837086 10/23/2018 09:04:35 AM [cvm.clients] [INFO]: Created Instance IP: b011dd8a-91e3-3d48-8b84-c4d698843e97 10/23/2018 09:04:35 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-right_vCenter-36892077-ctest-pt_svm0-41837086 10/23/2018 09:04:35 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3, display_name=vmi-ctest-right_vCenter-36892077-ctest-pt_svm0-41837086, vcenter_port=VCenterPort(mac_address=00:50:56:a4:53:c4, port_key=113310, portgroup_key=dvportgroup-48424, vlan_id=3), ip_address=25.142.22.251) 10/23/2018 09:04:35 AM [cvm.clients] [INFO]: Read vRouter port with uuid: ffa55db6-f45d-3490-b5d5-512eef35b9d5, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:04:35.297038', u'dns-server': u'28.96.129.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:1c:e2', u'rx-vlan-id': 1, u'tx-vlan-id': 1, u'gateway': u'28.96.129.254', u'system-name': u'ffa55db6-f45d-3490-b5d5-512eef35b9d5', u'ip-address': u'28.96.129.252', u'vn-id': u'5bec593d-6eac-4feb-8f21-a68cd8869c69', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'ffa55db6-f45d-3490-b5d5-512eef35b9d5', u'instance-id': u'5024b109-8611-b3bc-f302-d1680ff800c3'} 10/23/2018 09:04:35 AM [cvm.clients] [INFO]: Read vRouter port with uuid: 636c8df6-1398-30d2-9269-2014e3e6fd03, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:04:35.301672', u'dns-server': u'59.206.118.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:43:d9', u'rx-vlan-id': 2, u'tx-vlan-id': 2, u'gateway': u'59.206.118.254', u'system-name': u'636c8df6-1398-30d2-9269-2014e3e6fd03', u'ip-address': u'59.206.118.251', u'vn-id': u'825d5d15-825a-4725-be91-85aa6777c99a', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'636c8df6-1398-30d2-9269-2014e3e6fd03', u'instance-id': u'5024b109-8611-b3bc-f302-d1680ff800c3'} 10/23/2018 09:04:35 AM [cvm.clients] [INFO]: Read vRouter port with uuid: d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:04:35.306229', u'dns-server': u'25.142.22.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:53:c4', u'rx-vlan-id': 3, u'tx-vlan-id': 3, u'gateway': u'25.142.22.254', u'system-name': u'd0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3', u'ip-address': u'25.142.22.251', u'vn-id': u'dc3ff0cc-cf6c-4e92-8f67-ae5d79f85ca8', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'd0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3', u'instance-id': u'5024b109-8611-b3bc-f302-d1680ff800c3'} 10/23/2018 09:04:35 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: ffa55db6-f45d-3490-b5d5-512eef35b9d5 10/23/2018 09:04:35 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 636c8df6-1398-30d2-9269-2014e3e6fd03 10/23/2018 09:04:35 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3 10/23/2018 09:04:35 AM [cvm.services] [INFO]: VMware tools on VM ctest-pt_svm0-41837086 are running 10/23/2018 09:04:35 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-41837086 10/23/2018 09:04:35 AM [cvm.services] [INFO]: VM ctest-pt_svm0-41837086 was powered on 10/23/2018 09:04:35 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-41837086 10/23/2018 09:04:36 AM [cvm.clients] [INFO]: Enabled vRouter port with uuid: ffa55db6-f45d-3490-b5d5-512eef35b9d5 10/23/2018 09:04:36 AM [cvm.clients] [INFO]: Enabled vRouter port with uuid: 636c8df6-1398-30d2-9269-2014e3e6fd03 10/23/2018 09:04:36 AM [cvm.clients] [INFO]: Enabled vRouter port with uuid: d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3 10/23/2018 09:07:04 AM [cvm.services] [INFO]: VMware tools on VM ctest-pt_svm0-41837086 are not running 10/23/2018 09:07:04 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-41837086 10/23/2018 09:07:04 AM [cvm.services] [INFO]: VM ctest-pt_svm0-41837086 was powered off 10/23/2018 09:07:04 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-41837086 10/23/2018 09:07:04 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: ffa55db6-f45d-3490-b5d5-512eef35b9d5 10/23/2018 09:07:05 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 636c8df6-1398-30d2-9269-2014e3e6fd03 10/23/2018 09:07:05 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3 10/23/2018 09:07:06 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 09:07:06 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 171, in main gevent.spawn(vmware_monitor.start()), File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 13, in start self._controller.handle_update(update_set) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 31, in handle_update self._update_handler.handle_update(update_set) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 43, in handle_update handler.handle_change(object_update.obj, property_change) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 54, in handle_change self._handle_change(obj, value) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 89, in _handle_change self._handle_event(value) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 216, in _handle_event self._vmi_service.remove_vmis_for_vm_model(vm_name) File "/usr/lib/python2.7/site-packages/cvm/services.py", line 177, in remove_vmis_for_vm_model full_remove = self._vcenter_api_client.can_remove_vm(vm_model.uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 574, in __call__ return self.f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 394, in _InvokeAccessor return self._stub.InvokeAccessor(self, info) File "/usr/lib/python2.7/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor options=self._pcType.RetrieveOptions(maxObjects=1)) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 580, in self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 386, in _InvokeMethod return self._stub.InvokeMethod(self, info, args) File "/usr/lib/python2.7/site-packages/pyVmomi/SoapAdapter.py", line 1370, in InvokeMethod raise obj # pylint: disable-msg=E0702 vmodl.fault.ManagedObjectNotFound: (vmodl.fault.ManagedObjectNotFound) { dynamicType = , dynamicProperty = (vmodl.DynamicProperty) [], msg = "The object 'vim.VirtualMachine:vm-48427' has already been deleted or has not been completely created", faultCause = , faultMessage = (vmodl.LocalizableMessage) [], obj = 'vim.VirtualMachine:vm-48427' } 10/23/2018 09:07:08 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 09:07:08 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 09:07:08 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 09:07:08 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 09:07:08 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 09:07:08 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 09:07:08 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 09:07:08 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 09:07:08 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 09:07:08 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540285628676696 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.139:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 09:07:08 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 09:07:08 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 09:07:09 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 09:07:09 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48422 name: ctest-mgmt_vCenter-98202655 10/23/2018 09:07:09 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-mgmt_vCenter-98202655 already allowed. 10/23/2018 09:07:09 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-mgmt_vCenter-98202655 10/23/2018 09:07:09 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=5bec593d-6eac-4feb-8f21-a68cd8869c69, key=dvportgroup-48422, name=ctest-mgmt_vCenter-98202655) 10/23/2018 09:07:09 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48423 name: ctest-left_vCenter-81492784 10/23/2018 09:07:09 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-left_vCenter-81492784 already allowed. 10/23/2018 09:07:09 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-left_vCenter-81492784 10/23/2018 09:07:09 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=825d5d15-825a-4725-be91-85aa6777c99a, key=dvportgroup-48423, name=ctest-left_vCenter-81492784) 10/23/2018 09:07:09 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48424 name: ctest-right_vCenter-36892077 10/23/2018 09:07:09 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-right_vCenter-36892077 already allowed. 10/23/2018 09:07:09 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-right_vCenter-36892077 10/23/2018 09:07:09 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=dc3ff0cc-cf6c-4e92-8f67-ae5d79f85ca8, key=dvportgroup-48424, name=ctest-right_vCenter-36892077) 10/23/2018 09:07:11 AM [cvm.services] [INFO]: Deleting ffa55db6-f45d-3490-b5d5-512eef35b9d5 from VNC. 10/23/2018 09:07:12 AM [cvm.clients] [INFO]: Virtual Machine Interface ffa55db6-f45d-3490-b5d5-512eef35b9d5 removed from VNC 10/23/2018 09:07:12 AM [cvm] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/23/2018 09:07:12 AM [cvm] [INFO]: Session Event: TCP Connected 10/23/2018 09:07:12 AM [cvm] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/23/2018 09:07:12 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/23/2018 09:07:12 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ 10.204.217.140:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector connection down >>, ] >> 10/23/2018 09:07:12 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/23/2018 09:07:12 AM [cvm] [INFO]: Processing event[EvTcpConnected] in state[Connect] 10/23/2018 09:07:12 AM [cvm] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 10/23/2018 09:07:12 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 09:07:12 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 09:07:12 AM [cvm] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 10/23/2018 09:07:12 AM [cvm] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 10/23/2018 09:07:12 AM [cvm] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 10/23/2018 09:07:12 AM [cvm] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 10/23/2018 09:07:12 AM [cvm.services] [INFO]: Deleting 636c8df6-1398-30d2-9269-2014e3e6fd03 from VNC. 10/23/2018 09:07:12 AM [cvm.clients] [INFO]: Virtual Machine Interface 636c8df6-1398-30d2-9269-2014e3e6fd03 removed from VNC 10/23/2018 09:07:13 AM [cvm.services] [INFO]: Deleting d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3 from VNC. 10/23/2018 09:07:13 AM [cvm.clients] [INFO]: Virtual Machine Interface d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3 removed from VNC 10/23/2018 09:07:13 AM [cvm.database] [INFO]: Could not find VM model with uuid 502439ff-e7eb-3203-0595-f798b0fab1de. 10/23/2018 09:07:14 AM [cvm.database] [INFO]: Could not find VM model with uuid 50242a1d-212e-3235-01f3-63d057df0c79. 10/23/2018 09:07:15 AM [cvm.database] [INFO]: Could not find VM model with uuid 50247ee6-d6ac-2171-62d8-23c7162d42a3. 10/23/2018 09:07:16 AM [cvm.database] [INFO]: Could not find VM model with uuid 50265637-8829-fc50-89db-d80f49f1904a. 10/23/2018 09:07:16 AM [cvm.database] [INFO]: Could not find VM model with uuid 50244201-8627-5d16-d44a-0059f8a0985b. 10/23/2018 09:07:17 AM [cvm.database] [INFO]: Could not find VM model with uuid 5024b109-8611-b3bc-f302-d1680ff800c3. 10/23/2018 09:07:18 AM [cvm.services] [INFO]: Deleting 5024b109-8611-b3bc-f302-d1680ff800c3 from VNC 10/23/2018 09:07:18 AM [cvm.clients] [INFO]: Attempting to delete Virtual Machine 5024b109-8611-b3bc-f302-d1680ff800c3 from VNC... 10/23/2018 09:07:18 AM [cvm.clients] [INFO]: Virtual Machine 5024b109-8611-b3bc-f302-d1680ff800c3 removed from VNC 10/23/2018 09:07:18 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 8603aa6e-bd93-3f72-8be0-df7d0d51c599 10/23/2018 09:07:18 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 5d336238-f25e-3a02-89f4-f8fbf058164f 10/23/2018 09:07:18 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: ffa55db6-f45d-3490-b5d5-512eef35b9d5 10/23/2018 09:07:18 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 636c8df6-1398-30d2-9269-2014e3e6fd03 10/23/2018 09:07:19 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: d0ddc97e-a2b6-3b82-ae0d-4374ef6a25e3 10/23/2018 09:07:19 AM [cvm.controllers] [INFO]: Synchronization complete 10/23/2018 09:15:06 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 09:15:06 AM [cvm.clients] [INFO]: Read from ESXi API ctest-pt_svm0-82433525 properties: {u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '50247860-a938-9d5e-d509-cacdc50bb1af', u'runtime.powerState': 'poweredOff', u'name': 'ctest-pt_svm0-82433525', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 09:15:06 AM [cvm.database] [INFO]: Could not find VM model with uuid 50247860-a938-9d5e-d509-cacdc50bb1af. 10/23/2018 09:15:06 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine 50247860-a938-9d5e-d509-cacdc50bb1af in VNC 10/23/2018 09:15:06 AM [cvm.clients] [INFO]: Virtual Machine 50247860-a938-9d5e-d509-cacdc50bb1af not found in VNC - creating 10/23/2018 09:15:06 AM [cvm.clients] [INFO]: Virtual Machine 50247860-a938-9d5e-d509-cacdc50bb1af created in VNC 10/23/2018 09:15:06 AM [cvm.services] [INFO]: Created VirtualMachineModel(uuid=50247860-a938-9d5e-d509-cacdc50bb1af, name=ctest-pt_svm0-82433525, vm_properties={u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '50247860-a938-9d5e-d509-cacdc50bb1af', u'runtime.powerState': 'poweredOff', u'name': 'ctest-pt_svm0-82433525', u'summary.runtime.host': 'vim.HostSystem:ha-host'}, interfaces=[]) 10/23/2018 09:15:06 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-82433525 10/23/2018 09:15:07 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 09:15:07 AM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 09:15:07 AM [cvm.database] [INFO]: Could not find VN model with key dvportgroup-48428. 10/23/2018 09:15:07 AM [cvm.services] [INFO]: Fetching new portgroup for key: dvportgroup-48428 10/23/2018 09:15:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48428 name: ctest-mgmt_vCenter-29750405 10/23/2018 09:15:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-mgmt_vCenter-29750405 already allowed. 10/23/2018 09:15:07 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-mgmt_vCenter-29750405 10/23/2018 09:15:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=46b87b99-0e93-43de-a7e8-534e86dd24d0, key=dvportgroup-48428, name=ctest-mgmt_vCenter-29750405) 10/23/2018 09:15:07 AM [cvm.database] [INFO]: Could not find VN model with key dvportgroup-48429. 10/23/2018 09:15:07 AM [cvm.services] [INFO]: Fetching new portgroup for key: dvportgroup-48429 10/23/2018 09:15:08 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48429 name: ctest-left_vCenter-02203416 10/23/2018 09:15:08 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-left_vCenter-02203416 already allowed. 10/23/2018 09:15:08 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-left_vCenter-02203416 10/23/2018 09:15:08 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=43bfb4d2-ed84-4970-94a5-c5e4eec5ef20, key=dvportgroup-48429, name=ctest-left_vCenter-02203416) 10/23/2018 09:15:08 AM [cvm.database] [INFO]: Could not find VN model with key dvportgroup-48430. 10/23/2018 09:15:08 AM [cvm.services] [INFO]: Fetching new portgroup for key: dvportgroup-48430 10/23/2018 09:15:08 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48430 name: ctest-right_vCenter-04676770 10/23/2018 09:15:08 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-right_vCenter-04676770 already allowed. 10/23/2018 09:15:08 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-right_vCenter-04676770 10/23/2018 09:15:08 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6451cff1-fb8a-4564-84d7-5b40873b4667, key=dvportgroup-48430, name=ctest-right_vCenter-04676770) 10/23/2018 09:15:08 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=0e0bc3b2-3ac9-3a5a-b9af-02218287d03f, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:02:47, port_key=113439, portgroup_key=dvportgroup-48428, vlan_id=None), ip_address=unset) 10/23/2018 09:15:08 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113439 10/23/2018 09:15:08 AM [cvm.clients] [INFO]: Port: 113439 has no VLAN ID 10/23/2018 09:15:11 AM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 113439 to 1 10/23/2018 09:15:11 AM [cvm.clients] [INFO]: Successfully set VLAN ID: 1 for port: 113439 10/23/2018 09:15:11 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 0e0bc3b2-3ac9-3a5a-b9af-02218287d03f in VNC 10/23/2018 09:15:11 AM [cvm.clients] [INFO]: Virtual Machine Interface 0e0bc3b2-3ac9-3a5a-b9af-02218287d03f not found in VNC - creating 10/23/2018 09:15:11 AM [cvm.clients] [INFO]: Virtual Machine Interface 0e0bc3b2-3ac9-3a5a-b9af-02218287d03f created in VNC 10/23/2018 09:15:11 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-mgmt_vCenter-29750405-ctest-pt_svm0-82433525 10/23/2018 09:15:11 AM [cvm.clients] [INFO]: Created Instance IP: 4ee67ec8-ba07-3f99-a6d6-1443f834b249 10/23/2018 09:15:11 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-mgmt_vCenter-29750405-ctest-pt_svm0-82433525 10/23/2018 09:15:11 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=0e0bc3b2-3ac9-3a5a-b9af-02218287d03f, display_name=vmi-ctest-mgmt_vCenter-29750405-ctest-pt_svm0-82433525, vcenter_port=VCenterPort(mac_address=00:50:56:a4:02:47, port_key=113439, portgroup_key=dvportgroup-48428, vlan_id=1), ip_address=38.253.32.252) 10/23/2018 09:15:11 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=7a9e1013-0bcf-3e83-8c15-10f35b3064a7, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:f2:7b, port_key=113566, portgroup_key=dvportgroup-48429, vlan_id=None), ip_address=unset) 10/23/2018 09:15:12 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113566 10/23/2018 09:15:12 AM [cvm.clients] [INFO]: Port: 113566 has no VLAN ID 10/23/2018 09:15:15 AM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 113566 to 2 10/23/2018 09:15:15 AM [cvm.clients] [INFO]: Successfully set VLAN ID: 2 for port: 113566 10/23/2018 09:15:15 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 7a9e1013-0bcf-3e83-8c15-10f35b3064a7 in VNC 10/23/2018 09:15:15 AM [cvm.clients] [INFO]: Virtual Machine Interface 7a9e1013-0bcf-3e83-8c15-10f35b3064a7 not found in VNC - creating 10/23/2018 09:15:15 AM [cvm.clients] [INFO]: Virtual Machine Interface 7a9e1013-0bcf-3e83-8c15-10f35b3064a7 created in VNC 10/23/2018 09:15:15 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-left_vCenter-02203416-ctest-pt_svm0-82433525 10/23/2018 09:15:15 AM [cvm.clients] [INFO]: Created Instance IP: 7631e252-a6c4-3ef9-8d60-cac85aa65ac1 10/23/2018 09:15:15 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-left_vCenter-02203416-ctest-pt_svm0-82433525 10/23/2018 09:15:15 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=7a9e1013-0bcf-3e83-8c15-10f35b3064a7, display_name=vmi-ctest-left_vCenter-02203416-ctest-pt_svm0-82433525, vcenter_port=VCenterPort(mac_address=00:50:56:a4:f2:7b, port_key=113566, portgroup_key=dvportgroup-48429, vlan_id=2), ip_address=42.100.201.251) 10/23/2018 09:15:15 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=d9efd328-a23e-3a35-94c1-0f584835d577, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:93:a7, port_key=113694, portgroup_key=dvportgroup-48430, vlan_id=None), ip_address=unset) 10/23/2018 09:15:15 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113694 10/23/2018 09:15:15 AM [cvm.clients] [INFO]: Port: 113694 has no VLAN ID 10/23/2018 09:15:18 AM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 113694 to 3 10/23/2018 09:15:18 AM [cvm.clients] [INFO]: Successfully set VLAN ID: 3 for port: 113694 10/23/2018 09:15:18 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface d9efd328-a23e-3a35-94c1-0f584835d577 in VNC 10/23/2018 09:15:18 AM [cvm.clients] [INFO]: Virtual Machine Interface d9efd328-a23e-3a35-94c1-0f584835d577 not found in VNC - creating 10/23/2018 09:15:18 AM [cvm.clients] [INFO]: Virtual Machine Interface d9efd328-a23e-3a35-94c1-0f584835d577 created in VNC 10/23/2018 09:15:18 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-right_vCenter-04676770-ctest-pt_svm0-82433525 10/23/2018 09:15:18 AM [cvm.clients] [INFO]: Created Instance IP: 92e9aae2-74ba-3383-afcc-5a109c813eb5 10/23/2018 09:15:18 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-right_vCenter-04676770-ctest-pt_svm0-82433525 10/23/2018 09:15:18 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=d9efd328-a23e-3a35-94c1-0f584835d577, display_name=vmi-ctest-right_vCenter-04676770-ctest-pt_svm0-82433525, vcenter_port=VCenterPort(mac_address=00:50:56:a4:93:a7, port_key=113694, portgroup_key=dvportgroup-48430, vlan_id=3), ip_address=81.206.159.251) 10/23/2018 09:15:18 AM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: 0e0bc3b2-3ac9-3a5a-b9af-02218287d03f 10/23/2018 09:15:18 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 0e0bc3b2-3ac9-3a5a-b9af-02218287d03f 10/23/2018 09:15:19 AM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': '0e0bc3b2-3ac9-3a5a-b9af-02218287d03f', 'display_name': 'ctest-pt_svm0-82433525', 'rx_vlan': 1, 'vn_id': '46b87b99-0e93-43de-a7e8-534e86dd24d0', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 1, 'port_type': 2, 'mac_address': '00:50:56:a4:02:47', 'interface_name': '0e0bc3b2-3ac9-3a5a-b9af-02218287d03f', 'ip_address': u'38.253.32.252', 'vm_uuid_str': '50247860-a938-9d5e-d509-cacdc50bb1af'} 10/23/2018 09:15:19 AM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: 7a9e1013-0bcf-3e83-8c15-10f35b3064a7 10/23/2018 09:15:19 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 7a9e1013-0bcf-3e83-8c15-10f35b3064a7 10/23/2018 09:15:19 AM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': '7a9e1013-0bcf-3e83-8c15-10f35b3064a7', 'display_name': 'ctest-pt_svm0-82433525', 'rx_vlan': 2, 'vn_id': '43bfb4d2-ed84-4970-94a5-c5e4eec5ef20', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 2, 'port_type': 2, 'mac_address': '00:50:56:a4:f2:7b', 'interface_name': '7a9e1013-0bcf-3e83-8c15-10f35b3064a7', 'ip_address': u'42.100.201.251', 'vm_uuid_str': '50247860-a938-9d5e-d509-cacdc50bb1af'} 10/23/2018 09:15:19 AM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: d9efd328-a23e-3a35-94c1-0f584835d577 10/23/2018 09:15:19 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: d9efd328-a23e-3a35-94c1-0f584835d577 10/23/2018 09:15:19 AM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': 'd9efd328-a23e-3a35-94c1-0f584835d577', 'display_name': 'ctest-pt_svm0-82433525', 'rx_vlan': 3, 'vn_id': '6451cff1-fb8a-4564-84d7-5b40873b4667', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 3, 'port_type': 2, 'mac_address': '00:50:56:a4:93:a7', 'interface_name': 'd9efd328-a23e-3a35-94c1-0f584835d577', 'ip_address': u'81.206.159.251', 'vm_uuid_str': '50247860-a938-9d5e-d509-cacdc50bb1af'} 10/23/2018 09:15:19 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 0e0bc3b2-3ac9-3a5a-b9af-02218287d03f 10/23/2018 09:15:20 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 7a9e1013-0bcf-3e83-8c15-10f35b3064a7 10/23/2018 09:15:20 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: d9efd328-a23e-3a35-94c1-0f584835d577 10/23/2018 09:15:20 AM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 09:15:20 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=0e0bc3b2-3ac9-3a5a-b9af-02218287d03f, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:02:47, port_key=113439, portgroup_key=dvportgroup-48428, vlan_id=None), ip_address=unset) 10/23/2018 09:15:20 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113439 10/23/2018 09:15:20 AM [cvm.clients] [INFO]: Port: 113439 VLAN ID: 1 10/23/2018 09:15:20 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 0e0bc3b2-3ac9-3a5a-b9af-02218287d03f in VNC 10/23/2018 09:15:20 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface 0e0bc3b2-3ac9-3a5a-b9af-02218287d03f 10/23/2018 09:15:20 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface 0e0bc3b2-3ac9-3a5a-b9af-02218287d03f to ctest-mgmt_vCenter-29750405 10/23/2018 09:15:20 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-mgmt_vCenter-29750405-ctest-pt_svm0-82433525 10/23/2018 09:15:20 AM [cvm.clients] [INFO]: Created Instance IP: 4ee67ec8-ba07-3f99-a6d6-1443f834b249 10/23/2018 09:15:20 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-mgmt_vCenter-29750405-ctest-pt_svm0-82433525 10/23/2018 09:15:20 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=0e0bc3b2-3ac9-3a5a-b9af-02218287d03f, display_name=vmi-ctest-mgmt_vCenter-29750405-ctest-pt_svm0-82433525, vcenter_port=VCenterPort(mac_address=00:50:56:a4:02:47, port_key=113439, portgroup_key=dvportgroup-48428, vlan_id=1), ip_address=38.253.32.252) 10/23/2018 09:15:20 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=7a9e1013-0bcf-3e83-8c15-10f35b3064a7, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:f2:7b, port_key=113566, portgroup_key=dvportgroup-48429, vlan_id=None), ip_address=unset) 10/23/2018 09:15:20 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113566 10/23/2018 09:15:20 AM [cvm.clients] [INFO]: Port: 113566 VLAN ID: 2 10/23/2018 09:15:20 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 7a9e1013-0bcf-3e83-8c15-10f35b3064a7 in VNC 10/23/2018 09:15:20 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface 7a9e1013-0bcf-3e83-8c15-10f35b3064a7 10/23/2018 09:15:20 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface 7a9e1013-0bcf-3e83-8c15-10f35b3064a7 to ctest-left_vCenter-02203416 10/23/2018 09:15:21 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-left_vCenter-02203416-ctest-pt_svm0-82433525 10/23/2018 09:15:21 AM [cvm.clients] [INFO]: Created Instance IP: 7631e252-a6c4-3ef9-8d60-cac85aa65ac1 10/23/2018 09:15:21 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-left_vCenter-02203416-ctest-pt_svm0-82433525 10/23/2018 09:15:21 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=7a9e1013-0bcf-3e83-8c15-10f35b3064a7, display_name=vmi-ctest-left_vCenter-02203416-ctest-pt_svm0-82433525, vcenter_port=VCenterPort(mac_address=00:50:56:a4:f2:7b, port_key=113566, portgroup_key=dvportgroup-48429, vlan_id=2), ip_address=42.100.201.251) 10/23/2018 09:15:21 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=d9efd328-a23e-3a35-94c1-0f584835d577, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:93:a7, port_key=113694, portgroup_key=dvportgroup-48430, vlan_id=None), ip_address=unset) 10/23/2018 09:15:21 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113694 10/23/2018 09:15:21 AM [cvm.clients] [INFO]: Port: 113694 VLAN ID: 3 10/23/2018 09:15:21 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface d9efd328-a23e-3a35-94c1-0f584835d577 in VNC 10/23/2018 09:15:21 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface d9efd328-a23e-3a35-94c1-0f584835d577 10/23/2018 09:15:21 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface d9efd328-a23e-3a35-94c1-0f584835d577 to ctest-right_vCenter-04676770 10/23/2018 09:15:21 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-right_vCenter-04676770-ctest-pt_svm0-82433525 10/23/2018 09:15:21 AM [cvm.clients] [INFO]: Created Instance IP: 92e9aae2-74ba-3383-afcc-5a109c813eb5 10/23/2018 09:15:21 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-right_vCenter-04676770-ctest-pt_svm0-82433525 10/23/2018 09:15:21 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=d9efd328-a23e-3a35-94c1-0f584835d577, display_name=vmi-ctest-right_vCenter-04676770-ctest-pt_svm0-82433525, vcenter_port=VCenterPort(mac_address=00:50:56:a4:93:a7, port_key=113694, portgroup_key=dvportgroup-48430, vlan_id=3), ip_address=81.206.159.251) 10/23/2018 09:15:21 AM [cvm.clients] [INFO]: Read vRouter port with uuid: 0e0bc3b2-3ac9-3a5a-b9af-02218287d03f, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:15:21.502478', u'dns-server': u'38.253.32.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:02:47', u'rx-vlan-id': 1, u'tx-vlan-id': 1, u'gateway': u'38.253.32.254', u'system-name': u'0e0bc3b2-3ac9-3a5a-b9af-02218287d03f', u'ip-address': u'38.253.32.252', u'vn-id': u'46b87b99-0e93-43de-a7e8-534e86dd24d0', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'0e0bc3b2-3ac9-3a5a-b9af-02218287d03f', u'instance-id': u'50247860-a938-9d5e-d509-cacdc50bb1af'} 10/23/2018 09:15:21 AM [cvm.clients] [INFO]: Read vRouter port with uuid: 7a9e1013-0bcf-3e83-8c15-10f35b3064a7, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:15:21.507693', u'dns-server': u'42.100.201.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:f2:7b', u'rx-vlan-id': 2, u'tx-vlan-id': 2, u'gateway': u'42.100.201.254', u'system-name': u'7a9e1013-0bcf-3e83-8c15-10f35b3064a7', u'ip-address': u'42.100.201.251', u'vn-id': u'43bfb4d2-ed84-4970-94a5-c5e4eec5ef20', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'7a9e1013-0bcf-3e83-8c15-10f35b3064a7', u'instance-id': u'50247860-a938-9d5e-d509-cacdc50bb1af'} 10/23/2018 09:15:21 AM [cvm.clients] [INFO]: Read vRouter port with uuid: d9efd328-a23e-3a35-94c1-0f584835d577, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:15:21.512321', u'dns-server': u'81.206.159.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:93:a7', u'rx-vlan-id': 3, u'tx-vlan-id': 3, u'gateway': u'81.206.159.254', u'system-name': u'd9efd328-a23e-3a35-94c1-0f584835d577', u'ip-address': u'81.206.159.251', u'vn-id': u'6451cff1-fb8a-4564-84d7-5b40873b4667', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'd9efd328-a23e-3a35-94c1-0f584835d577', u'instance-id': u'50247860-a938-9d5e-d509-cacdc50bb1af'} 10/23/2018 09:15:21 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 0e0bc3b2-3ac9-3a5a-b9af-02218287d03f 10/23/2018 09:15:21 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 7a9e1013-0bcf-3e83-8c15-10f35b3064a7 10/23/2018 09:15:21 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: d9efd328-a23e-3a35-94c1-0f584835d577 10/23/2018 09:15:21 AM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 09:15:22 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=0e0bc3b2-3ac9-3a5a-b9af-02218287d03f, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:02:47, port_key=113439, portgroup_key=dvportgroup-48428, vlan_id=None), ip_address=unset) 10/23/2018 09:15:22 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113439 10/23/2018 09:15:22 AM [cvm.clients] [INFO]: Port: 113439 VLAN ID: 1 10/23/2018 09:15:22 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 0e0bc3b2-3ac9-3a5a-b9af-02218287d03f in VNC 10/23/2018 09:15:22 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface 0e0bc3b2-3ac9-3a5a-b9af-02218287d03f 10/23/2018 09:15:22 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface 0e0bc3b2-3ac9-3a5a-b9af-02218287d03f to ctest-mgmt_vCenter-29750405 10/23/2018 09:15:22 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-mgmt_vCenter-29750405-ctest-pt_svm0-82433525 10/23/2018 09:15:22 AM [cvm.clients] [INFO]: Created Instance IP: 4ee67ec8-ba07-3f99-a6d6-1443f834b249 10/23/2018 09:15:22 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-mgmt_vCenter-29750405-ctest-pt_svm0-82433525 10/23/2018 09:15:22 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=0e0bc3b2-3ac9-3a5a-b9af-02218287d03f, display_name=vmi-ctest-mgmt_vCenter-29750405-ctest-pt_svm0-82433525, vcenter_port=VCenterPort(mac_address=00:50:56:a4:02:47, port_key=113439, portgroup_key=dvportgroup-48428, vlan_id=1), ip_address=38.253.32.252) 10/23/2018 09:15:22 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=7a9e1013-0bcf-3e83-8c15-10f35b3064a7, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:f2:7b, port_key=113566, portgroup_key=dvportgroup-48429, vlan_id=None), ip_address=unset) 10/23/2018 09:15:22 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113566 10/23/2018 09:15:22 AM [cvm.clients] [INFO]: Port: 113566 VLAN ID: 2 10/23/2018 09:15:22 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 7a9e1013-0bcf-3e83-8c15-10f35b3064a7 in VNC 10/23/2018 09:15:22 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface 7a9e1013-0bcf-3e83-8c15-10f35b3064a7 10/23/2018 09:15:22 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface 7a9e1013-0bcf-3e83-8c15-10f35b3064a7 to ctest-left_vCenter-02203416 10/23/2018 09:15:22 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-left_vCenter-02203416-ctest-pt_svm0-82433525 10/23/2018 09:15:22 AM [cvm.clients] [INFO]: Created Instance IP: 7631e252-a6c4-3ef9-8d60-cac85aa65ac1 10/23/2018 09:15:22 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-left_vCenter-02203416-ctest-pt_svm0-82433525 10/23/2018 09:15:22 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=7a9e1013-0bcf-3e83-8c15-10f35b3064a7, display_name=vmi-ctest-left_vCenter-02203416-ctest-pt_svm0-82433525, vcenter_port=VCenterPort(mac_address=00:50:56:a4:f2:7b, port_key=113566, portgroup_key=dvportgroup-48429, vlan_id=2), ip_address=42.100.201.251) 10/23/2018 09:15:22 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=d9efd328-a23e-3a35-94c1-0f584835d577, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:93:a7, port_key=113694, portgroup_key=dvportgroup-48430, vlan_id=None), ip_address=unset) 10/23/2018 09:15:23 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113694 10/23/2018 09:15:23 AM [cvm.clients] [INFO]: Port: 113694 VLAN ID: 3 10/23/2018 09:15:23 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface d9efd328-a23e-3a35-94c1-0f584835d577 in VNC 10/23/2018 09:15:23 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface d9efd328-a23e-3a35-94c1-0f584835d577 10/23/2018 09:15:23 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface d9efd328-a23e-3a35-94c1-0f584835d577 to ctest-right_vCenter-04676770 10/23/2018 09:15:23 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-right_vCenter-04676770-ctest-pt_svm0-82433525 10/23/2018 09:15:23 AM [cvm.clients] [INFO]: Created Instance IP: 92e9aae2-74ba-3383-afcc-5a109c813eb5 10/23/2018 09:15:23 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-right_vCenter-04676770-ctest-pt_svm0-82433525 10/23/2018 09:15:23 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=d9efd328-a23e-3a35-94c1-0f584835d577, display_name=vmi-ctest-right_vCenter-04676770-ctest-pt_svm0-82433525, vcenter_port=VCenterPort(mac_address=00:50:56:a4:93:a7, port_key=113694, portgroup_key=dvportgroup-48430, vlan_id=3), ip_address=81.206.159.251) 10/23/2018 09:15:23 AM [cvm.clients] [INFO]: Read vRouter port with uuid: 0e0bc3b2-3ac9-3a5a-b9af-02218287d03f, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:15:23.310125', u'dns-server': u'38.253.32.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:02:47', u'rx-vlan-id': 1, u'tx-vlan-id': 1, u'gateway': u'38.253.32.254', u'system-name': u'0e0bc3b2-3ac9-3a5a-b9af-02218287d03f', u'ip-address': u'38.253.32.252', u'vn-id': u'46b87b99-0e93-43de-a7e8-534e86dd24d0', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'0e0bc3b2-3ac9-3a5a-b9af-02218287d03f', u'instance-id': u'50247860-a938-9d5e-d509-cacdc50bb1af'} 10/23/2018 09:15:23 AM [cvm.clients] [INFO]: Read vRouter port with uuid: 7a9e1013-0bcf-3e83-8c15-10f35b3064a7, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:15:23.315269', u'dns-server': u'42.100.201.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:f2:7b', u'rx-vlan-id': 2, u'tx-vlan-id': 2, u'gateway': u'42.100.201.254', u'system-name': u'7a9e1013-0bcf-3e83-8c15-10f35b3064a7', u'ip-address': u'42.100.201.251', u'vn-id': u'43bfb4d2-ed84-4970-94a5-c5e4eec5ef20', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'7a9e1013-0bcf-3e83-8c15-10f35b3064a7', u'instance-id': u'50247860-a938-9d5e-d509-cacdc50bb1af'} 10/23/2018 09:15:23 AM [cvm.clients] [INFO]: Read vRouter port with uuid: d9efd328-a23e-3a35-94c1-0f584835d577, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:15:23.320077', u'dns-server': u'81.206.159.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:93:a7', u'rx-vlan-id': 3, u'tx-vlan-id': 3, u'gateway': u'81.206.159.254', u'system-name': u'd9efd328-a23e-3a35-94c1-0f584835d577', u'ip-address': u'81.206.159.251', u'vn-id': u'6451cff1-fb8a-4564-84d7-5b40873b4667', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'd9efd328-a23e-3a35-94c1-0f584835d577', u'instance-id': u'50247860-a938-9d5e-d509-cacdc50bb1af'} 10/23/2018 09:15:23 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 0e0bc3b2-3ac9-3a5a-b9af-02218287d03f 10/23/2018 09:15:23 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 7a9e1013-0bcf-3e83-8c15-10f35b3064a7 10/23/2018 09:15:23 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: d9efd328-a23e-3a35-94c1-0f584835d577 10/23/2018 09:15:23 AM [cvm.services] [INFO]: VMware tools on VM ctest-pt_svm0-82433525 are running 10/23/2018 09:15:23 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-82433525 10/23/2018 09:15:23 AM [cvm.services] [INFO]: VM ctest-pt_svm0-82433525 was powered on 10/23/2018 09:15:23 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-82433525 10/23/2018 09:15:24 AM [cvm.clients] [INFO]: Enabled vRouter port with uuid: 0e0bc3b2-3ac9-3a5a-b9af-02218287d03f 10/23/2018 09:15:24 AM [cvm.clients] [INFO]: Enabled vRouter port with uuid: 7a9e1013-0bcf-3e83-8c15-10f35b3064a7 10/23/2018 09:15:24 AM [cvm.clients] [INFO]: Enabled vRouter port with uuid: d9efd328-a23e-3a35-94c1-0f584835d577 10/23/2018 09:20:59 AM [cvm.services] [INFO]: VMware tools on VM ctest-pt_svm0-82433525 are not running 10/23/2018 09:20:59 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-82433525 10/23/2018 09:20:59 AM [cvm.services] [INFO]: VM ctest-pt_svm0-82433525 was powered off 10/23/2018 09:20:59 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-82433525 10/23/2018 09:20:59 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 0e0bc3b2-3ac9-3a5a-b9af-02218287d03f 10/23/2018 09:20:59 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 7a9e1013-0bcf-3e83-8c15-10f35b3064a7 10/23/2018 09:20:59 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: d9efd328-a23e-3a35-94c1-0f584835d577 10/23/2018 09:21:00 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 09:21:01 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 171, in main gevent.spawn(vmware_monitor.start()), File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 13, in start self._controller.handle_update(update_set) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 31, in handle_update self._update_handler.handle_update(update_set) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 43, in handle_update handler.handle_change(object_update.obj, property_change) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 54, in handle_change self._handle_change(obj, value) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 89, in _handle_change self._handle_event(value) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 216, in _handle_event self._vmi_service.remove_vmis_for_vm_model(vm_name) File "/usr/lib/python2.7/site-packages/cvm/services.py", line 177, in remove_vmis_for_vm_model full_remove = self._vcenter_api_client.can_remove_vm(vm_model.uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 574, in __call__ return self.f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 394, in _InvokeAccessor return self._stub.InvokeAccessor(self, info) File "/usr/lib/python2.7/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor options=self._pcType.RetrieveOptions(maxObjects=1)) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 580, in self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 386, in _InvokeMethod return self._stub.InvokeMethod(self, info, args) File "/usr/lib/python2.7/site-packages/pyVmomi/SoapAdapter.py", line 1370, in InvokeMethod raise obj # pylint: disable-msg=E0702 vmodl.fault.ManagedObjectNotFound: (vmodl.fault.ManagedObjectNotFound) { dynamicType = , dynamicProperty = (vmodl.DynamicProperty) [], msg = "The object 'vim.VirtualMachine:vm-48433' has already been deleted or has not been completely created", faultCause = , faultMessage = (vmodl.LocalizableMessage) [], obj = 'vim.VirtualMachine:vm-48433' } 10/23/2018 09:21:03 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 09:21:03 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 09:21:03 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 09:21:03 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 09:21:03 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 09:21:03 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 09:21:03 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 09:21:03 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 09:21:03 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 09:21:03 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540286463091652 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.140:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 09:21:03 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 09:21:03 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 09:21:03 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 09:21:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48428 name: ctest-mgmt_vCenter-29750405 10/23/2018 09:21:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-mgmt_vCenter-29750405 already allowed. 10/23/2018 09:21:03 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-mgmt_vCenter-29750405 10/23/2018 09:21:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=46b87b99-0e93-43de-a7e8-534e86dd24d0, key=dvportgroup-48428, name=ctest-mgmt_vCenter-29750405) 10/23/2018 09:21:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48429 name: ctest-left_vCenter-02203416 10/23/2018 09:21:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-left_vCenter-02203416 already allowed. 10/23/2018 09:21:03 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-left_vCenter-02203416 10/23/2018 09:21:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=43bfb4d2-ed84-4970-94a5-c5e4eec5ef20, key=dvportgroup-48429, name=ctest-left_vCenter-02203416) 10/23/2018 09:21:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48430 name: ctest-right_vCenter-04676770 10/23/2018 09:21:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-right_vCenter-04676770 already allowed. 10/23/2018 09:21:03 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-right_vCenter-04676770 10/23/2018 09:21:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6451cff1-fb8a-4564-84d7-5b40873b4667, key=dvportgroup-48430, name=ctest-right_vCenter-04676770) 10/23/2018 09:21:06 AM [cvm.services] [INFO]: Deleting d9efd328-a23e-3a35-94c1-0f584835d577 from VNC. 10/23/2018 09:21:06 AM [cvm.clients] [ERROR]: Could not find VMI d9efd328-a23e-3a35-94c1-0f584835d577 in VNC 10/23/2018 09:21:06 AM [cvm.clients] [ERROR]: Virtual Machine Interface d9efd328-a23e-3a35-94c1-0f584835d577 not found in VNC. Unable to delete 10/23/2018 09:21:07 AM [cvm.services] [INFO]: Deleting 55786598-269c-3667-bfc2-9bbbbac34fbd from VNC. 10/23/2018 09:21:07 AM [cvm.clients] [ERROR]: Could not find VMI 55786598-269c-3667-bfc2-9bbbbac34fbd in VNC 10/23/2018 09:21:07 AM [cvm.clients] [ERROR]: Virtual Machine Interface 55786598-269c-3667-bfc2-9bbbbac34fbd not found in VNC. Unable to delete 10/23/2018 09:21:07 AM [cvm] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/23/2018 09:21:07 AM [cvm] [INFO]: Session Event: TCP Connected 10/23/2018 09:21:07 AM [cvm] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/23/2018 09:21:07 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/23/2018 09:21:07 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ 10.204.217.139:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector connection down >>, ] >> 10/23/2018 09:21:07 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/23/2018 09:21:07 AM [cvm] [INFO]: Processing event[EvTcpConnected] in state[Connect] 10/23/2018 09:21:07 AM [cvm] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 10/23/2018 09:21:07 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 09:21:07 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 09:21:07 AM [cvm] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 10/23/2018 09:21:07 AM [cvm] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 10/23/2018 09:21:07 AM [cvm] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 10/23/2018 09:21:07 AM [cvm] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 10/23/2018 09:21:07 AM [cvm.services] [INFO]: Deleting ac954a35-8a92-3ab2-a2e3-2cc588defa93 from VNC. 10/23/2018 09:21:07 AM [cvm.clients] [ERROR]: Could not find VMI ac954a35-8a92-3ab2-a2e3-2cc588defa93 in VNC 10/23/2018 09:21:07 AM [cvm.clients] [ERROR]: Virtual Machine Interface ac954a35-8a92-3ab2-a2e3-2cc588defa93 not found in VNC. Unable to delete 10/23/2018 09:21:08 AM [cvm.services] [INFO]: Deleting 1dcd786b-79c0-314f-bcaa-0250800957a6 from VNC. 10/23/2018 09:21:08 AM [cvm.clients] [ERROR]: Could not find VMI 1dcd786b-79c0-314f-bcaa-0250800957a6 in VNC 10/23/2018 09:21:08 AM [cvm.clients] [ERROR]: Virtual Machine Interface 1dcd786b-79c0-314f-bcaa-0250800957a6 not found in VNC. Unable to delete 10/23/2018 09:21:09 AM [cvm.services] [INFO]: Deleting 765aeedc-58f5-35c1-b234-a02dc79610bd from VNC. 10/23/2018 09:21:09 AM [cvm.clients] [ERROR]: Could not find VMI 765aeedc-58f5-35c1-b234-a02dc79610bd in VNC 10/23/2018 09:21:09 AM [cvm.clients] [ERROR]: Virtual Machine Interface 765aeedc-58f5-35c1-b234-a02dc79610bd not found in VNC. Unable to delete 10/23/2018 09:21:09 AM [cvm.services] [INFO]: Deleting 7036454d-fc6e-3f41-8303-4f915c31cf8d from VNC. 10/23/2018 09:21:09 AM [cvm.clients] [ERROR]: Could not find VMI 7036454d-fc6e-3f41-8303-4f915c31cf8d in VNC 10/23/2018 09:21:09 AM [cvm.clients] [ERROR]: Virtual Machine Interface 7036454d-fc6e-3f41-8303-4f915c31cf8d not found in VNC. Unable to delete 10/23/2018 09:21:10 AM [cvm.services] [INFO]: Deleting 95dcf71a-9178-3dca-a3ee-1427957d78a0 from VNC. 10/23/2018 09:21:10 AM [cvm.clients] [ERROR]: Could not find VMI 95dcf71a-9178-3dca-a3ee-1427957d78a0 in VNC 10/23/2018 09:21:10 AM [cvm.clients] [ERROR]: Virtual Machine Interface 95dcf71a-9178-3dca-a3ee-1427957d78a0 not found in VNC. Unable to delete 10/23/2018 09:21:10 AM [cvm.database] [INFO]: Could not find VM model with uuid 502439ff-e7eb-3203-0595-f798b0fab1de. 10/23/2018 09:21:11 AM [cvm.database] [INFO]: Could not find VM model with uuid 50242a1d-212e-3235-01f3-63d057df0c79. 10/23/2018 09:21:12 AM [cvm.database] [INFO]: Could not find VM model with uuid 50247860-a938-9d5e-d509-cacdc50bb1af. 10/23/2018 09:21:13 AM [cvm.services] [INFO]: Deleting 50247860-a938-9d5e-d509-cacdc50bb1af from VNC 10/23/2018 09:21:13 AM [cvm.clients] [INFO]: Attempting to delete Virtual Machine 50247860-a938-9d5e-d509-cacdc50bb1af from VNC... 10/23/2018 09:21:13 AM [cvm.clients] [ERROR]: Virtual Machine 50247860-a938-9d5e-d509-cacdc50bb1af not found in VNC. Unable to delete 10/23/2018 09:21:13 AM [cvm.database] [INFO]: Could not find VM model with uuid 50265637-8829-fc50-89db-d80f49f1904a. 10/23/2018 09:21:13 AM [cvm.database] [INFO]: Could not find VM model with uuid 50243b3b-1402-cbd7-d0ba-c09875301f52. 10/23/2018 09:21:14 AM [cvm.services] [INFO]: Deleting 50243b3b-1402-cbd7-d0ba-c09875301f52 from VNC 10/23/2018 09:21:14 AM [cvm.clients] [INFO]: Attempting to delete Virtual Machine 50243b3b-1402-cbd7-d0ba-c09875301f52 from VNC... 10/23/2018 09:21:14 AM [cvm.clients] [INFO]: Virtual Machine 50243b3b-1402-cbd7-d0ba-c09875301f52 removed from VNC 10/23/2018 09:21:14 AM [cvm.database] [INFO]: Could not find VM model with uuid 5024bab1-8ce8-fc4d-1158-c58a31105cdc. 10/23/2018 09:21:15 AM [cvm.database] [INFO]: Could not find VM model with uuid 50243993-ed7d-4647-e72d-823e31e268d1. 10/23/2018 09:21:16 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 5b3efe91-f284-3930-b2ef-0cdfc7bda3db 10/23/2018 09:21:16 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 296fd0ed-d4ef-362e-a185-03ee3df90ab3 10/23/2018 09:21:16 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: d9efd328-a23e-3a35-94c1-0f584835d577 10/23/2018 09:21:16 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 55786598-269c-3667-bfc2-9bbbbac34fbd 10/23/2018 09:21:16 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: ac954a35-8a92-3ab2-a2e3-2cc588defa93 10/23/2018 09:21:16 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 1dcd786b-79c0-314f-bcaa-0250800957a6 10/23/2018 09:21:16 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 765aeedc-58f5-35c1-b234-a02dc79610bd 10/23/2018 09:21:17 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 7036454d-fc6e-3f41-8303-4f915c31cf8d 10/23/2018 09:21:17 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 95dcf71a-9178-3dca-a3ee-1427957d78a0 10/23/2018 09:21:17 AM [cvm.controllers] [INFO]: Synchronization complete 10/23/2018 09:29:01 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 09:29:01 AM [cvm.clients] [INFO]: Read from ESXi API ctest-pt_svm0-24003645 properties: {u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '50240c65-7ade-da9b-806d-c30fedb2b02f', u'runtime.powerState': 'poweredOff', u'name': 'ctest-pt_svm0-24003645', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 09:29:01 AM [cvm.database] [INFO]: Could not find VM model with uuid 50240c65-7ade-da9b-806d-c30fedb2b02f. 10/23/2018 09:29:01 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine 50240c65-7ade-da9b-806d-c30fedb2b02f in VNC 10/23/2018 09:29:01 AM [cvm.clients] [INFO]: Virtual Machine 50240c65-7ade-da9b-806d-c30fedb2b02f not found in VNC - creating 10/23/2018 09:29:01 AM [cvm.clients] [INFO]: Virtual Machine 50240c65-7ade-da9b-806d-c30fedb2b02f created in VNC 10/23/2018 09:29:01 AM [cvm.services] [INFO]: Created VirtualMachineModel(uuid=50240c65-7ade-da9b-806d-c30fedb2b02f, name=ctest-pt_svm0-24003645, vm_properties={u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '50240c65-7ade-da9b-806d-c30fedb2b02f', u'runtime.powerState': 'poweredOff', u'name': 'ctest-pt_svm0-24003645', u'summary.runtime.host': 'vim.HostSystem:ha-host'}, interfaces=[]) 10/23/2018 09:29:01 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-24003645 10/23/2018 09:29:02 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 09:29:02 AM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 09:29:02 AM [cvm.database] [INFO]: Could not find VN model with key dvportgroup-48438. 10/23/2018 09:29:02 AM [cvm.services] [INFO]: Fetching new portgroup for key: dvportgroup-48438 10/23/2018 09:29:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48438 name: ctest-right_vCenter-70204404 10/23/2018 09:29:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-right_vCenter-70204404 already allowed. 10/23/2018 09:29:02 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-right_vCenter-70204404 10/23/2018 09:29:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=58e6d3ef-7761-4892-990a-7ba924faf886, key=dvportgroup-48438, name=ctest-right_vCenter-70204404) 10/23/2018 09:29:02 AM [cvm.database] [INFO]: Could not find VN model with key dvportgroup-48437. 10/23/2018 09:29:02 AM [cvm.services] [INFO]: Fetching new portgroup for key: dvportgroup-48437 10/23/2018 09:29:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48437 name: ctest-left_vCenter-74919226 10/23/2018 09:29:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-left_vCenter-74919226 already allowed. 10/23/2018 09:29:03 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-left_vCenter-74919226 10/23/2018 09:29:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=fc727d3b-e02c-4edf-bd06-7a340cbe9e6d, key=dvportgroup-48437, name=ctest-left_vCenter-74919226) 10/23/2018 09:29:03 AM [cvm.database] [INFO]: Could not find VN model with key dvportgroup-48436. 10/23/2018 09:29:03 AM [cvm.services] [INFO]: Fetching new portgroup for key: dvportgroup-48436 10/23/2018 09:29:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48436 name: ctest-mgmt_vCenter-17887276 10/23/2018 09:29:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-mgmt_vCenter-17887276 already allowed. 10/23/2018 09:29:03 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-mgmt_vCenter-17887276 10/23/2018 09:29:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=1c15b657-7185-41be-b70e-c6cf1a4e8b3a, key=dvportgroup-48436, name=ctest-mgmt_vCenter-17887276) 10/23/2018 09:29:03 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=d2f6ce0d-7142-378b-a159-75521cffbbd0, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:60:32, port_key=114078, portgroup_key=dvportgroup-48438, vlan_id=None), ip_address=unset) 10/23/2018 09:29:03 AM [cvm.clients] [INFO]: Reading VLAN ID of port 114078 10/23/2018 09:29:03 AM [cvm.clients] [INFO]: Port: 114078 has no VLAN ID 10/23/2018 09:29:06 AM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 114078 to 1 10/23/2018 09:29:06 AM [cvm.clients] [INFO]: Successfully set VLAN ID: 1 for port: 114078 10/23/2018 09:29:06 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface d2f6ce0d-7142-378b-a159-75521cffbbd0 in VNC 10/23/2018 09:29:06 AM [cvm.clients] [INFO]: Virtual Machine Interface d2f6ce0d-7142-378b-a159-75521cffbbd0 not found in VNC - creating 10/23/2018 09:29:06 AM [cvm.clients] [INFO]: Virtual Machine Interface d2f6ce0d-7142-378b-a159-75521cffbbd0 created in VNC 10/23/2018 09:29:06 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-right_vCenter-70204404-ctest-pt_svm0-24003645 10/23/2018 09:29:06 AM [cvm.clients] [INFO]: Created Instance IP: f974dbe9-b283-392b-8abd-3a4ead299e7f 10/23/2018 09:29:06 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-right_vCenter-70204404-ctest-pt_svm0-24003645 10/23/2018 09:29:06 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=d2f6ce0d-7142-378b-a159-75521cffbbd0, display_name=vmi-ctest-right_vCenter-70204404-ctest-pt_svm0-24003645, vcenter_port=VCenterPort(mac_address=00:50:56:a4:60:32, port_key=114078, portgroup_key=dvportgroup-48438, vlan_id=1), ip_address=26.38.206.251) 10/23/2018 09:29:06 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=d259967d-0661-3003-a917-35e45fd9eada, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:c2:a3, port_key=113950, portgroup_key=dvportgroup-48437, vlan_id=None), ip_address=unset) 10/23/2018 09:29:06 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113950 10/23/2018 09:29:06 AM [cvm.clients] [INFO]: Port: 113950 has no VLAN ID 10/23/2018 09:29:09 AM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 113950 to 2 10/23/2018 09:29:10 AM [cvm.clients] [INFO]: Successfully set VLAN ID: 2 for port: 113950 10/23/2018 09:29:10 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface d259967d-0661-3003-a917-35e45fd9eada in VNC 10/23/2018 09:29:10 AM [cvm.clients] [INFO]: Virtual Machine Interface d259967d-0661-3003-a917-35e45fd9eada not found in VNC - creating 10/23/2018 09:29:10 AM [cvm.clients] [INFO]: Virtual Machine Interface d259967d-0661-3003-a917-35e45fd9eada created in VNC 10/23/2018 09:29:10 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-left_vCenter-74919226-ctest-pt_svm0-24003645 10/23/2018 09:29:10 AM [cvm.clients] [INFO]: Created Instance IP: e2c7ed9c-1e11-3214-8e44-a90e19dd8cc4 10/23/2018 09:29:10 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-left_vCenter-74919226-ctest-pt_svm0-24003645 10/23/2018 09:29:10 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=d259967d-0661-3003-a917-35e45fd9eada, display_name=vmi-ctest-left_vCenter-74919226-ctest-pt_svm0-24003645, vcenter_port=VCenterPort(mac_address=00:50:56:a4:c2:a3, port_key=113950, portgroup_key=dvportgroup-48437, vlan_id=2), ip_address=139.159.99.251) 10/23/2018 09:29:10 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=74eb3e62-852f-3409-91ab-d54becadbbf3, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:62:3f, port_key=113823, portgroup_key=dvportgroup-48436, vlan_id=None), ip_address=unset) 10/23/2018 09:29:10 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113823 10/23/2018 09:29:10 AM [cvm.clients] [INFO]: Port: 113823 has no VLAN ID 10/23/2018 09:29:13 AM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 113823 to 3 10/23/2018 09:29:13 AM [cvm.clients] [INFO]: Successfully set VLAN ID: 3 for port: 113823 10/23/2018 09:29:13 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 74eb3e62-852f-3409-91ab-d54becadbbf3 in VNC 10/23/2018 09:29:13 AM [cvm.clients] [INFO]: Virtual Machine Interface 74eb3e62-852f-3409-91ab-d54becadbbf3 not found in VNC - creating 10/23/2018 09:29:13 AM [cvm.clients] [INFO]: Virtual Machine Interface 74eb3e62-852f-3409-91ab-d54becadbbf3 created in VNC 10/23/2018 09:29:13 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-mgmt_vCenter-17887276-ctest-pt_svm0-24003645 10/23/2018 09:29:13 AM [cvm.clients] [INFO]: Created Instance IP: 05d0d35c-a643-358d-a755-822acfdbe797 10/23/2018 09:29:13 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-mgmt_vCenter-17887276-ctest-pt_svm0-24003645 10/23/2018 09:29:13 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=74eb3e62-852f-3409-91ab-d54becadbbf3, display_name=vmi-ctest-mgmt_vCenter-17887276-ctest-pt_svm0-24003645, vcenter_port=VCenterPort(mac_address=00:50:56:a4:62:3f, port_key=113823, portgroup_key=dvportgroup-48436, vlan_id=3), ip_address=158.166.154.252) 10/23/2018 09:29:13 AM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: d2f6ce0d-7142-378b-a159-75521cffbbd0 10/23/2018 09:29:13 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: d2f6ce0d-7142-378b-a159-75521cffbbd0 10/23/2018 09:29:14 AM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': 'd2f6ce0d-7142-378b-a159-75521cffbbd0', 'display_name': 'ctest-pt_svm0-24003645', 'rx_vlan': 1, 'vn_id': '58e6d3ef-7761-4892-990a-7ba924faf886', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 1, 'port_type': 2, 'mac_address': '00:50:56:a4:60:32', 'interface_name': 'd2f6ce0d-7142-378b-a159-75521cffbbd0', 'ip_address': u'26.38.206.251', 'vm_uuid_str': '50240c65-7ade-da9b-806d-c30fedb2b02f'} 10/23/2018 09:29:14 AM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: d259967d-0661-3003-a917-35e45fd9eada 10/23/2018 09:29:14 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: d259967d-0661-3003-a917-35e45fd9eada 10/23/2018 09:29:14 AM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': 'd259967d-0661-3003-a917-35e45fd9eada', 'display_name': 'ctest-pt_svm0-24003645', 'rx_vlan': 2, 'vn_id': 'fc727d3b-e02c-4edf-bd06-7a340cbe9e6d', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 2, 'port_type': 2, 'mac_address': '00:50:56:a4:c2:a3', 'interface_name': 'd259967d-0661-3003-a917-35e45fd9eada', 'ip_address': u'139.159.99.251', 'vm_uuid_str': '50240c65-7ade-da9b-806d-c30fedb2b02f'} 10/23/2018 09:29:14 AM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: 74eb3e62-852f-3409-91ab-d54becadbbf3 10/23/2018 09:29:14 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 74eb3e62-852f-3409-91ab-d54becadbbf3 10/23/2018 09:29:14 AM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': '74eb3e62-852f-3409-91ab-d54becadbbf3', 'display_name': 'ctest-pt_svm0-24003645', 'rx_vlan': 3, 'vn_id': '1c15b657-7185-41be-b70e-c6cf1a4e8b3a', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 3, 'port_type': 2, 'mac_address': '00:50:56:a4:62:3f', 'interface_name': '74eb3e62-852f-3409-91ab-d54becadbbf3', 'ip_address': u'158.166.154.252', 'vm_uuid_str': '50240c65-7ade-da9b-806d-c30fedb2b02f'} 10/23/2018 09:29:14 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: d2f6ce0d-7142-378b-a159-75521cffbbd0 10/23/2018 09:29:15 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: d259967d-0661-3003-a917-35e45fd9eada 10/23/2018 09:29:15 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 74eb3e62-852f-3409-91ab-d54becadbbf3 10/23/2018 09:29:15 AM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 09:29:15 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=d2f6ce0d-7142-378b-a159-75521cffbbd0, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:60:32, port_key=114078, portgroup_key=dvportgroup-48438, vlan_id=None), ip_address=unset) 10/23/2018 09:29:15 AM [cvm.clients] [INFO]: Reading VLAN ID of port 114078 10/23/2018 09:29:15 AM [cvm.clients] [INFO]: Port: 114078 VLAN ID: 1 10/23/2018 09:29:15 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface d2f6ce0d-7142-378b-a159-75521cffbbd0 in VNC 10/23/2018 09:29:15 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface d2f6ce0d-7142-378b-a159-75521cffbbd0 10/23/2018 09:29:15 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface d2f6ce0d-7142-378b-a159-75521cffbbd0 to ctest-right_vCenter-70204404 10/23/2018 09:29:15 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-right_vCenter-70204404-ctest-pt_svm0-24003645 10/23/2018 09:29:15 AM [cvm.clients] [INFO]: Created Instance IP: f974dbe9-b283-392b-8abd-3a4ead299e7f 10/23/2018 09:29:15 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-right_vCenter-70204404-ctest-pt_svm0-24003645 10/23/2018 09:29:15 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=d2f6ce0d-7142-378b-a159-75521cffbbd0, display_name=vmi-ctest-right_vCenter-70204404-ctest-pt_svm0-24003645, vcenter_port=VCenterPort(mac_address=00:50:56:a4:60:32, port_key=114078, portgroup_key=dvportgroup-48438, vlan_id=1), ip_address=26.38.206.251) 10/23/2018 09:29:15 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=d259967d-0661-3003-a917-35e45fd9eada, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:c2:a3, port_key=113950, portgroup_key=dvportgroup-48437, vlan_id=None), ip_address=unset) 10/23/2018 09:29:15 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113950 10/23/2018 09:29:15 AM [cvm.clients] [INFO]: Port: 113950 VLAN ID: 2 10/23/2018 09:29:15 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface d259967d-0661-3003-a917-35e45fd9eada in VNC 10/23/2018 09:29:15 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface d259967d-0661-3003-a917-35e45fd9eada 10/23/2018 09:29:15 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface d259967d-0661-3003-a917-35e45fd9eada to ctest-left_vCenter-74919226 10/23/2018 09:29:15 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-left_vCenter-74919226-ctest-pt_svm0-24003645 10/23/2018 09:29:16 AM [cvm.clients] [INFO]: Created Instance IP: e2c7ed9c-1e11-3214-8e44-a90e19dd8cc4 10/23/2018 09:29:16 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-left_vCenter-74919226-ctest-pt_svm0-24003645 10/23/2018 09:29:16 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=d259967d-0661-3003-a917-35e45fd9eada, display_name=vmi-ctest-left_vCenter-74919226-ctest-pt_svm0-24003645, vcenter_port=VCenterPort(mac_address=00:50:56:a4:c2:a3, port_key=113950, portgroup_key=dvportgroup-48437, vlan_id=2), ip_address=139.159.99.251) 10/23/2018 09:29:16 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=74eb3e62-852f-3409-91ab-d54becadbbf3, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:62:3f, port_key=113823, portgroup_key=dvportgroup-48436, vlan_id=None), ip_address=unset) 10/23/2018 09:29:16 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113823 10/23/2018 09:29:16 AM [cvm.clients] [INFO]: Port: 113823 VLAN ID: 3 10/23/2018 09:29:16 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 74eb3e62-852f-3409-91ab-d54becadbbf3 in VNC 10/23/2018 09:29:16 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface 74eb3e62-852f-3409-91ab-d54becadbbf3 10/23/2018 09:29:16 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface 74eb3e62-852f-3409-91ab-d54becadbbf3 to ctest-mgmt_vCenter-17887276 10/23/2018 09:29:16 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-mgmt_vCenter-17887276-ctest-pt_svm0-24003645 10/23/2018 09:29:16 AM [cvm.clients] [INFO]: Created Instance IP: 05d0d35c-a643-358d-a755-822acfdbe797 10/23/2018 09:29:16 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-mgmt_vCenter-17887276-ctest-pt_svm0-24003645 10/23/2018 09:29:16 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=74eb3e62-852f-3409-91ab-d54becadbbf3, display_name=vmi-ctest-mgmt_vCenter-17887276-ctest-pt_svm0-24003645, vcenter_port=VCenterPort(mac_address=00:50:56:a4:62:3f, port_key=113823, portgroup_key=dvportgroup-48436, vlan_id=3), ip_address=158.166.154.252) 10/23/2018 09:29:16 AM [cvm.clients] [INFO]: Read vRouter port with uuid: d2f6ce0d-7142-378b-a159-75521cffbbd0, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:29:16.515457', u'dns-server': u'26.38.206.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:60:32', u'rx-vlan-id': 1, u'tx-vlan-id': 1, u'gateway': u'26.38.206.254', u'system-name': u'd2f6ce0d-7142-378b-a159-75521cffbbd0', u'ip-address': u'26.38.206.251', u'vn-id': u'58e6d3ef-7761-4892-990a-7ba924faf886', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'd2f6ce0d-7142-378b-a159-75521cffbbd0', u'instance-id': u'50240c65-7ade-da9b-806d-c30fedb2b02f'} 10/23/2018 09:29:16 AM [cvm.clients] [INFO]: Read vRouter port with uuid: d259967d-0661-3003-a917-35e45fd9eada, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:29:16.520418', u'dns-server': u'139.159.99.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:c2:a3', u'rx-vlan-id': 2, u'tx-vlan-id': 2, u'gateway': u'139.159.99.254', u'system-name': u'd259967d-0661-3003-a917-35e45fd9eada', u'ip-address': u'139.159.99.251', u'vn-id': u'fc727d3b-e02c-4edf-bd06-7a340cbe9e6d', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'd259967d-0661-3003-a917-35e45fd9eada', u'instance-id': u'50240c65-7ade-da9b-806d-c30fedb2b02f'} 10/23/2018 09:29:16 AM [cvm.clients] [INFO]: Read vRouter port with uuid: 74eb3e62-852f-3409-91ab-d54becadbbf3, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:29:16.524943', u'dns-server': u'158.166.154.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:62:3f', u'rx-vlan-id': 3, u'tx-vlan-id': 3, u'gateway': u'158.166.154.254', u'system-name': u'74eb3e62-852f-3409-91ab-d54becadbbf3', u'ip-address': u'158.166.154.252', u'vn-id': u'1c15b657-7185-41be-b70e-c6cf1a4e8b3a', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'74eb3e62-852f-3409-91ab-d54becadbbf3', u'instance-id': u'50240c65-7ade-da9b-806d-c30fedb2b02f'} 10/23/2018 09:29:16 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: d2f6ce0d-7142-378b-a159-75521cffbbd0 10/23/2018 09:29:16 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: d259967d-0661-3003-a917-35e45fd9eada 10/23/2018 09:29:17 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 74eb3e62-852f-3409-91ab-d54becadbbf3 10/23/2018 09:29:17 AM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 09:29:17 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=d2f6ce0d-7142-378b-a159-75521cffbbd0, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:60:32, port_key=114078, portgroup_key=dvportgroup-48438, vlan_id=None), ip_address=unset) 10/23/2018 09:29:17 AM [cvm.clients] [INFO]: Reading VLAN ID of port 114078 10/23/2018 09:29:17 AM [cvm.clients] [INFO]: Port: 114078 VLAN ID: 1 10/23/2018 09:29:17 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface d2f6ce0d-7142-378b-a159-75521cffbbd0 in VNC 10/23/2018 09:29:17 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface d2f6ce0d-7142-378b-a159-75521cffbbd0 10/23/2018 09:29:17 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface d2f6ce0d-7142-378b-a159-75521cffbbd0 to ctest-right_vCenter-70204404 10/23/2018 09:29:17 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-right_vCenter-70204404-ctest-pt_svm0-24003645 10/23/2018 09:29:17 AM [cvm.clients] [INFO]: Created Instance IP: f974dbe9-b283-392b-8abd-3a4ead299e7f 10/23/2018 09:29:17 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-right_vCenter-70204404-ctest-pt_svm0-24003645 10/23/2018 09:29:17 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=d2f6ce0d-7142-378b-a159-75521cffbbd0, display_name=vmi-ctest-right_vCenter-70204404-ctest-pt_svm0-24003645, vcenter_port=VCenterPort(mac_address=00:50:56:a4:60:32, port_key=114078, portgroup_key=dvportgroup-48438, vlan_id=1), ip_address=26.38.206.251) 10/23/2018 09:29:17 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=d259967d-0661-3003-a917-35e45fd9eada, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:c2:a3, port_key=113950, portgroup_key=dvportgroup-48437, vlan_id=None), ip_address=unset) 10/23/2018 09:29:17 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113950 10/23/2018 09:29:17 AM [cvm.clients] [INFO]: Port: 113950 VLAN ID: 2 10/23/2018 09:29:17 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface d259967d-0661-3003-a917-35e45fd9eada in VNC 10/23/2018 09:29:17 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface d259967d-0661-3003-a917-35e45fd9eada 10/23/2018 09:29:17 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface d259967d-0661-3003-a917-35e45fd9eada to ctest-left_vCenter-74919226 10/23/2018 09:29:17 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-left_vCenter-74919226-ctest-pt_svm0-24003645 10/23/2018 09:29:17 AM [cvm.clients] [INFO]: Created Instance IP: e2c7ed9c-1e11-3214-8e44-a90e19dd8cc4 10/23/2018 09:29:17 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-left_vCenter-74919226-ctest-pt_svm0-24003645 10/23/2018 09:29:17 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=d259967d-0661-3003-a917-35e45fd9eada, display_name=vmi-ctest-left_vCenter-74919226-ctest-pt_svm0-24003645, vcenter_port=VCenterPort(mac_address=00:50:56:a4:c2:a3, port_key=113950, portgroup_key=dvportgroup-48437, vlan_id=2), ip_address=139.159.99.251) 10/23/2018 09:29:17 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=74eb3e62-852f-3409-91ab-d54becadbbf3, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:62:3f, port_key=113823, portgroup_key=dvportgroup-48436, vlan_id=None), ip_address=unset) 10/23/2018 09:29:18 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113823 10/23/2018 09:29:18 AM [cvm.clients] [INFO]: Port: 113823 VLAN ID: 3 10/23/2018 09:29:18 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 74eb3e62-852f-3409-91ab-d54becadbbf3 in VNC 10/23/2018 09:29:18 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface 74eb3e62-852f-3409-91ab-d54becadbbf3 10/23/2018 09:29:18 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface 74eb3e62-852f-3409-91ab-d54becadbbf3 to ctest-mgmt_vCenter-17887276 10/23/2018 09:29:18 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-mgmt_vCenter-17887276-ctest-pt_svm0-24003645 10/23/2018 09:29:18 AM [cvm.clients] [INFO]: Created Instance IP: 05d0d35c-a643-358d-a755-822acfdbe797 10/23/2018 09:29:18 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-mgmt_vCenter-17887276-ctest-pt_svm0-24003645 10/23/2018 09:29:18 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=74eb3e62-852f-3409-91ab-d54becadbbf3, display_name=vmi-ctest-mgmt_vCenter-17887276-ctest-pt_svm0-24003645, vcenter_port=VCenterPort(mac_address=00:50:56:a4:62:3f, port_key=113823, portgroup_key=dvportgroup-48436, vlan_id=3), ip_address=158.166.154.252) 10/23/2018 09:29:18 AM [cvm.clients] [INFO]: Read vRouter port with uuid: d2f6ce0d-7142-378b-a159-75521cffbbd0, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:29:18.279574', u'dns-server': u'26.38.206.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:60:32', u'rx-vlan-id': 1, u'tx-vlan-id': 1, u'gateway': u'26.38.206.254', u'system-name': u'd2f6ce0d-7142-378b-a159-75521cffbbd0', u'ip-address': u'26.38.206.251', u'vn-id': u'58e6d3ef-7761-4892-990a-7ba924faf886', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'd2f6ce0d-7142-378b-a159-75521cffbbd0', u'instance-id': u'50240c65-7ade-da9b-806d-c30fedb2b02f'} 10/23/2018 09:29:18 AM [cvm.clients] [INFO]: Read vRouter port with uuid: d259967d-0661-3003-a917-35e45fd9eada, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:29:18.284411', u'dns-server': u'139.159.99.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:c2:a3', u'rx-vlan-id': 2, u'tx-vlan-id': 2, u'gateway': u'139.159.99.254', u'system-name': u'd259967d-0661-3003-a917-35e45fd9eada', u'ip-address': u'139.159.99.251', u'vn-id': u'fc727d3b-e02c-4edf-bd06-7a340cbe9e6d', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'd259967d-0661-3003-a917-35e45fd9eada', u'instance-id': u'50240c65-7ade-da9b-806d-c30fedb2b02f'} 10/23/2018 09:29:18 AM [cvm.clients] [INFO]: Read vRouter port with uuid: 74eb3e62-852f-3409-91ab-d54becadbbf3, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:29:18.289060', u'dns-server': u'158.166.154.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:62:3f', u'rx-vlan-id': 3, u'tx-vlan-id': 3, u'gateway': u'158.166.154.254', u'system-name': u'74eb3e62-852f-3409-91ab-d54becadbbf3', u'ip-address': u'158.166.154.252', u'vn-id': u'1c15b657-7185-41be-b70e-c6cf1a4e8b3a', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'74eb3e62-852f-3409-91ab-d54becadbbf3', u'instance-id': u'50240c65-7ade-da9b-806d-c30fedb2b02f'} 10/23/2018 09:29:18 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: d2f6ce0d-7142-378b-a159-75521cffbbd0 10/23/2018 09:29:18 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: d259967d-0661-3003-a917-35e45fd9eada 10/23/2018 09:29:18 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 74eb3e62-852f-3409-91ab-d54becadbbf3 10/23/2018 09:29:18 AM [cvm.services] [INFO]: VMware tools on VM ctest-pt_svm0-24003645 are running 10/23/2018 09:29:18 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-24003645 10/23/2018 09:29:18 AM [cvm.services] [INFO]: VM ctest-pt_svm0-24003645 was powered on 10/23/2018 09:29:18 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-24003645 10/23/2018 09:29:19 AM [cvm.clients] [INFO]: Enabled vRouter port with uuid: 74eb3e62-852f-3409-91ab-d54becadbbf3 10/23/2018 09:29:19 AM [cvm.clients] [INFO]: Enabled vRouter port with uuid: d259967d-0661-3003-a917-35e45fd9eada 10/23/2018 09:29:19 AM [cvm.clients] [INFO]: Enabled vRouter port with uuid: d2f6ce0d-7142-378b-a159-75521cffbbd0 10/23/2018 09:34:21 AM [cvm.services] [INFO]: VMware tools on VM ctest-pt_svm0-24003645 are not running 10/23/2018 09:34:21 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-24003645 10/23/2018 09:34:21 AM [cvm.services] [INFO]: VM ctest-pt_svm0-24003645 was powered off 10/23/2018 09:34:21 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-24003645 10/23/2018 09:34:21 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 74eb3e62-852f-3409-91ab-d54becadbbf3 10/23/2018 09:34:21 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: d259967d-0661-3003-a917-35e45fd9eada 10/23/2018 09:34:21 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: d2f6ce0d-7142-378b-a159-75521cffbbd0 10/23/2018 09:34:23 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 09:34:23 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 171, in main gevent.spawn(vmware_monitor.start()), File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 13, in start self._controller.handle_update(update_set) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 31, in handle_update self._update_handler.handle_update(update_set) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 43, in handle_update handler.handle_change(object_update.obj, property_change) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 54, in handle_change self._handle_change(obj, value) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 89, in _handle_change self._handle_event(value) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 216, in _handle_event self._vmi_service.remove_vmis_for_vm_model(vm_name) File "/usr/lib/python2.7/site-packages/cvm/services.py", line 177, in remove_vmis_for_vm_model full_remove = self._vcenter_api_client.can_remove_vm(vm_model.uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 574, in __call__ return self.f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 394, in _InvokeAccessor return self._stub.InvokeAccessor(self, info) File "/usr/lib/python2.7/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor options=self._pcType.RetrieveOptions(maxObjects=1)) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 580, in self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 386, in _InvokeMethod return self._stub.InvokeMethod(self, info, args) File "/usr/lib/python2.7/site-packages/pyVmomi/SoapAdapter.py", line 1370, in InvokeMethod raise obj # pylint: disable-msg=E0702 vmodl.fault.ManagedObjectNotFound: (vmodl.fault.ManagedObjectNotFound) { dynamicType = , dynamicProperty = (vmodl.DynamicProperty) [], msg = "The object 'vim.VirtualMachine:vm-48441' has already been deleted or has not been completely created", faultCause = , faultMessage = (vmodl.LocalizableMessage) [], obj = 'vim.VirtualMachine:vm-48441' } 10/23/2018 09:34:25 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 09:34:25 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 09:34:25 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 09:34:25 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 09:34:25 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 09:34:25 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 09:34:25 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 09:34:25 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 09:34:25 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 09:34:25 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540287265278834 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.139:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 09:34:25 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 09:34:25 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 09:34:25 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 09:34:25 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48436 name: ctest-mgmt_vCenter-17887276 10/23/2018 09:34:25 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-mgmt_vCenter-17887276 already allowed. 10/23/2018 09:34:25 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-mgmt_vCenter-17887276 10/23/2018 09:34:25 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=1c15b657-7185-41be-b70e-c6cf1a4e8b3a, key=dvportgroup-48436, name=ctest-mgmt_vCenter-17887276) 10/23/2018 09:34:25 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48437 name: ctest-left_vCenter-74919226 10/23/2018 09:34:25 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-left_vCenter-74919226 already allowed. 10/23/2018 09:34:25 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-left_vCenter-74919226 10/23/2018 09:34:25 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=fc727d3b-e02c-4edf-bd06-7a340cbe9e6d, key=dvportgroup-48437, name=ctest-left_vCenter-74919226) 10/23/2018 09:34:26 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48438 name: ctest-right_vCenter-70204404 10/23/2018 09:34:26 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-right_vCenter-70204404 already allowed. 10/23/2018 09:34:26 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-right_vCenter-70204404 10/23/2018 09:34:26 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=58e6d3ef-7761-4892-990a-7ba924faf886, key=dvportgroup-48438, name=ctest-right_vCenter-70204404) 10/23/2018 09:34:28 AM [cvm.services] [INFO]: Deleting 74eb3e62-852f-3409-91ab-d54becadbbf3 from VNC. 10/23/2018 09:34:28 AM [cvm.clients] [ERROR]: Could not find VMI 74eb3e62-852f-3409-91ab-d54becadbbf3 in VNC 10/23/2018 09:34:28 AM [cvm.clients] [ERROR]: Virtual Machine Interface 74eb3e62-852f-3409-91ab-d54becadbbf3 not found in VNC. Unable to delete 10/23/2018 09:34:28 AM [cvm.services] [INFO]: Deleting b58516f7-f421-3d85-af29-b996a452dda8 from VNC. 10/23/2018 09:34:28 AM [cvm.clients] [ERROR]: Could not find VMI b58516f7-f421-3d85-af29-b996a452dda8 in VNC 10/23/2018 09:34:28 AM [cvm.clients] [ERROR]: Virtual Machine Interface b58516f7-f421-3d85-af29-b996a452dda8 not found in VNC. Unable to delete 10/23/2018 09:34:29 AM [cvm] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/23/2018 09:34:29 AM [cvm] [INFO]: Session Event: TCP Connected 10/23/2018 09:34:29 AM [cvm] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/23/2018 09:34:29 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/23/2018 09:34:29 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ 10.204.217.144:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector connection down >>, ] >> 10/23/2018 09:34:29 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/23/2018 09:34:29 AM [cvm] [INFO]: Processing event[EvTcpConnected] in state[Connect] 10/23/2018 09:34:29 AM [cvm] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 10/23/2018 09:34:29 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 09:34:29 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 09:34:29 AM [cvm] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 10/23/2018 09:34:29 AM [cvm] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 10/23/2018 09:34:29 AM [cvm] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 10/23/2018 09:34:29 AM [cvm] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 10/23/2018 09:34:29 AM [cvm.services] [INFO]: Deleting 009224ac-42ad-35f1-a6b5-8a9be184631f from VNC. 10/23/2018 09:34:29 AM [cvm.clients] [ERROR]: Could not find VMI 009224ac-42ad-35f1-a6b5-8a9be184631f in VNC 10/23/2018 09:34:29 AM [cvm.clients] [ERROR]: Virtual Machine Interface 009224ac-42ad-35f1-a6b5-8a9be184631f not found in VNC. Unable to delete 10/23/2018 09:34:30 AM [cvm.services] [INFO]: Deleting 8667a1f6-29eb-39bd-a4ca-8dd41b953eeb from VNC. 10/23/2018 09:34:30 AM [cvm.clients] [ERROR]: Could not find VMI 8667a1f6-29eb-39bd-a4ca-8dd41b953eeb in VNC 10/23/2018 09:34:30 AM [cvm.clients] [ERROR]: Virtual Machine Interface 8667a1f6-29eb-39bd-a4ca-8dd41b953eeb not found in VNC. Unable to delete 10/23/2018 09:34:30 AM [cvm.database] [INFO]: Could not find VM model with uuid 502439ff-e7eb-3203-0595-f798b0fab1de. 10/23/2018 09:34:30 AM [cvm.database] [INFO]: Could not find VM model with uuid 5024ec26-7bb8-583e-e6c1-b4ebdb8fa956. 10/23/2018 09:34:31 AM [cvm.services] [INFO]: Deleting 5024ec26-7bb8-583e-e6c1-b4ebdb8fa956 from VNC 10/23/2018 09:34:31 AM [cvm.clients] [INFO]: Attempting to delete Virtual Machine 5024ec26-7bb8-583e-e6c1-b4ebdb8fa956 from VNC... 10/23/2018 09:34:31 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 25, in sync self._vm_service.delete_unused_vms_in_vnc() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 280, in delete_unused_vms_in_vnc self._vnc_api_client.delete_vm(vnc_vm.uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 320, in delete_vm vm = self.read_vm(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 350, in read_vm return self.vnc_lib.virtual_machine_read(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 670, in _object_read response = self._request_server(OP_GET, uri, query_params) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1085, in _request % (op, url, data, content)) NoIdError: Unknown id: Error: oper 2 url /virtual-machine/5024ec26-7bb8-583e-e6c1-b4ebdb8fa956 body {'exclude_back_refs': True, 'exclude_children': True} response Unknown id: 5024ec26-7bb8-583e-e6c1-b4ebdb8fa956 10/23/2018 09:34:33 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 09:34:33 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 09:34:33 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 09:34:33 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 09:34:33 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 09:34:33 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 09:34:33 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 09:34:33 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 09:34:33 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 09:34:33 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540287273241660 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.144:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 09:34:33 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 09:34:33 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 09:34:33 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 09:34:33 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48436 name: ctest-mgmt_vCenter-17887276 10/23/2018 09:34:33 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-mgmt_vCenter-17887276 already allowed. 10/23/2018 09:34:33 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-mgmt_vCenter-17887276 10/23/2018 09:34:33 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=1c15b657-7185-41be-b70e-c6cf1a4e8b3a, key=dvportgroup-48436, name=ctest-mgmt_vCenter-17887276) 10/23/2018 09:34:33 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48437 name: ctest-left_vCenter-74919226 10/23/2018 09:34:34 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-left_vCenter-74919226 already allowed. 10/23/2018 09:34:34 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-left_vCenter-74919226 10/23/2018 09:34:34 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=fc727d3b-e02c-4edf-bd06-7a340cbe9e6d, key=dvportgroup-48437, name=ctest-left_vCenter-74919226) 10/23/2018 09:34:34 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48438 name: ctest-right_vCenter-70204404 10/23/2018 09:34:34 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-right_vCenter-70204404 already allowed. 10/23/2018 09:34:34 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-right_vCenter-70204404 10/23/2018 09:34:34 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=58e6d3ef-7761-4892-990a-7ba924faf886, key=dvportgroup-48438, name=ctest-right_vCenter-70204404) 10/23/2018 09:34:35 AM [cvm.database] [INFO]: Could not find VM model with uuid 50244889-da8f-820c-75e6-5df4f8fe217e. 10/23/2018 09:34:36 AM [cvm.database] [INFO]: Could not find VM model with uuid 50265637-8829-fc50-89db-d80f49f1904a. 10/23/2018 09:34:37 AM [cvm] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/23/2018 09:34:37 AM [cvm] [INFO]: Session Event: TCP Connected 10/23/2018 09:34:37 AM [cvm] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/23/2018 09:34:37 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/23/2018 09:34:37 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ 10.204.217.140:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector connection down >>, ] >> 10/23/2018 09:34:37 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/23/2018 09:34:37 AM [cvm] [INFO]: Processing event[EvTcpConnected] in state[Connect] 10/23/2018 09:34:37 AM [cvm] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 10/23/2018 09:34:37 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 09:34:37 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 09:34:37 AM [cvm] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 10/23/2018 09:34:37 AM [cvm] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 10/23/2018 09:34:37 AM [cvm] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 10/23/2018 09:34:37 AM [cvm] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 10/23/2018 09:34:37 AM [cvm.database] [INFO]: Could not find VM model with uuid 502439ff-e7eb-3203-0595-f798b0fab1de. 10/23/2018 09:34:38 AM [cvm.database] [INFO]: Could not find VM model with uuid 50248132-39f7-d856-0080-234e60aad0fb. 10/23/2018 09:34:38 AM [cvm.database] [INFO]: Could not find VM model with uuid 50242a1d-212e-3235-01f3-63d057df0c79. 10/23/2018 09:34:39 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: bd1cd380-f4d9-3641-80ef-a6b5c86dfaad 10/23/2018 09:34:39 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 00021acc-c7f1-3f75-8b3d-fa01fb9610b3 10/23/2018 09:34:39 AM [cvm.controllers] [INFO]: Synchronization complete 10/23/2018 09:36:04 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 09:36:04 AM [cvm.clients] [INFO]: Read from ESXi API ctest-pt_svm0-85388690 properties: {u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '50248f70-ef18-e22c-d7c8-e3bd240208eb', u'runtime.powerState': 'poweredOff', u'name': 'ctest-pt_svm0-85388690', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 09:36:04 AM [cvm.database] [INFO]: Could not find VM model with uuid 50248f70-ef18-e22c-d7c8-e3bd240208eb. 10/23/2018 09:36:04 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine 50248f70-ef18-e22c-d7c8-e3bd240208eb in VNC 10/23/2018 09:36:04 AM [cvm.clients] [INFO]: Virtual Machine 50248f70-ef18-e22c-d7c8-e3bd240208eb not found in VNC - creating 10/23/2018 09:36:04 AM [cvm.clients] [INFO]: Virtual Machine 50248f70-ef18-e22c-d7c8-e3bd240208eb created in VNC 10/23/2018 09:36:04 AM [cvm.services] [INFO]: Created VirtualMachineModel(uuid=50248f70-ef18-e22c-d7c8-e3bd240208eb, name=ctest-pt_svm0-85388690, vm_properties={u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '50248f70-ef18-e22c-d7c8-e3bd240208eb', u'runtime.powerState': 'poweredOff', u'name': 'ctest-pt_svm0-85388690', u'summary.runtime.host': 'vim.HostSystem:ha-host'}, interfaces=[]) 10/23/2018 09:36:04 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-85388690 10/23/2018 09:36:05 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 09:36:05 AM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 09:36:05 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=a65fe91d-b220-3908-ae5f-84baf726dfa4, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:45:58, port_key=114077, portgroup_key=dvportgroup-48438, vlan_id=None), ip_address=unset) 10/23/2018 09:36:05 AM [cvm.clients] [INFO]: Reading VLAN ID of port 114077 10/23/2018 09:36:05 AM [cvm.clients] [INFO]: Port: 114077 VLAN ID: 2 10/23/2018 09:36:05 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface a65fe91d-b220-3908-ae5f-84baf726dfa4 in VNC 10/23/2018 09:36:05 AM [cvm.clients] [INFO]: Virtual Machine Interface a65fe91d-b220-3908-ae5f-84baf726dfa4 not found in VNC - creating 10/23/2018 09:36:05 AM [cvm.clients] [INFO]: Virtual Machine Interface a65fe91d-b220-3908-ae5f-84baf726dfa4 created in VNC 10/23/2018 09:36:05 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-right_vCenter-70204404-ctest-pt_svm0-85388690 10/23/2018 09:36:05 AM [cvm.clients] [INFO]: Created Instance IP: 32226e8f-1c1f-37ce-9a61-fd76057b2853 10/23/2018 09:36:05 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-right_vCenter-70204404-ctest-pt_svm0-85388690 10/23/2018 09:36:05 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=a65fe91d-b220-3908-ae5f-84baf726dfa4, display_name=vmi-ctest-right_vCenter-70204404-ctest-pt_svm0-85388690, vcenter_port=VCenterPort(mac_address=00:50:56:a4:45:58, port_key=114077, portgroup_key=dvportgroup-48438, vlan_id=2), ip_address=26.38.206.250) 10/23/2018 09:36:05 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=0ffeaa2f-9931-3289-8465-82525b36f74d, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:13:91, port_key=113949, portgroup_key=dvportgroup-48437, vlan_id=None), ip_address=unset) 10/23/2018 09:36:06 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113949 10/23/2018 09:36:06 AM [cvm.clients] [INFO]: Port: 113949 VLAN ID: 4 10/23/2018 09:36:06 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 0ffeaa2f-9931-3289-8465-82525b36f74d in VNC 10/23/2018 09:36:06 AM [cvm.clients] [INFO]: Virtual Machine Interface 0ffeaa2f-9931-3289-8465-82525b36f74d not found in VNC - creating 10/23/2018 09:36:06 AM [cvm.clients] [INFO]: Virtual Machine Interface 0ffeaa2f-9931-3289-8465-82525b36f74d created in VNC 10/23/2018 09:36:06 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-left_vCenter-74919226-ctest-pt_svm0-85388690 10/23/2018 09:36:06 AM [cvm.clients] [INFO]: Created Instance IP: 95dfb713-0ff4-3fe6-8299-38f20ba733d0 10/23/2018 09:36:06 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-left_vCenter-74919226-ctest-pt_svm0-85388690 10/23/2018 09:36:06 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=0ffeaa2f-9931-3289-8465-82525b36f74d, display_name=vmi-ctest-left_vCenter-74919226-ctest-pt_svm0-85388690, vcenter_port=VCenterPort(mac_address=00:50:56:a4:13:91, port_key=113949, portgroup_key=dvportgroup-48437, vlan_id=4), ip_address=139.159.99.250) 10/23/2018 09:36:06 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=abca9b3e-fe0f-30c0-a281-919222383c24, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:3c:f3, port_key=113822, portgroup_key=dvportgroup-48436, vlan_id=None), ip_address=unset) 10/23/2018 09:36:06 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113822 10/23/2018 09:36:06 AM [cvm.clients] [INFO]: Port: 113822 VLAN ID: 3 10/23/2018 09:36:06 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface abca9b3e-fe0f-30c0-a281-919222383c24 in VNC 10/23/2018 09:36:06 AM [cvm.clients] [INFO]: Virtual Machine Interface abca9b3e-fe0f-30c0-a281-919222383c24 not found in VNC - creating 10/23/2018 09:36:06 AM [cvm.clients] [INFO]: Virtual Machine Interface abca9b3e-fe0f-30c0-a281-919222383c24 created in VNC 10/23/2018 09:36:06 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-mgmt_vCenter-17887276-ctest-pt_svm0-85388690 10/23/2018 09:36:06 AM [cvm.clients] [INFO]: Created Instance IP: 55196a97-3c14-32ab-a5be-3c6662ea1626 10/23/2018 09:36:06 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-mgmt_vCenter-17887276-ctest-pt_svm0-85388690 10/23/2018 09:36:06 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=abca9b3e-fe0f-30c0-a281-919222383c24, display_name=vmi-ctest-mgmt_vCenter-17887276-ctest-pt_svm0-85388690, vcenter_port=VCenterPort(mac_address=00:50:56:a4:3c:f3, port_key=113822, portgroup_key=dvportgroup-48436, vlan_id=3), ip_address=158.166.154.251) 10/23/2018 09:36:06 AM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: a65fe91d-b220-3908-ae5f-84baf726dfa4 10/23/2018 09:36:06 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: a65fe91d-b220-3908-ae5f-84baf726dfa4 10/23/2018 09:36:06 AM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': 'a65fe91d-b220-3908-ae5f-84baf726dfa4', 'display_name': 'ctest-pt_svm0-85388690', 'rx_vlan': 2, 'vn_id': '58e6d3ef-7761-4892-990a-7ba924faf886', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 2, 'port_type': 2, 'mac_address': '00:50:56:a4:45:58', 'interface_name': 'a65fe91d-b220-3908-ae5f-84baf726dfa4', 'ip_address': u'26.38.206.250', 'vm_uuid_str': '50248f70-ef18-e22c-d7c8-e3bd240208eb'} 10/23/2018 09:36:06 AM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: 0ffeaa2f-9931-3289-8465-82525b36f74d 10/23/2018 09:36:07 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 0ffeaa2f-9931-3289-8465-82525b36f74d 10/23/2018 09:36:07 AM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': '0ffeaa2f-9931-3289-8465-82525b36f74d', 'display_name': 'ctest-pt_svm0-85388690', 'rx_vlan': 4, 'vn_id': 'fc727d3b-e02c-4edf-bd06-7a340cbe9e6d', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 4, 'port_type': 2, 'mac_address': '00:50:56:a4:13:91', 'interface_name': '0ffeaa2f-9931-3289-8465-82525b36f74d', 'ip_address': u'139.159.99.250', 'vm_uuid_str': '50248f70-ef18-e22c-d7c8-e3bd240208eb'} 10/23/2018 09:36:07 AM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: abca9b3e-fe0f-30c0-a281-919222383c24 10/23/2018 09:36:07 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: abca9b3e-fe0f-30c0-a281-919222383c24 10/23/2018 09:36:07 AM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': 'abca9b3e-fe0f-30c0-a281-919222383c24', 'display_name': 'ctest-pt_svm0-85388690', 'rx_vlan': 3, 'vn_id': '1c15b657-7185-41be-b70e-c6cf1a4e8b3a', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 3, 'port_type': 2, 'mac_address': '00:50:56:a4:3c:f3', 'interface_name': 'abca9b3e-fe0f-30c0-a281-919222383c24', 'ip_address': u'158.166.154.251', 'vm_uuid_str': '50248f70-ef18-e22c-d7c8-e3bd240208eb'} 10/23/2018 09:36:07 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: a65fe91d-b220-3908-ae5f-84baf726dfa4 10/23/2018 09:36:07 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 0ffeaa2f-9931-3289-8465-82525b36f74d 10/23/2018 09:36:08 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: abca9b3e-fe0f-30c0-a281-919222383c24 10/23/2018 09:36:08 AM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 09:36:08 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=a65fe91d-b220-3908-ae5f-84baf726dfa4, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:45:58, port_key=114077, portgroup_key=dvportgroup-48438, vlan_id=None), ip_address=unset) 10/23/2018 09:36:08 AM [cvm.clients] [INFO]: Reading VLAN ID of port 114077 10/23/2018 09:36:08 AM [cvm.clients] [INFO]: Port: 114077 VLAN ID: 2 10/23/2018 09:36:08 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface a65fe91d-b220-3908-ae5f-84baf726dfa4 in VNC 10/23/2018 09:36:08 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface a65fe91d-b220-3908-ae5f-84baf726dfa4 10/23/2018 09:36:08 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface a65fe91d-b220-3908-ae5f-84baf726dfa4 to ctest-right_vCenter-70204404 10/23/2018 09:36:08 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-right_vCenter-70204404-ctest-pt_svm0-85388690 10/23/2018 09:36:08 AM [cvm.clients] [INFO]: Created Instance IP: 32226e8f-1c1f-37ce-9a61-fd76057b2853 10/23/2018 09:36:08 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-right_vCenter-70204404-ctest-pt_svm0-85388690 10/23/2018 09:36:08 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=a65fe91d-b220-3908-ae5f-84baf726dfa4, display_name=vmi-ctest-right_vCenter-70204404-ctest-pt_svm0-85388690, vcenter_port=VCenterPort(mac_address=00:50:56:a4:45:58, port_key=114077, portgroup_key=dvportgroup-48438, vlan_id=2), ip_address=26.38.206.250) 10/23/2018 09:36:08 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=0ffeaa2f-9931-3289-8465-82525b36f74d, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:13:91, port_key=113949, portgroup_key=dvportgroup-48437, vlan_id=None), ip_address=unset) 10/23/2018 09:36:08 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113949 10/23/2018 09:36:08 AM [cvm.clients] [INFO]: Port: 113949 VLAN ID: 4 10/23/2018 09:36:08 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 0ffeaa2f-9931-3289-8465-82525b36f74d in VNC 10/23/2018 09:36:08 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface 0ffeaa2f-9931-3289-8465-82525b36f74d 10/23/2018 09:36:08 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface 0ffeaa2f-9931-3289-8465-82525b36f74d to ctest-left_vCenter-74919226 10/23/2018 09:36:08 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-left_vCenter-74919226-ctest-pt_svm0-85388690 10/23/2018 09:36:08 AM [cvm.clients] [INFO]: Created Instance IP: 95dfb713-0ff4-3fe6-8299-38f20ba733d0 10/23/2018 09:36:09 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-left_vCenter-74919226-ctest-pt_svm0-85388690 10/23/2018 09:36:09 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=0ffeaa2f-9931-3289-8465-82525b36f74d, display_name=vmi-ctest-left_vCenter-74919226-ctest-pt_svm0-85388690, vcenter_port=VCenterPort(mac_address=00:50:56:a4:13:91, port_key=113949, portgroup_key=dvportgroup-48437, vlan_id=4), ip_address=139.159.99.250) 10/23/2018 09:36:09 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=abca9b3e-fe0f-30c0-a281-919222383c24, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:3c:f3, port_key=113822, portgroup_key=dvportgroup-48436, vlan_id=None), ip_address=unset) 10/23/2018 09:36:09 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113822 10/23/2018 09:36:09 AM [cvm.clients] [INFO]: Port: 113822 VLAN ID: 3 10/23/2018 09:36:09 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface abca9b3e-fe0f-30c0-a281-919222383c24 in VNC 10/23/2018 09:36:09 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface abca9b3e-fe0f-30c0-a281-919222383c24 10/23/2018 09:36:09 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface abca9b3e-fe0f-30c0-a281-919222383c24 to ctest-mgmt_vCenter-17887276 10/23/2018 09:36:09 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-mgmt_vCenter-17887276-ctest-pt_svm0-85388690 10/23/2018 09:36:09 AM [cvm.clients] [INFO]: Created Instance IP: 55196a97-3c14-32ab-a5be-3c6662ea1626 10/23/2018 09:36:09 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-mgmt_vCenter-17887276-ctest-pt_svm0-85388690 10/23/2018 09:36:09 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=abca9b3e-fe0f-30c0-a281-919222383c24, display_name=vmi-ctest-mgmt_vCenter-17887276-ctest-pt_svm0-85388690, vcenter_port=VCenterPort(mac_address=00:50:56:a4:3c:f3, port_key=113822, portgroup_key=dvportgroup-48436, vlan_id=3), ip_address=158.166.154.251) 10/23/2018 09:36:09 AM [cvm.clients] [INFO]: Read vRouter port with uuid: a65fe91d-b220-3908-ae5f-84baf726dfa4, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:36:09.425162', u'dns-server': u'26.38.206.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:45:58', u'rx-vlan-id': 2, u'tx-vlan-id': 2, u'gateway': u'26.38.206.254', u'system-name': u'a65fe91d-b220-3908-ae5f-84baf726dfa4', u'ip-address': u'26.38.206.250', u'vn-id': u'58e6d3ef-7761-4892-990a-7ba924faf886', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'a65fe91d-b220-3908-ae5f-84baf726dfa4', u'instance-id': u'50248f70-ef18-e22c-d7c8-e3bd240208eb'} 10/23/2018 09:36:09 AM [cvm.clients] [INFO]: Read vRouter port with uuid: 0ffeaa2f-9931-3289-8465-82525b36f74d, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:36:09.430281', u'dns-server': u'139.159.99.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:13:91', u'rx-vlan-id': 4, u'tx-vlan-id': 4, u'gateway': u'139.159.99.254', u'system-name': u'0ffeaa2f-9931-3289-8465-82525b36f74d', u'ip-address': u'139.159.99.250', u'vn-id': u'fc727d3b-e02c-4edf-bd06-7a340cbe9e6d', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'0ffeaa2f-9931-3289-8465-82525b36f74d', u'instance-id': u'50248f70-ef18-e22c-d7c8-e3bd240208eb'} 10/23/2018 09:36:09 AM [cvm.clients] [INFO]: Read vRouter port with uuid: abca9b3e-fe0f-30c0-a281-919222383c24, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:36:09.435581', u'dns-server': u'158.166.154.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:3c:f3', u'rx-vlan-id': 3, u'tx-vlan-id': 3, u'gateway': u'158.166.154.254', u'system-name': u'abca9b3e-fe0f-30c0-a281-919222383c24', u'ip-address': u'158.166.154.251', u'vn-id': u'1c15b657-7185-41be-b70e-c6cf1a4e8b3a', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'abca9b3e-fe0f-30c0-a281-919222383c24', u'instance-id': u'50248f70-ef18-e22c-d7c8-e3bd240208eb'} 10/23/2018 09:36:09 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: a65fe91d-b220-3908-ae5f-84baf726dfa4 10/23/2018 09:36:09 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 0ffeaa2f-9931-3289-8465-82525b36f74d 10/23/2018 09:36:09 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: abca9b3e-fe0f-30c0-a281-919222383c24 10/23/2018 09:36:09 AM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 09:36:10 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=a65fe91d-b220-3908-ae5f-84baf726dfa4, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:45:58, port_key=114077, portgroup_key=dvportgroup-48438, vlan_id=None), ip_address=unset) 10/23/2018 09:36:10 AM [cvm.clients] [INFO]: Reading VLAN ID of port 114077 10/23/2018 09:36:10 AM [cvm.clients] [INFO]: Port: 114077 VLAN ID: 2 10/23/2018 09:36:10 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface a65fe91d-b220-3908-ae5f-84baf726dfa4 in VNC 10/23/2018 09:36:10 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface a65fe91d-b220-3908-ae5f-84baf726dfa4 10/23/2018 09:36:10 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface a65fe91d-b220-3908-ae5f-84baf726dfa4 to ctest-right_vCenter-70204404 10/23/2018 09:36:10 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-right_vCenter-70204404-ctest-pt_svm0-85388690 10/23/2018 09:36:10 AM [cvm.clients] [INFO]: Created Instance IP: 32226e8f-1c1f-37ce-9a61-fd76057b2853 10/23/2018 09:36:10 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-right_vCenter-70204404-ctest-pt_svm0-85388690 10/23/2018 09:36:10 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=a65fe91d-b220-3908-ae5f-84baf726dfa4, display_name=vmi-ctest-right_vCenter-70204404-ctest-pt_svm0-85388690, vcenter_port=VCenterPort(mac_address=00:50:56:a4:45:58, port_key=114077, portgroup_key=dvportgroup-48438, vlan_id=2), ip_address=26.38.206.250) 10/23/2018 09:36:10 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=0ffeaa2f-9931-3289-8465-82525b36f74d, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:13:91, port_key=113949, portgroup_key=dvportgroup-48437, vlan_id=None), ip_address=unset) 10/23/2018 09:36:10 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113949 10/23/2018 09:36:10 AM [cvm.clients] [INFO]: Port: 113949 VLAN ID: 4 10/23/2018 09:36:10 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 0ffeaa2f-9931-3289-8465-82525b36f74d in VNC 10/23/2018 09:36:10 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface 0ffeaa2f-9931-3289-8465-82525b36f74d 10/23/2018 09:36:10 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface 0ffeaa2f-9931-3289-8465-82525b36f74d to ctest-left_vCenter-74919226 10/23/2018 09:36:10 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-left_vCenter-74919226-ctest-pt_svm0-85388690 10/23/2018 09:36:10 AM [cvm.clients] [INFO]: Created Instance IP: 95dfb713-0ff4-3fe6-8299-38f20ba733d0 10/23/2018 09:36:10 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-left_vCenter-74919226-ctest-pt_svm0-85388690 10/23/2018 09:36:10 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=0ffeaa2f-9931-3289-8465-82525b36f74d, display_name=vmi-ctest-left_vCenter-74919226-ctest-pt_svm0-85388690, vcenter_port=VCenterPort(mac_address=00:50:56:a4:13:91, port_key=113949, portgroup_key=dvportgroup-48437, vlan_id=4), ip_address=139.159.99.250) 10/23/2018 09:36:10 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=abca9b3e-fe0f-30c0-a281-919222383c24, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:3c:f3, port_key=113822, portgroup_key=dvportgroup-48436, vlan_id=None), ip_address=unset) 10/23/2018 09:36:11 AM [cvm.clients] [INFO]: Reading VLAN ID of port 113822 10/23/2018 09:36:11 AM [cvm.clients] [INFO]: Port: 113822 VLAN ID: 3 10/23/2018 09:36:11 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface abca9b3e-fe0f-30c0-a281-919222383c24 in VNC 10/23/2018 09:36:11 AM [cvm.clients] [INFO]: Deleting old Instance IP for Interface abca9b3e-fe0f-30c0-a281-919222383c24 10/23/2018 09:36:11 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface abca9b3e-fe0f-30c0-a281-919222383c24 to ctest-mgmt_vCenter-17887276 10/23/2018 09:36:11 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-mgmt_vCenter-17887276-ctest-pt_svm0-85388690 10/23/2018 09:36:11 AM [cvm.clients] [INFO]: Created Instance IP: 55196a97-3c14-32ab-a5be-3c6662ea1626 10/23/2018 09:36:11 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-mgmt_vCenter-17887276-ctest-pt_svm0-85388690 10/23/2018 09:36:11 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=abca9b3e-fe0f-30c0-a281-919222383c24, display_name=vmi-ctest-mgmt_vCenter-17887276-ctest-pt_svm0-85388690, vcenter_port=VCenterPort(mac_address=00:50:56:a4:3c:f3, port_key=113822, portgroup_key=dvportgroup-48436, vlan_id=3), ip_address=158.166.154.251) 10/23/2018 09:36:11 AM [cvm.clients] [INFO]: Read vRouter port with uuid: a65fe91d-b220-3908-ae5f-84baf726dfa4, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:36:11.260253', u'dns-server': u'26.38.206.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:45:58', u'rx-vlan-id': 2, u'tx-vlan-id': 2, u'gateway': u'26.38.206.254', u'system-name': u'a65fe91d-b220-3908-ae5f-84baf726dfa4', u'ip-address': u'26.38.206.250', u'vn-id': u'58e6d3ef-7761-4892-990a-7ba924faf886', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'a65fe91d-b220-3908-ae5f-84baf726dfa4', u'instance-id': u'50248f70-ef18-e22c-d7c8-e3bd240208eb'} 10/23/2018 09:36:11 AM [cvm.clients] [INFO]: Read vRouter port with uuid: 0ffeaa2f-9931-3289-8465-82525b36f74d, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:36:11.266100', u'dns-server': u'139.159.99.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:13:91', u'rx-vlan-id': 4, u'tx-vlan-id': 4, u'gateway': u'139.159.99.254', u'system-name': u'0ffeaa2f-9931-3289-8465-82525b36f74d', u'ip-address': u'139.159.99.250', u'vn-id': u'fc727d3b-e02c-4edf-bd06-7a340cbe9e6d', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'0ffeaa2f-9931-3289-8465-82525b36f74d', u'instance-id': u'50248f70-ef18-e22c-d7c8-e3bd240208eb'} 10/23/2018 09:36:11 AM [cvm.clients] [INFO]: Read vRouter port with uuid: abca9b3e-fe0f-30c0-a281-919222383c24, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427857:36:11.270816', u'dns-server': u'158.166.154.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:3c:f3', u'rx-vlan-id': 3, u'tx-vlan-id': 3, u'gateway': u'158.166.154.254', u'system-name': u'abca9b3e-fe0f-30c0-a281-919222383c24', u'ip-address': u'158.166.154.251', u'vn-id': u'1c15b657-7185-41be-b70e-c6cf1a4e8b3a', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'abca9b3e-fe0f-30c0-a281-919222383c24', u'instance-id': u'50248f70-ef18-e22c-d7c8-e3bd240208eb'} 10/23/2018 09:36:11 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: a65fe91d-b220-3908-ae5f-84baf726dfa4 10/23/2018 09:36:11 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 0ffeaa2f-9931-3289-8465-82525b36f74d 10/23/2018 09:36:11 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: abca9b3e-fe0f-30c0-a281-919222383c24 10/23/2018 09:36:11 AM [cvm.services] [INFO]: VMware tools on VM ctest-pt_svm0-85388690 are running 10/23/2018 09:36:11 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-85388690 10/23/2018 09:36:11 AM [cvm.services] [INFO]: VM ctest-pt_svm0-85388690 was powered on 10/23/2018 09:36:11 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-85388690 10/23/2018 09:36:12 AM [cvm.clients] [INFO]: Enabled vRouter port with uuid: abca9b3e-fe0f-30c0-a281-919222383c24 10/23/2018 09:36:12 AM [cvm.clients] [INFO]: Enabled vRouter port with uuid: 0ffeaa2f-9931-3289-8465-82525b36f74d 10/23/2018 09:36:12 AM [cvm.clients] [INFO]: Enabled vRouter port with uuid: a65fe91d-b220-3908-ae5f-84baf726dfa4 10/23/2018 09:39:37 AM [cvm.services] [INFO]: VMware tools on VM ctest-pt_svm0-85388690 are not running 10/23/2018 09:39:37 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-85388690 10/23/2018 09:39:37 AM [cvm.services] [INFO]: VM ctest-pt_svm0-85388690 was powered off 10/23/2018 09:39:37 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-pt_svm0-85388690 10/23/2018 09:39:37 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: abca9b3e-fe0f-30c0-a281-919222383c24 10/23/2018 09:39:37 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 0ffeaa2f-9931-3289-8465-82525b36f74d 10/23/2018 09:39:38 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: a65fe91d-b220-3908-ae5f-84baf726dfa4 10/23/2018 09:39:39 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 09:39:39 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 171, in main gevent.spawn(vmware_monitor.start()), File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 13, in start self._controller.handle_update(update_set) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 31, in handle_update self._update_handler.handle_update(update_set) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 43, in handle_update handler.handle_change(object_update.obj, property_change) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 54, in handle_change self._handle_change(obj, value) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 89, in _handle_change self._handle_event(value) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 216, in _handle_event self._vmi_service.remove_vmis_for_vm_model(vm_name) File "/usr/lib/python2.7/site-packages/cvm/services.py", line 177, in remove_vmis_for_vm_model full_remove = self._vcenter_api_client.can_remove_vm(vm_model.uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 574, in __call__ return self.f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 394, in _InvokeAccessor return self._stub.InvokeAccessor(self, info) File "/usr/lib/python2.7/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor options=self._pcType.RetrieveOptions(maxObjects=1)) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 580, in self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 386, in _InvokeMethod return self._stub.InvokeMethod(self, info, args) File "/usr/lib/python2.7/site-packages/pyVmomi/SoapAdapter.py", line 1370, in InvokeMethod raise obj # pylint: disable-msg=E0702 vmodl.fault.ManagedObjectNotFound: (vmodl.fault.ManagedObjectNotFound) { dynamicType = , dynamicProperty = (vmodl.DynamicProperty) [], msg = "The object 'vim.VirtualMachine:vm-48444' has already been deleted or has not been completely created", faultCause = , faultMessage = (vmodl.LocalizableMessage) [], obj = 'vim.VirtualMachine:vm-48444' } 10/23/2018 09:39:41 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 09:39:41 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 09:39:41 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 09:39:41 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 09:39:41 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 09:39:41 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 09:39:41 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 09:39:41 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 09:39:41 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 09:39:41 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540287581580726 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.140:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 09:39:41 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 09:39:41 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 09:39:42 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 09:39:42 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48436 name: ctest-mgmt_vCenter-17887276 10/23/2018 09:39:42 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-mgmt_vCenter-17887276 already allowed. 10/23/2018 09:39:42 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-mgmt_vCenter-17887276 10/23/2018 09:39:42 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=1c15b657-7185-41be-b70e-c6cf1a4e8b3a, key=dvportgroup-48436, name=ctest-mgmt_vCenter-17887276) 10/23/2018 09:39:42 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48437 name: ctest-left_vCenter-74919226 10/23/2018 09:39:42 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-left_vCenter-74919226 already allowed. 10/23/2018 09:39:42 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-left_vCenter-74919226 10/23/2018 09:39:42 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=fc727d3b-e02c-4edf-bd06-7a340cbe9e6d, key=dvportgroup-48437, name=ctest-left_vCenter-74919226) 10/23/2018 09:39:42 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48438 name: ctest-right_vCenter-70204404 10/23/2018 09:39:42 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-right_vCenter-70204404 already allowed. 10/23/2018 09:39:42 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-right_vCenter-70204404 10/23/2018 09:39:42 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=58e6d3ef-7761-4892-990a-7ba924faf886, key=dvportgroup-48438, name=ctest-right_vCenter-70204404) 10/23/2018 09:39:45 AM [cvm.services] [INFO]: Deleting 034d88dd-8fbc-3083-bd58-5eb50f2345f6 from VNC. 10/23/2018 09:39:45 AM [cvm.clients] [INFO]: Virtual Machine Interface 034d88dd-8fbc-3083-bd58-5eb50f2345f6 removed from VNC 10/23/2018 09:39:45 AM [cvm] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/23/2018 09:39:45 AM [cvm] [INFO]: Session Event: TCP Connected 10/23/2018 09:39:45 AM [cvm] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/23/2018 09:39:45 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/23/2018 09:39:45 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ 10.204.217.139:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector connection down >>, ] >> 10/23/2018 09:39:45 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/23/2018 09:39:45 AM [cvm] [INFO]: Processing event[EvTcpConnected] in state[Connect] 10/23/2018 09:39:45 AM [cvm] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 10/23/2018 09:39:45 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 09:39:45 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 09:39:45 AM [cvm] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 10/23/2018 09:39:45 AM [cvm] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 10/23/2018 09:39:45 AM [cvm] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 10/23/2018 09:39:45 AM [cvm] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 10/23/2018 09:39:45 AM [cvm.services] [INFO]: Deleting bf8b1eb1-f085-3fc6-90ee-c8d7680da0d9 from VNC. 10/23/2018 09:39:46 AM [cvm.clients] [INFO]: Virtual Machine Interface bf8b1eb1-f085-3fc6-90ee-c8d7680da0d9 removed from VNC 10/23/2018 09:39:46 AM [cvm.services] [INFO]: Deleting 5d4e119f-1f1f-3b81-a594-ea5ecc21bdc7 from VNC. 10/23/2018 09:39:46 AM [cvm.clients] [INFO]: Virtual Machine Interface 5d4e119f-1f1f-3b81-a594-ea5ecc21bdc7 removed from VNC 10/23/2018 09:39:47 AM [cvm.services] [INFO]: Deleting a65fe91d-b220-3908-ae5f-84baf726dfa4 from VNC. 10/23/2018 09:39:47 AM [cvm.clients] [INFO]: Virtual Machine Interface a65fe91d-b220-3908-ae5f-84baf726dfa4 removed from VNC 10/23/2018 09:39:48 AM [cvm.services] [INFO]: Deleting 0ffeaa2f-9931-3289-8465-82525b36f74d from VNC. 10/23/2018 09:39:48 AM [cvm.clients] [INFO]: Virtual Machine Interface 0ffeaa2f-9931-3289-8465-82525b36f74d removed from VNC 10/23/2018 09:39:49 AM [cvm.services] [INFO]: Deleting abca9b3e-fe0f-30c0-a281-919222383c24 from VNC. 10/23/2018 09:39:49 AM [cvm.clients] [INFO]: Virtual Machine Interface abca9b3e-fe0f-30c0-a281-919222383c24 removed from VNC 10/23/2018 09:39:49 AM [cvm.database] [INFO]: Could not find VM model with uuid 50248f70-ef18-e22c-d7c8-e3bd240208eb. 10/23/2018 09:39:50 AM [cvm.services] [INFO]: Deleting 50248f70-ef18-e22c-d7c8-e3bd240208eb from VNC 10/23/2018 09:39:50 AM [cvm.clients] [INFO]: Attempting to delete Virtual Machine 50248f70-ef18-e22c-d7c8-e3bd240208eb from VNC... 10/23/2018 09:39:50 AM [cvm.clients] [INFO]: Virtual Machine 50248f70-ef18-e22c-d7c8-e3bd240208eb removed from VNC 10/23/2018 09:39:50 AM [cvm.database] [INFO]: Could not find VM model with uuid 502439ff-e7eb-3203-0595-f798b0fab1de. 10/23/2018 09:39:51 AM [cvm.database] [INFO]: Could not find VM model with uuid 50240e38-d616-6f88-db83-7b617ced59a2. 10/23/2018 09:39:51 AM [cvm.services] [INFO]: Deleting 50240e38-d616-6f88-db83-7b617ced59a2 from VNC 10/23/2018 09:39:51 AM [cvm.clients] [INFO]: Attempting to delete Virtual Machine 50240e38-d616-6f88-db83-7b617ced59a2 from VNC... 10/23/2018 09:39:52 AM [cvm.clients] [INFO]: Virtual Machine 50240e38-d616-6f88-db83-7b617ced59a2 removed from VNC 10/23/2018 09:39:52 AM [cvm.database] [INFO]: Could not find VM model with uuid 50248132-39f7-d856-0080-234e60aad0fb. 10/23/2018 09:39:52 AM [cvm.database] [INFO]: Could not find VM model with uuid 50242a1d-212e-3235-01f3-63d057df0c79. 10/23/2018 09:39:53 AM [cvm.database] [INFO]: Could not find VM model with uuid 50265637-8829-fc50-89db-d80f49f1904a. 10/23/2018 09:39:54 AM [cvm.database] [INFO]: Could not find VM model with uuid 50244889-da8f-820c-75e6-5df4f8fe217e. 10/23/2018 09:39:55 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: bd1cd380-f4d9-3641-80ef-a6b5c86dfaad 10/23/2018 09:39:55 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 00021acc-c7f1-3f75-8b3d-fa01fb9610b3 10/23/2018 09:39:55 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 034d88dd-8fbc-3083-bd58-5eb50f2345f6 10/23/2018 09:39:56 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: bf8b1eb1-f085-3fc6-90ee-c8d7680da0d9 10/23/2018 09:39:56 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 5d4e119f-1f1f-3b81-a594-ea5ecc21bdc7 10/23/2018 09:39:56 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: a65fe91d-b220-3908-ae5f-84baf726dfa4 10/23/2018 09:39:56 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 0ffeaa2f-9931-3289-8465-82525b36f74d 10/23/2018 09:39:56 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: abca9b3e-fe0f-30c0-a281-919222383c24 10/23/2018 09:39:56 AM [cvm.controllers] [INFO]: Synchronization complete 10/23/2018 09:59:17 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 09:59:17 AM [cvm.clients] [INFO]: Read from ESXi API ctest-vCenter-20435430 properties: {u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '5024cd1c-893c-bcc0-04c1-d5a7fa6665de', u'runtime.powerState': 'poweredOff', u'name': 'ctest-vCenter-20435430', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 09:59:17 AM [cvm.database] [INFO]: Could not find VM model with uuid 5024cd1c-893c-bcc0-04c1-d5a7fa6665de. 10/23/2018 09:59:17 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine 5024cd1c-893c-bcc0-04c1-d5a7fa6665de in VNC 10/23/2018 09:59:17 AM [cvm.clients] [INFO]: Virtual Machine 5024cd1c-893c-bcc0-04c1-d5a7fa6665de not found in VNC - creating 10/23/2018 09:59:17 AM [cvm.clients] [INFO]: Virtual Machine 5024cd1c-893c-bcc0-04c1-d5a7fa6665de created in VNC 10/23/2018 09:59:17 AM [cvm.services] [INFO]: Created VirtualMachineModel(uuid=5024cd1c-893c-bcc0-04c1-d5a7fa6665de, name=ctest-vCenter-20435430, vm_properties={u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '5024cd1c-893c-bcc0-04c1-d5a7fa6665de', u'runtime.powerState': 'poweredOff', u'name': 'ctest-vCenter-20435430', u'summary.runtime.host': 'vim.HostSystem:ha-host'}, interfaces=[]) 10/23/2018 09:59:17 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-vCenter-20435430 10/23/2018 09:59:18 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 09:59:18 AM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 09:59:18 AM [cvm.database] [INFO]: Could not find VN model with key dvportgroup-48454. 10/23/2018 09:59:18 AM [cvm.services] [INFO]: Fetching new portgroup for key: dvportgroup-48454 10/23/2018 09:59:18 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 09:59:18 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 09:59:18 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 09:59:18 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 09:59:18 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=2adec358-0f80-3d02-9859-fc3c245dcf31, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:1c:8b, port_key=114721, portgroup_key=dvportgroup-48454, vlan_id=None), ip_address=unset) 10/23/2018 09:59:18 AM [cvm.clients] [INFO]: Reading VLAN ID of port 114721 10/23/2018 09:59:18 AM [cvm.clients] [INFO]: Port: 114721 has no VLAN ID 10/23/2018 09:59:21 AM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 114721 to 1 10/23/2018 09:59:21 AM [cvm.clients] [INFO]: Successfully set VLAN ID: 1 for port: 114721 10/23/2018 09:59:21 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 2adec358-0f80-3d02-9859-fc3c245dcf31 in VNC 10/23/2018 09:59:22 AM [cvm.clients] [INFO]: Virtual Machine Interface 2adec358-0f80-3d02-9859-fc3c245dcf31 not found in VNC - creating 10/23/2018 09:59:22 AM [cvm.clients] [INFO]: Virtual Machine Interface 2adec358-0f80-3d02-9859-fc3c245dcf31 created in VNC 10/23/2018 09:59:22 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-vn-50026300-ctest-vCenter-20435430 10/23/2018 09:59:22 AM [cvm.clients] [INFO]: Created Instance IP: 98dd9edb-c6ec-37f5-8301-0eaaf86a53de 10/23/2018 09:59:22 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-vn-50026300-ctest-vCenter-20435430 10/23/2018 09:59:22 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=2adec358-0f80-3d02-9859-fc3c245dcf31, display_name=vmi-ctest-vn-50026300-ctest-vCenter-20435430, vcenter_port=VCenterPort(mac_address=00:50:56:a4:1c:8b, port_key=114721, portgroup_key=dvportgroup-48454, vlan_id=1), ip_address=92.229.130.251) 10/23/2018 09:59:22 AM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: 2adec358-0f80-3d02-9859-fc3c245dcf31 10/23/2018 09:59:22 AM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 2adec358-0f80-3d02-9859-fc3c245dcf31 10/23/2018 09:59:22 AM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': '2adec358-0f80-3d02-9859-fc3c245dcf31', 'display_name': 'ctest-vCenter-20435430', 'rx_vlan': 1, 'vn_id': 'ba1a71f8-6194-44a9-ab11-5b3c17cd3cca', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 1, 'port_type': 2, 'mac_address': '00:50:56:a4:1c:8b', 'interface_name': '2adec358-0f80-3d02-9859-fc3c245dcf31', 'ip_address': u'92.229.130.251', 'vm_uuid_str': '5024cd1c-893c-bcc0-04c1-d5a7fa6665de'} 10/23/2018 09:59:22 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 2adec358-0f80-3d02-9859-fc3c245dcf31 10/23/2018 09:59:22 AM [cvm.services] [INFO]: VM ctest-vCenter-20435430 was powered on 10/23/2018 09:59:22 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-vCenter-20435430 10/23/2018 09:59:22 AM [cvm.clients] [INFO]: Enabled vRouter port with uuid: 2adec358-0f80-3d02-9859-fc3c245dcf31 10/23/2018 10:01:29 AM [cvm.services] [INFO]: VMware tools on VM ctest-vCenter-20435430 are running 10/23/2018 10:01:29 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-vCenter-20435430 10/23/2018 10:09:30 AM [cvm.services] [INFO]: VMware tools on VM ctest-vCenter-20435430 are not running 10/23/2018 10:09:30 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-vCenter-20435430 10/23/2018 10:09:30 AM [cvm.services] [INFO]: VM ctest-vCenter-20435430 was powered off 10/23/2018 10:09:30 AM [cvm.database] [INFO]: Saved Virtual Machine model for ctest-vCenter-20435430 10/23/2018 10:09:30 AM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 2adec358-0f80-3d02-9859-fc3c245dcf31 10/23/2018 10:09:32 AM [cvm.controllers] [INFO]: Detected event: 10/23/2018 10:09:32 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 171, in main gevent.spawn(vmware_monitor.start()), File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 13, in start self._controller.handle_update(update_set) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 31, in handle_update self._update_handler.handle_update(update_set) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 43, in handle_update handler.handle_change(object_update.obj, property_change) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 54, in handle_change self._handle_change(obj, value) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 89, in _handle_change self._handle_event(value) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 216, in _handle_event self._vmi_service.remove_vmis_for_vm_model(vm_name) File "/usr/lib/python2.7/site-packages/cvm/services.py", line 177, in remove_vmis_for_vm_model full_remove = self._vcenter_api_client.can_remove_vm(vm_model.uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 574, in __call__ return self.f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 394, in _InvokeAccessor return self._stub.InvokeAccessor(self, info) File "/usr/lib/python2.7/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor options=self._pcType.RetrieveOptions(maxObjects=1)) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 580, in self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 386, in _InvokeMethod return self._stub.InvokeMethod(self, info, args) File "/usr/lib/python2.7/site-packages/pyVmomi/SoapAdapter.py", line 1370, in InvokeMethod raise obj # pylint: disable-msg=E0702 vmodl.fault.ManagedObjectNotFound: (vmodl.fault.ManagedObjectNotFound) { dynamicType = , dynamicProperty = (vmodl.DynamicProperty) [], msg = "The object 'vim.VirtualMachine:vm-48456' has already been deleted or has not been completely created", faultCause = , faultMessage = (vmodl.LocalizableMessage) [], obj = 'vim.VirtualMachine:vm-48456' } 10/23/2018 10:09:34 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:09:34 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:09:34 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:09:34 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:09:34 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:09:34 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:09:34 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:09:34 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:09:34 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:09:34 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540289374670045 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.140:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:09:34 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:09:34 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:09:35 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:09:35 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:09:35 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:09:35 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:09:35 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:09:35 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:09:35 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:09:35 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:09:35 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:09:35 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:09:35 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:09:35 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:09:35 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:09:35 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:09:35 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:09:35 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:09:35 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:09:37 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 574, in __call__ return self.f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 394, in _InvokeAccessor return self._stub.InvokeAccessor(self, info) File "/usr/lib/python2.7/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor options=self._pcType.RetrieveOptions(maxObjects=1)) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 580, in self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 386, in _InvokeMethod return self._stub.InvokeMethod(self, info, args) File "/usr/lib/python2.7/site-packages/pyVmomi/SoapAdapter.py", line 1370, in InvokeMethod raise obj # pylint: disable-msg=E0702 vmodl.fault.ManagedObjectNotFound: (vmodl.fault.ManagedObjectNotFound) { dynamicType = , dynamicProperty = (vmodl.DynamicProperty) [], msg = "The object 'vim.VirtualMachine:vm-48455' has already been deleted or has not been completely created", faultCause = , faultMessage = (vmodl.LocalizableMessage) [], obj = 'vim.VirtualMachine:vm-48455' } 10/23/2018 10:09:38 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:09:38 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:09:38 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:09:38 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:09:38 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:09:38 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:09:38 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:09:38 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:09:38 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:09:38 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540289378822092 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.139:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:09:38 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:09:39 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:09:39 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:09:39 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:09:39 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:09:39 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:09:39 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:09:39 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:09:39 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:09:39 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:09:39 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:09:39 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:09:39 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:09:39 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:09:39 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:09:39 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:09:39 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:09:39 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:09:39 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:09:41 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:09:41 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:09:43 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:09:43 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:09:43 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:09:43 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:09:43 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:09:43 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:09:43 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:09:43 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:09:43 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:09:43 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540289383472337 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.140:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:09:43 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:09:43 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:09:43 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:09:44 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:09:44 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:09:44 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:09:44 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:09:44 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:09:44 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:09:44 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:09:44 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:09:44 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:09:44 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:09:44 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:09:44 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:09:44 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:09:44 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:09:44 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:09:44 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:09:46 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:09:46 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:09:48 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:09:48 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:09:48 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:09:48 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:09:48 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:09:48 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:09:48 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:09:48 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:09:48 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:09:48 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540289388612319 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.140:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:09:48 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:09:48 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:09:49 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:09:49 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:09:49 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:09:49 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:09:49 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:09:49 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:09:49 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:09:49 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:09:49 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:09:49 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:09:49 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:09:49 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:09:49 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:09:49 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:09:49 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:09:49 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:09:49 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:09:51 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:09:51 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:09:54 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:09:54 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:09:54 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:09:54 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:09:54 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:09:54 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:09:54 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:09:54 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:09:54 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:09:54 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540289394505537 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.140:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:09:54 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:09:54 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:09:54 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:09:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:09:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:09:55 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:09:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:09:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:09:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:09:55 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:09:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:09:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:09:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:09:55 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:09:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:09:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:09:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:09:55 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:09:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:09:57 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:09:57 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.144:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:10:01 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:10:01 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:10:01 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:10:01 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:10:01 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:10:01 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:10:01 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:10:01 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:10:01 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:10:01 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540289401975121 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.139:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:10:01 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:10:02 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:10:02 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:10:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:10:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:10:02 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:10:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:10:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:10:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:10:02 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:10:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:10:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:10:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:10:02 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:10:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:10:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:10:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:10:02 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:10:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:10:04 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:10:04 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:10:12 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:10:12 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:10:12 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:10:12 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:10:12 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:10:12 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:10:12 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:10:12 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:10:12 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:10:12 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540289412662503 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.144:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:10:12 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:10:12 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:10:13 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:10:13 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:10:13 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:10:13 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:10:13 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:10:13 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:10:13 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:10:13 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:10:13 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:10:13 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:10:13 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:10:13 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:10:13 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:10:13 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:10:13 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:10:13 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:10:13 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:10:15 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:10:15 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:10:29 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:10:29 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:10:29 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:10:29 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:10:29 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:10:29 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:10:29 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:10:29 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:10:29 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:10:29 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540289429548941 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.139:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:10:29 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:10:29 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:10:29 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:10:30 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:10:30 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:10:30 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:10:30 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:10:30 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:10:30 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:10:30 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:10:30 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:10:30 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:10:30 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:10:30 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:10:30 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:10:30 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:10:30 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:10:30 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:10:30 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:10:32 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:10:32 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:10:59 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:10:59 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:10:59 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:10:59 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:10:59 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:10:59 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:10:59 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:10:59 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:10:59 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:10:59 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540289459402204 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.139:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:10:59 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:10:59 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:10:59 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:11:00 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:11:00 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:11:00 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:11:00 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:11:00 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:11:00 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:11:00 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:11:00 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:11:00 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:11:00 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:11:00 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:11:00 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:11:00 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:11:00 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:11:00 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:11:00 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:11:01 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:11:01 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:11:54 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:11:54 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:11:54 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:11:54 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:11:54 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:11:54 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:11:54 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:11:54 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:11:54 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:11:54 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540289514185260 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.140:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:11:54 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:11:54 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:11:54 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:11:54 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:11:54 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:11:54 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:11:54 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:11:54 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:11:54 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:11:54 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:11:54 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:11:54 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:11:54 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:11:54 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:11:54 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:11:54 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:11:54 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:11:54 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:11:54 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:11:55 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:11:56 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.144:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:12:57 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:12:57 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:12:57 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:12:57 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:12:57 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:12:57 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:12:57 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:12:57 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:12:57 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:12:57 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540289577672771 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.140:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:12:57 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:12:57 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:12:58 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:12:58 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:12:58 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:12:58 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:12:58 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:12:58 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:12:58 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:12:58 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:12:58 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:12:58 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:12:58 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:12:58 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:12:58 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:12:58 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:12:58 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:12:58 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:12:58 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:12:58 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:12:58 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:12:58 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:12:58 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:12:58 AM [cvm.services] [ERROR]: Unable to fetch new portgroup for name: VN2 10/23/2018 10:12:59 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:12:59 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:14:01 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:14:01 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:14:01 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:14:01 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:14:01 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:14:01 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:14:01 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:14:01 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:14:01 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:14:01 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540289641314216 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.139:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:14:01 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:14:01 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:14:01 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:14:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:14:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:14:02 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:14:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:14:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:14:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:14:02 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:14:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:14:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:14:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:14:02 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:14:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:14:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:14:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:14:02 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:14:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:14:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:14:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:14:02 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:14:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:14:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:14:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:14:02 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:14:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:14:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:14:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:14:02 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:14:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:14:03 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:14:03 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.144:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:15:05 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:15:05 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:15:05 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:15:05 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:15:05 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:15:05 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:15:05 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:15:05 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:15:05 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:15:05 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540289705118089 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.140:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:15:05 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:15:05 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:15:05 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:15:05 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:15:05 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:15:05 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:15:05 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:15:05 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:15:05 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:15:05 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:15:05 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:15:05 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:15:05 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:15:05 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:15:05 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:15:05 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:15:05 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:15:05 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:15:05 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:15:06 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:15:06 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:15:06 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:15:06 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:15:06 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:15:06 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:15:06 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:15:06 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:15:06 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:15:06 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:15:06 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:15:06 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:15:07 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] AttributeError: 'NoneType' object has no attribute 'instanceUuid' 10/23/2018 10:16:08 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:16:08 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:16:08 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:16:08 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:16:08 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:16:08 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:16:08 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:16:08 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:16:08 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:16:08 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540289768774036 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.144:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:16:08 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:16:08 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:16:09 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:16:09 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:16:09 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:16:09 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:16:09 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:16:09 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:16:09 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:16:09 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:16:09 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:16:09 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:16:09 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:16:09 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:16:09 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:16:09 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:16:09 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:16:09 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:16:09 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:16:09 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:16:09 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:16:09 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:16:09 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:16:09 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:16:09 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:16:09 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:16:09 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:16:09 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:16:09 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:16:09 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:16:09 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:16:10 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] AttributeError: 'NoneType' object has no attribute 'instanceUuid' 10/23/2018 10:17:12 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:17:12 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:17:12 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:17:12 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:17:12 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:17:12 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:17:12 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:17:12 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:17:12 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:17:12 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540289832459705 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.139:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:17:12 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:17:12 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:17:12 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:17:13 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:17:13 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:17:13 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:17:13 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:17:13 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:17:13 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:17:13 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:17:13 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:17:13 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:17:13 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:17:13 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:17:13 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:17:13 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:17:13 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:17:13 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:17:13 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:17:13 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:17:13 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:17:13 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:17:13 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:17:13 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:17:13 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:17:13 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:17:13 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:17:13 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:17:13 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:17:13 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:17:13 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:17:14 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:17:14 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:18:16 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:18:16 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:18:16 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:18:16 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:18:16 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:18:16 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:18:16 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:18:16 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:18:16 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:18:16 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540289896669717 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.139:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:18:16 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:18:16 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:18:17 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:18:17 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:18:17 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:18:17 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:18:17 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:18:17 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:18:17 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:18:17 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:18:17 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:18:17 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:18:17 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:18:17 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:18:17 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:18:17 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:18:17 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:18:17 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:18:17 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:18:17 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:18:17 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:18:17 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:18:17 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:18:17 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:18:17 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:18:17 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:18:17 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:18:17 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:18:17 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:18:17 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:18:17 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:18:18 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] AttributeError: 'NoneType' object has no attribute 'instanceUuid' 10/23/2018 10:19:20 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:19:20 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:19:20 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:19:20 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:19:20 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:19:20 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:19:20 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:19:20 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:19:20 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:19:20 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540289960391175 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.139:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:19:20 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:19:20 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:19:20 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:19:20 AM [cvm.clients] [INFO]: Read from ESXi API VM31 properties: {u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '50248768-3ed2-8c5e-c524-1845afb8f952', u'runtime.powerState': 'poweredOff', u'name': 'VM31', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:19:20 AM [cvm.database] [INFO]: Could not find VM model with uuid 50248768-3ed2-8c5e-c524-1845afb8f952. 10/23/2018 10:19:20 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine 50248768-3ed2-8c5e-c524-1845afb8f952 in VNC 10/23/2018 10:19:21 AM [cvm.clients] [INFO]: Virtual Machine 50248768-3ed2-8c5e-c524-1845afb8f952 not found in VNC - creating 10/23/2018 10:19:21 AM [cvm.clients] [INFO]: Virtual Machine 50248768-3ed2-8c5e-c524-1845afb8f952 created in VNC 10/23/2018 10:19:21 AM [cvm.services] [INFO]: Created VirtualMachineModel(uuid=50248768-3ed2-8c5e-c524-1845afb8f952, name=VM31, vm_properties={u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '50248768-3ed2-8c5e-c524-1845afb8f952', u'runtime.powerState': 'poweredOff', u'name': 'VM31', u'summary.runtime.host': 'vim.HostSystem:ha-host'}, interfaces=['53a8aac3-a51d-3f43-a94c-075b16b95285']) 10/23/2018 10:19:21 AM [cvm.database] [INFO]: Saved Virtual Machine model for VM31 10/23/2018 10:19:21 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:19:21 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:19:21 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:19:21 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:19:21 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:19:21 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:19:21 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:19:21 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:19:21 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:19:21 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:19:21 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:19:21 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:19:21 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:19:21 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:19:21 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:19:21 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:19:21 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:19:21 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:19:21 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:19:21 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:19:21 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:19:21 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:19:21 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:19:21 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:19:21 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:19:21 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:19:21 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:19:21 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:19:21 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=53a8aac3-a51d-3f43-a94c-075b16b95285, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:a4:f2, port_key=115247, portgroup_key=dvportgroup-48462, vlan_id=None), ip_address=unset) 10/23/2018 10:19:21 AM [cvm.clients] [INFO]: Reading VLAN ID of port 115247 10/23/2018 10:19:21 AM [cvm.clients] [INFO]: Port: 115247 has no VLAN ID 10/23/2018 10:19:24 AM [cvm] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/23/2018 10:19:24 AM [cvm] [INFO]: Session Event: TCP Connected 10/23/2018 10:19:24 AM [cvm] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/23/2018 10:19:24 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/23/2018 10:19:24 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ 10.204.217.144:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:19:24 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/23/2018 10:19:24 AM [cvm] [INFO]: Processing event[EvTcpConnected] in state[Connect] 10/23/2018 10:19:24 AM [cvm] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 10/23/2018 10:19:24 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 10:19:24 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 10:19:24 AM [cvm] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 10/23/2018 10:19:24 AM [cvm] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 10/23/2018 10:19:24 AM [cvm] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 10/23/2018 10:19:24 AM [cvm] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 10/23/2018 10:19:24 AM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 115247 to 1 10/23/2018 10:19:24 AM [cvm.clients] [INFO]: Successfully set VLAN ID: 1 for port: 115247 10/23/2018 10:19:24 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 53a8aac3-a51d-3f43-a94c-075b16b95285 in VNC 10/23/2018 10:19:24 AM [cvm.clients] [INFO]: Virtual Machine Interface 53a8aac3-a51d-3f43-a94c-075b16b95285 not found in VNC - creating 10/23/2018 10:19:25 AM [cvm.clients] [INFO]: Virtual Machine Interface 53a8aac3-a51d-3f43-a94c-075b16b95285 created in VNC 10/23/2018 10:19:25 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-VN3-VM31 10/23/2018 10:19:25 AM [cvm.clients] [INFO]: Created Instance IP: a2da5b93-c8b5-3939-80a6-c00ac3881ae6 10/23/2018 10:19:25 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-VN3-VM31 10/23/2018 10:19:25 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=53a8aac3-a51d-3f43-a94c-075b16b95285, display_name=vmi-VN3-VM31, vcenter_port=VCenterPort(mac_address=00:50:56:a4:a4:f2, port_key=115247, portgroup_key=dvportgroup-48462, vlan_id=1), ip_address=10.3.1.252) 10/23/2018 10:19:26 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:19:26 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:20:27 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:20:27 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:20:27 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:20:27 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:20:27 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:20:27 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:20:27 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:20:27 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:20:27 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:20:27 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540290027986019 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.139:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:20:27 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:20:28 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:20:28 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:20:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:20:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:20:28 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:20:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:20:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:20:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:20:28 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:20:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:20:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:20:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:20:28 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:20:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:20:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:20:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:20:28 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:20:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:20:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:20:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:20:28 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:20:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:20:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:20:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:20:28 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:20:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:20:29 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:20:29 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:20:29 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:20:29 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:20:30 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:20:30 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:21:31 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:21:31 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:21:31 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:21:31 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:21:31 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:21:31 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:21:31 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:21:31 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:21:31 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:21:31 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540290091782651 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.139:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:21:31 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:21:31 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:21:32 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:21:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:21:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:21:32 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:21:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:21:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:21:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:21:32 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:21:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:21:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:21:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:21:32 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:21:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:21:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:21:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:21:32 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:21:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:21:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:21:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:21:32 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:21:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:21:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:21:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:21:32 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:21:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:21:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:21:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:21:32 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:21:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:21:33 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:21:33 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.144:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:22:35 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:22:35 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:22:35 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:22:35 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:22:35 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:22:35 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:22:35 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:22:35 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:22:35 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:22:35 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540290155635073 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.144:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:22:35 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:22:35 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:22:35 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:22:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:22:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:22:36 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:22:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:22:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:22:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:22:36 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:22:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:22:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:22:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:22:36 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:22:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:22:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:22:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:22:36 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:22:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:22:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:22:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:22:36 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:22:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:22:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:22:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:22:36 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:22:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:22:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:22:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:22:36 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:22:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:22:37 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:22:37 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:23:39 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:23:39 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:23:39 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:23:39 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:23:39 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:23:39 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:23:39 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:23:39 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:23:39 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:23:39 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540290219517847 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.139:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:23:39 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:23:39 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:23:39 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:23:40 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:23:40 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:23:40 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:23:40 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:23:40 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:23:40 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:23:40 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:23:40 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:23:40 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:23:40 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:23:40 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:23:40 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:23:40 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:23:40 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:23:40 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:23:40 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:23:40 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:23:40 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:23:40 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:23:40 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:23:40 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:23:40 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:23:40 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:23:40 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:23:40 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:23:40 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:23:40 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:23:40 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:23:41 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:23:41 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:24:43 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:24:43 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:24:43 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:24:43 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:24:43 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:24:43 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:24:43 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:24:43 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:24:43 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:24:43 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540290283313575 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.139:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:24:43 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:24:43 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:24:43 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:24:43 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:24:44 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:24:44 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:24:44 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:24:44 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:24:44 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:24:44 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:24:44 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:24:44 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:24:44 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:24:44 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:24:44 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:24:44 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:24:44 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:24:44 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:24:44 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:24:44 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:24:44 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:24:44 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:24:44 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:24:44 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:24:44 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:24:44 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:24:44 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:24:44 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:24:44 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:24:44 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:24:44 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:24:45 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:24:45 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.144:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:25:47 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:25:47 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:25:47 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:25:47 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:25:47 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:25:47 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:25:47 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:25:47 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:25:47 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:25:47 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540290347136822 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.139:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:25:47 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:25:47 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:25:47 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:25:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:25:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:25:47 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:25:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:25:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:25:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:25:47 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:25:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:25:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:25:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:25:47 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:25:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:25:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:25:48 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:25:48 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:25:48 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:25:48 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:25:48 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:25:48 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:25:48 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:25:48 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:25:48 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:25:48 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:25:48 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:25:48 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:25:48 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:25:48 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:25:48 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:25:49 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:25:49 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:26:50 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:26:51 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:26:51 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:26:51 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:26:51 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:26:51 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:26:51 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:26:51 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:26:51 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:26:51 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540290411004043 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.139:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:26:51 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:26:51 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:26:51 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:26:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:26:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:26:51 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:26:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:26:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:26:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:26:51 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:26:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:26:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:26:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:26:51 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:26:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:26:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:26:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:26:51 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:26:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:26:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:26:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:26:51 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:26:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:26:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:26:52 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:26:52 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:26:52 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:26:52 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:26:52 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:26:52 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:26:52 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:26:53 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] AttributeError: 'NoneType' object has no attribute 'instanceUuid' 10/23/2018 10:27:54 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:27:54 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:27:54 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:27:54 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:27:54 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:27:54 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:27:54 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:27:54 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:27:54 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:27:54 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540290474789309 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.139:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:27:54 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:27:55 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:27:55 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:27:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:27:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:27:55 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:27:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:27:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:27:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:27:55 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:27:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:27:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:27:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:27:55 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:27:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:27:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:27:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:27:55 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:27:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:27:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:27:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:27:55 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:27:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:27:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:27:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:27:55 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:27:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:27:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:27:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:27:55 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:27:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:27:57 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:27:57 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:28:58 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:28:58 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:28:58 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:28:58 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:28:58 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:28:58 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:28:58 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:28:58 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:28:58 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:28:58 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540290538741899 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.144:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:28:58 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:28:58 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:28:59 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:28:59 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:28:59 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:28:59 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:28:59 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:28:59 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:28:59 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:28:59 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:28:59 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:28:59 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:28:59 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:28:59 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:28:59 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:28:59 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:28:59 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:28:59 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:28:59 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:28:59 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:28:59 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:28:59 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:28:59 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:28:59 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:28:59 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:28:59 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:28:59 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:28:59 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:28:59 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:28:59 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:28:59 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:29:00 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] AttributeError: 'NoneType' object has no attribute 'instanceUuid' 10/23/2018 10:30:02 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:30:02 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:30:02 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:30:02 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:30:02 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:30:02 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:30:02 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:30:02 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:30:02 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:30:02 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540290602477049 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.140:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:30:02 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:30:02 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:30:02 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:30:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:30:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:30:03 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:30:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:30:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:30:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:30:03 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:30:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:30:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:30:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:30:03 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:30:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:30:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:30:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:30:03 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:30:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:30:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:30:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:30:03 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:30:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:30:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:30:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:30:03 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:30:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:30:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:30:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:30:03 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:30:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:30:04 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:30:04 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.144:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:31:06 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:31:06 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:31:06 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:31:06 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:31:06 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:31:06 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:31:06 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:31:06 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:31:06 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:31:06 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540290666549144 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.140:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:31:06 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:31:06 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:31:07 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:31:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:31:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:31:07 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:31:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:31:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:31:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:31:07 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:31:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:31:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:31:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:31:07 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:31:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:31:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:31:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:31:07 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:31:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:31:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:31:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:31:07 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:31:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:31:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:31:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:31:07 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:31:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:31:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:31:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:31:07 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:31:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:31:08 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] AttributeError: 'NoneType' object has no attribute 'instanceUuid' 10/23/2018 10:32:10 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:32:10 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:32:10 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:32:10 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:32:10 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:32:10 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:32:10 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:32:10 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:32:10 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:32:10 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540290730406068 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.139:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:32:10 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:32:10 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:32:10 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:32:10 AM [cvm.clients] [INFO]: Read from ESXi API VM31 properties: {u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '502443d3-0daa-acde-d812-e23efeaa70a7', u'runtime.powerState': 'poweredOn', u'name': 'VM31', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:32:10 AM [cvm.database] [INFO]: Could not find VM model with uuid 502443d3-0daa-acde-d812-e23efeaa70a7. 10/23/2018 10:32:11 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine 502443d3-0daa-acde-d812-e23efeaa70a7 in VNC 10/23/2018 10:32:11 AM [cvm.clients] [INFO]: Virtual Machine 502443d3-0daa-acde-d812-e23efeaa70a7 not found in VNC - creating 10/23/2018 10:32:11 AM [cvm.clients] [INFO]: Virtual Machine 502443d3-0daa-acde-d812-e23efeaa70a7 created in VNC 10/23/2018 10:32:11 AM [cvm.services] [INFO]: Created VirtualMachineModel(uuid=502443d3-0daa-acde-d812-e23efeaa70a7, name=VM31, vm_properties={u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '502443d3-0daa-acde-d812-e23efeaa70a7', u'runtime.powerState': 'poweredOn', u'name': 'VM31', u'summary.runtime.host': 'vim.HostSystem:ha-host'}, interfaces=['02950786-2014-333f-92bc-454f041659fe']) 10/23/2018 10:32:11 AM [cvm.database] [INFO]: Saved Virtual Machine model for VM31 10/23/2018 10:32:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:32:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:32:11 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:32:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:32:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:32:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:32:11 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:32:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:32:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:32:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:32:11 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:32:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:32:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:32:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:32:11 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:32:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:32:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:32:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:32:11 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:32:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:32:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:32:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:32:11 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:32:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:32:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:32:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:32:11 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:32:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:32:11 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=02950786-2014-333f-92bc-454f041659fe, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:73:bd, port_key=115247, portgroup_key=dvportgroup-48462, vlan_id=None), ip_address=unset) 10/23/2018 10:32:11 AM [cvm.clients] [INFO]: Reading VLAN ID of port 115247 10/23/2018 10:32:11 AM [cvm.clients] [INFO]: Port: 115247 VLAN ID: 1 10/23/2018 10:32:11 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 02950786-2014-333f-92bc-454f041659fe in VNC 10/23/2018 10:32:11 AM [cvm.clients] [INFO]: Virtual Machine Interface 02950786-2014-333f-92bc-454f041659fe not found in VNC - creating 10/23/2018 10:32:12 AM [cvm.clients] [INFO]: Virtual Machine Interface 02950786-2014-333f-92bc-454f041659fe created in VNC 10/23/2018 10:32:12 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-VN3-VM31 10/23/2018 10:32:12 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-VN3-VM31 10/23/2018 10:32:12 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=02950786-2014-333f-92bc-454f041659fe, display_name=vmi-VN3-VM31, vcenter_port=VCenterPort(mac_address=00:50:56:a4:73:bd, port_key=115247, portgroup_key=dvportgroup-48462, vlan_id=1), ip_address=10.3.1.252) 10/23/2018 10:32:13 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:32:13 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:33:15 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:33:15 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:33:15 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:33:15 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:33:15 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:33:15 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:33:15 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:33:15 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:33:15 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:33:15 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540290795078279 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.144:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:33:15 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:33:15 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:33:15 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:33:15 AM [cvm.clients] [INFO]: Read from ESXi API VM31 properties: {u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '502443d3-0daa-acde-d812-e23efeaa70a7', u'runtime.powerState': 'poweredOn', u'name': 'VM31', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:33:15 AM [cvm.database] [INFO]: Could not find VM model with uuid 502443d3-0daa-acde-d812-e23efeaa70a7. 10/23/2018 10:33:15 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine 502443d3-0daa-acde-d812-e23efeaa70a7 in VNC 10/23/2018 10:33:15 AM [cvm.clients] [INFO]: Virtual Machine 502443d3-0daa-acde-d812-e23efeaa70a7 updated in VNC 10/23/2018 10:33:15 AM [cvm.services] [INFO]: Created VirtualMachineModel(uuid=502443d3-0daa-acde-d812-e23efeaa70a7, name=VM31, vm_properties={u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '502443d3-0daa-acde-d812-e23efeaa70a7', u'runtime.powerState': 'poweredOn', u'name': 'VM31', u'summary.runtime.host': 'vim.HostSystem:ha-host'}, interfaces=['02950786-2014-333f-92bc-454f041659fe']) 10/23/2018 10:33:15 AM [cvm.database] [INFO]: Saved Virtual Machine model for VM31 10/23/2018 10:33:16 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:33:16 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:33:16 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:33:16 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:33:16 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:33:16 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:33:16 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:33:16 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:33:16 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:33:16 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:33:16 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:33:16 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:33:16 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:33:16 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:33:16 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:33:16 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:33:16 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:33:16 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:33:16 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:33:16 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:33:16 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:33:16 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:33:16 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:33:16 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:33:16 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:33:16 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:33:16 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:33:16 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:33:16 AM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=02950786-2014-333f-92bc-454f041659fe, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:73:bd, port_key=115247, portgroup_key=dvportgroup-48462, vlan_id=None), ip_address=unset) 10/23/2018 10:33:16 AM [cvm.clients] [INFO]: Reading VLAN ID of port 115247 10/23/2018 10:33:16 AM [cvm.clients] [INFO]: Port: 115247 VLAN ID: 1 10/23/2018 10:33:16 AM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 02950786-2014-333f-92bc-454f041659fe in VNC 10/23/2018 10:33:16 AM [cvm.clients] [INFO]: Updating Virtual Network of Interface 02950786-2014-333f-92bc-454f041659fe to VN3 10/23/2018 10:33:16 AM [cvm.models] [INFO]: Constructing Instance IP for vmi-VN3-VM31 10/23/2018 10:33:16 AM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-VN3-VM31 10/23/2018 10:33:16 AM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=02950786-2014-333f-92bc-454f041659fe, display_name=vmi-VN3-VM31, vcenter_port=VCenterPort(mac_address=00:50:56:a4:73:bd, port_key=115247, portgroup_key=dvportgroup-48462, vlan_id=1), ip_address=10.3.1.252) 10/23/2018 10:33:17 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:33:18 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:34:19 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:34:19 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:34:19 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:34:19 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:34:19 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:34:19 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:34:19 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:34:19 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:34:19 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:34:19 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540290859642637 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.144:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:34:19 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:34:19 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:34:20 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:34:20 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:34:20 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:34:20 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:34:20 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:34:20 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:34:20 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:34:20 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:34:20 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:34:20 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:34:20 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:34:20 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:34:20 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:34:20 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:34:20 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:34:20 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:34:20 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:34:20 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:34:20 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:34:20 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:34:20 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:34:20 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:34:20 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:34:20 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:34:20 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:34:20 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:34:20 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:34:20 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:34:20 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:34:21 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:34:21 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:35:23 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:35:23 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:35:23 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:35:23 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:35:23 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:35:23 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:35:23 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:35:23 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:35:23 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:35:23 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540290923551466 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.144:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:35:23 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:35:23 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:35:23 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:35:24 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:35:24 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:35:24 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:35:24 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:35:24 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:35:24 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:35:24 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:35:24 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:35:24 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:35:24 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:35:24 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:35:24 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:35:24 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:35:24 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:35:24 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:35:24 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:35:24 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:35:24 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:35:24 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:35:24 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:35:24 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:35:24 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:35:24 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:35:24 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:35:24 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:35:24 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:35:24 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:35:24 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:35:25 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:35:25 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.144:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:36:27 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:36:27 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:36:27 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:36:27 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:36:27 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:36:27 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:36:27 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:36:27 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:36:27 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:36:27 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540290987451726 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.144:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:36:27 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:36:27 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:36:27 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:36:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:36:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:36:28 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:36:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:36:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:36:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:36:28 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:36:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:36:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:36:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:36:28 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:36:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:36:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:36:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:36:28 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:36:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:36:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:36:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:36:28 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:36:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:36:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:36:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:36:28 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:36:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:36:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:36:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:36:28 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:36:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:36:29 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:36:29 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:37:31 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:37:31 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:37:31 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:37:31 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:37:31 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:37:31 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:37:31 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:37:31 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:37:31 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:37:31 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540291051364965 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.144:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:37:31 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:37:31 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:37:31 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:37:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:37:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:37:32 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:37:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:37:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:37:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:37:32 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:37:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:37:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:37:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:37:32 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:37:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:37:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:37:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:37:32 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:37:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:37:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:37:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:37:32 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:37:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:37:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:37:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:37:32 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:37:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:37:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:37:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:37:32 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:37:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:37:33 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:37:33 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:38:35 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:38:35 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:38:35 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:38:35 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:38:35 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:38:35 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:38:35 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:38:35 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:38:35 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:38:35 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540291115246213 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.140:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:38:35 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:38:35 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:38:35 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:38:35 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:38:35 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:38:35 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:38:35 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:38:35 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:38:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:38:36 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:38:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:38:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:38:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:38:36 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:38:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:38:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:38:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:38:36 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:38:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:38:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:38:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:38:36 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:38:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:38:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:38:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:38:36 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:38:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:38:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:38:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:38:36 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:38:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:38:37 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:38:37 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:39:39 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:39:39 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:39:39 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:39:39 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:39:39 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:39:39 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:39:39 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:39:39 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:39:39 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:39:39 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540291179153981 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.139:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:39:39 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:39:39 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:39:39 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:39:39 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:39:39 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:39:39 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:39:39 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:39:39 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:39:39 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:39:39 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:39:39 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:39:39 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:39:39 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:39:39 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:39:39 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:39:40 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:39:40 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:39:40 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:39:40 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:39:40 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:39:40 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:39:40 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:39:40 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:39:40 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:39:40 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:39:40 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:39:40 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:39:40 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:39:40 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:39:40 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:39:40 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:39:41 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:39:41 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:40:42 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:40:42 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:40:42 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:40:42 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:40:42 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:40:42 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:40:42 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:40:42 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:40:42 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:40:43 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540291242998140 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.144:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:40:43 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:40:43 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:40:43 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:40:43 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:40:43 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:40:43 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:40:43 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:40:43 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:40:43 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:40:43 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:40:43 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:40:43 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:40:43 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:40:43 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:40:43 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:40:43 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:40:43 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:40:43 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:40:43 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:40:43 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:40:43 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:40:43 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:40:43 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:40:43 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:40:43 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:40:43 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:40:43 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:40:44 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:40:44 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:40:44 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:40:44 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:40:45 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] AttributeError: 'NoneType' object has no attribute 'instanceUuid' 10/23/2018 10:41:46 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:41:46 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:41:46 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:41:46 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:41:46 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:41:46 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:41:46 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:41:46 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:41:46 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:41:46 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540291306881702 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.144:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:41:46 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:41:47 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:41:47 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:41:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:41:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:41:47 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:41:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:41:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:41:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:41:47 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:41:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:41:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:41:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:41:47 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:41:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:41:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:41:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:41:47 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:41:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:41:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:41:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:41:47 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:41:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:41:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:41:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:41:47 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:41:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:41:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:41:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:41:47 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:41:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:41:49 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] AttributeError: 'NoneType' object has no attribute 'instanceUuid' 10/23/2018 10:42:50 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:42:50 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:42:50 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:42:50 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:42:50 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:42:50 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:42:50 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:42:50 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:42:50 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:42:50 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540291370755752 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.140:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:42:50 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:42:50 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:42:51 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:42:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:42:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:42:51 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:42:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:42:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:42:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:42:51 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:42:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:42:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:42:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:42:51 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:42:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:42:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:42:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:42:51 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:42:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:42:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:42:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:42:51 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:42:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:42:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:42:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:42:51 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:42:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:42:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:42:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:42:51 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:42:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:42:52 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] AttributeError: 'NoneType' object has no attribute 'instanceUuid' 10/23/2018 10:43:54 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:43:54 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:43:54 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:43:54 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:43:54 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:43:54 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:43:54 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:43:54 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:43:54 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:43:54 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540291434580549 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.139:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:43:54 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:43:54 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:43:55 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:43:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:43:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:43:55 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:43:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:43:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:43:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:43:55 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:43:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:43:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:43:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:43:55 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:43:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:43:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:43:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:43:55 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:43:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:43:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:43:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:43:55 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:43:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:43:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:43:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:43:55 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:43:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:43:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:43:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:43:55 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:43:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:43:56 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:43:57 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.144:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:44:58 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:44:58 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:44:58 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:44:58 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:44:58 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:44:58 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:44:58 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:44:58 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:44:58 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:44:58 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540291498944888 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.144:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:44:58 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:44:59 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:44:59 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:44:59 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:44:59 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:44:59 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:44:59 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:44:59 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:44:59 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:44:59 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:44:59 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:44:59 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:44:59 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:44:59 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:44:59 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:44:59 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:44:59 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:44:59 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:44:59 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:44:59 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:44:59 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:44:59 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:44:59 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:45:00 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:45:00 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:45:00 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:45:00 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:45:00 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:45:00 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:45:00 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:45:00 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:45:01 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] AttributeError: 'NoneType' object has no attribute 'instanceUuid' 10/23/2018 10:46:02 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:46:02 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:46:02 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:46:02 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:46:02 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:46:02 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:46:02 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:46:02 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:46:02 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:46:02 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540291562682068 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.140:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:46:02 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:46:02 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:46:03 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:46:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:46:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:46:03 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:46:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:46:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:46:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:46:03 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:46:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:46:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:46:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:46:03 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:46:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:46:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:46:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:46:03 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:46:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:46:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:46:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:46:03 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:46:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:46:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:46:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:46:03 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:46:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:46:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:46:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:46:03 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:46:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:46:04 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:46:05 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:47:06 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:47:06 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:47:06 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:47:06 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:47:06 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:47:06 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:47:06 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:47:06 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:47:06 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:47:06 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540291626634225 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.139:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:47:06 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:47:06 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:47:06 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:47:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:47:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:47:07 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:47:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:47:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:47:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:47:07 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:47:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:47:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:47:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:47:07 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:47:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:47:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:47:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:47:07 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:47:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:47:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:47:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:47:07 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:47:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:47:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:47:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:47:07 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:47:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:47:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:47:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:47:07 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:47:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:47:08 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:47:08 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:48:10 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:48:10 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:48:10 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:48:10 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:48:10 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:48:10 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:48:10 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:48:10 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:48:10 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:48:10 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540291690566609 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.139:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:48:10 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:48:10 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:48:10 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:48:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:48:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:48:11 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:48:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:48:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:48:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:48:11 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:48:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:48:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:48:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:48:11 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:48:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:48:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:48:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:48:11 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:48:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:48:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:48:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:48:11 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:48:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:48:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:48:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:48:11 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:48:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:48:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:48:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:48:11 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:48:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:48:12 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:48:12 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:49:14 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:49:14 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:49:14 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:49:14 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:49:14 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:49:14 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:49:14 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:49:14 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:49:14 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:49:14 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540291754461757 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.144:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:49:14 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:49:14 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:49:14 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:49:15 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:49:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:49:15 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:49:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:49:15 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:49:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:49:15 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:49:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:49:15 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:49:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:49:15 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:49:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:49:15 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:49:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:49:15 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:49:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:49:15 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:49:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:49:15 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:49:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:49:15 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:49:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:49:15 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:49:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:49:15 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:49:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:49:15 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:49:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:49:16 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:49:16 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:50:18 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:50:18 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:50:18 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:50:18 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:50:18 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:50:18 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:50:18 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:50:18 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:50:18 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:50:18 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540291818409710 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.139:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:50:18 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:50:18 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:50:18 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:50:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:50:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:50:19 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:50:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:50:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:50:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:50:19 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:50:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:50:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:50:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:50:19 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:50:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:50:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:50:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:50:19 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:50:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:50:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:50:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:50:19 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:50:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:50:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:50:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:50:19 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:50:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:50:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:50:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:50:19 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:50:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:50:20 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:50:20 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:51:22 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:51:22 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:51:22 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:51:22 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:51:22 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:51:22 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:51:22 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:51:22 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:51:22 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:51:22 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540291882305429 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.144:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:51:22 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:51:22 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:51:22 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:51:22 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:51:23 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:51:23 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:51:23 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:51:23 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:51:23 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:51:23 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:51:23 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:51:23 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:51:23 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:51:23 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:51:23 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:51:23 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:51:23 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:51:23 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:51:23 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:51:23 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:51:23 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:51:23 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:51:23 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:51:23 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:51:23 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:51:23 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:51:23 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:51:23 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:51:23 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:51:23 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:51:23 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:51:24 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:51:24 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:52:26 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:52:26 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:52:26 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:52:26 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:52:26 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:52:26 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:52:26 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:52:26 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:52:26 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:52:26 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540291946195818 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.140:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:52:26 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:52:26 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:52:26 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:52:26 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:52:26 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:52:26 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:52:26 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:52:26 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:52:26 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:52:26 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:52:26 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:52:26 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:52:27 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:52:27 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:52:27 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:52:27 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:52:27 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:52:27 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:52:27 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:52:27 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:52:27 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:52:27 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:52:27 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:52:27 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:52:27 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:52:27 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:52:27 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:52:27 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:52:27 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:52:27 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:52:27 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:52:28 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:52:28 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.144:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:53:30 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:53:30 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:53:30 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:53:30 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:53:30 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:53:30 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:53:30 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:53:30 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:53:30 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:53:30 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540292010152430 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.144:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:53:30 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:53:30 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:53:30 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:53:30 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:53:30 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:53:30 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:53:30 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:53:30 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:53:30 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:53:30 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:53:30 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:53:30 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:53:30 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:53:30 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:53:30 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:53:31 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:53:31 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:53:31 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:53:31 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:53:31 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:53:31 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:53:31 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:53:31 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:53:31 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:53:31 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:53:31 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:53:31 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:53:31 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:53:31 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:53:31 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:53:31 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:53:32 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] AttributeError: 'NoneType' object has no attribute 'instanceUuid' 10/23/2018 10:54:34 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:54:34 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:54:34 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:54:34 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:54:34 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:54:34 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:54:34 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:54:34 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:54:34 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:54:34 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540292074038514 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.139:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:54:34 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:54:34 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:54:34 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:54:34 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:54:34 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:54:34 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:54:34 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:54:34 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:54:34 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:54:34 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:54:34 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:54:34 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:54:34 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:54:34 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:54:34 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:54:34 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:54:34 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:54:34 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:54:34 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:54:34 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:54:35 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:54:35 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:54:35 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:54:35 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:54:35 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:54:35 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:54:35 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:54:35 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:54:35 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:54:35 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:54:35 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:54:36 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:54:36 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:55:38 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:55:38 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:55:38 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:55:38 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:55:38 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:55:38 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:55:38 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:55:38 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:55:38 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:55:38 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540292138064611 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.140:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:55:38 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:55:38 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:55:38 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:55:38 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:55:38 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:55:38 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:55:38 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:55:38 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:55:38 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:55:38 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:55:38 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:55:38 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:55:38 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:55:38 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:55:38 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:55:38 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:55:38 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:55:38 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:55:38 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:55:39 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:55:39 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:55:39 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:55:39 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:55:39 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:55:39 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:55:39 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:55:39 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:55:39 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:55:39 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:55:39 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:55:39 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:55:40 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] AttributeError: 'NoneType' object has no attribute 'instanceUuid' 10/23/2018 10:56:42 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:56:42 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:56:42 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:56:42 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:56:42 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:56:42 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:56:42 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:56:42 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:56:42 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:56:42 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540292202440192 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.140:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:56:42 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:56:42 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:56:42 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:56:43 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:56:43 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:56:43 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:56:43 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:56:43 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:56:43 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:56:43 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:56:43 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:56:43 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:56:43 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:56:43 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:56:43 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:56:43 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:56:43 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:56:43 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:56:43 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:56:43 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:56:43 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:56:43 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:56:43 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:56:43 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:56:43 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:56:43 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:56:43 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:56:43 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:56:43 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:56:43 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:56:43 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:56:44 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] AttributeError: 'NoneType' object has no attribute 'instanceUuid' 10/23/2018 10:57:46 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:57:46 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:57:46 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:57:46 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:57:46 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:57:46 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:57:46 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:57:46 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:57:46 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:57:46 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540292266267639 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.144:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:57:46 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:57:46 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:57:46 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:57:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:57:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:57:47 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:57:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:57:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:57:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:57:47 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:57:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:57:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:57:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:57:47 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:57:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:57:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:57:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:57:47 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:57:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:57:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:57:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:57:47 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:57:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:57:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:57:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:57:47 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:57:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:57:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:57:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:57:47 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:57:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:57:48 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] AttributeError: 'NoneType' object has no attribute 'instanceUuid' 10/23/2018 10:58:50 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:58:50 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:58:50 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:58:50 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:58:50 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:58:50 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:58:50 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:58:50 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:58:50 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:58:50 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540292330203176 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.140:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:58:50 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:58:50 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:58:50 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:58:50 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:58:50 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:58:50 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:58:50 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:58:50 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:58:50 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:58:50 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:58:50 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:58:50 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:58:50 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:58:51 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:58:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:58:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:58:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:58:51 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:58:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:58:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:58:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:58:51 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:58:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:58:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:58:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:58:51 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:58:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:58:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:58:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:58:51 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:58:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:58:52 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:58:52 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 10:59:54 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 10:59:54 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 10:59:54 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 10:59:54 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 10:59:54 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 10:59:54 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:59:54 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 10:59:54 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:59:54 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:59:54 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540292394173557 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.144:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 10:59:54 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 10:59:54 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 10:59:54 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 10:59:54 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 10:59:54 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 10:59:54 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 10:59:54 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 10:59:54 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 10:59:54 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 10:59:54 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 10:59:54 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 10:59:54 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 10:59:54 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 10:59:54 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 10:59:54 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 10:59:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 10:59:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 10:59:55 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 10:59:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 10:59:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 10:59:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 10:59:55 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 10:59:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 10:59:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 10:59:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 10:59:55 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 10:59:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 10:59:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 10:59:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 10:59:55 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 10:59:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 10:59:56 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 10:59:56 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.144:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:00:58 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:00:58 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:00:58 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:00:58 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:00:58 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:00:58 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:00:58 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:00:58 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:00:58 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:00:58 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540292458157547 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.144:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:00:58 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:00:58 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:00:58 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:00:58 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:00:58 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:00:58 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:00:58 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:00:58 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:00:58 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:00:58 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:00:58 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:00:58 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:00:58 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:00:58 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:00:58 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:00:58 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:00:59 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:00:59 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:00:59 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:00:59 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:00:59 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:00:59 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:00:59 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:00:59 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:00:59 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:00:59 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:00:59 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:00:59 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:00:59 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:00:59 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:00:59 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:01:00 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:01:00 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.144:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:02:02 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:02:02 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:02:02 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:02:02 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:02:02 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:02:02 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:02:02 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:02:02 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:02:02 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:02:02 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540292522133065 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.144:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:02:02 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:02:02 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:02:02 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:02:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:02:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:02:02 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:02:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:02:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:02:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:02:02 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:02:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:02:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:02:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:02:02 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:02:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:02:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:02:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:02:03 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:02:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:02:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:02:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:02:03 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:02:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:02:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:02:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:02:03 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:02:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:02:03 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:02:03 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:02:03 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:02:03 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:02:04 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:02:04 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:03:06 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:03:06 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:03:06 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:03:06 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:03:06 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:03:06 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:03:06 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:03:06 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:03:06 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:03:06 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540292586124894 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.140:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:03:06 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:03:06 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:03:06 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:03:06 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:03:06 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:03:06 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:03:06 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:03:06 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:03:06 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:03:06 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:03:06 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:03:06 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:03:06 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:03:06 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:03:06 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:03:06 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:03:06 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:03:06 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:03:06 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:03:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:03:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:03:07 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:03:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:03:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:03:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:03:07 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:03:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:03:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:03:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:03:07 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:03:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:03:08 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:03:08 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:04:10 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:04:10 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:04:10 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:04:10 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:04:10 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:04:10 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:04:10 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:04:10 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:04:10 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:04:10 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540292650043746 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.140:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:04:10 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:04:10 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:04:10 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:04:10 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:04:10 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:04:10 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:04:10 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:04:10 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:04:10 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:04:10 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:04:10 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:04:10 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:04:10 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:04:10 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:04:10 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:04:10 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:04:10 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:04:10 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:04:10 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:04:10 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:04:10 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:04:10 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:04:10 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:04:10 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:04:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:04:11 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:04:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:04:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:04:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:04:11 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:04:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:04:12 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:04:12 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:05:14 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:05:14 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:05:14 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:05:14 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:05:14 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:05:14 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:05:14 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:05:14 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:05:14 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:05:14 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540292714053083 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.140:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:05:14 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:05:14 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:05:14 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:05:14 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:05:14 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:05:14 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:05:14 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:05:14 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:05:14 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:05:14 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:05:14 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:05:14 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:05:14 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:05:14 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:05:14 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:05:14 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:05:14 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:05:14 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:05:14 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:05:14 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:05:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:05:15 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:05:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:05:15 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:05:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:05:15 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:05:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:05:15 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:05:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:05:15 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:05:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:05:15 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:05:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:05:15 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:05:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:05:15 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:05:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:05:15 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:05:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:05:16 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:05:16 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:06:18 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:06:18 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:06:18 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:06:18 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:06:18 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:06:18 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:06:18 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:06:18 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:06:18 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:06:18 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540292778213634 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.144:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:06:18 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:06:18 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:06:18 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:06:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:06:19 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:06:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:06:19 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:06:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:06:19 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:06:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:06:19 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:06:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:06:19 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:06:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:06:19 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:06:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:06:19 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:06:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:06:19 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:06:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:06:19 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48477 name: ctest-vnet2-90776466 10/23/2018 11:06:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet2-90776466 already allowed. 10/23/2018 11:06:19 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet2-90776466 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=077cb348-b19e-4568-a118-0dd0c8272ed9, key=dvportgroup-48477, name=ctest-vnet2-90776466) 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48478 name: ctest-vnet3-95983105 10/23/2018 11:06:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet3-95983105 already allowed. 10/23/2018 11:06:19 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet3-95983105 10/23/2018 11:06:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=0aa52d82-8865-4e2b-9012-20084da9b417, key=dvportgroup-48478, name=ctest-vnet3-95983105) 10/23/2018 11:06:20 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:06:20 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.144:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:07:22 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:07:22 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:07:22 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:07:22 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:07:22 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:07:22 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:07:22 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:07:22 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:07:22 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:07:22 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540292842599940 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.140:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:07:22 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:07:22 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:07:22 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:07:23 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:07:23 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:07:23 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:07:23 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:07:23 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:07:23 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:07:23 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:07:23 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:07:23 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:07:23 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:07:23 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:07:23 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:07:23 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:07:23 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:07:23 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:07:23 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:07:23 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:07:23 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:07:23 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:07:23 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:07:23 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:07:23 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:07:23 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:07:23 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:07:23 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:07:23 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:07:23 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:07:23 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:07:23 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:07:23 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:07:23 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:07:23 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:07:23 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:07:23 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:07:23 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:07:23 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:07:23 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48477 name: ctest-vnet2-90776466 10/23/2018 11:07:23 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet2-90776466 already allowed. 10/23/2018 11:07:23 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet2-90776466 10/23/2018 11:07:23 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=077cb348-b19e-4568-a118-0dd0c8272ed9, key=dvportgroup-48477, name=ctest-vnet2-90776466) 10/23/2018 11:07:24 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48478 name: ctest-vnet3-95983105 10/23/2018 11:07:24 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet3-95983105 already allowed. 10/23/2018 11:07:24 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet3-95983105 10/23/2018 11:07:24 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=0aa52d82-8865-4e2b-9012-20084da9b417, key=dvportgroup-48478, name=ctest-vnet3-95983105) 10/23/2018 11:07:25 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] AttributeError: 'NoneType' object has no attribute 'instanceUuid' 10/23/2018 11:08:26 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:08:26 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:08:26 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:08:26 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:08:26 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:08:26 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:08:26 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:08:26 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:08:26 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:08:26 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540292906761837 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.144:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:08:26 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:08:26 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:08:27 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:08:27 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:08:27 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:08:27 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:08:27 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:08:27 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:08:27 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:08:27 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:08:27 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:08:27 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:08:27 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:08:27 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:08:27 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:08:27 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:08:27 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:08:27 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:08:27 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:08:27 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:08:27 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:08:27 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:08:27 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:08:27 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:08:27 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:08:27 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:08:27 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:08:27 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:08:27 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:08:27 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:08:27 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:08:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:08:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:08:28 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:08:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:08:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:08:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:08:28 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:08:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:08:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48477 name: ctest-vnet2-90776466 10/23/2018 11:08:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet2-90776466 already allowed. 10/23/2018 11:08:28 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet2-90776466 10/23/2018 11:08:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=077cb348-b19e-4568-a118-0dd0c8272ed9, key=dvportgroup-48477, name=ctest-vnet2-90776466) 10/23/2018 11:08:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48478 name: ctest-vnet3-95983105 10/23/2018 11:08:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet3-95983105 already allowed. 10/23/2018 11:08:28 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet3-95983105 10/23/2018 11:08:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=0aa52d82-8865-4e2b-9012-20084da9b417, key=dvportgroup-48478, name=ctest-vnet3-95983105) 10/23/2018 11:08:29 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:08:29 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:09:31 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:09:31 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:09:31 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:09:31 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:09:31 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:09:31 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:09:31 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:09:31 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:09:31 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:09:31 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540292971236094 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.140:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:09:31 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:09:31 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:09:31 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:09:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:09:32 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:09:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:09:32 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:09:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:09:32 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:09:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:09:32 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:09:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:09:32 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:09:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:09:32 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:09:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:09:32 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:09:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:09:32 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:09:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:09:32 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48477 name: ctest-vnet2-90776466 10/23/2018 11:09:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet2-90776466 already allowed. 10/23/2018 11:09:32 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet2-90776466 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=077cb348-b19e-4568-a118-0dd0c8272ed9, key=dvportgroup-48477, name=ctest-vnet2-90776466) 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48478 name: ctest-vnet3-95983105 10/23/2018 11:09:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet3-95983105 already allowed. 10/23/2018 11:09:32 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet3-95983105 10/23/2018 11:09:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=0aa52d82-8865-4e2b-9012-20084da9b417, key=dvportgroup-48478, name=ctest-vnet3-95983105) 10/23/2018 11:09:33 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] AttributeError: 'NoneType' object has no attribute 'instanceUuid' 10/23/2018 11:10:35 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:10:35 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:10:35 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:10:35 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:10:35 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:10:35 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:10:35 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:10:35 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:10:35 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:10:35 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540293035593299 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.144:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:10:35 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:10:35 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:10:36 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:10:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:10:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:10:36 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:10:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:10:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:10:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:10:36 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:10:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:10:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:10:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:10:36 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:10:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:10:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:10:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:10:36 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:10:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:10:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:10:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:10:36 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:10:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:10:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:10:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:10:36 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:10:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:10:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:10:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:10:36 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:10:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:10:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:10:36 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:10:36 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:10:36 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:10:36 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:10:37 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:10:37 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:10:37 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:10:37 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48477 name: ctest-vnet2-90776466 10/23/2018 11:10:37 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet2-90776466 already allowed. 10/23/2018 11:10:37 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet2-90776466 10/23/2018 11:10:37 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=077cb348-b19e-4568-a118-0dd0c8272ed9, key=dvportgroup-48477, name=ctest-vnet2-90776466) 10/23/2018 11:10:37 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48478 name: ctest-vnet3-95983105 10/23/2018 11:10:37 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet3-95983105 already allowed. 10/23/2018 11:10:37 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet3-95983105 10/23/2018 11:10:37 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=0aa52d82-8865-4e2b-9012-20084da9b417, key=dvportgroup-48478, name=ctest-vnet3-95983105) 10/23/2018 11:10:38 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:10:38 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:11:40 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:11:40 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:11:40 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:11:40 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:11:40 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:11:40 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:11:40 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:11:40 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:11:40 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:11:40 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540293100219192 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.140:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:11:40 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:11:40 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:11:40 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:11:41 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:11:41 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:11:41 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:11:41 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:11:41 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:11:41 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:11:41 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:11:41 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:11:41 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:11:41 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:11:41 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:11:41 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:11:41 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:11:41 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:11:41 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:11:41 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:11:41 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:11:41 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48477 name: ctest-vnet2-90776466 10/23/2018 11:11:41 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet2-90776466 already allowed. 10/23/2018 11:11:41 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet2-90776466 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=077cb348-b19e-4568-a118-0dd0c8272ed9, key=dvportgroup-48477, name=ctest-vnet2-90776466) 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48478 name: ctest-vnet3-95983105 10/23/2018 11:11:41 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet3-95983105 already allowed. 10/23/2018 11:11:41 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet3-95983105 10/23/2018 11:11:41 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=0aa52d82-8865-4e2b-9012-20084da9b417, key=dvportgroup-48478, name=ctest-vnet3-95983105) 10/23/2018 11:11:43 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:11:43 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.144:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:12:44 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:12:44 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:12:44 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:12:44 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:12:44 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:12:44 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:12:44 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:12:44 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:12:44 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:12:44 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540293164733615 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.144:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:12:44 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:12:44 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:12:45 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:12:45 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:12:45 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:12:45 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:12:45 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:12:45 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:12:45 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:12:45 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:12:45 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:12:45 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:12:45 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:12:45 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:12:45 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:12:45 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:12:45 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:12:45 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:12:45 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:12:45 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:12:45 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:12:45 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:12:45 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:12:45 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:12:45 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:12:45 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:12:45 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:12:45 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:12:45 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:12:45 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:12:45 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:12:45 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:12:45 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:12:45 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:12:45 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:12:45 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:12:46 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:12:46 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:12:46 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:12:47 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:12:47 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.144:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:13:48 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:13:48 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:13:48 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:13:48 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:13:48 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:13:48 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:13:48 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:13:48 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:13:48 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:13:48 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540293228946275 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.144:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:13:48 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:13:49 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:13:49 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:13:49 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:13:49 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:13:49 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:13:49 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:13:49 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:13:49 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:13:49 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:13:49 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:13:49 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:13:49 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:13:49 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:13:49 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:13:49 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:13:49 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:13:49 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:13:49 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:13:49 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:13:49 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:13:49 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:13:49 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:13:49 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:13:50 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:13:50 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:13:50 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:13:50 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:13:50 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:13:50 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:13:50 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:13:50 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:13:50 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:13:50 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:13:50 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:13:50 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:13:50 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:13:50 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:13:50 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:13:51 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:13:51 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:14:53 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:14:53 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:14:53 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:14:53 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:14:53 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:14:53 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:14:53 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:14:53 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:14:53 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:14:53 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540293293172517 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.144:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:14:53 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:14:53 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:14:53 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:14:53 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:14:53 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:14:53 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:14:53 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:14:53 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:14:53 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:14:53 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:14:53 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:14:54 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:14:54 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:14:54 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:14:54 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:14:54 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:14:54 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:14:54 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:14:54 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:14:54 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:14:54 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:14:54 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:14:54 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:14:54 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:14:54 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:14:54 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:14:54 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:14:54 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:14:54 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:14:54 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:14:54 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:14:54 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:14:54 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:14:54 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:14:54 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:14:54 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:14:54 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:14:54 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:14:54 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:14:55 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:14:55 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.144:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:15:57 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:15:57 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:15:57 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:15:57 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:15:57 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:15:57 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:15:57 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:15:57 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:15:57 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:15:57 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540293357320091 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.144:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:15:57 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:15:57 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:15:57 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:15:58 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:15:58 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:15:58 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:15:58 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:15:58 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:15:58 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:15:58 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:15:58 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:15:58 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:15:58 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:15:58 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:15:58 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:15:58 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:15:58 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:15:58 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:15:58 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:15:58 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:15:58 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:15:58 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:15:58 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:15:58 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:15:58 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:15:58 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:15:58 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:15:58 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:15:58 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:15:58 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:15:58 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:15:58 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:15:58 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:15:58 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:15:58 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:15:58 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:15:58 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:15:58 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:15:58 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:15:59 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:15:59 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:17:01 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:17:01 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:17:01 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:17:01 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:17:01 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:17:01 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:17:01 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:17:01 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:17:01 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:17:01 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540293421556259 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.144:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:17:01 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:17:01 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:17:01 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:17:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:17:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:17:02 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:17:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:17:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:17:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:17:02 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:17:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:17:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:17:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:17:02 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:17:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:17:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:17:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:17:02 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:17:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:17:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:17:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:17:02 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:17:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:17:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:17:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:17:02 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:17:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:17:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:17:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:17:02 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:17:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:17:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:17:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:17:02 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:17:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:17:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:17:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:17:02 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:17:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:17:02 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48481 name: ctest-vnet0-72006042 10/23/2018 11:17:02 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-72006042 already allowed. 10/23/2018 11:17:02 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-72006042 10/23/2018 11:17:02 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=a0173704-97df-49ea-b8fe-f8efd6e51f5f, key=dvportgroup-48481, name=ctest-vnet0-72006042) 10/23/2018 11:17:04 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] AttributeError: 'NoneType' object has no attribute 'instanceUuid' 10/23/2018 11:18:05 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:18:05 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:18:05 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:18:05 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:18:05 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:18:05 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:18:05 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:18:05 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:18:05 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:18:05 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540293485861216 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.139:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:18:05 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:18:06 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:18:06 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:18:06 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:18:06 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:18:06 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:18:06 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:18:06 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:18:06 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:18:06 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:18:06 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:18:06 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:18:06 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:18:06 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:18:06 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:18:06 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:18:06 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:18:06 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:18:06 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:18:06 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:18:06 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:18:06 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:18:06 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:18:06 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:18:06 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:18:06 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:18:06 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:18:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:18:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:18:07 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:18:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:18:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:18:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:18:07 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:18:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:18:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:18:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:18:07 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:18:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:18:07 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48481 name: ctest-vnet0-72006042 10/23/2018 11:18:07 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-72006042 already allowed. 10/23/2018 11:18:07 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-72006042 10/23/2018 11:18:07 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=a0173704-97df-49ea-b8fe-f8efd6e51f5f, key=dvportgroup-48481, name=ctest-vnet0-72006042) 10/23/2018 11:18:08 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] AttributeError: 'NoneType' object has no attribute 'instanceUuid' 10/23/2018 11:19:10 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:19:10 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:19:10 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:19:10 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:19:10 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:19:10 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:19:10 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:19:10 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:19:10 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:19:10 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540293550230321 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.140:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:19:10 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:19:10 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:19:10 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:19:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:19:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:19:11 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:19:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:19:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:19:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:19:11 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:19:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:19:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:19:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:19:11 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:19:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:19:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:19:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:19:11 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:19:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:19:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:19:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:19:11 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:19:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:19:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:19:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:19:11 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:19:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:19:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:19:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:19:11 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:19:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:19:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:19:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:19:11 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:19:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:19:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:19:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:19:11 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:19:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:19:11 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48481 name: ctest-vnet0-72006042 10/23/2018 11:19:11 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-72006042 already allowed. 10/23/2018 11:19:11 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-72006042 10/23/2018 11:19:11 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=a0173704-97df-49ea-b8fe-f8efd6e51f5f, key=dvportgroup-48481, name=ctest-vnet0-72006042) 10/23/2018 11:19:12 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:19:13 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:20:14 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:20:14 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:20:14 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:20:14 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:20:14 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:20:14 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:20:14 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:20:14 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:20:14 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:20:14 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540293614658333 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.139:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:20:14 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:20:14 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:20:15 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:20:15 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:20:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:20:15 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:20:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:20:15 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:20:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:20:15 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:20:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:20:15 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:20:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:20:15 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:20:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:20:15 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:20:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:20:15 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:20:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:20:15 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:20:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:20:15 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:20:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:20:15 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:20:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:20:15 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:20:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:20:15 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:20:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:20:15 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:20:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:20:15 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:20:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:20:15 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:20:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:20:15 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:20:15 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:20:15 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:20:15 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:20:16 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48481 name: ctest-vnet0-72006042 10/23/2018 11:20:16 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-72006042 already allowed. 10/23/2018 11:20:16 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-72006042 10/23/2018 11:20:16 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=a0173704-97df-49ea-b8fe-f8efd6e51f5f, key=dvportgroup-48481, name=ctest-vnet0-72006042) 10/23/2018 11:20:17 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:20:17 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:21:18 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:21:18 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:21:18 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:21:18 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:21:18 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:21:18 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:21:18 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:21:18 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:21:18 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:21:18 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540293678993402 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.144:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:21:18 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:21:19 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:21:19 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:21:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:21:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:21:19 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:21:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:21:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:21:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:21:19 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:21:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:21:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:21:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:21:19 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:21:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:21:19 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:21:19 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:21:19 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:21:19 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:21:20 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:21:20 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:21:20 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:21:20 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:21:20 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:21:20 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:21:20 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:21:20 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:21:20 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:21:20 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:21:20 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:21:20 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:21:20 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:21:20 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:21:20 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:21:20 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:21:20 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:21:20 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:21:20 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:21:20 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:21:20 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48481 name: ctest-vnet0-72006042 10/23/2018 11:21:20 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-72006042 already allowed. 10/23/2018 11:21:20 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-72006042 10/23/2018 11:21:20 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=a0173704-97df-49ea-b8fe-f8efd6e51f5f, key=dvportgroup-48481, name=ctest-vnet0-72006042) 10/23/2018 11:21:21 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:21:21 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:22:23 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:22:23 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:22:23 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:22:23 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:22:23 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:22:23 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:22:23 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:22:23 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:22:23 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:22:23 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540293743336520 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.144:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:22:23 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:22:23 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:22:23 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:22:24 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:22:24 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:22:24 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:22:24 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:22:24 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:22:24 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:22:24 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:22:24 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:22:24 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:22:24 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:22:24 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:22:24 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:22:24 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:22:24 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:22:24 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:22:24 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:22:24 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:22:24 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:22:24 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:22:24 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:22:24 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:22:24 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:22:24 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:22:24 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:22:24 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:22:24 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:22:24 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:22:24 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:22:24 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:22:24 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:22:24 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:22:24 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:22:24 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:22:24 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:22:24 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:22:24 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:22:24 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48481 name: ctest-vnet0-72006042 10/23/2018 11:22:24 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-72006042 already allowed. 10/23/2018 11:22:24 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-72006042 10/23/2018 11:22:24 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=a0173704-97df-49ea-b8fe-f8efd6e51f5f, key=dvportgroup-48481, name=ctest-vnet0-72006042) 10/23/2018 11:22:25 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:22:26 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:23:27 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:23:27 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:23:27 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:23:27 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:23:27 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:23:27 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:23:27 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:23:27 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:23:27 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:23:27 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540293807672889 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.139:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:23:27 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:23:27 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:23:28 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:23:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:23:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:23:28 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:23:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:23:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:23:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:23:28 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:23:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:23:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:23:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:23:28 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:23:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:23:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:23:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:23:28 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:23:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:23:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:23:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:23:28 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:23:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:23:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:23:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:23:28 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:23:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:23:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:23:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:23:28 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:23:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:23:28 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:23:28 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:23:28 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:23:28 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:23:29 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:23:29 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:23:29 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:23:29 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:23:29 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48481 name: ctest-vnet0-72006042 10/23/2018 11:23:29 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-72006042 already allowed. 10/23/2018 11:23:29 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-72006042 10/23/2018 11:23:29 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=a0173704-97df-49ea-b8fe-f8efd6e51f5f, key=dvportgroup-48481, name=ctest-vnet0-72006042) 10/23/2018 11:23:29 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48484 name: ctest-vnet0-73495412 10/23/2018 11:23:29 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-73495412 already allowed. 10/23/2018 11:23:29 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-73495412 10/23/2018 11:23:29 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=1f8a5d3a-2910-430b-9a76-2d2504da12ff, key=dvportgroup-48484, name=ctest-vnet0-73495412) 10/23/2018 11:23:30 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:23:30 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:24:32 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:24:32 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:24:32 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:24:32 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:24:32 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:24:32 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:24:32 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:24:32 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:24:32 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:24:32 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540293872140661 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.139:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:24:32 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:24:32 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:24:32 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:24:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:24:32 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:24:32 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:24:32 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:24:32 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:24:33 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:24:33 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:24:33 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:24:33 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:24:33 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:24:33 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:24:33 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:24:33 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:24:33 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:24:33 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:24:33 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:24:33 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:24:33 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:24:33 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:24:33 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:24:33 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:24:33 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:24:33 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:24:33 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:24:33 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:24:33 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:24:33 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:24:33 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:24:33 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:24:33 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:24:33 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:24:33 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:24:33 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:24:33 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:24:33 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:24:33 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:24:33 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48481 name: ctest-vnet0-72006042 10/23/2018 11:24:33 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-72006042 already allowed. 10/23/2018 11:24:33 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-72006042 10/23/2018 11:24:33 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=a0173704-97df-49ea-b8fe-f8efd6e51f5f, key=dvportgroup-48481, name=ctest-vnet0-72006042) 10/23/2018 11:24:33 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48484 name: ctest-vnet0-73495412 10/23/2018 11:24:33 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-73495412 already allowed. 10/23/2018 11:24:33 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-73495412 10/23/2018 11:24:33 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=1f8a5d3a-2910-430b-9a76-2d2504da12ff, key=dvportgroup-48484, name=ctest-vnet0-73495412) 10/23/2018 11:24:34 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] AttributeError: 'NoneType' object has no attribute 'instanceUuid' 10/23/2018 11:25:36 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:25:36 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:25:36 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:25:36 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:25:36 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:25:36 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:25:36 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:25:36 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:25:36 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:25:36 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540293936613456 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.140:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:25:36 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:25:36 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:25:37 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:25:37 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:25:37 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:25:37 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:25:37 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:25:37 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:25:37 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:25:37 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:25:37 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:25:37 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:25:37 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:25:37 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:25:37 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:25:37 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:25:37 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:25:37 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:25:37 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:25:37 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:25:37 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:25:37 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:25:37 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:25:37 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:25:37 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:25:37 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:25:37 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:25:37 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:25:37 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:25:37 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:25:37 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:25:37 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:25:37 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:25:37 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:25:37 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:25:37 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:25:38 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:25:38 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:25:38 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:25:38 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48481 name: ctest-vnet0-72006042 10/23/2018 11:25:38 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-72006042 already allowed. 10/23/2018 11:25:38 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-72006042 10/23/2018 11:25:38 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=a0173704-97df-49ea-b8fe-f8efd6e51f5f, key=dvportgroup-48481, name=ctest-vnet0-72006042) 10/23/2018 11:25:38 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48484 name: ctest-vnet0-73495412 10/23/2018 11:25:38 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-73495412 already allowed. 10/23/2018 11:25:38 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-73495412 10/23/2018 11:25:38 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=1f8a5d3a-2910-430b-9a76-2d2504da12ff, key=dvportgroup-48484, name=ctest-vnet0-73495412) 10/23/2018 11:25:39 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:25:39 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.140:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:26:41 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:26:41 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:26:41 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:26:41 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:26:41 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:26:41 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:26:41 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:26:41 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:26:41 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:26:41 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540294001179049 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.140:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:26:41 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:26:41 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:26:41 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:26:42 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:26:42 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:26:42 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:26:42 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:26:42 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:26:42 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:26:42 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:26:42 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:26:42 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:26:42 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:26:42 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:26:42 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:26:42 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:26:42 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:26:42 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:26:42 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:26:42 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:26:42 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48481 name: ctest-vnet0-72006042 10/23/2018 11:26:42 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-72006042 already allowed. 10/23/2018 11:26:42 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-72006042 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=a0173704-97df-49ea-b8fe-f8efd6e51f5f, key=dvportgroup-48481, name=ctest-vnet0-72006042) 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48484 name: ctest-vnet0-73495412 10/23/2018 11:26:42 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-73495412 already allowed. 10/23/2018 11:26:42 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-73495412 10/23/2018 11:26:42 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=1f8a5d3a-2910-430b-9a76-2d2504da12ff, key=dvportgroup-48484, name=ctest-vnet0-73495412) 10/23/2018 11:26:44 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:26:44 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.144:8082/floating-ip/255052c3-71e7-4e5c-a1f2-6523a18d14ac'] 10/23/2018 11:27:45 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:27:45 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:27:45 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:27:45 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:27:45 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:27:45 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:27:45 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:27:45 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:27:45 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:27:45 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540294065826108 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.140:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:27:45 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:27:46 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:27:46 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:27:46 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:27:46 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:27:46 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:27:46 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:27:46 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:27:46 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:27:46 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:27:46 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:27:46 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:27:46 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:27:46 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:27:46 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:27:46 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:27:46 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:27:46 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:27:46 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:27:46 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:27:46 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:27:46 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:27:46 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:27:46 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:27:46 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:27:46 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:27:46 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:27:46 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:27:46 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:27:46 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:27:46 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:27:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:27:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:27:47 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:27:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:27:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:27:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:27:47 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:27:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:27:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48481 name: ctest-vnet0-72006042 10/23/2018 11:27:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-72006042 already allowed. 10/23/2018 11:27:47 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-72006042 10/23/2018 11:27:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=a0173704-97df-49ea-b8fe-f8efd6e51f5f, key=dvportgroup-48481, name=ctest-vnet0-72006042) 10/23/2018 11:27:47 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48484 name: ctest-vnet0-73495412 10/23/2018 11:27:47 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-73495412 already allowed. 10/23/2018 11:27:47 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-73495412 10/23/2018 11:27:47 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=1f8a5d3a-2910-430b-9a76-2d2504da12ff, key=dvportgroup-48484, name=ctest-vnet0-73495412) 10/23/2018 11:27:48 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:27:48 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/loadbalancer/ef9890ca-345f-4d8f-904e-4d90a3b01d06'] 10/23/2018 11:28:50 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:28:50 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:28:50 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:28:50 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:28:50 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:28:50 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:28:50 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:28:50 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:28:50 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:28:50 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540294130283223 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.144:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:28:50 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:28:50 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:28:50 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:28:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:28:51 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:28:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:28:51 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:28:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:28:51 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:28:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:28:51 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:28:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:28:51 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:28:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:28:51 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:28:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:28:51 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:28:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:28:51 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:28:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:28:51 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48481 name: ctest-vnet0-72006042 10/23/2018 11:28:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-72006042 already allowed. 10/23/2018 11:28:51 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-72006042 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=a0173704-97df-49ea-b8fe-f8efd6e51f5f, key=dvportgroup-48481, name=ctest-vnet0-72006042) 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48484 name: ctest-vnet0-73495412 10/23/2018 11:28:51 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-73495412 already allowed. 10/23/2018 11:28:51 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-73495412 10/23/2018 11:28:51 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=1f8a5d3a-2910-430b-9a76-2d2504da12ff, key=dvportgroup-48484, name=ctest-vnet0-73495412) 10/23/2018 11:28:53 AM [cvm.services] [INFO]: Deleting ef9890ca-345f-4d8f-904e-4d90a3b01d06 from VNC. 10/23/2018 11:28:53 AM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1091, in _request raise RefsExistError(content) RefsExistError: Delete when resource still referred: ['http://10.204.217.139:8082/loadbalancer/ef9890ca-345f-4d8f-904e-4d90a3b01d06'] 10/23/2018 11:29:54 AM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 11:29:54 AM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 11:29:54 AM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 11:29:54 AM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 11:29:54 AM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 11:29:54 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:29:54 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:29:54 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:29:54 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:29:54 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540294194695084 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.139:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 11:29:54 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 11:29:54 AM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 11:29:55 AM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 11:29:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 11:29:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 11:29:55 AM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 11:29:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 11:29:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48452 name: ctest-vCenter-70487290 10/23/2018 11:29:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-70487290 already allowed. 10/23/2018 11:29:55 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-70487290 10/23/2018 11:29:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=d8532105-7700-47a0-862a-895c57d51203, key=dvportgroup-48452, name=ctest-vCenter-70487290) 10/23/2018 11:29:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48454 name: ctest-vn-50026300 10/23/2018 11:29:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-50026300 already allowed. 10/23/2018 11:29:55 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-50026300 10/23/2018 11:29:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=ba1a71f8-6194-44a9-ab11-5b3c17cd3cca, key=dvportgroup-48454, name=ctest-vn-50026300) 10/23/2018 11:29:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48458 name: ctest-vCenter-16414629 10/23/2018 11:29:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vCenter-16414629 already allowed. 10/23/2018 11:29:55 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vCenter-16414629 10/23/2018 11:29:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=eaefe938-645e-473a-9fc7-aed774e17e48, key=dvportgroup-48458, name=ctest-vCenter-16414629) 10/23/2018 11:29:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48460 name: VN1 10/23/2018 11:29:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN1 already allowed. 10/23/2018 11:29:55 AM [cvm.database] [INFO]: Saved Virtual Network model for VN1 10/23/2018 11:29:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c9026219-2dcb-45b5-a4fc-4237a9ab5e2e, key=dvportgroup-48460, name=VN1) 10/23/2018 11:29:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48461 name: VN2 10/23/2018 11:29:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN2 already allowed. 10/23/2018 11:29:55 AM [cvm.database] [INFO]: Saved Virtual Network model for VN2 10/23/2018 11:29:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=6f4f4253-b045-4d63-90b8-7fe20d329c98, key=dvportgroup-48461, name=VN2) 10/23/2018 11:29:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48462 name: VN3 10/23/2018 11:29:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup VN3 already allowed. 10/23/2018 11:29:55 AM [cvm.database] [INFO]: Saved Virtual Network model for VN3 10/23/2018 11:29:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=9f864193-4449-4d8b-a156-41eb0dec2bf1, key=dvportgroup-48462, name=VN3) 10/23/2018 11:29:55 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48475 name: ctest-vnet0-96646780 10/23/2018 11:29:55 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-96646780 already allowed. 10/23/2018 11:29:55 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-96646780 10/23/2018 11:29:55 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=85686022-a557-4148-bd8e-19281a03ae94, key=dvportgroup-48475, name=ctest-vnet0-96646780) 10/23/2018 11:29:56 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48476 name: ctest-vnet1-13049088 10/23/2018 11:29:56 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet1-13049088 already allowed. 10/23/2018 11:29:56 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet1-13049088 10/23/2018 11:29:56 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=30009d94-9f12-4278-b0fb-068bcd989d82, key=dvportgroup-48476, name=ctest-vnet1-13049088) 10/23/2018 11:29:56 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48481 name: ctest-vnet0-72006042 10/23/2018 11:29:56 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-72006042 already allowed. 10/23/2018 11:29:56 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-72006042 10/23/2018 11:29:56 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=a0173704-97df-49ea-b8fe-f8efd6e51f5f, key=dvportgroup-48481, name=ctest-vnet0-72006042) 10/23/2018 11:29:56 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48484 name: ctest-vnet0-73495412 10/23/2018 11:29:56 AM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vnet0-73495412 already allowed. 10/23/2018 11:29:56 AM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vnet0-73495412 10/23/2018 11:29:56 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=1f8a5d3a-2910-430b-9a76-2d2504da12ff, key=dvportgroup-48484, name=ctest-vnet0-73495412) 10/23/2018 11:29:56 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48486 name: vnet0 10/23/2018 11:29:56 AM [cvm.clients] [INFO]: VLAN Override for portgroup vnet0 already allowed. 10/23/2018 11:29:56 AM [cvm.database] [INFO]: Saved Virtual Network model for vnet0 10/23/2018 11:29:56 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=19d7528d-3cc0-487c-ae50-772601bbf8da, key=dvportgroup-48486, name=vnet0) 10/23/2018 11:29:56 AM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48487 name: vnet1 10/23/2018 11:29:56 AM [cvm.clients] [INFO]: VLAN Override for portgroup vnet1 already allowed. 10/23/2018 11:29:56 AM [cvm.database] [INFO]: Saved Virtual Network model for vnet1 10/23/2018 11:29:56 AM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=e4ca4233-7483-4088-9661-0e62df363c38, key=dvportgroup-48487, name=vnet1) 10/23/2018 11:29:56 AM [cvm.database] [INFO]: Could not find VM model with uuid 50265637-8829-fc50-89db-d80f49f1904a. 10/23/2018 11:29:57 AM [cvm.database] [INFO]: Could not find VM model with uuid 502439ff-e7eb-3203-0595-f798b0fab1de. 10/23/2018 11:29:58 AM [cvm.database] [INFO]: Could not find VM model with uuid 50242a1d-212e-3235-01f3-63d057df0c79. 10/23/2018 11:29:58 AM [cvm] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/23/2018 11:29:58 AM [cvm] [INFO]: Session Event: TCP Connected 10/23/2018 11:29:58 AM [cvm] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/23/2018 11:29:58 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/23/2018 11:29:58 AM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ 10.204.217.144:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector connection down >>, ] >> 10/23/2018 11:29:58 AM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/23/2018 11:29:58 AM [cvm] [INFO]: Processing event[EvTcpConnected] in state[Connect] 10/23/2018 11:29:58 AM [cvm] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 10/23/2018 11:29:58 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 11:29:58 AM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 11:29:58 AM [cvm] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 10/23/2018 11:29:58 AM [cvm] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 10/23/2018 11:29:58 AM [cvm] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 10/23/2018 11:29:58 AM [cvm] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 10/23/2018 11:29:59 AM [cvm.controllers] [INFO]: Synchronization complete 10/23/2018 11:35:59 AM [cvm.database] [INFO]: Could not find VM model with name vcenter_tiny_vm. 10/23/2018 11:36:01 AM [cvm.database] [INFO]: Could not find VM model with name vcenter_tiny_vm. 10/23/2018 11:36:03 AM [cvm.database] [INFO]: Could not find VM model with name vcenter_tiny_vm. 10/23/2018 11:36:05 AM [cvm.database] [INFO]: Could not find VM model with name vcenter_tiny_vm. 10/23/2018 11:36:07 AM [cvm.database] [INFO]: Could not find VM model with name vcenter_tiny_vm. 10/23/2018 11:36:10 AM [cvm.database] [INFO]: Could not find VM model with name vcenter_tiny_vm. 10/23/2018 12:09:27 PM [cvm.controllers] [INFO]: Detected event: 10/23/2018 12:09:27 PM [cvm.clients] [INFO]: Read from ESXi API middle_vm1 properties: {u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '5024e8e9-453d-f30a-dc7e-80f1d719d443', u'runtime.powerState': 'poweredOff', u'name': 'middle_vm1', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 12:09:27 PM [cvm.database] [INFO]: Could not find VM model with uuid 5024e8e9-453d-f30a-dc7e-80f1d719d443. 10/23/2018 12:09:27 PM [cvm.clients] [INFO]: Attempting to update Virtual Machine 5024e8e9-453d-f30a-dc7e-80f1d719d443 in VNC 10/23/2018 12:09:27 PM [cvm.clients] [INFO]: Virtual Machine 5024e8e9-453d-f30a-dc7e-80f1d719d443 not found in VNC - creating 10/23/2018 12:09:27 PM [cvm.clients] [INFO]: Virtual Machine 5024e8e9-453d-f30a-dc7e-80f1d719d443 created in VNC 10/23/2018 12:09:27 PM [cvm.services] [INFO]: Created VirtualMachineModel(uuid=5024e8e9-453d-f30a-dc7e-80f1d719d443, name=middle_vm1, vm_properties={u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '5024e8e9-453d-f30a-dc7e-80f1d719d443', u'runtime.powerState': 'poweredOff', u'name': 'middle_vm1', u'summary.runtime.host': 'vim.HostSystem:ha-host'}, interfaces=[]) 10/23/2018 12:09:27 PM [cvm.database] [INFO]: Saved Virtual Machine model for middle_vm1 10/23/2018 12:09:28 PM [cvm.controllers] [INFO]: Detected event: 10/23/2018 12:09:28 PM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 12:09:28 PM [cvm.database] [INFO]: Could not find VN model with key dvportgroup-48498. 10/23/2018 12:09:28 PM [cvm.services] [INFO]: Fetching new portgroup for key: dvportgroup-48498 10/23/2018 12:09:28 PM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48498 name: ctest-vn-44962720 10/23/2018 12:09:28 PM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-44962720 already allowed. 10/23/2018 12:09:28 PM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-44962720 10/23/2018 12:09:28 PM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=f67e7e34-4d45-4d43-ad9f-bb8d8f0f605b, key=dvportgroup-48498, name=ctest-vn-44962720) 10/23/2018 12:09:28 PM [cvm.database] [INFO]: Could not find VN model with key dvportgroup-48499. 10/23/2018 12:09:28 PM [cvm.services] [INFO]: Fetching new portgroup for key: dvportgroup-48499 10/23/2018 12:09:28 PM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48499 name: ctest-vn-56597770 10/23/2018 12:09:28 PM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-56597770 already allowed. 10/23/2018 12:09:28 PM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-56597770 10/23/2018 12:09:28 PM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c2a71d4d-7cb1-44cb-ae77-e6310eb27133, key=dvportgroup-48499, name=ctest-vn-56597770) 10/23/2018 12:09:28 PM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=0e38274c-4042-30c6-bf38-41b798877af2, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:3d:79, port_key=117022, portgroup_key=dvportgroup-48498, vlan_id=None), ip_address=unset) 10/23/2018 12:09:28 PM [cvm.clients] [INFO]: Reading VLAN ID of port 117022 10/23/2018 12:09:28 PM [cvm.clients] [INFO]: Port: 117022 has no VLAN ID 10/23/2018 12:09:31 PM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 117022 to 1 10/23/2018 12:09:31 PM [cvm.clients] [INFO]: Successfully set VLAN ID: 1 for port: 117022 10/23/2018 12:09:31 PM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 0e38274c-4042-30c6-bf38-41b798877af2 in VNC 10/23/2018 12:09:31 PM [cvm.clients] [INFO]: Virtual Machine Interface 0e38274c-4042-30c6-bf38-41b798877af2 not found in VNC - creating 10/23/2018 12:09:32 PM [cvm.clients] [INFO]: Virtual Machine Interface 0e38274c-4042-30c6-bf38-41b798877af2 created in VNC 10/23/2018 12:09:32 PM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-vn-44962720-middle_vm1 10/23/2018 12:09:32 PM [cvm.clients] [INFO]: Created Instance IP: d12c2962-2b86-346a-9e74-dea6b4aea05b 10/23/2018 12:09:32 PM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-vn-44962720-middle_vm1 10/23/2018 12:09:32 PM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=0e38274c-4042-30c6-bf38-41b798877af2, display_name=vmi-ctest-vn-44962720-middle_vm1, vcenter_port=VCenterPort(mac_address=00:50:56:a4:3d:79, port_key=117022, portgroup_key=dvportgroup-48498, vlan_id=1), ip_address=74.184.116.251) 10/23/2018 12:09:32 PM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=49d9738f-4227-36b7-b94d-35bd1122cab5, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:c5:8e, port_key=117150, portgroup_key=dvportgroup-48499, vlan_id=None), ip_address=unset) 10/23/2018 12:09:32 PM [cvm.clients] [INFO]: Reading VLAN ID of port 117150 10/23/2018 12:09:32 PM [cvm.clients] [INFO]: Port: 117150 has no VLAN ID 10/23/2018 12:09:35 PM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 117150 to 2 10/23/2018 12:09:35 PM [cvm.clients] [ERROR]: vCenter task in unknown state: queued 10/23/2018 12:09:35 PM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 49d9738f-4227-36b7-b94d-35bd1122cab5 in VNC 10/23/2018 12:09:35 PM [cvm.clients] [INFO]: Virtual Machine Interface 49d9738f-4227-36b7-b94d-35bd1122cab5 not found in VNC - creating 10/23/2018 12:09:35 PM [cvm.clients] [INFO]: Virtual Machine Interface 49d9738f-4227-36b7-b94d-35bd1122cab5 created in VNC 10/23/2018 12:09:35 PM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-vn-56597770-middle_vm1 10/23/2018 12:09:35 PM [cvm.clients] [INFO]: Created Instance IP: 97a218df-6378-375e-b3cc-bfcb3dc4f8dd 10/23/2018 12:09:35 PM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-vn-56597770-middle_vm1 10/23/2018 12:09:35 PM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=49d9738f-4227-36b7-b94d-35bd1122cab5, display_name=vmi-ctest-vn-56597770-middle_vm1, vcenter_port=VCenterPort(mac_address=00:50:56:a4:c5:8e, port_key=117150, portgroup_key=dvportgroup-48499, vlan_id=2), ip_address=36.172.65.251) 10/23/2018 12:09:35 PM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: 0e38274c-4042-30c6-bf38-41b798877af2 10/23/2018 12:09:35 PM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 0e38274c-4042-30c6-bf38-41b798877af2 10/23/2018 12:09:35 PM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': '0e38274c-4042-30c6-bf38-41b798877af2', 'display_name': 'middle_vm1', 'rx_vlan': 1, 'vn_id': 'f67e7e34-4d45-4d43-ad9f-bb8d8f0f605b', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 1, 'port_type': 2, 'mac_address': '00:50:56:a4:3d:79', 'interface_name': '0e38274c-4042-30c6-bf38-41b798877af2', 'ip_address': u'74.184.116.251', 'vm_uuid_str': '5024e8e9-453d-f30a-dc7e-80f1d719d443'} 10/23/2018 12:09:35 PM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: 49d9738f-4227-36b7-b94d-35bd1122cab5 10/23/2018 12:09:36 PM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 49d9738f-4227-36b7-b94d-35bd1122cab5 10/23/2018 12:09:36 PM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': '49d9738f-4227-36b7-b94d-35bd1122cab5', 'display_name': 'middle_vm1', 'rx_vlan': 2, 'vn_id': 'c2a71d4d-7cb1-44cb-ae77-e6310eb27133', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 2, 'port_type': 2, 'mac_address': '00:50:56:a4:c5:8e', 'interface_name': '49d9738f-4227-36b7-b94d-35bd1122cab5', 'ip_address': u'36.172.65.251', 'vm_uuid_str': '5024e8e9-453d-f30a-dc7e-80f1d719d443'} 10/23/2018 12:09:36 PM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 0e38274c-4042-30c6-bf38-41b798877af2 10/23/2018 12:09:36 PM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 49d9738f-4227-36b7-b94d-35bd1122cab5 10/23/2018 12:09:36 PM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 12:09:36 PM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=0e38274c-4042-30c6-bf38-41b798877af2, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:3d:79, port_key=117022, portgroup_key=dvportgroup-48498, vlan_id=None), ip_address=unset) 10/23/2018 12:09:36 PM [cvm.clients] [INFO]: Reading VLAN ID of port 117022 10/23/2018 12:09:36 PM [cvm.clients] [INFO]: Port: 117022 VLAN ID: 1 10/23/2018 12:09:36 PM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 0e38274c-4042-30c6-bf38-41b798877af2 in VNC 10/23/2018 12:09:36 PM [cvm.clients] [INFO]: Deleting old Instance IP for Interface 0e38274c-4042-30c6-bf38-41b798877af2 10/23/2018 12:09:36 PM [cvm.clients] [INFO]: Updating Virtual Network of Interface 0e38274c-4042-30c6-bf38-41b798877af2 to ctest-vn-44962720 10/23/2018 12:09:36 PM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-vn-44962720-middle_vm1 10/23/2018 12:09:37 PM [cvm.clients] [INFO]: Created Instance IP: d12c2962-2b86-346a-9e74-dea6b4aea05b 10/23/2018 12:09:37 PM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-vn-44962720-middle_vm1 10/23/2018 12:09:37 PM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=0e38274c-4042-30c6-bf38-41b798877af2, display_name=vmi-ctest-vn-44962720-middle_vm1, vcenter_port=VCenterPort(mac_address=00:50:56:a4:3d:79, port_key=117022, portgroup_key=dvportgroup-48498, vlan_id=1), ip_address=74.184.116.251) 10/23/2018 12:09:37 PM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=49d9738f-4227-36b7-b94d-35bd1122cab5, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:c5:8e, port_key=117150, portgroup_key=dvportgroup-48499, vlan_id=None), ip_address=unset) 10/23/2018 12:09:37 PM [cvm.clients] [INFO]: Reading VLAN ID of port 117150 10/23/2018 12:09:37 PM [cvm.clients] [INFO]: Port: 117150 VLAN ID: 2 10/23/2018 12:09:37 PM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 49d9738f-4227-36b7-b94d-35bd1122cab5 in VNC 10/23/2018 12:09:37 PM [cvm.clients] [INFO]: Deleting old Instance IP for Interface 49d9738f-4227-36b7-b94d-35bd1122cab5 10/23/2018 12:09:37 PM [cvm.clients] [INFO]: Updating Virtual Network of Interface 49d9738f-4227-36b7-b94d-35bd1122cab5 to ctest-vn-56597770 10/23/2018 12:09:37 PM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-vn-56597770-middle_vm1 10/23/2018 12:09:37 PM [cvm.clients] [INFO]: Created Instance IP: 97a218df-6378-375e-b3cc-bfcb3dc4f8dd 10/23/2018 12:09:37 PM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-vn-56597770-middle_vm1 10/23/2018 12:09:37 PM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=49d9738f-4227-36b7-b94d-35bd1122cab5, display_name=vmi-ctest-vn-56597770-middle_vm1, vcenter_port=VCenterPort(mac_address=00:50:56:a4:c5:8e, port_key=117150, portgroup_key=dvportgroup-48499, vlan_id=2), ip_address=36.172.65.251) 10/23/2018 12:09:37 PM [cvm.clients] [INFO]: Read vRouter port with uuid: 0e38274c-4042-30c6-bf38-41b798877af2, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427860:09:37.478870', u'dns-server': u'74.184.116.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:3d:79', u'rx-vlan-id': 1, u'tx-vlan-id': 1, u'gateway': u'74.184.116.254', u'system-name': u'0e38274c-4042-30c6-bf38-41b798877af2', u'ip-address': u'74.184.116.251', u'vn-id': u'f67e7e34-4d45-4d43-ad9f-bb8d8f0f605b', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'0e38274c-4042-30c6-bf38-41b798877af2', u'instance-id': u'5024e8e9-453d-f30a-dc7e-80f1d719d443'} 10/23/2018 12:09:37 PM [cvm.clients] [INFO]: Read vRouter port with uuid: 49d9738f-4227-36b7-b94d-35bd1122cab5, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427860:09:37.485972', u'dns-server': u'36.172.65.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:c5:8e', u'rx-vlan-id': 2, u'tx-vlan-id': 2, u'gateway': u'36.172.65.254', u'system-name': u'49d9738f-4227-36b7-b94d-35bd1122cab5', u'ip-address': u'36.172.65.251', u'vn-id': u'c2a71d4d-7cb1-44cb-ae77-e6310eb27133', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'49d9738f-4227-36b7-b94d-35bd1122cab5', u'instance-id': u'5024e8e9-453d-f30a-dc7e-80f1d719d443'} 10/23/2018 12:09:37 PM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 0e38274c-4042-30c6-bf38-41b798877af2 10/23/2018 12:09:37 PM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 49d9738f-4227-36b7-b94d-35bd1122cab5 10/23/2018 12:09:37 PM [cvm.services] [INFO]: VM middle_vm1 was powered on 10/23/2018 12:09:37 PM [cvm.database] [INFO]: Saved Virtual Machine model for middle_vm1 10/23/2018 12:09:38 PM [cvm.clients] [INFO]: Enabled vRouter port with uuid: 0e38274c-4042-30c6-bf38-41b798877af2 10/23/2018 12:09:38 PM [cvm.clients] [INFO]: Enabled vRouter port with uuid: 49d9738f-4227-36b7-b94d-35bd1122cab5 10/23/2018 12:11:39 PM [cvm.services] [INFO]: VMware tools on VM middle_vm1 are running 10/23/2018 12:11:39 PM [cvm.database] [INFO]: Saved Virtual Machine model for middle_vm1 10/23/2018 12:12:20 PM [cvm.services] [INFO]: VMware tools on VM middle_vm1 are not running 10/23/2018 12:12:20 PM [cvm.database] [INFO]: Saved Virtual Machine model for middle_vm1 10/23/2018 12:12:20 PM [cvm.services] [INFO]: VM middle_vm1 was powered off 10/23/2018 12:12:20 PM [cvm.database] [INFO]: Saved Virtual Machine model for middle_vm1 10/23/2018 12:12:20 PM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 0e38274c-4042-30c6-bf38-41b798877af2 10/23/2018 12:12:20 PM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 49d9738f-4227-36b7-b94d-35bd1122cab5 10/23/2018 12:12:21 PM [cvm.controllers] [INFO]: Detected event: 10/23/2018 12:12:22 PM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 171, in main gevent.spawn(vmware_monitor.start()), File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 13, in start self._controller.handle_update(update_set) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 31, in handle_update self._update_handler.handle_update(update_set) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 43, in handle_update handler.handle_change(object_update.obj, property_change) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 54, in handle_change self._handle_change(obj, value) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 89, in _handle_change self._handle_event(value) File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 216, in _handle_event self._vmi_service.remove_vmis_for_vm_model(vm_name) File "/usr/lib/python2.7/site-packages/cvm/services.py", line 177, in remove_vmis_for_vm_model full_remove = self._vcenter_api_client.can_remove_vm(vm_model.uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 574, in __call__ return self.f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 394, in _InvokeAccessor return self._stub.InvokeAccessor(self, info) File "/usr/lib/python2.7/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor options=self._pcType.RetrieveOptions(maxObjects=1)) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 580, in self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 386, in _InvokeMethod return self._stub.InvokeMethod(self, info, args) File "/usr/lib/python2.7/site-packages/pyVmomi/SoapAdapter.py", line 1370, in InvokeMethod raise obj # pylint: disable-msg=E0702 vmodl.fault.ManagedObjectNotFound: (vmodl.fault.ManagedObjectNotFound) { dynamicType = , dynamicProperty = (vmodl.DynamicProperty) [], msg = "The object 'vim.VirtualMachine:vm-48502' has already been deleted or has not been completely created", faultCause = , faultMessage = (vmodl.LocalizableMessage) [], obj = 'vim.VirtualMachine:vm-48502' } 10/23/2018 12:12:24 PM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 12:12:24 PM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 12:12:24 PM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 12:12:24 PM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 12:12:24 PM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 12:12:24 PM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 12:12:24 PM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 12:12:24 PM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 12:12:24 PM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 12:12:24 PM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540296744211268 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.140:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 12:12:24 PM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 12:12:24 PM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 12:12:24 PM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 12:12:24 PM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 12:12:24 PM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 12:12:24 PM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 12:12:24 PM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 12:12:24 PM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48498 name: ctest-vn-44962720 10/23/2018 12:12:24 PM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-44962720 already allowed. 10/23/2018 12:12:24 PM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-44962720 10/23/2018 12:12:24 PM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=f67e7e34-4d45-4d43-ad9f-bb8d8f0f605b, key=dvportgroup-48498, name=ctest-vn-44962720) 10/23/2018 12:12:25 PM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48499 name: ctest-vn-56597770 10/23/2018 12:12:25 PM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-56597770 already allowed. 10/23/2018 12:12:25 PM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-56597770 10/23/2018 12:12:25 PM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c2a71d4d-7cb1-44cb-ae77-e6310eb27133, key=dvportgroup-48499, name=ctest-vn-56597770) 10/23/2018 12:12:26 PM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 574, in __call__ return self.f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 394, in _InvokeAccessor return self._stub.InvokeAccessor(self, info) File "/usr/lib/python2.7/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor options=self._pcType.RetrieveOptions(maxObjects=1)) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 580, in self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 386, in _InvokeMethod return self._stub.InvokeMethod(self, info, args) File "/usr/lib/python2.7/site-packages/pyVmomi/SoapAdapter.py", line 1370, in InvokeMethod raise obj # pylint: disable-msg=E0702 vmodl.fault.ManagedObjectNotFound: (vmodl.fault.ManagedObjectNotFound) { dynamicType = , dynamicProperty = (vmodl.DynamicProperty) [], msg = "The object 'vim.VirtualMachine:vm-48501' has already been deleted or has not been completely created", faultCause = , faultMessage = (vmodl.LocalizableMessage) [], obj = 'vim.VirtualMachine:vm-48501' } 10/23/2018 12:12:28 PM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 12:12:28 PM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 12:12:28 PM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 12:12:28 PM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 12:12:28 PM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 12:12:28 PM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 12:12:28 PM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 12:12:28 PM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 12:12:28 PM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 12:12:28 PM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540296748507433 collector_name = collector_ip = collector_list = [ 10.204.217.139:8086, 10.204.217.140:8086, 10.204.217.144:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 12:12:28 PM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 12:12:28 PM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 12:12:28 PM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 12:12:29 PM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 12:12:29 PM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 12:12:29 PM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 12:12:29 PM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 12:12:29 PM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48498 name: ctest-vn-44962720 10/23/2018 12:12:29 PM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-44962720 already allowed. 10/23/2018 12:12:29 PM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-44962720 10/23/2018 12:12:29 PM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=f67e7e34-4d45-4d43-ad9f-bb8d8f0f605b, key=dvportgroup-48498, name=ctest-vn-44962720) 10/23/2018 12:12:29 PM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48499 name: ctest-vn-56597770 10/23/2018 12:12:29 PM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-56597770 already allowed. 10/23/2018 12:12:29 PM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-56597770 10/23/2018 12:12:29 PM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c2a71d4d-7cb1-44cb-ae77-e6310eb27133, key=dvportgroup-48499, name=ctest-vn-56597770) 10/23/2018 12:12:30 PM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 129, in _delete_unused_vmis if self._vcenter_api_client.can_remove_vmi(vnc_vmi): File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 245, in can_remove_vmi return self.can_remove_vm(uuid=vm_uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 237, in can_remove_vm return not self._get_vm_by_uuid(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 40, in _get_vm_by_uuid return [vm for vm in container.view if vm.config.instanceUuid == uuid][0] File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 574, in __call__ return self.f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 394, in _InvokeAccessor return self._stub.InvokeAccessor(self, info) File "/usr/lib/python2.7/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor options=self._pcType.RetrieveOptions(maxObjects=1)) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 580, in self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 386, in _InvokeMethod return self._stub.InvokeMethod(self, info, args) File "/usr/lib/python2.7/site-packages/pyVmomi/SoapAdapter.py", line 1370, in InvokeMethod raise obj # pylint: disable-msg=E0702 vmodl.fault.ManagedObjectNotFound: (vmodl.fault.ManagedObjectNotFound) { dynamicType = , dynamicProperty = (vmodl.DynamicProperty) [], msg = "The object 'vim.VirtualMachine:vm-48500' has already been deleted or has not been completely created", faultCause = , faultMessage = (vmodl.LocalizableMessage) [], obj = 'vim.VirtualMachine:vm-48500' } 10/23/2018 12:12:32 PM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 12:12:32 PM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 12:12:32 PM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 12:12:32 PM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 12:12:32 PM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 12:12:32 PM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 12:12:32 PM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 12:12:32 PM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 12:12:32 PM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 12:12:32 PM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540296752852550 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.139:8086, 10.204.217.140:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 12:12:32 PM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 12:12:33 PM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 12:12:33 PM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 12:12:33 PM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 12:12:33 PM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 12:12:33 PM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 12:12:33 PM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 12:12:33 PM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48498 name: ctest-vn-44962720 10/23/2018 12:12:33 PM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-44962720 already allowed. 10/23/2018 12:12:33 PM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-44962720 10/23/2018 12:12:33 PM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=f67e7e34-4d45-4d43-ad9f-bb8d8f0f605b, key=dvportgroup-48498, name=ctest-vn-44962720) 10/23/2018 12:12:33 PM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48499 name: ctest-vn-56597770 10/23/2018 12:12:33 PM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-56597770 already allowed. 10/23/2018 12:12:33 PM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-56597770 10/23/2018 12:12:33 PM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c2a71d4d-7cb1-44cb-ae77-e6310eb27133, key=dvportgroup-48499, name=ctest-vn-56597770) 10/23/2018 12:12:34 PM [cvm.services] [INFO]: Deleting efb27f19-10b6-3f1c-ab36-474750fe2251 from VNC. 10/23/2018 12:12:34 PM [cvm.clients] [ERROR]: Could not find VMI efb27f19-10b6-3f1c-ab36-474750fe2251 in VNC 10/23/2018 12:12:34 PM [cvm.clients] [ERROR]: Virtual Machine Interface efb27f19-10b6-3f1c-ab36-474750fe2251 not found in VNC. Unable to delete 10/23/2018 12:12:35 PM [cvm.services] [INFO]: Deleting 230bb430-c8d1-3add-94f5-5204dd4189dc from VNC. 10/23/2018 12:12:35 PM [cvm.clients] [INFO]: Virtual Machine Interface 230bb430-c8d1-3add-94f5-5204dd4189dc removed from VNC 10/23/2018 12:12:35 PM [cvm.services] [INFO]: Deleting 0e38274c-4042-30c6-bf38-41b798877af2 from VNC. 10/23/2018 12:12:35 PM [cvm.clients] [INFO]: Virtual Machine Interface 0e38274c-4042-30c6-bf38-41b798877af2 removed from VNC 10/23/2018 12:12:36 PM [cvm.services] [INFO]: Deleting 49d9738f-4227-36b7-b94d-35bd1122cab5 from VNC. 10/23/2018 12:12:36 PM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 24, in sync self._vmi_service.sync_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 39, in sync_vmis self._delete_unused_vmis() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 131, in _delete_unused_vmis self._vnc_api_client.delete_vmi(vnc_vmi.get_uuid()) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 395, in delete_vmi self.vnc_lib.virtual_machine_interface_delete(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 805, in _object_delete self._request_server(OP_DELETE, uri) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1085, in _request % (op, url, data, content)) NoIdError: Unknown id: Error: oper 4 url /virtual-machine-interface/49d9738f-4227-36b7-b94d-35bd1122cab5 body None response No virtual-machine-interface object found for id 49d9738f-4227-36b7-b94d-35bd1122cab5 10/23/2018 12:12:39 PM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 12:12:39 PM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 12:12:39 PM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 12:12:39 PM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 12:12:39 PM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 12:12:39 PM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 12:12:39 PM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 12:12:39 PM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 12:12:39 PM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 12:12:39 PM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540296759142404 collector_name = collector_ip = collector_list = [ 10.204.217.144:8086, 10.204.217.140:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 12:12:39 PM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 12:12:39 PM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 12:12:39 PM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 12:12:39 PM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 12:12:39 PM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 12:12:39 PM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 12:12:39 PM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 12:12:40 PM [cvm.database] [INFO]: Could not find VM model with uuid 50265637-8829-fc50-89db-d80f49f1904a. 10/23/2018 12:12:40 PM [cvm.database] [INFO]: Could not find VM model with uuid 502439ff-e7eb-3203-0595-f798b0fab1de. 10/23/2018 12:12:41 PM [cvm.database] [INFO]: Could not find VM model with uuid 50248c3d-fd84-9f09-9d05-e18a264bd946. 10/23/2018 12:12:42 PM [cvm.services] [INFO]: Deleting 50248c3d-fd84-9f09-9d05-e18a264bd946 from VNC 10/23/2018 12:12:42 PM [cvm.clients] [INFO]: Attempting to delete Virtual Machine 50248c3d-fd84-9f09-9d05-e18a264bd946 from VNC... 10/23/2018 12:12:42 PM [cvm] [CRITICAL]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 182, in main(parsed_args) File "/usr/lib/python2.7/site-packages/cvm/__main__.py", line 169, in main vmware_monitor.sync() File "/usr/lib/python2.7/site-packages/cvm/monitors.py", line 7, in sync self._controller.sync() File "/usr/lib/python2.7/site-packages/cvm/controllers.py", line 25, in sync self._vm_service.delete_unused_vms_in_vnc() File "/usr/lib/python2.7/site-packages/cvm/services.py", line 280, in delete_unused_vms_in_vnc self._vnc_api_client.delete_vm(vnc_vm.uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 320, in delete_vm vm = self.read_vm(uuid) File "/usr/lib/python2.7/site-packages/cvm/clients.py", line 350, in read_vm return self.vnc_lib.virtual_machine_read(id=uuid) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 51, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 670, in _object_read response = self._request_server(OP_GET, uri, query_params) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1032, in _request_server retry_after_authn=retry_after_authn, retry_count=retry_count) File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1085, in _request % (op, url, data, content)) NoIdError: Unknown id: Error: oper 2 url /virtual-machine/50248c3d-fd84-9f09-9d05-e18a264bd946 body {'exclude_back_refs': True, 'exclude_children': True} response Unknown id: 50248c3d-fd84-9f09-9d05-e18a264bd946 10/23/2018 12:12:45 PM [cvm.controllers] [INFO]: Synchronizing CVM... 10/23/2018 12:12:45 PM [cvm] [ERROR]: Cannot write http_port 9090 to /tmp/cvm.0.http_port 10/23/2018 12:12:45 PM [cvm] [ERROR]: Starting Introspect on HTTP Port 9090 10/23/2018 12:12:45 PM [cvm] [INFO]: Processing event[EvStart] in state[none] 10/23/2018 12:12:45 PM [cvm] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/23/2018 12:12:45 PM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 12:12:45 PM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> 10/23/2018 12:12:45 PM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 12:12:45 PM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/23/2018 12:12:45 PM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei35-compute-vm:Compute:cvm:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 9090 start_time = 1540296765645286 collector_name = collector_ip = collector_list = [ 10.204.217.140:8086, 10.204.217.144:8086, 10.204.217.139:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/23/2018 12:12:45 PM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/23/2018 12:12:45 PM [cvm.clients] [INFO]: Retrieving reserved VLAN IDs 10/23/2018 12:12:45 PM [cvm.clients] [INFO]: Read from ESXi API ContrailVM-i27_datacenter11-10.204.217.147 properties: {u'guest.toolsRunningStatus': 'guestToolsRunning', u'config.instanceUuid': '50248a13-623c-3c74-0f79-e7230292505a', u'runtime.powerState': 'poweredOn', u'name': 'ContrailVM-i27_datacenter11-10.204.217.147', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 12:12:46 PM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48445 name: public_network 10/23/2018 12:12:46 PM [cvm.clients] [INFO]: VLAN Override for portgroup public_network already allowed. 10/23/2018 12:12:46 PM [cvm.database] [INFO]: Saved Virtual Network model for public_network 10/23/2018 12:12:46 PM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=93e7cae7-92a1-4128-aad3-1aed0aafd3b7, key=dvportgroup-48445, name=public_network) 10/23/2018 12:12:46 PM [cvm.database] [INFO]: Could not find VM model with uuid 50265637-8829-fc50-89db-d80f49f1904a. 10/23/2018 12:12:47 PM [cvm.database] [INFO]: Could not find VM model with uuid 502439ff-e7eb-3203-0595-f798b0fab1de. 10/23/2018 12:12:48 PM [cvm.database] [INFO]: Could not find VM model with uuid 50242a1d-212e-3235-01f3-63d057df0c79. 10/23/2018 12:12:48 PM [cvm.controllers] [INFO]: Synchronization complete 10/23/2018 12:12:49 PM [cvm] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/23/2018 12:12:49 PM [cvm] [INFO]: Session Event: TCP Connected 10/23/2018 12:12:49 PM [cvm] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/23/2018 12:12:49 PM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/23/2018 12:12:49 PM [cvm] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei35-compute-vm process_status = [ << module_id = contrail-vcenter-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ 10.204.217.140:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector connection down >>, ] >> 10/23/2018 12:12:49 PM [cvm] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/23/2018 12:12:49 PM [cvm] [INFO]: Processing event[EvTcpConnected] in state[Connect] 10/23/2018 12:12:49 PM [cvm] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 10/23/2018 12:12:49 PM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 12:12:49 PM [cvm] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 10/23/2018 12:12:49 PM [cvm] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 10/23/2018 12:12:49 PM [cvm] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 10/23/2018 12:12:49 PM [cvm] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 10/23/2018 12:12:49 PM [cvm] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 10/23/2018 12:20:49 PM [cvm.controllers] [INFO]: Detected event: 10/23/2018 12:20:49 PM [cvm.clients] [INFO]: Read from ESXi API middle_vm1 properties: {u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '502445c5-7aa2-a1c6-6450-445e4ffe0c18', u'runtime.powerState': 'poweredOff', u'name': 'middle_vm1', u'summary.runtime.host': 'vim.HostSystem:ha-host'} 10/23/2018 12:20:49 PM [cvm.database] [INFO]: Could not find VM model with uuid 502445c5-7aa2-a1c6-6450-445e4ffe0c18. 10/23/2018 12:20:49 PM [cvm.clients] [INFO]: Attempting to update Virtual Machine 502445c5-7aa2-a1c6-6450-445e4ffe0c18 in VNC 10/23/2018 12:20:49 PM [cvm.clients] [INFO]: Virtual Machine 502445c5-7aa2-a1c6-6450-445e4ffe0c18 not found in VNC - creating 10/23/2018 12:20:49 PM [cvm.clients] [INFO]: Virtual Machine 502445c5-7aa2-a1c6-6450-445e4ffe0c18 created in VNC 10/23/2018 12:20:49 PM [cvm.services] [INFO]: Created VirtualMachineModel(uuid=502445c5-7aa2-a1c6-6450-445e4ffe0c18, name=middle_vm1, vm_properties={u'guest.toolsRunningStatus': 'guestToolsNotRunning', u'config.instanceUuid': '502445c5-7aa2-a1c6-6450-445e4ffe0c18', u'runtime.powerState': 'poweredOff', u'name': 'middle_vm1', u'summary.runtime.host': 'vim.HostSystem:ha-host'}, interfaces=[]) 10/23/2018 12:20:49 PM [cvm.database] [INFO]: Saved Virtual Machine model for middle_vm1 10/23/2018 12:20:49 PM [cvm.controllers] [INFO]: Detected event: 10/23/2018 12:20:50 PM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 12:20:50 PM [cvm.database] [INFO]: Could not find VN model with key dvportgroup-48503. 10/23/2018 12:20:50 PM [cvm.services] [INFO]: Fetching new portgroup for key: dvportgroup-48503 10/23/2018 12:20:50 PM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48503 name: ctest-vn-27624927 10/23/2018 12:20:50 PM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-27624927 already allowed. 10/23/2018 12:20:50 PM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-27624927 10/23/2018 12:20:50 PM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=c319c2eb-d7dc-477f-9b76-f92e7e2d10bd, key=dvportgroup-48503, name=ctest-vn-27624927) 10/23/2018 12:20:50 PM [cvm.database] [INFO]: Could not find VN model with key dvportgroup-48504. 10/23/2018 12:20:50 PM [cvm.services] [INFO]: Fetching new portgroup for key: dvportgroup-48504 10/23/2018 12:20:50 PM [cvm.services] [INFO]: Fetched new portgroup key: dvportgroup-48504 name: ctest-vn-31929771 10/23/2018 12:20:50 PM [cvm.clients] [INFO]: VLAN Override for portgroup ctest-vn-31929771 already allowed. 10/23/2018 12:20:50 PM [cvm.database] [INFO]: Saved Virtual Network model for ctest-vn-31929771 10/23/2018 12:20:50 PM [cvm.services] [INFO]: Created VirtualNetworkModel(uuid=bff60414-8da0-409a-b299-ad9050aeafc4, key=dvportgroup-48504, name=ctest-vn-31929771) 10/23/2018 12:20:50 PM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=39366cd7-260d-3692-b8a3-f99e7814a907, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:36:d6, port_key=117278, portgroup_key=dvportgroup-48503, vlan_id=None), ip_address=unset) 10/23/2018 12:20:50 PM [cvm.clients] [INFO]: Reading VLAN ID of port 117278 10/23/2018 12:20:50 PM [cvm.clients] [INFO]: Port: 117278 has no VLAN ID 10/23/2018 12:20:53 PM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 117278 to 1 10/23/2018 12:20:53 PM [cvm.clients] [INFO]: Successfully set VLAN ID: 1 for port: 117278 10/23/2018 12:20:53 PM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 39366cd7-260d-3692-b8a3-f99e7814a907 in VNC 10/23/2018 12:20:53 PM [cvm.clients] [INFO]: Virtual Machine Interface 39366cd7-260d-3692-b8a3-f99e7814a907 not found in VNC - creating 10/23/2018 12:20:54 PM [cvm.clients] [INFO]: Virtual Machine Interface 39366cd7-260d-3692-b8a3-f99e7814a907 created in VNC 10/23/2018 12:20:54 PM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-vn-27624927-middle_vm1 10/23/2018 12:20:54 PM [cvm.clients] [INFO]: Created Instance IP: 1c58ac6a-64e9-3fad-805c-ab64b0bdbafc 10/23/2018 12:20:54 PM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-vn-27624927-middle_vm1 10/23/2018 12:20:54 PM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=39366cd7-260d-3692-b8a3-f99e7814a907, display_name=vmi-ctest-vn-27624927-middle_vm1, vcenter_port=VCenterPort(mac_address=00:50:56:a4:36:d6, port_key=117278, portgroup_key=dvportgroup-48503, vlan_id=1), ip_address=153.2.136.251) 10/23/2018 12:20:54 PM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=641e4a93-432d-3dd3-970e-a12ea9285889, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:bb:84, port_key=117406, portgroup_key=dvportgroup-48504, vlan_id=None), ip_address=unset) 10/23/2018 12:20:54 PM [cvm.clients] [INFO]: Reading VLAN ID of port 117406 10/23/2018 12:20:54 PM [cvm.clients] [INFO]: Port: 117406 has no VLAN ID 10/23/2018 12:20:57 PM [cvm.clients] [INFO]: Setting vCenter VLAN ID of port 117406 to 2 10/23/2018 12:20:57 PM [cvm.clients] [INFO]: Successfully set VLAN ID: 2 for port: 117406 10/23/2018 12:20:57 PM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 641e4a93-432d-3dd3-970e-a12ea9285889 in VNC 10/23/2018 12:20:57 PM [cvm.clients] [INFO]: Virtual Machine Interface 641e4a93-432d-3dd3-970e-a12ea9285889 not found in VNC - creating 10/23/2018 12:20:57 PM [cvm.clients] [INFO]: Virtual Machine Interface 641e4a93-432d-3dd3-970e-a12ea9285889 created in VNC 10/23/2018 12:20:57 PM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-vn-31929771-middle_vm1 10/23/2018 12:20:57 PM [cvm.clients] [INFO]: Created Instance IP: 889cb126-792a-3575-a2a0-0384f78d8708 10/23/2018 12:20:57 PM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-vn-31929771-middle_vm1 10/23/2018 12:20:57 PM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=641e4a93-432d-3dd3-970e-a12ea9285889, display_name=vmi-ctest-vn-31929771-middle_vm1, vcenter_port=VCenterPort(mac_address=00:50:56:a4:bb:84, port_key=117406, portgroup_key=dvportgroup-48504, vlan_id=2), ip_address=49.33.32.251) 10/23/2018 12:20:57 PM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: 39366cd7-260d-3692-b8a3-f99e7814a907 10/23/2018 12:20:57 PM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 39366cd7-260d-3692-b8a3-f99e7814a907 10/23/2018 12:20:58 PM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': '39366cd7-260d-3692-b8a3-f99e7814a907', 'display_name': 'middle_vm1', 'rx_vlan': 1, 'vn_id': 'c319c2eb-d7dc-477f-9b76-f92e7e2d10bd', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 1, 'port_type': 2, 'mac_address': '00:50:56:a4:36:d6', 'interface_name': '39366cd7-260d-3692-b8a3-f99e7814a907', 'ip_address': u'153.2.136.251', 'vm_uuid_str': '502445c5-7aa2-a1c6-6450-445e4ffe0c18'} 10/23/2018 12:20:58 PM [cvm.clients] [INFO]: Unable to read vRouter port with uuid: 641e4a93-432d-3dd3-970e-a12ea9285889 10/23/2018 12:20:58 PM [cvm.clients] [INFO]: Removed port from vRouter with uuid: 641e4a93-432d-3dd3-970e-a12ea9285889 10/23/2018 12:20:58 PM [cvm.clients] [INFO]: Added port to vRouter with parameters: {'vif_uuid_str': '641e4a93-432d-3dd3-970e-a12ea9285889', 'display_name': 'middle_vm1', 'rx_vlan': 2, 'vn_id': 'bff60414-8da0-409a-b299-ad9050aeafc4', 'vm_project_id': u'009912e165374bc6a1aa105d31928848', 'vlan': 2, 'port_type': 2, 'mac_address': '00:50:56:a4:bb:84', 'interface_name': '641e4a93-432d-3dd3-970e-a12ea9285889', 'ip_address': u'49.33.32.251', 'vm_uuid_str': '502445c5-7aa2-a1c6-6450-445e4ffe0c18'} 10/23/2018 12:20:58 PM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 39366cd7-260d-3692-b8a3-f99e7814a907 10/23/2018 12:20:58 PM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 641e4a93-432d-3dd3-970e-a12ea9285889 10/23/2018 12:20:58 PM [cvm.controllers] [INFO]: Detected VmReconfiguredEvent with device 10/23/2018 12:20:58 PM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=39366cd7-260d-3692-b8a3-f99e7814a907, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:36:d6, port_key=117278, portgroup_key=dvportgroup-48503, vlan_id=None), ip_address=unset) 10/23/2018 12:20:59 PM [cvm.clients] [INFO]: Reading VLAN ID of port 117278 10/23/2018 12:20:59 PM [cvm.clients] [INFO]: Port: 117278 VLAN ID: 1 10/23/2018 12:20:59 PM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 39366cd7-260d-3692-b8a3-f99e7814a907 in VNC 10/23/2018 12:20:59 PM [cvm.clients] [INFO]: Deleting old Instance IP for Interface 39366cd7-260d-3692-b8a3-f99e7814a907 10/23/2018 12:20:59 PM [cvm.clients] [INFO]: Updating Virtual Network of Interface 39366cd7-260d-3692-b8a3-f99e7814a907 to ctest-vn-27624927 10/23/2018 12:20:59 PM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-vn-27624927-middle_vm1 10/23/2018 12:20:59 PM [cvm.clients] [INFO]: Created Instance IP: 1c58ac6a-64e9-3fad-805c-ab64b0bdbafc 10/23/2018 12:20:59 PM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-vn-27624927-middle_vm1 10/23/2018 12:20:59 PM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=39366cd7-260d-3692-b8a3-f99e7814a907, display_name=vmi-ctest-vn-27624927-middle_vm1, vcenter_port=VCenterPort(mac_address=00:50:56:a4:36:d6, port_key=117278, portgroup_key=dvportgroup-48503, vlan_id=1), ip_address=153.2.136.251) 10/23/2018 12:20:59 PM [cvm.services] [INFO]: Updating VirtualMachineInterfaceModel(uuid=641e4a93-432d-3dd3-970e-a12ea9285889, display_name=None, vcenter_port=VCenterPort(mac_address=00:50:56:a4:bb:84, port_key=117406, portgroup_key=dvportgroup-48504, vlan_id=None), ip_address=unset) 10/23/2018 12:20:59 PM [cvm.clients] [INFO]: Reading VLAN ID of port 117406 10/23/2018 12:20:59 PM [cvm.clients] [INFO]: Port: 117406 VLAN ID: 2 10/23/2018 12:20:59 PM [cvm.clients] [INFO]: Attempting to update Virtual Machine Interface 641e4a93-432d-3dd3-970e-a12ea9285889 in VNC 10/23/2018 12:20:59 PM [cvm.clients] [INFO]: Deleting old Instance IP for Interface 641e4a93-432d-3dd3-970e-a12ea9285889 10/23/2018 12:20:59 PM [cvm.clients] [INFO]: Updating Virtual Network of Interface 641e4a93-432d-3dd3-970e-a12ea9285889 to ctest-vn-31929771 10/23/2018 12:20:59 PM [cvm.models] [INFO]: Constructing Instance IP for vmi-ctest-vn-31929771-middle_vm1 10/23/2018 12:20:59 PM [cvm.clients] [INFO]: Created Instance IP: 889cb126-792a-3575-a2a0-0384f78d8708 10/23/2018 12:20:59 PM [cvm.database] [INFO]: Saved Virtual Machine Interface model for vmi-ctest-vn-31929771-middle_vm1 10/23/2018 12:20:59 PM [cvm.services] [INFO]: Updated VirtualMachineInterfaceModel(uuid=641e4a93-432d-3dd3-970e-a12ea9285889, display_name=vmi-ctest-vn-31929771-middle_vm1, vcenter_port=VCenterPort(mac_address=00:50:56:a4:bb:84, port_key=117406, portgroup_key=dvportgroup-48504, vlan_id=2), ip_address=49.33.32.251) 10/23/2018 12:20:59 PM [cvm.clients] [INFO]: Read vRouter port with uuid: 39366cd7-260d-3692-b8a3-f99e7814a907, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427860:20:59.653701', u'dns-server': u'153.2.136.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:36:d6', u'rx-vlan-id': 1, u'tx-vlan-id': 1, u'gateway': u'153.2.136.254', u'system-name': u'39366cd7-260d-3692-b8a3-f99e7814a907', u'ip-address': u'153.2.136.251', u'vn-id': u'c319c2eb-d7dc-477f-9b76-f92e7e2d10bd', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'39366cd7-260d-3692-b8a3-f99e7814a907', u'instance-id': u'502445c5-7aa2-a1c6-6450-445e4ffe0c18'} 10/23/2018 12:20:59 PM [cvm.clients] [INFO]: Read vRouter port with uuid: 641e4a93-432d-3dd3-970e-a12ea9285889, port properties: {u'vhostuser-mode': 0, u'ip6-address': u'::', u'time': u'427860:20:59.658311', u'dns-server': u'49.33.32.253', u'author': u'/usr/bin/contrail-vrouter-agent', u'mac-address': u'00:50:56:a4:bb:84', u'rx-vlan-id': 2, u'tx-vlan-id': 2, u'gateway': u'49.33.32.254', u'system-name': u'641e4a93-432d-3dd3-970e-a12ea9285889', u'ip-address': u'49.33.32.251', u'vn-id': u'bff60414-8da0-409a-b299-ad9050aeafc4', u'plen': 24, u'vm-project-id': u'009912e1-6537-4bc6-a1aa-105d31928848', u'id': u'641e4a93-432d-3dd3-970e-a12ea9285889', u'instance-id': u'502445c5-7aa2-a1c6-6450-445e4ffe0c18'} 10/23/2018 12:20:59 PM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 39366cd7-260d-3692-b8a3-f99e7814a907 10/23/2018 12:20:59 PM [cvm.clients] [INFO]: Disabled vRouter port with uuid: 641e4a93-432d-3dd3-970e-a12ea9285889 10/23/2018 12:21:00 PM [cvm.services] [INFO]: VM middle_vm1 was powered on 10/23/2018 12:21:00 PM [cvm.database] [INFO]: Saved Virtual Machine model for middle_vm1 10/23/2018 12:21:00 PM [cvm.clients] [INFO]: Enabled vRouter port with uuid: 39366cd7-260d-3692-b8a3-f99e7814a907 10/23/2018 12:21:00 PM [cvm.clients] [INFO]: Enabled vRouter port with uuid: 641e4a93-432d-3dd3-970e-a12ea9285889 10/23/2018 12:23:02 PM [cvm.services] [INFO]: VMware tools on VM middle_vm1 are running 10/23/2018 12:23:02 PM [cvm.database] [INFO]: Saved Virtual Machine model for middle_vm1