ping between VMs over management and data networks failed after cold migrate

[Peng Peng]

Brief Description
-----------------
Boot up VMs and verify ping between VM over management and data networks is good. After cold migrate, ping between VM over management and data networks failed.

Severity
--------
Major

Steps to Reproduce
------------------
Boot up VMs
verify ping between VM over management and data networks is good.
cold migrate,
verify ping between VM over management and data networks

TC-name: networking/test_multiple_ports.py::TestMutiPortsBasic::()::test_multiports_on_same_network_vm_actions[virtio_x4]

Expected Behavior
------------------
ping is good

Actual Behavior
----------------
ping failed

Reproducibility
---------------
Intermittent (5/10)

System Configuration
--------------------
Two node system

Lab-name: WCP_76-77

Branch/Pull Time/Commit
-----------------------
20200127T000002Z

Last Pass
---------
20200121T000002Z

Timestamp/Logs
--------------
[2020-02-05 13:06:26,446] 2562 INFO MainThread network_helper.ping_server:: All packets received by 172.16.1.148
[2020-02-05 13:06:26,447] 2527 INFO MainThread network_helper.ping_server:: Ping 172.16.1.147 from host 192.168.141.37
[2020-02-05 13:06:26,447] 479 DEBUG MainThread ssh.exec_cmd:: Executing command...
[2020-02-05 13:06:26,447] 314 DEBUG MainThread ssh.send :: Send 'ping -c 5 172.16.1.147'
[2020-02-05 13:06:30,556] 436 DEBUG MainThread ssh.expect :: Output:
PING 172.16.1.147 (172.16.1.147) 56(84) bytes of data.
64 bytes from 172.16.1.147: icmp_seq=1 ttl=64 time=0.333 ms
64 bytes from 172.16.1.147: icmp_seq=2 ttl=64 time=0.198 ms
64 bytes from 172.16.1.147: icmp_seq=3 ttl=64 time=0.143 ms
64 bytes from 172.16.1.147: icmp_seq=4 ttl=64 time=0.161 ms
64 bytes from 172.16.1.147: icmp_seq=5 ttl=64 time=0.154 ms

[2020-02-05 13:06:47,914] 314 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne migrate --poll 134c1cc4-5111-4932-b448-14f12defa75d'

[2020-02-05 13:10:47,534] 479 DEBUG MainThread ssh.exec_cmd:: Executing command...
[2020-02-05 13:10:47,534] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server list --a'
[2020-02-05 13:10:56,415] 436 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+---------------------------+--------+------------------------------------------------------------------------------------------------------+-------+-----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+---------------------------+--------+------------------------------------------------------------------------------------------------------+-------+-----------+
| 134c1cc4-5111-4932-b448-14f12defa75d | tenant1-multiports-4 | ACTIVE | tenant1-mgmt-net=192.168.141.53; tenant1-net1=172.16.1.148, 172.16.1.147, 172.16.1.176, 172.16.1.180 | | dedicated |
| e335953a-d90d-4f4c-b9c0-c5ec41fe29e8 | tenant1-multiports_base-3 | ACTIVE | internal0-net0=10.0.0.167; tenant1-mgmt-net=192.168.141.37; tenant1-net1=172.16.1.169 | | dedicated |
+--------------------------------------+---------------------------+--------+------------------------------------------------------------------------------------------------------+-------+-----------+

[2020-02-05 13:12:30,096] 2527 INFO MainThread network_helper.ping_server:: Ping 172.16.1.147 from host 192.168.141.37
[2020-02-05 13:12:30,096] 479 DEBUG MainThread ssh.exec_cmd:: Executing command...
[2020-02-05 13:12:30,096] 314 DEBUG MainThread ssh.send :: Send 'ping -c 5 172.16.1.147'
[2020-02-05 13:12:37,212] 436 DEBUG MainThread ssh.expect :: Output:
PING 172.16.1.147 (172.16.1.147) 56(84) bytes of data.
From 172.16.1.169 icmp_seq=1 Destination Host Unreachable
From 172.16.1.169 icmp_seq=2 Destination Host Unreachable
From 172.16.1.169 icmp_seq=3 Destination Host Unreachable
From 172.16.1.169 icmp_seq=4 Destination Host Unreachable
From 172.16.1.169 icmp_seq=5 Destination Host Unreachable

Test Activity
-------------
Sanity

[Peng Peng (ppeng)]

Log @
https://files.starlingx.kube.cengn.ca/launchpad/1862067

[Ghada Khalil (gkhalil)]

@Peng, what is the vswitch type? ovs-dpdk or none?

[Peng Peng (ppeng)]

VSWITCH_TYPE ovs-dpdk

http://128.224.150.21/jenkins/job/cgcs-wildcat-76_77_k8s/223/parameters/

[Ghada Khalil (gkhalil)]

Marking as Low since the issue is intermittent and possibly related to ovs-dpdk.