Containers: two VMs not able to reach each other

[Peng Peng]

Brief Description
-----------------
Bring up two VMs from tis-centos-guest image, make sure both of them are reachable from outside (external network), try to ping VMs between them, but failed

Severity
--------
Minor

Steps to Reproduce
------------------
1. Boot up 2 VMs from tis-centos-guest image
2. ping them from external network
3. Ping between them

Expected Behaviour
------------------
Ping go through

Actual Behaviour
----------------
Ping failed

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Multi-node system

Branch/Pull Time/Commit
-----------------------
master as of 20190305T060000Z

Timestamp/Logs
--------------
[2019-03-07 09:19:21,231] 2626 WARNING MainThread network_helper.ping_server:: Ping from 192.168.211.72 to 192.168.211.77 failed.
[2019-03-07 09:19:21,231] 1589 INFO MainThread vm_helper._ping_vms:: Retry in 5 seconds
[2019-03-07 09:19:26,237] 2590 INFO MainThread network_helper.ping_server:: Ping 192.168.211.77 from host 192.168.211.72
[2019-03-07 09:19:26,237] 423 DEBUG MainThread ssh.exec_cmd:: Executing command...
[2019-03-07 09:19:26,237] 262 DEBUG MainThread ssh.send :: Send 'ping -c 5 192.168.211.77'
[2019-03-07 09:19:32,003] 387 DEBUG MainThread ssh.expect :: Output:
PING 192.168.211.77 (192.168.211.77) 56(84) bytes of data.
From 192.168.211.72 icmp_seq=1 Destination Host Unreachable
From 192.168.211.72 icmp_seq=2 Destination Host Unreachable

[Ghada Khalil (gkhalil)]

Requested information from the reporter on how reproducible this issue is and whether it was seen on only one system or multiple systems.

[Joseph Richard (josephrichard)]

When each VM pings the other VM at the same time, do the pings go through?

[Bruce Jones (brucej)]

Forrest has verified that this has been reproduced on 4 different setups in his lab. I'm going to ask Cindy Xie to have someone reproduce it as well.

[Ghada Khalil (gkhalil)]

From Numan sent to the stx discuss mailing list:
We have not seen https://bugs.launchpad.net/starlingx/+bug/1819031 issue in other labs (AIO-SX, AIO-DX and Multi-node with storage). We have successful VM connectivity in 20190305T060000Z build.

[Ghada Khalil (gkhalil)]

From Peng:
On load 20190315T013000Z, we did not see this issue on SM-2, WP_1-2, IP_7-12 & WCP_71-75.

Based on info from the reporter, this was a one time occurrence on one particular system. Lowering the priority accordingly.

I don't believe this is the same issue reported by Forrest's team which is now tracked separately via:
https://bugs.launchpad.net/starlingx/+bug/1820378

[Ghada Khalil (gkhalil)]

As per agreement with the reporter, this issue is not reproducible. Closing