tenant-mgmt-net is not reachable after updating router to DVR

[Peng Peng]

Brief Description
-----------------
After updating route to distributed, boot up VMs vms with server group policy anti-affinity; it is failed to ping tenant-mgmt-net

Severity
--------
Major

Steps to Reproduce
------------------
- Update router to distributed
   neutron router-update --distributed True
- Boot vm with server group policy anti-affinity
- ping tenant-mgmt-net

Expected Behavior
------------------
ping successful

Actual Behavior
----------------
ping failed

Reproducibility
---------------
Reproducible (7/10)

System Configuration
--------------------
Multi-node system

Branch/Pull Time/Commit
-----------------------
master as of 2018-09-19_20-18-00

Timestamp/Logs
--------------
[2018-09-20 10:21:44,201] 262 DEBUG MainThread ssh.send :: Send 'neutron --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne router-update 8f6e84c1-19ec-4f41-bd11-ac909b9137ae --distributed True'

[2018-09-20 10:22:33,152] 691 INFO MainThread vm_helper.boot_vm :: nova boot --key-name keypair-tenant2 --boot-volume 03eb693b-0a7c-4d5c-b7f2-5b6ff85c3a0b --hint group=802267e5-c5ad-409e-ac3c-8ef64d7696b0 --flavor a214df1f-46b6-4528-88f0-e8a72cad2705 --nic net-id=9edafbcb-d951-40a6-9c1a-42bf692e457c,vif-model=virtio --nic net-id=d528326a-3b52-49d5-9fb4-ec09285336ee,vif-model=virtio --nic net-id=18319041-2916-4021-b52c-92185a142ffa,vif-model=e1000 tenant2-dvr_ew_traffic-3 --poll

[2018-09-20 10:26:13,255] 262 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne list --all-tenants'
[2018-09-20 10:26:15,099] 382 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+--------------------------+----------------------------------+--------+------------+-------------+-----------------------------------------------------------------------------------------+
| ID | Name | Tenant ID | Status | Task State | Power State | Networks |
+--------------------------------------+--------------------------+----------------------------------+--------+------------+-------------+-----------------------------------------------------------------------------------------+
| 59a3c116-4e79-4550-975d-aee9d9d11f7f | tenant2-dvr_ew_traffic-3 | 5695b33a72cb481aaa41b776d6668d1c | ACTIVE | - | Running | tenant2-net0=172.18.0.130; internal0-net0-3=10.3.3.132; tenant2-mgmt-net=192.168.247.45 |
+--------------------------------------+--------------------------+----------------------------------+--------+------------+-------------+-----------------------------------------------------------------------------------------+
controller-1:~$
[2018-09-20 10:26:15,099] 262 DEBUG MainThread ssh.send :: Send 'echo $?'
[2018-09-20 10:26:15,202] 382 DEBUG MainThread ssh.expect :: Output:
0
controller-1:~$
[2018-09-20 10:26:15,202] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for id: 59a3c116-4e79-4550-975d-aee9d9d11f7f: [0]
[2018-09-20 10:26:15,203] 1442 DEBUG MainThread network_helper._get_net_ips_for_vms:: targeted_ip_str: tenant2-mgmt-net=192.168.247.45, ips for vm: ['192.168.247.45']
[2018-09-20 10:26:15,203] 1459 INFO MainThread network_helper._get_net_ips_for_vms:: IPs dict: {'59a3c116-4e79-4550-975d-aee9d9d11f7f': ['192.168.247.45']}
[2018-09-20 10:26:15,203] 419 DEBUG MainThread ssh.exec_cmd:: Executing command...
[2018-09-20 10:26:15,203] 262 DEBUG MainThread ssh.send :: Send 'ping -c 3 192.168.247.45'
[2018-09-20 10:26:18,314] 382 DEBUG MainThread ssh.expect :: Output:
PING 192.168.247.45 (192.168.247.45) 56(84) bytes of data.
From 192.168.47.3 icmp_seq=1 Destination Host Unreachable
From 192.168.47.3 icmp_seq=2 Destination Host Unreachable
From 192.168.47.3 icmp_seq=3 Destination Host Unreachable

--- 192.168.247.45 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2003ms

[Ghada Khalil (gkhalil)]

Targeting stx.2019.03 as this is related to DVR which is not a critical deployment for STX

[Ghada Khalil (gkhalil)]

This issue was reported on pike. StarlingX has rebased to openstack master/stein. Please re-test and open a new bug if the issue is still reproducible.

** Please also note that StarlingX is no longer carrying any custom code for neutron, so issues found may be related to standard openstack.