Router interfaces down and VM unreachable after changing neutron router to DVR then back to centralized

[Peng Peng]

Brief Description
-----------------
update neutron to DVR, then update it to centralized. The router interfaces and VM are not reachable.

Severity
--------
Major

Steps to Reproduce
------------------
Create router as centralized
update router to DVR
update route to centralized.
tried to reach tenant1-mgmt-net, but failed

TC-name:
test_dvr_vms_network_connection

Expected Behavior
------------------

Actual Behavior
----------------

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Multi-node system

Lab-name: WCP_3-6

Branch/Pull Time/Commit
-----------------------
2019-11-21_20-00-00

Last Pass
---------
2019-08-23_20-59-00

Timestamp/Logs
--------------
[2019-11-23 17:56:48,357] 311 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne router set --disable --distributed 66af863c-c34e-4ee4-8c40-395a4e7c7b34'
[2019-11-23 17:56:53,524] 311 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne router set --enable 66af863c-c34e-4ee4-8c40-395a4e7c7b34'

[2019-11-23 18:14:49,567] 311 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne router set --centralized --disable 66af863c-c34e-4ee4-8c40-395a4e7c7b34'
[2019-11-23 18:14:54,518] 311 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne router set --enable 66af863c-c34e-4ee4-8c40-395a4e7c7b34'

+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------+--------+-----------------+--------------------------+------+
| ID | Name | MAC Address | Fixed IP Addresses | Status | Security Groups | Device Owner | Tags |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------+--------+-----------------+--------------------------+------+
| 108f184a-869a-44c9-bdf3-1070c90850eb | | fa:16:3e:e3:96:52 | ip_address='192.168.120.33', subnet_id='d76fbaa1-b7ed-4554-b54a-9f7cbec5917a' | DOWN | | network:router_interface | |
| 58454d50-4289-46d9-8970-f343eac115bb | | fa:16:3e:ae:85:0a | ip_address='192.168.120.65', subnet_id='e6abb2ac-84b0-4462-8cf6-46f485d2927e' | DOWN | | network:router_interface | |
| 65496f91-c4d9-4efb-b1b7-46d55a16de74 | | fa:16:3e:11:73:88 | ip_address='192.168.120.1', subnet_id='51fb4d93-2310-4b89-8e58-89530edd31ca' | DOWN | | network:router_interface | |
| f8405df0-1a47-4268-b6a2-a8f964a8f95b | | fa:16:3e:25:35:fa | ip_address='192.168.20.2', subnet_id='de25fcbb-6d74-4847-a04c-fd15a60a072f' | ACTIVE | | network:router_gateway | |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------+--------+-----------------+--------------------------+------+

[2019-11-23 18:17:42,525] 311 DEBUG MainThread ssh.send :: Send 'ssh -v -o RSAAuthentication=no -o PubkeyAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@192.168.120.15'
[2019-11-23 18:17:45,572] 433 DEBUG MainThread ssh.expect :: Output:
OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 192.168.120.15 [192.168.120.15] port 22.
debug1: connect to address 192.168.120.15 port 22: No route to host
ssh: connect to host 192.168.120.15 port 22: No route to host
svc-cgcsauto@tis-lab-nat-box:~$
[2019-11-23 18:17:45,573] 1241 INFO MainThread ssh.connect :: Unable to ssh to 192.168.120.15

Test Activity
-------------
Regression Testing

[Ghada Khalil (gkhalil)]

Assigning to networking PL for follow-up

[Yang Liu (yliu12)]

OpenStack configuration for WR lab is attached.
The neutron routers were created as centralized by default, and they were working fine initially until router update.

[marvin Yu (marvin-yu)]

Due to device limitation, I have tried to reproduce this bug at stx-aio-dx.

Steps to Reproduce
------------------
Create router as centralized (pass)
update router to DVR (pass)
update route to centralized (pass)
ping to vm (pass)

It`s seems work well and no exception occur. if I’ve missed some steps, Let me know.

By the way, EXEC the command "openstack router set --disable --distributed a3ec2cca-c1ec-4f2b-b7f2-d53453e1f10c" will make an error. I have to EXEC --disable first, and then do --distributed.

controller-0:/home/sysadmin# openstack router list
+--------------------------------------+-----------------+--------+-------+----------------------------------+-------------+-------+
| ID | Name | Status | State | Project | Distributed | HA |
+--------------------------------------+-----------------+--------+-------+----------------------------------+-------------+-------+
| 43e237e1-dcb2-4bc1-8859-e7556dfe1e22 | private-router0 | ACTIVE | UP | 76a69c24405a4ac1a98502219d674c48 | False | False |
| a3ec2cca-c1ec-4f2b-b7f2-d53453e1f10c | public-router0 | ACTIVE | UP | 76a69c24405a4ac1a98502219d674c48 | False | False |
+--------------------------------------+-----------------+--------+-------+----------------------------------+-------------+-------+
controller-0:/home/sysadmin# openstack router set --disable --distributed a3ec2cca-c1ec-4f2b-b7f2-d53453e1f10c
BadRequestException: 400: Client Error for url: http://neutron.openstack.svc.cluster.local:80/v2.0/routers/a3ec2cca-c1ec-4f2b-b7f2-d53453e1f10c, Bad router request: Cannot change the 'distributed' attribute of active routers. Please set router admin_state_up to False prior to upgrade.

[Ghada Khalil (gkhalil)]

As discussed in the stx networking meeting, reproduction should be attempted on a multi-node system. This issue could be specific to this configuration.

[marvin Yu (marvin-yu)]

I have deployed a multi-node env to test, the ISO was created on 11/25. But Refer to the following steps, I can`t reproduce this bug.

Steps to Reproduce
------------------
Create router as centralized (pass)
update router to DVR (pass)
update route to centralized (pass)
ssh to vm (pass)

@pengpeng, could you help to check the detail steps as below?
if I miss some steps important, please Let me know.

controller-0:~/ctn_provisioning$ openstack router list
+--------------------------------------+-----------------+--------+-------+----------------------------------+-------------+-------+
| ID | Name | Status | State | Project | Distributed | HA |
+--------------------------------------+-----------------+--------+-------+----------------------------------+-------------+-------+
| 1ec156a2-26b2-4906-8d45-5bc242db0c01 | public-router0 | ACTIVE | UP | 78ce0b32ed4d421796358060c82fb27c | False | False |
| 6db39de6-bfa8-4c80-bb9f-d5519ecd7e5a | private-router0 | ACTIVE | UP | 78ce0b32ed4d421796358060c82fb27c | False | False |
+--------------------------------------+-----------------+--------+-------+----------------------------------+-------------+-------+
controller-0:~/ctn_provisioning$ openstack router set --disable 1ec156a2-26b2-4906-8d45-5bc242db0c01
controller-0:~/ctn_provisioning$ openstack router set --distributed 1ec156a2-26b2-4906-8d45-5bc242db0c01
controller-0:~/ctn_provisioning$ openstack router set --enable 1ec156a2-26b2-4906-8d45-5bc242db0c01
controller-0:~/ctn_provisioning$ openstack router list
+--------------------------------------+-----------------+--------+-------+----------------------------------+-------------+-------+
| ID | Name | Status | State | Project | Distributed | HA |
+--------------------------------------+-----------------+--------+-------+----------------------------------+-------------+-------+
| 1ec156a2-26b2-4906-8d45-5bc242db0c01 | public-router0 | ACTIVE | UP | 78ce0b32ed4d421796358060c82fb27c | True | False |
| 6db39de6-bfa8-4c80-bb9f-d5519ecd7e5a | private-router0 | ACTIVE | UP | 78ce0b32ed4d421796358060c82fb27c | False | False |
+--------------------------------------+-----------------+--------+-------+----------------------------------+-------------+-------+
controller-0:~/ctn_provisioning$ openstack router set --disable 1ec156a2-26b2-4906-8d45-5bc242db0c01
controller-0:~/ctn_provisioning$ openstack router set --centralized 1ec156a2-26b2-4906-8d45-5bc242db0c01
controller-0:~/ctn_provisioning$ openstack router set --enable 1ec156a2-26b2-4906-8d45-5bc242db0c01
controller-0:~/ctn_provisioning$ openstack router list
+--------------------------------------+-----------------+--------+-------+----------------------------------+-------------+-------+
| ID | Name | Status | State | Project | Distributed | HA |
+--------------------------------------+-----------------+--------+-------+----------------------------------+-------------+-------+
| 1ec156a2-26b2-4906-8d...

[Yang Liu (yliu12)]

Hi Marvin,

How did you configure your neutron router?
Could you please check the ports for the router?

For peng, the internal interfaces for the router went down after the router update, that is the cause for vm not reachable. The vm may still have an ip assigned in this state.