Brief Description
-----------------
During sanity run, many TCs were failed by ping tenant management network. After investigation, we observed that actually system router external gateway is not reachable from natbox.
Severity
--------
Critical
Steps to Reproduce
------------------
ping route external gateway from natbox
TC-name: mtc/test_services_persists_over_reboot.py::test_system_persist_over_host_reboot[controller]
Expected Behavior
------------------
Actual Behavior
----------------
Reproducibility
---------------
Seen once
System Configuration
--------------------
Multi-node system
Lab-name: WCP_63-66
Branch/Pull Time/Commit
-----------------------
stx 2.0 as of 2019-08-26_20-59-00
Last Pass
---------
on master 2019-08-20_20-59-00
Timestamp/Logs
--------------
[2019-08-27 09:00:16,712] 301 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server list --a'
[2019-08-27 09:00:19,163] 423 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+--------------+--------+-------------------------------------------------------------+-------+----------------------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+--------------+--------+-------------------------------------------------------------+-------+----------------------+
| 9dd1ee5e-271d-4fec-85e8-fd3d6fea4938 | tenant1-vm-1 | ACTIVE | tenant1-mgmt-net=192.168.100.219; tenant1-net1=172.16.1.141 | | flavor-default-size2 |
+--------------------------------------+--------------+--------+-------------------------------------------------------------+-------+----------------------+
[sysadmin@controller-1 ~(keystone_admin)]$
[2019-08-27 09:00:19,163] 301 DEBUG MainThread ssh.send :: Send 'echo $?'
[2019-08-27 09:00:19,266] 423 DEBUG MainThread ssh.expect :: Output:
0
[sysadmin@controller-1 ~(keystone_admin)]$
[2019-08-27 09:00:19,267] 1654 DEBUG MainThread network_helper._get_net_ips_for_vms:: targeted ips for vm: ['192.168.100.219']
[2019-08-27 09:00:19,267] 1666 INFO MainThread network_helper._get_net_ips_for_vms:: IPs dict: {'9dd1ee5e-271d-4fec-85e8-fd3d6fea4938': ['192.168.100.219']}
[2019-08-27 09:00:19,267] 2525 INFO MainThread network_helper.ping_server:: Ping 192.168.100.219 from host 128.224.186.181
[2019-08-27 09:00:19,267] 466 DEBUG MainThread ssh.exec_cmd:: Executing command...
[2019-08-27 09:00:19,267] 301 DEBUG MainThread ssh.send :: Send 'ping -c 3 192.168.100.219'
[2019-08-27 09:00:22,369] 423 DEBUG MainThread ssh.expect :: Output:
PING 192.168.100.219 (192.168.100.219) 56(84) bytes of data.
From 10.10.100.1 icmp_seq=1 Destination Host Unreachable
From 10.10.100.1 icmp_seq=2 Destination Host Unreachable
From 10.10.100.1 icmp_seq=3 Destination Host Unreachable
[sysadmin@controller-1 ~(keystone_admin)]$ source ./openrc.admin
[sysadmin@controller-1 ~(keystone_admin)]$ neutron router-list
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+----------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+
| id | name | tenant_id | external_gateway_info | distributed | ha |
+--------------------------------------+----------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+
| 7992dd8e-6243-4cce-a3c7-1d6a3aeb9145 | tenant1-router | a56940f0c987491bb3c04369ad8e6c44 | {"network_id": "fe0f0fc0-309f-4b9e-8906-71be1d2cbe65", "enable_snat": false, "external_fixed_ips": [{"subnet_id": "b7450285-2b4c-4cb6-aa3e-67fbb6a39efb", "ip_address": "10.10.100.2"}]} | True | False |
| a44ac5e4-e2af-47db-80f7-e7cc7e8623a1 | tenant2-router | 4b452ac33f5b47e1a8a422851671fd44 | {"network_id": "fe0f0fc0-309f-4b9e-8906-71be1d2cbe65", "enable_snat": false, "external_fixed_ips": [{"subnet_id": "b7450285-2b4c-4cb6-aa3e-67fbb6a39efb", "ip_address": "10.10.100.3"}]} | False | False |
+--------------------------------------+----------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+
ppeng@tis-lab-nat-box:~$ ping 10.10.100.2
PING 10.10.100.2 (10.10.100.2) 56(84) bytes of data.
From 10.10.100.1 icmp_seq=1 Destination Host Unreachable
From 10.10.100.1 icmp_seq=2 Destination Host Unreachable
From 10.10.100.1 icmp_seq=3 Destination Host Unreachable
--- 10.10.100.2 ping statistics ---
6 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4999ms
pipe 3
ppeng@tis-lab-nat-box:~$ ping 10.10.100.3
PING 10.10.100.3 (10.10.100.3) 56(84) bytes of data.
--- 10.10.100.3 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1008ms
Test Activity
-------------
Sanity
A similar issue observed in BUILD_ID=
Lab: R720-1_2
ygebrema@
PING 192.168.10.3 (192.168.10.3) 56(84) bytes of data.
From 192.168.10.1 icmp_seq=1 Destination Host Unreachable
From 192.168.10.1 icmp_seq=2 Destination Host Unreachable
From 192.168.10.1 icmp_seq=3 Destination Host Unreachable
Logs are attached.