Neutron UDP traffic continue to get lost after adding security group rule to allow udp traffic

[Yvonne Ding]

Brief Description
-----------------
Enable neutron port security and create two security groups for each tenant. Apply this security group to vm pair with udp traffic. The delta value should not be changed between two fetches. But the delta becomes unstable with stable is set to true.

Severity
--------
Major

Steps to Reproduce
------------------
1. Setup traffic between VMs with ixncfg is configured as udp
2. Verify UDP traffic is not allowed
3. Allow UDP traffic on the fly and ensure the delta values is not changed
in between two fetches

TC-name:
TestPacketTypeSecurity::test_packet_type_security_modify_running_vm

Expected Behavior
-----------------
The delta value is not changed

Actual Behavior
----------------
The delta value is changed

Reproducibility
---------------
reproducible

System Configuration
--------------------
Regular standard 2+2

Lab-name:
wcp_7_10

Branch/Pull Time/Commit
-----------------------
BUILD_ID="r/stx.4.0"

Timestamp/Logs
--------------
[2020-07-17 05:41:28,165] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne security group rule create --remote-ip=0.0.0.0/0 --protocol=udp --ingress f0b79b88-d126-4906-8bc7-055e71df74f5'
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at | 2020-07-17T05:40:36Z |
| description | |
| direction | ingress |
| ether_type | IPv4 |
| id | 43104567-c0ae-4b09-b259-b8bff4a85f4c |
| location | cloud='', project.domain_id=, project.domain_name='Default', project.id='3c468d05132e441aa08e163828ddb639', project.name='admin', region_name='RegionOne', zone= |
| name | None |
| port_range_max | None |
| port_range_min | None |
| project_id | 3c468d05132e441aa08e163828ddb639 |
| protocol | udp |
| remote_group_id | None |
| remote_ip_prefix | 0.0.0.0/0 |
| revision_number | 0 |
| security_group_id | f0b79b88-d126-4906-8bc7-055e71df74f5 |
| tags | [] |
| updated_at | 2020-07-17T05:40:36Z |
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+

[2020-07-17 05:41:30,633] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne security group rule create --remote-ip=0.0.0.0/0 --protocol=udp --ingress 8f34c879-a7b4-47fc-8826-6d763de2f1ef'
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at | 2020-07-17T05:40:39Z |
| description | |
| direction | ingress |
| ether_type | IPv4 |
| id | e73823e9-27ca-4f43-842f-7203d630b44b |
| location | cloud='', project.domain_id=, project.domain_name='Default', project.id='3c468d05132e441aa08e163828ddb639', project.name='admin', region_name='RegionOne', zone= |
| name | None |
| port_range_max | None |
| port_range_min | None |
| project_id | 3c468d05132e441aa08e163828ddb639 |
| protocol | udp |
| remote_group_id | None |
| remote_ip_prefix | 0.0.0.0/0 |
| revision_number | 0 |
| security_group_id | 8f34c879-a7b4-47fc-8826-6d763de2f1ef |
| tags | [] |
| updated_at | 2020-07-17T05:40:39Z |
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[2020-07-17 05:41:33,031] 941 INFO MainThread ixia_helper.get_statistics:: matched with view ::ixNet::OBJ-/statistics/view:"Traffic Item Statistics"
[2020-07-17 05:41:34,038] 1015 INFO MainThread ixia_helper._get_delta:: Frames Delta=35872
[2020-07-17 05:41:44,049] 941 INFO MainThread ixia_helper.get_statistics:: matched with view ::ixNet::OBJ-/statistics/view:"Traffic Item Statistics"
[2020-07-17 05:41:45,056] 1015 INFO MainThread ixia_helper._get_delta:: Frames Delta=47872

......
[2020-07-17 05:45:57,464] 941 INFO MainThread ixia_helper.get_statistics:: matched with view ::ixNet::OBJ-/statistics/view:"Traffic Item Statistics"
[2020-07-17 05:45:58,472] 1015 INFO MainThread ixia_helper._get_delta:: Frames Delta=301872
[2020-07-17 05:46:08,483] 941 INFO MainThread ixia_helper.get_statistics:: matched with view ::ixNet::OBJ-/statistics/view:"Traffic Item Statistics"
[2020-07-17 05:46:09,491] 1015 INFO MainThread ixia_helper._get_delta:: Frames Delta=311872
[2020-07-17 05:46:19,495] 941 INFO MainThread ixia_helper.get_statistics:: matched with view ::ixNet::OBJ-/statistics/view:"Traffic Item Statistics"
[2020-07-17 05:46:20,501] 1015 INFO MainThread ixia_helper._get_delta:: Frames Delta=323872
[2020-07-17 05:46:30,513] 941 INFO MainThread ixia_helper.get_statistics:: matched with view ::ixNet::OBJ-/statistics/view:"Traffic Item Statistics"
[2020-07-17 05:46:31,520] 1015 INFO MainThread ixia_helper._get_delta:: Frames Delta=335872
[2020-07-17 05:46:41,532] 941 INFO MainThread ixia_helper.get_statistics:: matched with view ::ixNet::OBJ-/statistics/view:"Traffic Item Statistics"
[2020-07-17 05:46:42,540] 1015 INFO MainThread ixia_helper._get_delta:: Frames Delta=345872

[2020-07-17 05:46:54,837] 61 DEBUG MainThread conftest.update_results:: ***Failure at test call: /home/svc-cgcsauto/wassp-repos.new/testcases/cgcs/CGCSAuto/keywords/ixia_helper.py:1022: utils.exceptions.IxiaError: Ixia error.
***Details: self = <CGCSAuto.testcases.functional.neutron.test_qos_security.TestPacketTypeSecurity object at 0x7f16eb5f0710>
vm_type = 'virtio'
security_groups = ['f0b79b88-d126-4906-8bc7-055e71df74f5', '8f34c879-a7b4-47fc-8826-6d763de2f1ef']

Logs of .tar and automation log as below,
https://files.starlingx.kube.cengn.ca/launchpad/1888171

Test Activity
-------------
Test neutron function with the openstack install system

[Ghada Khalil (gkhalil)]

Issue w/ openstack regression after the rebase to Ussuri; maybe related to the rebase. Marking as stx.4.0 gating for now until further investigation by the distro.openstack team.

[Yan Chen (ychen2u)]

May need to upgrade openstack client for ussuri. There are new flags for security group.

https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/security-group.html

The test case need to be updated too.

[Yan Chen (ychen2u)]

May need to upgrade openstack client for ussuri.
The test case need to be updated too. Can we defer it to the maintenance release?

There are new flags for security group.
https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/security-group.html
Ussuri Release Notes also mentioned several changes on security group:
https://docs.openstack.org/releasenotes/neutron/ussuri.html

[Yan Chen (ychen2u)]

After went through the whole test log, I found something strange:

ixia_helper.configure:: configure: ::ixNet::OBJ-/vport:1/interface:2/ipv4:L1033 ['-gateway', '172.16.4.220', '-maskWidth', 24, '-ip', IPv4Address('172.16.4.1')]
ixia_helper.configure:: configure: ::ixNet::OBJ-/vport:2/interface:2/ipv4:L1035 ['-gateway', '172.18.1.246', '-maskWidth', 24, '-ip', IPv4Address('172.18.1.1')]

In this log, I think the IP addresses and gateway addresses are reversed, because '172.16.4.220' and '172.18.1.246' are actually the IP addresses of the vms.

| 4b082e70-07f6-4424-9b01-4ee5b26082dd | tenant2-virtio-0-29 | ACTIVE | internal0-net0-1=10.1.1.234; tenant2-mgmt-net=192.168.221.42; tenant2-net1=172.18.1.246 | | virtio-4 |
| d518345e-5ed6-45f0-a583-29000f75c8f5 | tenant1-virtio-0-28 | ACTIVE | internal0-net0-1=10.1.1.217; tenant1-mgmt-net=192.168.121.94; tenant1-net4=172.16.4.220 | | virtio |

And see from the whole log, the traffic statistics never passed, it seems that the connection is always failing.
Can you help to check if the test config is right? Or is there any log for a pass case?

[Yan Chen (ychen2u)]

And from my local test, cannot reproduce this issue.

The following test steps can pass:
1. Create 2 vms applied with a security group;
2. When there's no rule for UDP ingress, the UDP connection between the 2 vms failed;
3. Create a rule for UDP ingress over the security group, the UDP connection between the 2 vms succeeded.

Same test applied and passed on 2 vms with 2 different security group, or TCP/ICMP protocols.

See attached file security_group_rule_test.png.

[Yvonne Ding (yding)]

The issue is 100% reproduced with openstack STX master 4.0. The delta value keeps increasing between two fetches other than unchanged.

[2020-07-29 17:35:35,024] 941 INFO MainThread ixia_helper.get_statistics:: matched with view ::ixNet::OBJ-/statistics/view:"Traffic Item Statistics"
[2020-07-29 17:35:36,034] 1015 INFO MainThread ixia_helper._get_delta:: Frames Delta=35870
[2020-07-29 17:35:46,044] 941 INFO MainThread ixia_helper.get_statistics:: matched with view ::ixNet::OBJ-/statistics/view:"Traffic Item Statistics"
[2020-07-29 17:35:47,051] 1015 INFO MainThread ixia_helper._get_delta:: Frames Delta=47870
[2020-07-29 17:35:57,060] 941 INFO MainThread ixia_helper.get_statistics:: matched with view ::ixNet::OBJ-/statistics/view:"Traffic Item Statistics"
[2020-07-29 17:35:58,066] 1015 INFO MainThread ixia_helper._get_delta:: Frames Delta=59870
[2020-07-29 17:36:08,078] 941 INFO MainThread ixia_helper.get_statistics:: matched with view ::ixNet::OBJ-/statistics/view:"Traffic Item Statistics"
[2020-07-29 17:36:09,087] 1015 INFO MainThread ixia_helper._get_delta:: Frames Delta=69870
[2020-07-29 17:36:19,098] 941 INFO MainThread ixia_helper.get_statistics:: matched with view ::ixNet::OBJ-/statistics/view:"Traffic Item Statistics"
[2020-07-29 17:36:20,105] 1015 INFO MainThread ixia_helper._get_delta:: Frames Delta=81870
[2020-07-29 17:36:30,116] 941 INFO MainThread ixia_helper.get_statistics:: matched with view ::ixNet::OBJ-/statistics/view:"Traffic Item Statistics"
[2020-07-29 17:36:31,125] 1015 INFO MainThread ixia_helper._get_delta:: Frames Delta=91870
[2020-07-29 17:36:41,136] 941 INFO MainThread ixia_helper.get_statistics:: matched with view ::ixNet::OBJ-/statistics/view:"Traffic Item Statistics"
[2020-07-29 17:36:42,144] 1015 INFO MainThread ixia_helper._get_delta:: Frames Delta=103870
[2020-07-29 17:36:52,153] 941 INFO MainThread ixia_helper.get_statistics:: matched with view ::ixNet::OBJ-/statistics/view:"Traffic Item Statistics"
[2020-07-29 17:36:53,162] 1015 INFO MainThread ixia_helper._get_delta:: Frames Delta=113870
[2020-07-29 17:37:03,173] 941 INFO MainThread ixia_helper.get_statistics:: matched with view ::ixNet::OBJ-/statistics/view:"Traffic Item Statistics"
[2020-07-29 17:37:04,181] 1015 INFO MainThread ixia_helper._get_delta:: Frames Delta=125870
[2020-07-29 17:37:14,188] 941 INFO MainThread ixia_helper.get_statistics:: matched with view ::ixNet::OBJ-/statistics/view:"Traffic Item Statistics"
[2020-07-29 17:37:15,202] 1015 INFO MainThread ixia_helper._get_delta:: Frames Delta=135870
[2020-07-29 17:37:25,215] 941 INFO MainThread ixia_helper.get_statistics:: matched with view ::ixNet::OBJ-/statistics/view:"Traffic Item Statistics"
[2020-07-29 17:37:26,223] 1015 INFO MainThread ixia_helper._get_delta:: Frames Delta=147870
[2020-07-29 17:37:36,236] 941 INFO MainThread ixia_helper.get_statistics:: matched with view ::ixNet::OBJ-/statistics/view:"Traffic Item Statistics"
[2020-07-29 17:37:37,244] 1015 INFO MainThread ixia_helper._get_delta:: Frames Delta=157870
[2020-07-29 17:37:47,256] 941 INFO MainThread ixia_helper.get_statistics:: matched with view ::ixNet::OBJ-/statistics/view:"Traffic Item Sta...

[Yvonne Ding (yding)]

test image tis-centos-guest.img can be found here in case you need,
https://drive.google.com/drive/folders/1a4lHQ_op7qJ2ohXhsOYae-tJlOKC4dEi

[yong hu (yhu6)]

this issue might be related to currently used OpenStack client ("Train" version) with OpenStack services ("Ussuri" version). Considering it's impact, we push this LP to stx.5.0.

[Yan Chen (ychen2u)]

@yding, Yes, I will check your new log for lab setup, but please confirm the following log I found in your full automation log: Is the gateway/IP address set correctly?

ixia_helper.configure:: configure: ::ixNet::OBJ-/vport:1/interface:2/ipv4:L1033 ['-gateway', '172.16.4.220', '-maskWidth', 24, '-ip', IPv4Address('172.16.4.1')]
ixia_helper.configure:: configure: ::ixNet::OBJ-/vport:2/interface:2/ipv4:L1035 ['-gateway', '172.18.1.246', '-maskWidth', 24, '-ip', IPv4Address('172.18.1.1')]

[Austin Sun (sunausti)]

Hi, Yvonne:
   any update ? if not, I will close this bug

[Yvonne Ding (yding)]

Hi Austin,

Sure. Please close it.

Yvonne

On 2021-01-05 8:58 a.m., Austin Sun wrote:
> Hi, Yvonne:
> any update ? if not, I will close this bug
>

[Austin Sun (sunausti)]

As aligned, we will close this issue.