Egress UDP traffic is dropped

[Annie Melen]

Hello!

I've encountered the error while becnhmarking UDP traffic between two instances. Any help would be apreciated.

I've got two instances (test1: 172.31.200.232, test2: 172.31.200.234) located in one external network on separate compute nodes. Testing via iperf3 brought me to fail:

root@test1:~# iperf3 -s
root@test2:~# iperf3 -c 172.31.200.234 -u
Connecting to host 172.31.200.234, port 5201
iperf3: error - unable to read from stream socket: Resource temporarily unavailable

While, when I put together these instances on the same compute node, everything is fine:

root@test1:~# iperf3 -s
root@test2:~# iperf3 -c 172.31.200.234 -u
Connecting to host 172.31.200.234, port 5201
[ 4] local 172.31.200.232 port 57176 connected to 172.31.200.234 port 5201
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-1.00 sec 120 KBytes 983 Kbits/sec 15
[ 4] 1.00-2.00 sec 128 KBytes 1.05 Mbits/sec 16
[ 4] 2.00-3.00 sec 128 KBytes 1.05 Mbits/sec 16
[ 4] 3.00-3.22 sec 40.0 KBytes 1.46 Mbits/sec 5

My security groups allow UDP and TCP traffic.

Openstack Queens, openvswitch v.2.9
Firewall_driver is 'openvswitch', can it be the cause?

Thanks,
Annie

[Brian Haley (brian-haley)]

UDP egress should always be allowed unless you have modified them. Are you sure you have the ingress rules set correctly for all the ports iperf might be using?

Also, to rule out the openvswitch firewall you can always change to iptables_hybrid and re-test to make sure.

[Annie Melen (anniemelen)]

1. Default security group, which I use for now, has the following rules:

direction='ingress', ethertype='IPv4', port_range_max='65535', port_range_min='1', protocol='tcp', remote_ip_prefix='0.0.0.0/0',
direction='egress', ethertype='IPv4',
direction='ingress', ethertype='IPv4', protocol='icmp', remote_ip_prefix='0.0.0.0/0'
direction='ingress', ethertype='IPv4', port_range_max='65535', port_range_min='1', protocol='udp', remote_ip_prefix='0.0.0.0/0'

They should be enough, I think?

2. I should change firewall_driver only on compute nodes or on control nodes too?..

[Annie Melen (anniemelen)]

2. I've changed firewall_driver from 'openvswitch' to 'iptables_hybrid' on compute nodes, and udp works fine.
It seems there is some kind of bug with ovs fw?

[Slawek Kaplonski (slaweq)]

Hi,

Changing fw driver don't means anything in fact. When You spawned vms You had "hybrid_connection" set to False in binding profile so libvirt connected vms directly to br-int.
iptables fw driver require hybrid connection to be set to True so basically now You disabled security groups at all.
It may be that problem is in openvswitch firewall driver somewhere but it also can be something different.
Can You maybe turn openvswitch fw driver again and send list of all openflow rules from br-int on both nodes then?
Also some informations about what kind of network it is (vlan, flat) and some debugging exactly on which interface Your packets are dropped would be useful.

[Annie Melen (anniemelen)]

Slawek,
thanks for being envolved!

Dump-flows from compute nodes are attached:
compute01: https://paste.ubuntu.com/p/wKr8JPDT6g/
compute02: https://paste.ubuntu.com/p/T6s329CBZN/

About test network:
root@control02:~# openstack network show e1834c28-ab3a-4a8e-accd-c9aadc1d64c6
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | ts-demo |
| created_at | 2018-09-18T13:34:49Z |
| description | |
| dns_domain | None |
| id | e1834c28-ab3a-4a8e-accd-c9aadc1d64c6 |
| ipv4_address_scope | None |
| ipv6_address_scope | None |
| is_default | False |
| is_vlan_transparent | None |
| mtu | 1500 |
| name | External_for_UDP_test |
| port_security_enabled | True |
| project_id | 6ca1ee2afa534e87bc71881ef5567acb |
| provider:network_type | vlan |
| provider:physical_network | external |
| provider:segmentation_id | 400 |
| qos_policy_id | None |
| revision_number | 7 |
| router:external | External |
| segments | None |
| shared | True |
| status | ACTIVE |
| subnets | 387b23a5-0057-4c0b-91a7-cf55ee9c8a2e |
| tags | |
| updated_at | 2018-09-18T15:34:32Z |
+---------------------------+--------------------------------------+

[Brian Haley (brian-haley)]

Just a follow-on to Slawek's comment. If you changed the firewall driver and started new VMs, then they should be plugged differently and be using iptables, and not possibly bypassing rules.

But the flows might show something anyways, thanks for those.

[Annie Melen (anniemelen)]

Guys,

I have a question for you:
whenI've changed firewall_driver from 'openvswitch' to 'iptables_hybrid', my vms (which were created during ovs fw) have 'ovs_hybrid_plug=False' status, and new ones have 'ovs_hybrid_plug=True'. Can it affect my deployment in future somehow?

Thanks,
Annie

[Slawek Kaplonski (slaweq)]

@Annie: if Your vm was created when ovs fw driver was used and has ovs_hybrid_plug=False then if You switch to iptables firewall driver, this vm will not have security groups at all. That will be the only impact of such vm later.

[Slawek Kaplonski (slaweq)]

Hi, I tried to reproduce this issue on master branch on single node devstack. All worked fine for me.
Can You check exactly where traffic is blocked in Your case? Is it really on some openflow rules on destination host?
Also another question - does both Your vms using same security group rule? If so, all traffic should be accepted between them by default so adding other security group rules for upd traffic shouldn't be necessary at all.