security groups iptables can block legitimate traffic as INVALID

Bug #1460741 reported by Mike Dorman
22
This bug affects 4 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Kevin Benton
Juno
Fix Released
Undecided
Unassigned
Kilo
Fix Released
Undecided
Unassigned

Bug Description

The iptables implementation of security groups includes a default rule to drop any INVALID packets (according to the Linux connection state tracking system.) It looks like this:

-A neutron-openvswi-od0518220-e -m state --state INVALID -j DROP

This is placed near the top of the rule stack, before any security group rules added by the user. See:

https://github.com/openstack/neutron/blob/stable/kilo/neutron/agent/linux/iptables_firewall.py#L495
https://github.com/openstack/neutron/blob/stable/kilo/neutron/agent/linux/iptables_firewall.py#L506-L510

However, there are some cases where you would not want traffic marked as INVALID to be dropped here. Specifically, our use case:

We have a load balancing scheme where requests from the LB are tunneled as IP-in-IP encapsulation between the LB and the VM. Response traffic is configured for DSR, so the responses go directly out the default gateway of the VM.

The results of this are iptables on the hypervisor does not see the initial SYN from the LB to VM (because it is encapsulated in IP-in-IP), and thus it does not make it into the connection table. The response that comes out of the VM (not encapsulated) hits iptables on the hypervisor and is dropped as invalid.

I'd like to see a Neutron option to enable/disable the population of this INVALID state rule, so that operators (such as us) can disable it if desired. Obviously it's better in general to keep it in there to drop invalid packets, but there are cases where you would like to not do this.

Revision history for this message
Kent Nickell (kent-nickell) wrote :

This is also an issue in Juno. We have a different use case than the original (sctp instead of IP-IP), but the same underlying issue is causing valid SCTP packets to be dropped.

Revision history for this message
Anton Aksola (aakso) wrote :

Got customer report on this too. In our case there was a Load Balancer and missing RST packets greatly affect the fault detection.

Revision history for this message
Kevin Benton (kevinbenton) wrote :

Is it acceptable to use the port-security option to disable filtering on the port? Or do you still want rule enforcement just without the INVALID rule?

Revision history for this message
Mike Dorman (mdorman-m) wrote :

We would still want the rule enforcement.

Revision history for this message
Kevin Benton (kevinbenton) wrote :

Hi Mike,

Can you do me a favor and try out https://review.openstack.org/218517 ?

It moves the INVALID drop below the user-defined rules. With that patch applied, you should be able to add explicit allow rules that match the headers of the bad packets so they are permitted before the INVALID drop.

Changed in neutron:
assignee: nobody → Kevin Benton (kevinbenton)
status: New → Confirmed
Changed in neutron:
status: Confirmed → In Progress
Changed in neutron:
importance: Undecided → Medium
milestone: none → liberty-rc1
Revision history for this message
Mike Dorman (mdorman-m) wrote :

Yeah, I will give it a shot (just back from vacation, so not ignoring you here.) On the face of it it looks good though.

Revision history for this message
Kyle Mestery (mestery) wrote :

Leaving in Liberty-RC1 for now.

Revision history for this message
Mike Dorman (mdorman-m) wrote :

Brought in that patch and LGTM.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/218517
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0a258afc7ee3c03974dffa2c0dd0b7b367034cc7
Submitter: Jenkins
Branch: master

commit 0a258afc7ee3c03974dffa2c0dd0b7b367034cc7
Author: Kevin Benton <email address hidden>
Date: Fri Aug 28 00:50:59 2015 -0700

    Process user iptables rules before INVALID

    Process user-defined iptables rules before the INVALID DROP
    rule. This is to allow scenarios where the VMs need to
    legitimately receive packets that conntrack doesn't have an
    entry for (e.g. SYN-ACK where the SYN wasn't sent by the VM).
    A user can accomplish this by adding an allow rule that matches
    the headers of these INVALID packets so they get permitted before
    they hit the INVALID DROP rule.

    Closes-Bug: #1460741
    Change-Id: Ie6ce5f3fa688f1bf25b77db5955211922d9fe85b

Changed in neutron:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (feature/pecan)

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/224334

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/224357

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (feature/pecan)
Download full text (73.6 KiB)

Reviewed: https://review.openstack.org/224357
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fdc3431ccd219accf6a795079d9b67b8656eed8e
Submitter: Jenkins
Branch: feature/pecan

commit fe236bdaadb949661a0bfb9b62ddbe432b4cf5f1
Author: Miguel Angel Ajo <email address hidden>
Date: Thu Sep 3 15:40:12 2015 +0200

    No network devices on network attached qos policies

    Network devices, like internal router legs, or dhcp ports
    should not be affected by bandwidth limiting rules.

    This patch disables application of network attached policies
    to network/neutron owned ports.

    Closes-bug: #1486039
    DocImpact

    Change-Id: I75d80227f1e6c4b3f5fa7762b8dc3b0c0f1abd46

commit db4a06f7caa20a4c7879b58b20e95b223ed8eeaf
Author: Ken'ichi Ohmichi <email address hidden>
Date: Wed Sep 16 10:04:32 2015 +0000

    Use tempest-lib's token_client

    Now tempest-lib provides token_client modules as library and the
    interface is stable. So neutron repogitory doesn't need to contain
    these modules.
    This patch makes neutron use tempest-lib's token_client and removes
    the own modules for the maintenance.

    Change-Id: Ieff7eb003f6e8257d83368dbc80e332aa66a156c

commit 78aed58edbe6eb8a71339c7add491fe9de9a0546
Author: Jakub Libosvar <email address hidden>
Date: Thu Aug 13 09:08:20 2015 +0000

    Fix establishing UDP connection

    Previously, in establish_connection() for UDP protocol data were sent
    but never read on peer socket. That lead to successful read on peer side
    if this connection was filtered. Having constant testing string masked
    this issue as we can't distinguish to which test of connectivity data
    belong.

    This patch makes unique data string per test_connectivity() and
    also makes establish_connection() to create an ASSURED entry in
    conntrack table. Finally, in last test after firewall filter was
    removed, connection is re-established in order to avoid troubles with
    terminated processes or TCP continuing sending packets which weren't
    successfully delivered.

    Closes-Bug: 1478847
    Change-Id: I2920d587d8df8d96dc1c752c28f48ba495f3cf0f

commit e6292fcdd6262434a7b713ad8802db6bc8a6d3dc
Author: YAMAMOTO Takashi <email address hidden>
Date: Wed Sep 16 13:20:51 2015 +0900

    ovsdb: Fix a few docstring

    Change-Id: I53e1e21655b28fe5da60e58aeeb7cbbd103ae014

commit c22949a4449d96a67caa616290cf76b67b182917
Author: fumihiko kakuma <email address hidden>
Date: Wed Sep 16 11:52:59 2015 +0900

    Remove requirements.txt for the ofagent mechanism driver

    It is no longer used.

    Related-Blueprint: core-vendor-decomposition
    https://blueprints.launchpad.net/neutron/+spec/core-vendor-decomposition

    Change-Id: Ib31fb3febf8968e50d86dd66e1e6e1ea2313f8ac

commit d1d4de19d85f961d388c91e70f31b3bafec418c5
Author: Kevin Benton <email address hidden>
Date: Thu Sep 3 20:25:57 2015 -0700

    Always return iterables in L3 get_candidates

    The caller of this function expects iterables.

    Closes-Bug: #1494996
    Change-Id: I3d103e63f4e127a77268502415c0ddb0d804b54a

commit 1ad6ac448067306...

tags: added: in-feature-pecan
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (feature/pecan)

Change abandoned by Doug Wiegley (<email address hidden>) on branch: feature/pecan
Review: https://review.openstack.org/224334

Thierry Carrez (ttx)
Changed in neutron:
status: Fix Committed → Fix Released
tags: added: kilo-backport-potential
tags: added: juno-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/231503

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/kilo)

Reviewed: https://review.openstack.org/231503
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2f85b22d84244e8785389144232f446afd24db6b
Submitter: Jenkins
Branch: stable/kilo

commit 2f85b22d84244e8785389144232f446afd24db6b
Author: Kevin Benton <email address hidden>
Date: Fri Aug 28 00:50:59 2015 -0700

    Process user iptables rules before INVALID

    Process user-defined iptables rules before the INVALID DROP
    rule. This is to allow scenarios where the VMs need to
    legitimately receive packets that conntrack doesn't have an
    entry for (e.g. SYN-ACK where the SYN wasn't sent by the VM).
    A user can accomplish this by adding an allow rule that matches
    the headers of these INVALID packets so they get permitted before
    they hit the INVALID DROP rule.

    Conflicts:
         neutron/tests/unit/agent/test_securitygroups_rpc.py

    Closes-Bug: #1460741
    Change-Id: Ie6ce5f3fa688f1bf25b77db5955211922d9fe85b

tags: added: in-stable-kilo
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/231928

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/juno)

Reviewed: https://review.openstack.org/231928
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=58904f3626cdf09006bbb8ac9e60f9a24298f01e
Submitter: Jenkins
Branch: stable/juno

commit 58904f3626cdf09006bbb8ac9e60f9a24298f01e
Author: Kevin Benton <email address hidden>
Date: Fri Aug 28 00:50:59 2015 -0700

    Process user iptables rules before INVALID

    Process user-defined iptables rules before the INVALID DROP
    rule. This is to allow scenarios where the VMs need to
    legitimately receive packets that conntrack doesn't have an
    entry for (e.g. SYN-ACK where the SYN wasn't sent by the VM).
    A user can accomplish this by adding an allow rule that matches
    the headers of these INVALID packets so they get permitted before
    they hit the INVALID DROP rule.

    Conflicts:
     neutron/agent/linux/iptables_firewall.py
     neutron/tests/unit/test_iptables_firewall.py
     neutron/tests/unit/test_security_groups_rpc.py

    Closes-Bug: #1460741
    Change-Id: Ie6ce5f3fa688f1bf25b77db5955211922d9fe85b

tags: added: in-stable-juno
Thierry Carrez (ttx)
Changed in neutron:
milestone: liberty-rc1 → 7.0.0
Revision history for this message
BALAJI SRINIVASAN (balaji-vasan) wrote :

Hi Mike,

After installing kilo release, we found that SCTP packets VM were being dropped at the host.
After installing this patch. SCTP packets were not dropped. They were transmitted.
But they were not being NATed. SCTP packets go out of host with local IP Address of the VM(192.168.x.x) only for SCTP packets.

UDP packets are being NATed and have no problem.

Is this a known issue and fix is available?

Revision history for this message
Mike Dorman (mdorman-m) wrote :

I don't know of a fix, but IIRC the iptables rules only apply to udp, tcp, and icmp, and I think sctp is an entirely different one, right?

I would take a look at what iptables rules are being generated on the host for this VM and see what the particular nat table rules are. I am not sure what all differences there are from juno to kilo other than this.

Maybe it's a connection state tracking thing? I don't know if iptables and the kernel specifically supports connection tracking for sctp, but it's possible if you could turn that on, it would work.

Revision history for this message
BALAJI SRINIVASAN (balaji-vasan) wrote :

Found out that packets dropped is a known issue https://bugs.launchpad.net/neutron/+bug/1460741

and downloaded the neutron patch neutron 2015.1.2 and applied the same.

Centos security hardening had disabled iptables for sctp.

I did
echo "install sctp /bin/true" > /etc/modprobe.d/sctp.conf.

to make sctp to work.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.