proc/sys/net/ipv4/tcp_syncookies=1 should be seriously considered to permit SYN flood defense...

[enyc]

This is intended to be a 'wishlist' wulnerability -- w.r.t. procps and Edgy.

In my opinion,the /etc/sysctl.conf should have 'proc/sys/net/ipv4/tcp_syncookies=1' in order to permit the linux SYNcookies syn-flood trivial DoS attack to be mitigated as-necessary, by default.

Note that the disadvantages of connections initiated w/ SYNcookies enabled only apply when the system is under attack (SYN queue getting rather full), as the syncookies reply-with-only-one-SYN+ACK behaviour only 'kicks in' when the system has a SYN_RECVD backlog problem. (If SYNcookies were not permitted incoming TCP connections have a very low chance of succeeding at all while under SYN-flood attack).

Without this setting enabled, any TCP services on the machine can be DoSed from a dial-up line sending a stream of SYN packets from weird source addresses to open TCP ports like Samba/VNC/http/whatever....

Does anybody have any legitimate reason tcp_syncookies should be disabled?

Some people claimed that SYNcookies break some RFCs once but I have not seen any evidence to this effect, only notes from djb saying that this is not true.

Comments wanted please ;-)
Thankyou in advance,
-- enyc

[Jeremy Vies (jeremy.vies)]

Hi enyc,

I think "tcp_syncookies" is considered as part of the FW mechanism of the kernel.
As Dapper (and previous releases) does not provide any FW out of the box, it is normal that tcp_syncookies are not activated by default.
Your bug repport should be put as a wish for next release, and maybe linked to bug about the "missing FW" in Ubuntu.

[enyc (enyc)]

On Mon, 21 Aug 2006, Jeremy Vies wrote:
> I think "tcp_syncookies" is considered as part of the FW mechanism of the kernel.
> As Dapper (and previous releases) does not provide any FW out of the box, it is normal that tcp_syncookies are not activated by default.
> Your bug repport should be put as a wish for next release, and maybe linked to bug about the "missing FW" in Ubuntu.
Urrm... Well a firewall addon is another matter...
That is for blocking ports and particular hosts and soforth.....

Ubuntu (sensibly) starts with no 'open ports' (except on 127.0.0.1)
  unless you add a service or install a LAMP server...

It doesnt need a firewall for a lot of cases -- firewall just adds
  needless extra complexity.... Just dont start services you dont
  want. Only need to add a firewall if you want to control access
  of particular IP addresses and soforth...

But w/o syncookies your VNC or SSH or Samba-shares or whatever can
  be trivially DoSed from low-bandwidth-connection which is rather
  silly really. I understand they dont actually change anything
  about TCP behaviour until there are too many SYN_RECVD entries,
  at which point the syncookies 'kick in' permitting access to
  your TCP servers which under continuing SYN flood....

--enyc <email address hidden>

[Jeremy Vies (jeremy.vies)]

For me syncookies is the same problem as FW is.
As you said, as long as you don't start a network service, your computer is safe. If you start a SSH server or whatever, you have to protect your system from DoS or other attacks...

(By the way, if your server is reachable from the internet, as soon as you open a network service, you will need some iptables rules to filter some attacks, as ssh scans for example.)

[enyc (enyc)]

Firstly...

You do not need to start a 'server program' as-such to get a listening port. Using bittorrent client or 'ftp' will easily create listening ports, making the system vulnerable to SYN-flood-DoS on those tcp-sockets. I.e. users can be easily 'vulnerable' without having to set up a service as-such.

Iptables does not directly solve the problem of SYN DoS.... Note that the DoS can/does happen using forged-source IP addresses and hence blocking 'source addresses' w/ iptables can only seek to mitigate the issue but not prevent it.

I believe that firemalls are a separate consideration really....
This is a TCP configuration parameter that permits/denies the use of a syncookies defense under SYN-flood-attack scenarios.

This is a 'TCP stack vulnerable to resource-exhaustion DoS by configuration/design' rather than 'service needs covering by firewall' sort of matter...

I think I see this "issue" in a different way Jeremy Vies.

The way I see this, there is no reason to disallow the use of SYNcookies when under SYN DoS. I understand the TCP stack in linux continues to answer TCP SYN w/ options etc. in the same way as without syncookies=1 until SYN queue is full -- i.e. no difference except when under DoS.

*** Any facts / details to confirm or contradict anything stated above would be appreciated ;-) ***

[Jeremy Vies (jeremy.vies)]

Sorry, I didn't know that ftp was not a server program...

My point of view is that it should not be activated by default, but should be easily configurable with a GUI, probably the same GUI that should configure the FW.

I add the ubuntu network team and the ubuntu security team to the bug.

[enyc (enyc)]

Jeremy,
I can confirm that SYNcookies are NOT part of the firewall mechanism of the kernel.

CONFIG_NETFILTER option in linux 2.6 is the toggle for linux packet filtering support called 'netfilter'(iptables)... There are many sub-choices/options for netfilter.

CONFIG_SYN_COOKIES however is a different choice, that allows you to enable/disable compiling support for SYNcookies SYN-flood-defense support.

Please also note that you generally cannot properly 'firewall out' a typical spoofed-source SYN flood without preventing legitimate access to your server.

[Matt Zimmerman (mdz)]

SYN cookies are disabled by default in Ubuntu for the same reason they are disabled by default in the kernel. According to the kernel documentation, use of this option causes the system to violate the TCP standard, and so is only intended to be used to mitigate an attack in progress.

[alecm3 (alecm-chatango)]

We installed 2 production servers and suddenly we started getting strange connection problems, with no errors in the application or system logs. The problems were highly intermittent, but amounted to being unable to connect to a port our TCP server was receiving client internet connections on.

After 3 days of debugging (netfilter, the server application, writing custom bash/awk programs to poll and graph netstat, doing tcpdumps) the problem what traced to random SYN attacks.

It turns out that net.ipv4.tcp_syncookies=1 is commented out in the *server* edition of Ubuntu 8.04!

After all this wasted time (and upset users), my only reaction is "WTF...?" We have many SuSE production servers, starting from 9.0 and they all came with syn cookies enabled. Messages like

possible SYN flooding on port 80. Sending cookies.

are *very* common in /var/log/messages, anybody who has run a heavily loaded server with many connections has seen tons of them.

A developer above seems to answer that "use of this option causes the system to violate the TCP standard". I guess SuSE developers understood better that a server-intended Linux distribution is not a computer science exercise, but an operating system that is *actually used* for production servers.

[Kees Cook (kees)]

Enabling syncookies disables TCP window scaling[1], and in most situations, existing SYN-flood protections in the kernel
already address most sorts of those attacks. In some situations (perhaps like what alecm3 was experiencing) there are situations it might be needed, but for a default, I am against[2][3] it if for no other reason than keeping window scaling working.

[1] http://lkml.org/lkml/2008/2/5/167
[2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=495884
[3] http://launchpadlibrarian.net/16972932/procps_1%3A3.2.7-8ubuntu2_1%3A3.2.7-9ubuntu1.diff.gz

[Simon Iremonger (ubuntu-iremonger)]

On Fri, 12 Sep 2008, Kees Cook wrote:
> Enabling syncookies disables TCP window scaling[1],

I think this is incorrect as-stated.... But this should be
  confirmed/proved/disproved.

As far as I have found out elsewhere, the syn-cookies support
  in Linux is adaptive, and does NOT come into play unless
  there is an overflow of SYN_RECVD ...

I.e. tcp window scaling DOES work with syncookies=1 -- just
  not when there is a real syn-flood-problem ... but...
  if syncookies was not enabled, such a connection would
  likely not succeed at all! -- what is better? ;-).
  (but --see below -- situation is now different with latest
  kernel!)

> and in most situations,
> existing SYN-flood protections in the kernel
> already address most sorts of those attacks.

What are these 'existing SYN-flood protections'
  and how do they work?

Inceasing the backlog is simply increasing a finite limit --
  randomly dropping SYN_RECVD entries also makes syn-flooding
  slightly less effective relateve to forged-syn-traffic -- but
  -- it still should not actually take much traffic to overload
  the finite limits on SYN_RECVD thereby making new legitimate
  connections unlikely to succeed.

The crptographic cookie approach avoids the need for the syn
  packet backlog... and stops the repetition of syn+ack
  packehs in those cases.

> In some situations (perhaps like what alecm3 was experiencing)
> there are situations it might be needed,
I suspect that... with a busy server with many clients connecting
  a lot and connecting from slow links, it may be necessary to
  raise net.ipv4.tcp_max_syn_backlog because of legitimate
  rate/number of such not-yet-completed incoming-connections.

Its' worth reading this article:-
http://lwn.net/Articles/277146/

Seemingly 2.6.26 now supports syncookies on ipv6 too, and
  now supports connections with window-scaling even
  if connection was saved by syncookies.

Rather than having arguments over the value of the setting
  etc... -- How do we get this properly investigated
  and sorted out?

--Simon

[KimOlsen (kimfolsen80)]

>"...option causes the system to violate the TCP standard..."

I do not think this is the case. If you check RFC4732 they list this as a possible way to help against DoS attacks.

I also believe that window scaling is not affected, but large windows are. But accepting legit traffic without large windows is better than dropping the connections.

So if the implementation is an adaptive one that only use SYN cookies when under huge load, then I am all for this. At least in the server edition.

[Simon Iremonger (ubuntu-iremonger)]

On Thu, 23 Oct 2008, KimOlsen wrote:
>> "...option causes the system to violate the TCP standard..."
> I do not think this is the case. If you check RFC4732 they list this as
> a possible way to help against DoS attacks.

> I also believe that window scaling is not affected, but large windows
> are. But accepting legit traffic without large windows is better than
> dropping the connections.
Note, that, seemingly, as of Linux 2.6.26, tcp connections with
   "large windows" can now be accepted under syn-flood too! So,
   even that, no longer matters, seemingly...

> So if the implementation is an adaptive one that only use SYN
> cookies when under huge load, then I am all for this.
Yes, it is.
Linux produces messages on the kernel log, to say "sending cookies"
   when this happens. I.e. SYN-cookies do NOT come into play unless
   there is a high load of incoming connections.

I can understand that some systems receiving a legitimately high
   number of connections, it may be necessary to increase the
   net.ipv4.tcp_max_syn_backlog (or whatever it is, exactly) to
   avoid the use of cookies... but that *still* does not create
   any reason not to have set tcp_syncookies=1 !!

> At least in the server edition.
I don't see why the install CD type matters, myself...
Any install can result in some use of TCP listening sockets
   somewhere! Also... that then means extra work to setup
   different sysctl settings based upon install-disk...

But thats' only my thoughts...

It would be good to get this sorted-out properly... But I don't
   know what other information is needed. I guess the problem is
   not information.. in this world of information-overload ;-).

If Ubuntu networking team, don't want to change the setting, they
   don't want to change the setting... puzzling...

--Simon

[Kees Cook (kees)]

procps (1:3.2.7-11ubuntu1) jaunty; urgency=low

  * Merge from debian unstable, remaining changes:
    - debian/{postinst,rules}: init script to priority 17, remove on upgrade.
    - debian/rules (Ubuntu-specific):
      - install sysctl files from new sysctl.d directory.
      - append debian/sysctl.d/*.conf.$DEB_HOST_ARCH to 10-arch-specific.conf
    - debian/sysctl.d (Ubuntu-specific):
      - 10-console-messages.conf: stop low-level kernel messages on console.
      - 10-network-security.conf: enable "rp_filter" by default.
      - 10-process-security.conf: block lower 64k allocations to protect
        kernel from NULL deref attacks.
      - 10-keyboard.conf.powerpc: mouse button emulation on PowerPC.
  * procps-3.2.7/debian/{preinst,postinst,postrm}: drop
    sysctl.d/10-tcp-timestamps-workaround.conf again now that we have a
    fixed kernel, and make sure it gets removed on upgrade to this version
    (LP: #264019, duplicated from 1:3.2.7-9ubuntu2.1).
  * debian/sysctl.d/10-network-security.conf: enable SYN-flood protection
    by default (LP: #57091).

[pablomme (pablomme)]

I think this may have introduced a regression. While using aMule on my amd64 Jaunty desktop, there is a point at which the screen freezes and X stops responding to input (the mouse pointer moves, but it does not interact with anything). Ctrl-Shift-F1-6 won't drop me to a TTY. I believe the hang affects X exclusively, since the system continues logging and SysRq keys work fine.

All four times that this has happened I get this line repeatedly in /var/log/messages:

May 18 22:34:27 tinymme kernel: [ 9389.660132] possible SYN flooding on port 34443. Sending cookies.

at a rate of one per minute. These messages start at the same time as X hangs. Port 34443 is the TCP port I configured for use by aMule.

Two issues here:
 - SYN flood protection may be being mis-triggered by legitimate network load
 - the protection itself seems to be causing a problem with X (directly or indirectly)

I wouldn't think this is an interaction between the kernel and X if it weren't for the fact that I've had multiple instances of the problem with the same log entries, but I may as well be missing something (even though other log files are clean). Should I open a new bug report with this?

[Kees Cook (kees)]

On Tue, May 19, 2009 at 04:11:18PM -0000, pablomme wrote:
> Should I open a new bug report with this?

Yes please. I would initially assume that some other issue has caused the
kernel to stop handling network traffic rather than high network traffic
stopping the kernel.

[pablomme (pablomme)]

> Yes please.

Ok.

> I would initially assume that some other issue has caused the
> kernel to stop handling network traffic rather than high network traffic
> stopping the kernel.

The kernel did not stop, nor did the networking or anything else other than X.

[Olaf van der Spek (olafvdspek)]

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=520668

[Olaf van der Spek (olafvdspek)]

Are there any updates on this issue?
I don't see any counter arguments to the fact syn cookies only take effect after the queue is full.
Ideally this would be changed upstream, maybe an Ubuntu kernel dev could contact upstream about this?

[Kees Cook (kees)]

Olaf: that's why it is "fix released". :) It is enabled in Ubuntu now.

[Olaf van der Spek (olafvdspek)]

On Fri, Sep 25, 2009 at 4:56 PM, Kees Cook <email address hidden> wrote:
> Olaf: that's why it is "fix released".  :)  It is enabled in Ubuntu now.

Ah, nice. I kinda expected a link to the package version in which it got fixed.

[Olaf van der Spek (olafvdspek)]

Ah, nevermind, I can't read, it's at the bottom of that message.

On Fri, Sep 25, 2009 at 5:18 PM, Olaf van der Spek <email address hidden> wrote:
> On Fri, Sep 25, 2009 at 4:56 PM, Kees Cook <email address hidden> wrote:
>> Olaf: that's why it is "fix released".  :)  It is enabled in Ubuntu now.
>
> Ah, nice. I kinda expected a link to the package version in which it got fixed.
>

[Simon Iremonger (ubuntu-iremonger)]

>> Ah, nice. I kinda expected a link to the package version in which it got fixed.

The silly thing is....
There is misinformation in the /etc/sysctl.conf now!

It says:-
"# This disables TCP Window Scaling (http://lkml.org/lkml/2008/2/5/167)"
First of all that is incorrect as a blanket statement.
A connection 'saved by syncookies' used to not allow window scaling.
But, it always worked fine solong as there was not a synflood going on!

Secondly, its' completely wrong now, because newer kernel
   SynCookies, will ALWAYS allow window scaling, regardless
   of syncookies having 'kicked in' or not!

That could do with just being removed.

--Simon

[Olaf van der Spek (olafvdspek)]

Has this request been forwarded upstream (lkml)?

[Simon Iremonger (ubuntu-iremonger)]

On Fri, 9 Oct 2009, Olaf van der Spek wrote:
> Has this request been forwarded upstream (lkml)?

Not that I am aware of.

It would be good for this confusion/misinformation to get sorted
   out properly.
Why is it that some wish to make sweeping statements and not
   understand the whole situation?

What do you do in this circumstance?

--Simon

[Simon Iremonger (ubuntu-iremonger)]

Upstream kernel have decided to enable syncookies by default (according to that debian bug, since Linux 2.6.37!).
This makes sense, as the main downsides have already been resolved (especially window scaling even under syncookies-activation), and this feature only kicks-in if the SYN-queue is overloaded.

We might now consider taking out this (now superfluous) tcp_syncookies entry from /etc/sysctl.d/10-network-security.conf ...

I think, a similar situation has now arisen with respect to the "tcp_ecn" setting, where the (conservative) (enabled by default) fallback mechanism in the kernel, along with the rarity of ecn-intolerance, along with the wide ECN-adoption in practice in Apple ios / MAC OS X now, along with the importance of ECN for smooth responsive internet in the face of congestion, means that this tcp_ecn setting should similarly be seriously considered. This should be the subject of new bug report right-soon-now =).

[antisa (antisa)]

Here is the entry from ...10-network-security.conf from 16.04 (although from Desktop edition)

"
# Turn on SYN-flood protections. Starting with 2.6.26, there is no loss
# of TCP functionality/features under normal conditions. When flood
# protections kick in under high unanswered-SYN load, the system
# should remain more stable, with a trade off of some loss of TCP
# functionality/features (e.g. TCP Window scaling).
net.ipv4.tcp_syncookies=1

"

Guess it hasn't been removed.

[Matthew Caron (matt-mattcaron)]

Well, and it gets more interesting.

Bog standard 16.04 has it turned on (from the above referenced 10-network-security.conf).

But, if you then enabled ufw, it gets disabled, due to the default setting in /etc/ufw/sysctl.conf.

There seems to be serious debate as to whether or not enabling it is correct.

What I know is that I just spent two hours trying to figure out why SANE took forever to detect my network scanner, and this syslog entry clued me in:

Oct 6 22:54:26 hiro kernel: [48562.817258] TCP: request_sock_TCP: Possible SYN flooding on port 34029. Dropping request. Check SNMP counters.

The dropped request was responsible for the delay. If I enable syn cookies, I get:

Oct 6 22:57:28 hiro kernel: [48744.796029] TCP: request_sock_TCP: Possible SYN flooding on port 42041. Sending cookies. Check SNMP counters.

and it's basically instant.

On top of all of this, there isn't a lot of traffic - this is SANE talking to a vendor-provided scanner backend over localhost. If I capture it, there's ONE SYN request and the kernel thinks it's a "flood".. which makes no sense.

[Simon Iremonger (ubuntu-iremonger)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

> Bog standard 16.04 has it turned on (from the above referenced 10
> -network-security.conf).
> But, if you then enabled ufw, it gets disabled, due to the default
> setting in /etc/ufw/sysctl.conf.

> There seems to be serious debate as to whether or not enabling it is
> correct.

I haven't seen why not to enable use of adaptive syncookies, aiui
this creates no _disadvantage_ if not being triggered...

I CAN understand that for some scenarios the 'right thing to do'
is Increase the tcp_max_syn_backlog as cookies are triggering too
easily, even then it won't stop connections being accepted albeit
with less tcp options possible, but then without syncookies
the connections would be dropped as the syn queue fills...

> What I know is that I just spent two hours trying to figure out why SANE
> took forever to detect my network scanner, and this syslog entry clued
> me in:
> Oct 6 22:54:26 hiro kernel: [48562.817258] TCP: request_sock_TCP:
> Possible SYN flooding on port 34029. Dropping request. Check SNMP
> The dropped request was responsible for the delay. If I enable syn
> cookies, I get:
> Oct 6 22:57:28 hiro kernel: [48744.796029] TCP: request_sock_TCP:
> Possible SYN flooding on port 42041. Sending cookies. Check SNMP
> capture it, there's ONE SYN request and the kernel thinks it's a
> "flood".. which makes no sense.

Weird :).
I can't say I'm familiar with uwf, but I wonder if it is somehow
oversensitive in its' own ip(6)tables or they are fiddling with:-

/proc/sys/net/ipv4/tcp_max_syn_backlog

Do raise bug in the ufw // ufw sysctl.conf .... Also email me
separately the relevant bug numbers etc., be curious to see!!

- --Simon
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Topal (http://freshmeat.net/projects/topal)

iF4EAREIAAYFAlf3SqEACgkQA62i3HuJ2aHNCwEAnK4NvLNm/tKHzFNSEK+KRNMB
6hZOZ6tcnbecljP1+dAA/3C0bmEHFXEzeLF3xYNSco+py2TbD2bNPzXbG0NKsupb
=Fh0+
-----END PGP SIGNATURE-----

[Matthew Caron (matt-mattcaron)]

Will do, Simon.

[Nils Toedtmann (m-launchpad-net-mail-nils-toedtmann-net)]

I filed a request for ufw not to override https://bugs.launchpad.net/ubuntu/+source/ufw/+bug/1737585

[Simon Iremonger (ubuntu-iremonger)]

FWIW Although syncookies has long-since been enabled upstream, the outdated comments in sysctl about syncookies still persist, I have now created new ubuntu bug #1773157 [please comment there].

[This also requests ECN-on-outgoing enablement which has similarly matured etc.].