TCP socket backlog set too low ("request_sock_TCP: Possible SYN flooding on port ...")

[Nils Toedtmann]

[Impact]

Having the listen backlog (the TCP pending connections queue) set to 1, means that only one connection can be in a "waiting for an 'accept' call" state. If more connections are coming through in that window, i.e., before the connection in the queue is accepted, these additional connections will be ignored.

While the short time a system usually takes to clear that queue combined with the possibility of a connection being reset and retried by a client will diminish the impact of the queue maximum size on the users, special cases where too many connections are being received in a short time span may result in many of those connection requests to either take a long time to be processed or to fail. One example of such cases is when the openvpn server is restarted.

The proposed patch increases the listen backlog size to an arbitrary, larger size proposed and accepted upstream in https://community.openvpn.net/openvpn/changeset/ec0ca68f4ed1e6aa6f08f470b18e0198b7e5a4da

[Test Plan]

* I am attaching scripts to reproduce/test the bug and the fix, since the scripts may be a bit long. Below is a short summary on how to test (describing the steps taken by the attached scripts).

- Configure and start an openvpn server
- Perform several TCP connections to that server in a short time span, while measuring the time the whole connection batch takes to be accepted. You can also count the number of connections that get reset.
- Apply the proposed patch
- Verify how the time for the same operation described above is drastically reduced. You can also verify that the number of reset connections dropped (possibly to zero, depending on the amount of connections being performed).

Here is a python script that could be used to perform the tests on a running server:

```
import socket
import threading

HOST = 'localhost'
PORT = 1194

def run(name):
    for i in range(5):
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        try:
            s.connect((HOST, PORT))
            s.sendall(b'hello')
            data = s.recv(1024)
        except ConnectionResetError:
            print('Connection reset')

threads = []
for i in range(500):
    t = threading.Thread(target=run, args=(i,))
    threads.append(t)

for t in threads:
    t.start()

for t in threads:
    t.join()
```

[Where problems could occur]

When this fix was proposed upstream, the opening line from the submitter read

'For reasons historically unknown, OpenVPN sets the listen() backlog
queue to "1"'...

While changing the queue size did not cause regressions in the version of OpenVPN where the patch was applied (nor in the subsequent versions), backporting the patch to an arbitrary openVPN version could reveal a "historically unknown reason", forcing us to either fix it or revert this patch.

Such regressions may include networking errors, mostly related to the TCP features of the openvpn server. One symptom that can occur when the requests are not being processed in time is having the following message in demesg output: "Possible SYN flooding on port ${PORT}".

[Other Info]

Upstream bug report with relevant discussion: https://community.openvpn.net/openvpn/ticket/1208

Upstream patch and more relevant discussion: https://<email address hidden>/msg18758.html

[Original report]
See upstream bug reports:
- https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=968942
- https://community.openvpn.net/openvpn/ticket/1208

Openvpn < 2.4.8 opens the TCP port with a too small backlog, and on kernels > 4.3 that leads to incoming connections being dropped. This kernel message is a symptom:

  TCP: request_sock_TCP: Possible SYN flooding on port 1194. Dropping request. Check SNMP counters.

I experienced this on a Bionic 18.04.5 (after having upgraded from Xenial) with openvpn 2.4.4-2ubuntu1.5

Fixed upstream.

[Sergio Durigan Junior (sergiodj)]

Thank you for the bug report.

I did not try to reproduce this bug, but I can confirm that the problematic code is in the Bionic and Focal versions of OpenVPN. Therefore, I am marking this as Triaged.

I will talk to someone from our team about SRU'ing this.

[Athos Ribeiro (athos-ribeiro)]

Attaching focal reproducer for SRU testing purposes. This was based on the autopkgtest suite available in impish

[Athos Ribeiro (athos-ribeiro)]

Attaching bionic reproducer for SRU testing purposes. This was based on the autopkgtest suite available in impish

[Robie Basak (racb)]

Hello Nils, or anyone else affected,

Accepted openvpn into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvpn/2.4.7-1ubuntu2.20.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robie Basak (racb)]

Hello Nils, or anyone else affected,

Accepted openvpn into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvpn/2.4.4-2ubuntu1.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Athos Ribeiro (athos-ribeiro)]

As per the test plan, I measured the time an openvpn server takes to complete a fixed amount of TCP connection requests.

Times are formatted in minutes:seconts

test for bionic before patch:
3:26.10

test for bionic after patch:
0:02.19

test for focal before patch:
2:07.78

test for focal after patch:
0:02.17

Which shows the patch significantly improves the waiting time for part of the connection requests.

I am also attaching a full reproducer set of scripts for verification.

Note that this reproducer contains some interactive bits which will require user input.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openvpn - 2.4.7-1ubuntu2.20.04.3

---------------
openvpn (2.4.7-1ubuntu2.20.04.3) focal; urgency=medium

  * d/p/increase-listen-backlog-queue-to-32.patch: Increase listen backlog queue
    to 32 (LP: #1934781)

 -- Athos Ribeiro <email address hidden> Mon, 19 Jul 2021 16:26:19 -0300

[Robie Basak (racb)]

The verification of the Stable Release Update for openvpn has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openvpn - 2.4.4-2ubuntu1.6

---------------
openvpn (2.4.4-2ubuntu1.6) bionic; urgency=medium

  * d/p/increase-listen-backlog-queue-to-32.patch: Increase listen backlog queue
    to 32 (LP: #1934781)

 -- Athos Ribeiro <email address hidden> Mon, 19 Jul 2021 19:53:26 -0300