`http_upgrade_request_protocols WebSocket deny all` does not block websocket

[Chuan Li]

I want to block all WebSocket connections using Squid. Here are the configurations:

a) a WebSocket server is running on port 8765 by Python on 10.5.2.132/16 [1]
b) a WebSocket client is running on 10.5.0.204/16 [2]
c) Squid (Jammy 5.2-1ubuntu4.3) is running on 10.5.1.201/16 with configs as [3]

I expected that `http_upgrade_request_protocols WebSocket deny all` would block all WebSocket connections, but it did not work.

squid still can allow upgrade to websocket

access.log:

1682301945.941 14 10.5.0.204 TCP_TUNNEL/200 285 CONNECT 10.5.2.132:8765 - HIER_DIRECT/10.5.2.132 -

When I executed a WebSocket connection from the client to the server and did a tcpdump on the client, I can see the tcpdump results are as [4].

"
GET / HTTP/1.1
Upgrade: websocket
Host: 10.5.2.132:8765
Origin: http://10.5.2.132:8765
Sec-WebSocket-Key: N8tBxe1BeAIoxuP0J6A3bA==
Sec-WebSocket-Version: 13
Connection: Upgrade

2V.4HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: loR4dW7PF7ylnYzp92wOrA2fyr0=
Date: Mon, 24 Apr 2023 02:00:59 GMT
Server: Python/3.6 websockets/9.1
"

Also, the cache.log is as shown in [5].

This should not be a cache issue because the result is the same whether or not I stop Squid or remove `/var/spool/squid/netdb.state`.

[1] https://paste.ubuntu.com/p/jD3BnfmDPZ/
[2] https://paste.ubuntu.com/p/qhxj8s32t4/
[3] https://paste.ubuntu.com/p/YZnY8n64nG/
[4] https://paste.ubuntu.com/p/ZZTdfTFDmk/
[5] https://paste.ubuntu.com/p/CkPtncXwFx/

[Lucas Kanashiro (lucaskanashiro)]

Hi Chuan,

Thanks for reporting this bug and trying to make Ubuntu better.

First, thank you for providing the scripts/configs/logs you were using in your tests, that's really helpful. I took a quick look to see if there is no issue with your config, and apparently (after reading the upstream documentation) your squid config seems correct. After that I tried to reproduce the issue using your scripts. Set up the server and client in separate containers but when executing the code I am getting this traceback:

--- request header ---
GET / HTTP/1.1
Upgrade: websocket
Host: 10.96.142.135:8765
Origin: http://10.96.142.135:8765
Sec-WebSocket-Key: YPwIzSwWL9qXAJ/SwWPXxg==
Sec-WebSocket-Version: 13
Connection: Upgrade

-----------------------
--- response header ---
Traceback (most recent call last):
  File "/root/2.py", line 17, in <module>
    ws.connect("ws://10.96.142.135:8765")
  File "/usr/lib/python3/dist-packages/websocket/_core.py", line 253, in connect
    self.handshake_response = handshake(self.sock, *addrs, **options)
  File "/usr/lib/python3/dist-packages/websocket/_handshake.py", line 57, in handshake
    status, resp = _get_resp_headers(sock)
  File "/usr/lib/python3/dist-packages/websocket/_handshake.py", line 141, in _get_resp_headers
    status, resp_headers, status_message = read_headers(sock)
  File "/usr/lib/python3/dist-packages/websocket/_http.py", line 315, in read_headers
    line = recv_line(sock)
  File "/usr/lib/python3/dist-packages/websocket/_socket.py", line 134, in recv_line
    c = recv(sock, 1)
  File "/usr/lib/python3/dist-packages/websocket/_socket.py", line 113, in recv
    bytes_ = _recv()
  File "/usr/lib/python3/dist-packages/websocket/_socket.py", line 90, in _recv
    return sock.recv(bufsize)
ConnectionResetError: [Errno 104] Connection reset by peer

So I was not able to reproduce what you described locally. However, after checking the package, I believe none of the changes in the package should impact the scenario you are trying to implement, if this is a real issue I think you should try to report upstream and see what squid maintainers have to say. In case you do that, please link the upstream bug report here so we can track it.

[Lucas Kanashiro (lucaskanashiro)]

Just found out that the issue I was facing is an actual bug in python3-websockets in Jammy:

https://bugs.launchpad.net/ubuntu/+source/python-websockets/+bug/2017704

I suppose you were using a version from PyPi, am I right? However, I still think this squid issue is unrelated to Ubuntu and should be discussed with upstream.

[Sergio Durigan Junior (sergiodj)]

I couldn't let this one rest, so I kept poking at it and was able to reproduce the problem in the end.

I had to resort to a Perl websocket server. You can install it by doing:

# apt install cpanminus
# cpanm Net::WebSocket::Server

Then, paste the following code into a file:

# cat > ws-server.pl << __EOF__
use Net::WebSocket::Server;

Net::WebSocket::Server->new(
    listen => 8765,
    on_connect => sub {
        my ($serv, $conn) = @_;
        $conn->on(
            utf8 => sub {
                my ($conn, $msg) = @_;
                $conn->send_utf8($msg);
            },
        );
    },
)->start;
__EOF__

And then you can start the server:

# perl ./ws-server.pl

Then, inside the client container, you can do:

# cat > ws-client.py << __EOF__
import asyncio
import websocket

websocket.enableTrace(True)
ws = websocket.WebSocket()
ws.connect("ws://10.96.142.135:8765", http_proxy_host="10.96.142.184", http_proxy_port=3128)
ws.send("Hello, Server")
print(ws.recv())
ws.close()
__EOF__

Make sure to adjust the http_proxy_host accordingly. I also set the shell variable http_proxy, just in case.

After that, when you run ws-client.py, you see:

# python ./ws-client.py
Connecting proxy...
--- request header ---
CONNECT 10.96.142.135:8765 HTTP/1.1
Host: 10.96.142.135:8765

-----------------------
--- response header ---
HTTP/1.1 200 Connection established
-----------------------
--- request header ---
GET / HTTP/1.1
Upgrade: websocket
Host: 10.96.142.135:8765
Origin: http://10.96.142.135:8765
Sec-WebSocket-Key: vOgPzKUs+rVIi0vIhptBIg==
Sec-WebSocket-Version: 13
Connection: Upgrade

-----------------------
--- response header ---
HTTP/1.1 101 WebSocket Protocol Handshake
Upgrade: WebSocket
Connection: Upgrade
Sec-WebSocket-Accept: 0gtw0Pg+byjzP49Vu48h0D/yiSw=
-----------------------
++Sent raw: b'\x81\x8d\xb7\x00\x9f\x18\xffe\xf3t\xd8,\xbfK\xd2r\xe9}\xc5'
++Sent decoded: fin=1 opcode=1 data=b'Hello, Server'
++Rcv raw: b'\x81\rHello, Server'
++Rcv decoded: fin=1 opcode=1 data=b'Hello, Server'
Hello, Server
++Sent raw: b'\x88\x82ve:\xfdu\x8d'
++Sent decoded: fin=1 opcode=8 data=b'\x03\xe8'

And if you look at the squid access.log:

1682463720.216 2 10.96.142.115 TCP_TUNNEL/200 196 CONNECT 10.96.142.135:8765 - HIER_DIRECT/10.96.142.135 -

I don't understand why it isn't blocking the upgrade request.

I tried searching upstream's bug database, to no avail. There is https://bugs.squid-cache.org/show_bug.cgi?id=5201 which looks a bit similar, but it's not really the same problem.

I haven't tried reproducing this problem on Lunar, which has a more recent squid package.

[Chuan Li (lccn)]

Thank you very much for triaging the bug.
I would appreciate it if you could provide other methods to block websocket requests, besides using http_upgrade_request_protocols. Actually in the customer's environment, websocket server uses port 80 or 443, so simply blocking ports 80 or 443 is not possible as it would affect other accesses targeting those ports.

[Amos Jeffries (yadi)]

The reason Squid is not blocking the Upgrade is that it never sees that request.
Only these CONNECT tunnel headers are visible to Squid:
"
--- request header ---
CONNECT 10.96.142.135:8765 HTTP/1.1
Host: 10.96.142.135:8765

-----------------------
--- response header ---
HTTP/1.1 200 Connection established
-----------------------
"

as you can see there is no Upgrade header on the CONNECT request.

Those GET requests with Upgrade header you see in the tcpdump and python output are hidden inside the CONNECT tunnel as payload. To Squid that is all just opaque binary flowing between client and server.