UDP data corruption caused by buggy udp_recvmsg() -> skb_copy_and_csum_datagram_msg()

Bug #1888715 reported by Dexuan Cui
264
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Incomplete
Undecided
Unassigned
Xenial
Fix Released
Undecided
Unassigned

Bug Description

The Xenial v4.4 kernel (https://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/tag/?h=Ubuntu-4.4.0-184.214) lacks this upstream bug fix("make skb_copy_datagram_msg() et.al. preserve ->msg_iter on erro" https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3278682123811dd8ef07de5eb701fc4548fcebf2); as a result, the v4.4 kernel can deliver corrupt data to the application when a corrupt packet is closely followed by a valid packet, and actually the UDP payload of the corrupt packet is delivered to the application with the "from IP/Port" of the valid packet, so this is actually a security vulnerability that can be used to trick the application to think the corrupt packet’s payload is sent from the valid packet’s IP address/port, i.e. a source IP based security authentication mechanism can be bypassed.

Details:

For a packet longer than 76 bytes (see line 3951, https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/linux/skbuff.h?h=v5.8-rc6#n3948), Linux delays the UDP checksum verification until the application invokes the syscall recvmsg().

In the recvmsg() syscall handler, while Linux is copying the UDP payload to the application’s memory, it calculates the UDP checksum. If the calculated checksum doesn’t match the received checksum, Linux drops the corrupt UDP packet, and then starts to process the next packet (if any), and if the next packet is valid (i.e. the checksum is correct), Linux will copy the valid UDP packet's payload to the application’s receiver buffer.

The bug is: before Linux starts to copy the valid UDP packet, the data structure used to track how many more bytes should be copied to the application memory is not reset to what it was when the application just entered the kernel by the syscall! Consequently, only a small portion or none of the valid packet’s payload is copied to the application’s receive buffer, and later when the application exits from the kernel, actually most of the application’s receive buffer contains the payload of the corrupt packet while recvmsg() returns the size of the UDP payload of the valid packet. Note: this is actually a security vulnerability that can be used to trick the application to think the corrupt packet’s payload is sent from the valid packet’s IP address/port -- so a source IP based security authentication mechanism can be bypassed.

The bug was fixed in this 2017 patch “make skb_copy_datagram_msg() et.al. preserve ->msg_iter on error (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3278682123811dd8ef07de5eb701fc4548fcebf2)”, but unluckily the patch is only backported to the upstream v4.9+ kernels. I'm reporting this bug to request the bugfix to be backported to the v4.4 Xenial kernel, which is still used by some users and has not been EOL'ed (https://ubuntu.com/about/release-cycle).

I have the below one-line workaround patch to force the recvmsg() syscall handler to return to the userspace when Linux detects a corrupt UDP packet, so the application will invoke the syscall again to receive the next good UDP packet:
--- a/net/ipv4/udp.c
+++ b/net/ipv4/udp.c
@@ -1367,6 +1367,7 @@ csum_copy_err:
        /* starting over for a new packet, but check if we need to yield */
        cond_resched();
        msg->msg_flags &= ~MSG_TRUNC;
+ return -EAGAIN;
        goto try_again;
}

Note: the patch may not work well with blocking sockets, for which typically the application doesn’t expect an error of -EAGAIN. I guess it would be safer to return -EINTR instead.

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Hello Dexuan, excellent report, thanks.

Am I reading correctly that this affects strictly UDP? Without the three-way handshake and TCP sequence numbers, UDP sources are relatively easy to spoof in the first place.

If I've read this correctly, I think this is a bug that ought to be fixed, but may not itself allow crossing a security boundary.

Did I misread?

Thanks

Revision history for this message
Dexuan Cui (decui) wrote :

Any kernel code that needs to copy data to the userspace via recvmsg() may have the same bug, if the code has some "try_again upon error" logic (please check udp_recvmsg()) but doesn't properly reset the "struct iov_iter *to" when doing retry_again.

Here it looks TCP is not affected, but when I check the latest mainline kernel, iov_iter_revert() is called in a lot of files, so IMO it would be a good idea to audit every usage of iov_iter_revert() in the latest mainline and see if we should backport it to the v4.4.

Due to the bug, recvmsg() can pass corrupt UDP data payload to the application, and hence can trick the application this way:
1. an attacker sends a UDP packet with an incorrect UDP checksum from an untrusted IP_A.
2. a legit users send a valid UDP packet from an trusted IP_B.
3. assuming the 2 UDP packets arrive to the receiver VM one after one, the rceiver VM's kernel copies the payload of the first packet (and maybe some portion of the second packet) to the application, and the "from IP" returned by recvmsg() is IP_B, so the application incorrectly thinks the payload is
 from IP_B, while in fact it's basically from IP_A -- this is a security vulnerability, though IMO it's hard to be exploited in practice. The main concern is that the application receives incorrect UDP payload sliently.

Revision history for this message
Dexuan Cui (decui) wrote :

BTW, since the upstream v4.4 stable kernel also has the issue, I reported the bug to <email address hidden> as well just now.

Revision history for this message
Dexuan Cui (decui) wrote :

People from <email address hidden> also think the vulnerability should be pretty hard to be exploited in practice, but we should address the data corruption, so I reported the issue to the stable and netdev lists:
https://<email address hidden>/T/#u

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Hello Dexuan, wonderful, thanks;

Eric's proposed patch confuses me on one point:

- if (rcu_access_pointer(sk->sk_filter) &&
- udp_lib_checksum_complete(skb))
+ if (udp_lib_checksum_complete(skb))
                goto csum_error;

Do you know why the rcu_access_pointer() call has been removed?

Thanks

information type: Private Security → Public Security
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1888715

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Dexuan Cui (decui) wrote :

rcu_access_pointer(sk->sk_filter) is basically the same as sk->sk_filter.

If sk->sk_filter is true, the change makes no difference.
If sk->sk_filter is false, the change also drops a UDP packet with incorrect UDP checksum by "goto csum_error;". Without the change, the packet is dropped in udp_recvmsg(); with the change, the packet is dropped earlier.

Revision history for this message
Dexuan Cui (decui) wrote :

https://<email address hidden>/

Revision history for this message
Dexuan Cui (decui) wrote :
Revision history for this message
Marcelo Cerri (mhcerri) wrote :

The fix was already released as part of 4.4.0-190.220 for the upstream stable update from bug #1892822.

Changed in linux (Ubuntu Xenial):
status: New → Fix Released
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.