SUNRPC: Use after free when GSSD credentials are invalid causes oops
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Disco |
Fix Released
|
Medium
|
Matthew Ruffell |
Bug Description
BugLink: https:/
[Impact]
There is a use after free which normally causes a null pointer dereference when NFS clients send invalid credentials via GSSD to a NFS server which has shares protected by kerberos krb5* security.
The call trace is below:
BUG: unable to handle kernel NULL pointer dereference at 0000000000000098
Call Trace:
rpc_check_
call_
__rpc_
rpc_execute+
rpc_run_
nfs4_
nfs41_
? nfs4_realloc_
? nfs4_setup_
nfs41_
nfs41_
nfs4_
? nfs4_handle_
nfs4_
? kernel_
nfs4_
kthread+
? nfs4_state_
? __kthread_
ret_from_
This then causes outages on a heavily trafficked NFS server and no one can access their shares until the server is rebooted.
[Fix]
The fix comes in the following two commits:
commit cea57789e408187
Author: Trond Myklebust <email address hidden>
Date: Sat Mar 9 16:06:47 2019 -0500
Subject: SUNRPC: Clean up
commit 7987b694ade8cc4
Author: Trond Myklebust <email address hidden>
Date: Wed May 29 12:49:52 2019 -0400
Subject: SUNRPC: Fix a use after free when a server rejects the RPCSEC_GSS credential
There is a small subtlety to be addressed here.
7987b694ade8cc4
Please note, both commits have been backported.
cea57789e408187
7987b694ade8cc4
cea57789e408187
7987b694ade8cc4
7987b694ade8cc4
[Testcase]
This is difficult to reproduce on test systems, and has instead been verified on a production NFS v4.1 system in a customer environment. This server is heavily trafficked and has a large number of different NFS clients connected to it.
I have built a test kernel that contains the above patch, and also patches for Bug 1828978. It is available here:
https:/
Note that the above kernel is for bionic HWE, and not explicitly disco.
Discussion about the patch validation can be found at the bottom of Bug 1842037.
On unpatched kernels, expect to see the symptoms mentioned in Impact, and on patched systems, everything working as intended.
[Regression Potential]
The changes are limited to users of sunrpc and the change itself is limited to cases where the RPCSEC_GSS credential is rejected. Under blue skies scenarios, the code should only be triggered when misbehaving clients do not keep their authentication tickets up to date.
In case of regression, misbehaving clients may cause outages on services which use sunrpc. In which case, the server administrator would have to revert to an earlier kernel in order to restore those services, as you cannot stop misbehaving clients from connecting.
Since the fix was selected for upstream stable, it has been vetted and widely accepted by the community. The backport I performed should have no functional difference at all.
CVE References
description: | updated |
no longer affects: | linux-meta-hwe (Ubuntu) |
no longer affects: | linux-meta-hwe (Ubuntu Disco) |
summary: |
- Oops when Kerberos credentials are invalid + SUNRPC: Use after free when GSSD credentials are invalid causes oops |
description: | updated |
Changed in linux (Ubuntu): | |
status: | Incomplete → Fix Released |
Changed in linux (Ubuntu Disco): | |
status: | In Progress → Fix Committed |
tags: |
added: verification-done-disco removed: verification-needed-disco |
Hi Frank,
I have been reading up on this particular bug for the last few days now, and I
think I understand what is going on.
Firstly, the commit you requested:
commit 7987b694ade8cc4 65ce10fb3dceaa6 14f13ceaf3
Author: Trond Myklebust <email address hidden>
Date: Wed May 29 12:49:52 2019 -0400
Subject: SUNRPC: Fix a use after free when a server rejects the RPCSEC_GSS credential
This commit fixes a problem which was introduced by an earlier commit:
commit cea57789e408187 0ac3498fbefabbb d0d0fd8434
Author: Trond Myklebust <email address hidden>
Date: Sat Mar 9 16:06:47 2019 -0500
Subject: SUNRPC: Clean up
Now, this earlier commit "SUNRPC: Clean up" is not present in the current bionic 5.0 HWE kernel, which would mean that you might be facing a different bug entirely.
Is there any chance that you can provide more information about your system?
Can you provide a stack trace output by dmesg, or in /var/log/kern.log for this
problem, or better still, can you provide a sosreport?
https:/ /support. canonical. com/ua/ s/article/ canonical- support- data-collection -sosreport
Since the bug can still be triggered in different paths, I am in the process of backporting the commit you requested to the 5.0 kernel. The code has changed a lot between 5.2 where the patch was introduced, and 5.0 where I am backporting it to.
I think I have created a good backport, which are below if you are interested:
https:/ /paste. ubuntu. com/p/BPD3PzmZ9 7/ /paste. ubuntu. com/p/bmHMD7SPG X/
https:/
I am currently building a test kernel with the above patches applied for bionic, and I will let you know once it is ready, likely tomorrow. Once it has finished building I will ask you to install it to validate that it solves the problem you are facing.
In the meantime, can you get me a sosreport from the affected system, and I will let you know when the test kernel is ready.
Please do not post the sosreport publicly, and instead attach it to the SF case.
Let me know if you have any questions,
Thanks,
Matthew