SUNRPC: Use after free when GSSD credentials are invalid causes oops

Bug #1842037 reported by Frank Burkhardt
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Undecided
Unassigned
Disco
Fix Released
Medium
Matthew Ruffell

Bug Description

BugLink: https://bugs.launchpad.net/bugs/1842037

[Impact]

There is a use after free which normally causes a null pointer dereference when NFS clients send invalid credentials via GSSD to a NFS server which has shares protected by kerberos krb5* security.

The call trace is below:

BUG: unable to handle kernel NULL pointer dereference at 0000000000000098
Call Trace:
  rpc_check_timeout+0x22/0xe0 [sunrpc]
  call_decode+0x12c/0x190 [sunrpc]
  __rpc_execute+0x7a/0x340 [sunrpc]
  rpc_execute+0xa3/0xb0 [sunrpc]
  rpc_run_task+0xf7/0x140 [sunrpc]
  nfs4_proc_get_lease_time+0xf3/0x130 [nfsv4]
  nfs41_setup_state_renewal+0x3d/0x90 [nfsv4]
  ? nfs4_realloc_slot_table+0x5b/0x130 [nfsv4]
  ? nfs4_setup_session_slot_tables+0x77/0xc0 [nfsv4]
  nfs41_finish_session_reset+0x26/0x30 [nfsv4]
  nfs41_init_clientid+0x44/0x70 [nfsv4]
  nfs4_establish_lease+0x61/0xa0 [nfsv4]
  ? nfs4_handle_reclaim_lease_error+0x108/0x130 [nfsv4]
  nfs4_state_manager+0x1b1/0x750 [nfsv4]
  ? kernel_sigaction+0x43/0xe0
  nfs4_run_state_manager+0x24/0x40 [nfsv4]
  kthread+0x120/0x140
  ? nfs4_state_manager+0x750/0x750 [nfsv4]
  ? __kthread_parkme+0x70/0x70
  ret_from_fork+0x35/0x40

This then causes outages on a heavily trafficked NFS server and no one can access their shares until the server is rebooted.

[Fix]

The fix comes in the following two commits:

commit cea57789e4081870ac3498fbefabbbd0d0fd8434
Author: Trond Myklebust <email address hidden>
Date: Sat Mar 9 16:06:47 2019 -0500
Subject: SUNRPC: Clean up

commit 7987b694ade8cc465ce10fb3dceaa614f13ceaf3
Author: Trond Myklebust <email address hidden>
Date: Wed May 29 12:49:52 2019 -0400
Subject: SUNRPC: Fix a use after free when a server rejects the RPCSEC_GSS credential

There is a small subtlety to be addressed here.
7987b694ade8cc465ce10fb3dceaa614f13ceaf3 is marked as "Fixes" cea57789e4081870ac3498fbefabbbd0d0fd8434, and cea57789e4081870ac3498fbefabbbd0d0fd8434 is not present in the disco kernel. The code path which triggers the use after free can still be reached without cea57789e4081870ac3498fbefabbbd0d0fd8434 being present, and applying both commits resolves the problem, while still maintaining as little backporting as necessary.

Please note, both commits have been backported.

cea57789e4081870ac3498fbefabbbd0d0fd8434 required a small change to the final comment, as well as some minor context adjustments in the final hunk.

7987b694ade8cc465ce10fb3dceaa614f13ceaf3 required medium to heavy backporting. The upstream patch uses a switch statement based on error codes, while the disco kernel uses a goto and label type architecture. The commits in the middle were numerous and completely unrelated, so I backported the commit to use goto and labels. Please review this backport a little more closely than you normally do, but it has been tested, and I believe the code to be sound.

cea57789e4081870ac3498fbefabbbd0d0fd8434 landed in 5.1 upstream.
7987b694ade8cc465ce10fb3dceaa614f13ceaf3 landed in 5.2 upstream.

7987b694ade8cc465ce10fb3dceaa614f13ceaf3 was also selected for upstream -stable, in 5.1.9, and was omitted from disco stable updates process, probably because of the patch not cleanly applying and requiring the backport and infrastructure provided in cea57789e4081870ac3498fbefabbbd0d0fd8434.

[Testcase]

This is difficult to reproduce on test systems, and has instead been verified on a production NFS v4.1 system in a customer environment. This server is heavily trafficked and has a large number of different NFS clients connected to it.

I have built a test kernel that contains the above patch, and also patches for Bug 1828978. It is available here:

https://launchpad.net/~mruffell/+archive/ubuntu/sf241068-test

Note that the above kernel is for bionic HWE, and not explicitly disco.

Discussion about the patch validation can be found at the bottom of Bug 1842037.

On unpatched kernels, expect to see the symptoms mentioned in Impact, and on patched systems, everything working as intended.

[Regression Potential]

The changes are limited to users of sunrpc and the change itself is limited to cases where the RPCSEC_GSS credential is rejected. Under blue skies scenarios, the code should only be triggered when misbehaving clients do not keep their authentication tickets up to date.

In case of regression, misbehaving clients may cause outages on services which use sunrpc. In which case, the server administrator would have to revert to an earlier kernel in order to restore those services, as you cannot stop misbehaving clients from connecting.

Since the fix was selected for upstream stable, it has been vetted and widely accepted by the community. The backport I performed should have no functional difference at all.

CVE References

Frank Burkhardt (fbo)
description: updated
Revision history for this message
Matthew Ruffell (mruffell) wrote :

Hi Frank,

I have been reading up on this particular bug for the last few days now, and I
think I understand what is going on.

Firstly, the commit you requested:

commit 7987b694ade8cc465ce10fb3dceaa614f13ceaf3
Author: Trond Myklebust <email address hidden>
Date: Wed May 29 12:49:52 2019 -0400
Subject: SUNRPC: Fix a use after free when a server rejects the RPCSEC_GSS credential

This commit fixes a problem which was introduced by an earlier commit:

commit cea57789e4081870ac3498fbefabbbd0d0fd8434
Author: Trond Myklebust <email address hidden>
Date: Sat Mar 9 16:06:47 2019 -0500
Subject: SUNRPC: Clean up

Now, this earlier commit "SUNRPC: Clean up" is not present in the current bionic 5.0 HWE kernel, which would mean that you might be facing a different bug entirely.

Is there any chance that you can provide more information about your system?

Can you provide a stack trace output by dmesg, or in /var/log/kern.log for this
problem, or better still, can you provide a sosreport?

https://support.canonical.com/ua/s/article/canonical-support-data-collection-sosreport

Since the bug can still be triggered in different paths, I am in the process of backporting the commit you requested to the 5.0 kernel. The code has changed a lot between 5.2 where the patch was introduced, and 5.0 where I am backporting it to.

I think I have created a good backport, which are below if you are interested:

https://paste.ubuntu.com/p/BPD3PzmZ97/
https://paste.ubuntu.com/p/bmHMD7SPGX/

I am currently building a test kernel with the above patches applied for bionic, and I will let you know once it is ready, likely tomorrow. Once it has finished building I will ask you to install it to validate that it solves the problem you are facing.

In the meantime, can you get me a sosreport from the affected system, and I will let you know when the test kernel is ready.

Please do not post the sosreport publicly, and instead attach it to the SF case.

Let me know if you have any questions,

Thanks,
Matthew

tags: added: sts
Revision history for this message
Matthew Ruffell (mruffell) wrote :

Hi Frank,

The test kernel has built, and I have tested it to ensure it boots, but I now need you to verify that the backported patches fix the issue that you are having.

Can you please install the test kernel onto a system which is having problems with NFS clients that are connecting with invalid kerberos credentials?

Please note this kernel is NOT SUPPORTED by Canonical, and is for TESTING PURPOSES ONLY. ONLY Install in a dedicated test environment.

Instructions to install (on a bionic system):

1) sudo add-apt-repository ppa:mruffell/sf241068-test
2) sudo apt-get update
3) sudo apt install linux-image-unsigned-5.0.0-27-generic linux-headers-5.0.0-27-generic linux-headers-5.0.0-27 linux-modules-5.0.0-27-generic linux-modules-extra-5.0.0-27-generic
4) reboot
5) uname -rv
5.0.0-27-generic #28~18.04.1+hf241068v20190910b2-Ubuntu SMP Tue Sep 10 06:53:53 U

If you get output different from above then you need to edit grub configuration to boot the correct kernel. If you are asked to abort the kernel install due to removing the running kernel, say no.

This kernel contains the below patches you requested:

https://paste.ubuntu.com/p/BPD3PzmZ97/
https://paste.ubuntu.com/p/74n7nq5Wfj/

Please test the kernel and let me know if it fixes your problem. If you could also provide a stack trace or sosreport of the original issue, that would be good as well.

Let me know if you have any questions,

Thanks,
Matthew

Changed in linux-meta-hwe (Ubuntu):
status: New → Invalid
Changed in linux-meta-hwe (Ubuntu Disco):
status: New → Invalid
Changed in linux (Ubuntu Disco):
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Matthew Ruffell (mruffell)
no longer affects: linux-meta-hwe (Ubuntu)
no longer affects: linux-meta-hwe (Ubuntu Disco)
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1842037

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Frank Burkhardt (fbo) wrote : Re: [Bug 1842037] Re: Oops when Kerberos credentials are invalid

Hi Matthey,

----- Ursprüngliche Mail -----
> Von: "Matthew Ruffell" <email address hidden>
> An: "burk" <email address hidden>
> Gesendet: Mittwoch, 11. September 2019 02:44:11
> Betreff: [Bug 1842037] Re: Oops when Kerberos credentials are invalid

> Hi Frank,
>
> The test kernel has built, and I have tested it to ensure it boots, but
> I now need you to verify that the backported patches fix the issue that
> you are having.
>
> Can you please install the test kernel onto a system which is having
> problems with NFS clients that are connecting with invalid kerberos
> credentials?
>
> Please note this kernel is NOT SUPPORTED by Canonical, and is for
> TESTING PURPOSES ONLY. ONLY Install in a dedicated test environment.

thank you for the quick response. I'm deploying the kernel to
a test system right now.

Best regards,

Frank

--
Frank Burkhardt <email address hidden>
IT Dept., Max Planck Institute for Human Cognitive
and Brain Sciences, Leipzig, Germany

Revision history for this message
Frank Burkhardt (fbo) wrote :

Hi,

----- Ursprüngliche Mail -----
> Von: "Bug 1842037" <email address hidden>
> An: "burk" <email address hidden>
> Gesendet: Mittwoch, 11. September 2019 02:44:11
> Betreff: [Bug 1842037] Re: Oops when Kerberos credentials are invalid

> Hi Frank,
>
> The test kernel has built, and I have tested it to ensure it boots, but
> I now need you to verify that the backported patches fix the issue that
> you are having.

unfortunately, I have been unable to reproduce the problem in my test
environment until now. I attached the stack trace which I took when
opening the bug. I'll open the compute server with the patched kernel to some
test users now.

Best,

Frank

--
Frank Burkhardt <email address hidden>
IT Dept., Max Planck Institute for Human Cognitive
and Brain Sciences, Leipzig, Germany

Revision history for this message
Matthew Ruffell (mruffell) wrote : Re: Oops when Kerberos credentials are invalid

Hi Frank,

Thanks for the update. Let me know how the test kernel goes when users are on the system.

I looked into your stack trace and found the below link:

https://<email address hidden>/T/

I'm sure you have probably found it too when researching the bug, and the commit needed to fix it. Can you try running the reproducer?

> OK, I think I found something that triggers this fault. This happens
> when certain local users try to stat a file or directory on an nfs
> mount. Presumably these UIDs do not have appropriate permissions on
> the server but I'm not sure exactly (I do not control the server).
>
> I can reproduce the oops with a command like this:
>
> # su -s/bin/sh -c 'stat /path/to/nfs/file' problematic_user
>
> which oopes every time (and SIGKILLs the stat command).

Otherwise, let me know how the test kernel goes when opened up to users.

Thanks,
Matthew

Revision history for this message
Matthew Ruffell (mruffell) wrote :

Hi Frank,

Just checking in to see how the test kernel is going. Does it fix your problem of the kernel crashing when users have invalid kerberos credentials?

Did you try it on the original server which crashes frequently? Has it made things more stable?

Did you have an opportunity to try the reproducer I linked you in my previous message?

Let me know how things are going, when you have had a chance to test the kernel.

Thanks,
Matthew

Revision history for this message
Frank Burkhardt (fbo) wrote : Re: [Bug 1842037] Re: Oops when Kerberos credentials are invalid

Hi Matthew,

----- Ursprüngliche Mail -----
> Von: "Bug 1842037" <email address hidden>
> An: "burk" <email address hidden>
> Gesendet: Donnerstag, 26. September 2019 01:27:15
> Betreff: [Bug 1842037] Re: Oops when Kerberos credentials are invalid

> Hi Frank,
>
> Just checking in to see how the test kernel is going. Does it fix your
> problem of the kernel crashing when users have invalid kerberos
> credentials?

I've not been able to reproduce the problem. The suggested method of
triggering it doesn't seem to work because:

   * If there's no kerberos identity (just a local user), the server
     won't let me enter the mounted folder.
   * If the local user is not known network wide but has a Kerberos
     identity and the remote idmapd cannot resolve it, it's mapped to
     nobody on the server.
   * No Oopps in both cases.

> Did you try it on the original server which crashes frequently? Has it
> made things more stable?

I tried on several of the affected servers. They are not more stable but
suffer from different NFS related problems now.

> Did you have an opportunity to try the reproducer I linked you in my
> previous message?
>
> Let me know how things are going, when you have had a chance to test the
> kernel.

NFS is still unstable but the problem seems to be in GSSD now plus in
Bug 1828978 . However, I can't tell you, if 1828978 happens in Xenial, only.
Both problems are triggered relatively seldom and only cause headaches because
they happen on very crowded computer servers. If in doubt, the admins here
try to get the servers running again ASAP which makes analysis very difficult.

However, bug 1842037 is very clearly visible in the logs so I'm quite sure,
I didn't miss it. I think the best solution for now is to park the ticket on
your side and I'll provide feedback as soon as it happens again.

Thank you very much.

Best,

Frank

--
Frank Burkhardt <email address hidden>
IT Dept., Max Planck Institute for Human Cognitive
and Brain Sciences, Leipzig, Germany

Revision history for this message
Matthew Ruffell (mruffell) wrote : Re: Oops when Kerberos credentials are invalid

Hi Frank,

Thanks for the update.

> > Did you try it on the original server which crashes frequently? Has it
> > made things more stable?
>
> I tried on several of the affected servers. They are not more stable but
> suffer from different NFS related problems now.

> However, bug 1842037 is very clearly visible in the logs so I'm quite sure,
> I didn't miss it. I think the best solution for now is to park the ticket on
> your side and I'll provide feedback as soon as it happens again.

For Bug 1842037, how often would you normally see it happen? Would it crash your server once a day, every few days, or very seldom? By being clearly visible in the logs, did an oops appear in the logs each time you had to reboot your server?

What happened when you installed the test kernel? Do you still see the oops for bug 1842037 appear in the logs? Or is it not present at all anymore, but you now suffer from bug 1828978 instead? If it was present before, but it is no longer present with the test kernel, it might mean the test kernel fixed bug 1842037.

> NFS is still unstable but the problem seems to be in GSSD now plus in
> Bug 1828978 . However, I can't tell you, if 1828978 happens in Xenial, only.

I had a look at Bug 1829878, and I have started building a kernel which contains the commit you requested:

commit 068feb42581a81de45ec49618c554243c07c800e
Author: Trond Myklebust <email address hidden>
Date: Wed Jun 20 17:53:34 2018 -0400
Subject: NFSv4.1: Avoid false retries when RPC calls are interrupted

The test kernel will be available from the same ppa as the initial test kernel, https://launchpad.net/~mruffell/+archive/ubuntu/sf241068-test in a few hours time, once it has compiled.

The kernel which is currently building contains all three commits we were talking about. The two commits for Bug 1842037, and the above commit for Bug 1829878.

Note, for 1829878, the 4.18 HWE kernel is no longer supported, so any fixes will be made available for the 5.0 HWE kernel.

I will write to you tomorrow with instructions on how to install the test kernel, but if you are eager, they are the exact same as the previous test kernel, as the kernel version has not changed, as I just added the extra commit ontop of the last test kernel.

Let me know if you have any questions, and let me know if you still see symptoms of bug 1842037 with the previous test kernel.

Thanks,
Matthew

Revision history for this message
Matthew Ruffell (mruffell) wrote :

Hi Frank,

I have tested the new test kernel and it boots successfully.

Please note this kernel is NOT SUPPORTED by Canonical, and is for TESTING PURPOSES ONLY. ONLY Install in a dedicated test environment.

Instructions to install (on a bionic system):

1) sudo add-apt-repository ppa:mruffell/sf241068-test
2) sudo apt-get update
3) sudo apt install linux-image-unsigned-5.0.0-27-generic linux-headers-5.0.0-27-generic linux-headers-5.0.0-27 linux-modules-5.0.0-27-generic linux-modules-extra-5.0.0-27-generic
4) reboot
5) uname -rv
5.0.0-27-generic #28~18.04.1+hf241068v20191002b1-Ubuntu SMP Wed Oct 2 04:04:05 UTC

If you get different output from uname -rv, you might be booted into the wrong kernel, and you will need to change your grub configuration. Let me know if you need help with this.

Otherwise, this test kernel contains the requested patches for Bug 1842037 and Bug 1829878.

Let me know how this test kernel goes, and if it fixes your NFS problems. If it doesn't, it would be good to upload some logs so we can have a better look.

Thanks,
Matthew

Revision history for this message
Frank Burkhardt (fbo) wrote : Re: [Bug 1842037] Re: Oops when Kerberos credentials are invalid

Hi,

----- Ursprüngliche Mail -----
> Von: "Bug 1842037" <email address hidden>
> An: "burk" <email address hidden>
> Gesendet: Donnerstag, 3. Oktober 2019 01:22:16
> Betreff: [Bug 1842037] Re: Oops when Kerberos credentials are invalid

> Hi Frank,
>
> I have tested the new test kernel and it boots successfully.
>
> Please note this kernel is NOT SUPPORTED by Canonical, and is for
> TESTING PURPOSES ONLY. ONLY Install in a dedicated test environment.
>
> Instructions to install (on a bionic system):
>
> 1) sudo add-apt-repository ppa:mruffell/sf241068-test
> 2) sudo apt-get update
> 3) sudo apt install linux-image-unsigned-5.0.0-27-generic
> linux-headers-5.0.0-27-generic linux-headers-5.0.0-27
> linux-modules-5.0.0-27-generic linux-modules-extra-5.0.0-27-generic
> 4) reboot
> 5) uname -rv
> 5.0.0-27-generic #28~18.04.1+hf241068v20191002b1-Ubuntu SMP Wed Oct 2 04:04:05
> UTC
>
> If you get different output from uname -rv, you might be booted into the
> wrong kernel, and you will need to change your grub configuration. Let
> me know if you need help with this.
>
> Otherwise, this test kernel contains the requested patches for Bug
> 1842037 and Bug 1829878.

Thank you very much.

> Let me know how this test kernel goes, and if it fixes your NFS
> problems. If it doesn't, it would be good to upload some logs so we can
> have a better look.

I applied the test kernel to 3 of our production systems (since there's no
way to trigger either bug artificially). I know I shouldn't but I'm willing
to take the heat for problems arising from doing that.

I'll write you as soon as there's a hint of a problem on these machines.

Best,

Frank

--
Frank Burkhardt <email address hidden>
IT Dept., Max Planck Institute for Human Cognitive
and Brain Sciences, Leipzig, Germany

Revision history for this message
Matthew Ruffell (mruffell) wrote : Re: Oops when Kerberos credentials are invalid

Hi Frank,

It has been 9 days, or slightly over a week since you applied the respun test kernel to your systems.

Are your systems stable now? Are they suffering any symptoms of Bug 1842037 or Bug 1828978?

Is the test kernel more stable than regular released Ubuntu kernels? Do you think the patches you requested fix the problems you were having?

Does the test kernel still crash when mis-behaving NFS clients connect?

Let me know how the respun test kernel is going. If things are good, I can begin the process of getting the patches into an official Ubuntu kernel. If things are not good, if you provide some logs of call traces you are seeing, we can keep working on solving your problem.

Thanks,
Matthew

Revision history for this message
Frank Burkhardt (fbo) wrote : Re: [Bug 1842037] Re: Oops when Kerberos credentials are invalid

Hi Matthew,

----- Ursprüngliche Mail -----
> Von: "Bug 1842037" <email address hidden>
> An: "burk" <email address hidden>
> Gesendet: Samstag, 19. Oktober 2019 09:56:12
> Betreff: [Bug 1842037] Re: Oops when Kerberos credentials are invalid

> Hi Frank,
>
> It has been 9 days, or slightly over a week since you applied the respun
> test kernel to your systems.

It looks very promising. We had no more problems on the test systems.

> Are your systems stable now?

Yes, they are.

> Are they suffering any symptoms of Bug 1842037 or Bug 1828978?

No, they're not.

> Is the test kernel more stable than regular released Ubuntu kernels? Do
> you think the patches you requested fix the problems you were having?

I do think so. Please integrated the patches.

Thank you very much.

Best regards,

Frank

--
Frank Burkhardt <email address hidden>
IT Dept., Max Planck Institute for Human Cognitive
and Brain Sciences, Leipzig, Germany

summary: - Oops when Kerberos credentials are invalid
+ SUNRPC: Use after free when GSSD credentials are invalid causes oops
description: updated
Changed in linux (Ubuntu):
status: Incomplete → Fix Released
Revision history for this message
Matthew Ruffell (mruffell) wrote :

Hi Frank,

Great to hear that the respun test kernel solves your NFS issues.

I have submitted the patches to be included in the Ubuntu Bionic 5.0 HWE kernel.

You can see the patches on the kernel mailing list if you are interested:

Bug 1842037:

https://lists.ubuntu.com/archives/kernel-team/2019-October/105115.html
https://lists.ubuntu.com/archives/kernel-team/2019-October/105116.html
https://lists.ubuntu.com/archives/kernel-team/2019-October/105117.html

Bug 1828978:

https://lists.ubuntu.com/archives/kernel-team/2019-October/105113.html
https://lists.ubuntu.com/archives/kernel-team/2019-October/105114.html

The next step is for the kernel team to review the patches, and if they receive two acks, they will be built into the next kernel update. From there, the kernel team will build the update, and release it for testing in -proposed. When the time comes, you will then need to verify that the kernel in -proposes fixes your NFS problems. If all goes well, we will mark this bug as verified and it will be released a week or so later.

To give you an idea of the timeframes, https://kernel.ubuntu.com/ shows the next SRU cycle stops accepting commits on the 6th of November, the test kernel will be built between the 11th and 15th, and we will need to test the -proposed kernel between the 18th and 29th. If everything goes well, the kernel will be released early December.

I will keep you informed as we progress to each stage, and I will let you know when it is time to test the new update in -proposed.

Let me know if you have any questions.

Thanks,
Matthew

Revision history for this message
Matthew Ruffell (mruffell) wrote :

Hi Frank,

Just giving you an update on the status of these bugs. The kernel team has reviewed the patches I submitted, and each set of patches have received two acks each, meaning they will be built into the next kernel update.

Bug 1842037:

https://lists.ubuntu.com/archives/kernel-team/2019-October/105125.html
https://lists.ubuntu.com/archives/kernel-team/2019-November/105308.html

Bug 1828978:

https://lists.ubuntu.com/archives/kernel-team/2019-November/105304.html
https://lists.ubuntu.com/archives/kernel-team/2019-November/105322.html

The next step is for the kernel team to apply the patches to the kernel source trees, and build the kernel update, and release it for testing in -proposed. This will happen over the next week.

Once the kernel has been built and released to -proposed, we will need to verify that the kernel in -proposed fixes your NFS problem, so I will ask you to do some testing.

I will let you know when it is time to test the new update in -proposed. Likely in the next week or so.

As always, let me know if you have any questions.

Thanks,
Matthew

Changed in linux (Ubuntu Disco):
status: In Progress → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-disco' to 'verification-done-disco'. If the problem still exists, change the tag 'verification-needed-disco' to 'verification-failed-disco'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-disco
Revision history for this message
Frank Burkhardt (fbo) wrote :

I installed the patched kernel on 6 machines with several of my useds hitting them hard. The problem can no longer be reproduced.

Revision history for this message
Frank Burkhardt (fbo) wrote :

tags:added: verification-done-disco

Revision history for this message
Frank Burkhardt (fbo) wrote : Re: [Bug 1842037] Re: SUNRPC: Use after free when GSSD credentials are invalid causes oops

hi,

----- Ursprüngliche Mail -----
> Von: "Bug 1842037" <email address hidden>
> An: "burk" <email address hidden>
> Gesendet: Freitag, 8. November 2019 00:37:27
> Betreff: [Bug 1842037] Re: SUNRPC: Use after free when GSSD credentials are invalid causes oops

> Hi Frank,
>
> Just giving you an update on the status of these bugs. The kernel team
> has reviewed the patches I submitted, and each set of patches have
> received two acks each, meaning they will be built into the next kernel
> update.

thank you very much. I managed to install the proposed kernel on several
machines and had a lot of users testing them successfully. However, I can't
find a way to add a tag to a bug. Same goes for bug 1828978 which is fixed
in the same kernel.

Best,

Frank

--
Frank Burkhardt <email address hidden>
IT Dept., Max Planck Institute for Human Cognitive
and Brain Sciences, Leipzig, Germany

tags: added: verification-done-disco
removed: verification-needed-disco
Revision history for this message
Matthew Ruffell (mruffell) wrote :

Hi Frank,

Thanks for testing the kernel in -proposed, it lets us mark the bug as verified so the patches make it into the release, which will be happening in about two weeks going by https://kernel.ubuntu.com/

Just to be clear, you tested the bionic-hwe kernel 5.0.0-37~18.04.1 from bionic -proposed?

I have updated the tag, which in the future, can be done by clicking the pencil icon at the Tags: section, at the end of the summary at the top of the page, right before the first comment.

Thanks again for reporting the bugs and helping us test. There is nothing more to do now but wait for the SRU complete. The kernel will be released somewhere around the 2nd of December, give or take a few days in case any CVEs turn up.

Thanks,
Matthew

Revision history for this message
Frank Burkhardt (fbo) wrote :

Hi,

----- Ursprüngliche Mail -----
> Von: "Bug 1842037" <email address hidden>
> An: "burk" <email address hidden>
> Gesendet: Montag, 18. November 2019 07:41:30
> Betreff: [Bug 1842037] Re: SUNRPC: Use after free when GSSD credentials are invalid causes oops

> Hi Frank,
>
> Thanks for testing the kernel in -proposed, it lets us mark the bug as
> verified so the patches make it into the release, which will be
> happening in about two weeks going by https://kernel.ubuntu.com/
>
> Just to be clear, you tested the bionic-hwe kernel 5.0.0-37~18.04.1 from
> bionic -proposed?

root@styx:~ > uname -a
Linux styx 5.0.0-37-generic #40~18.04.1-Ubuntu SMP Thu Nov 14 12:06:39 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Yes, I'm certain.

> I have updated the tag, which in the future, can be done by clicking the
> pencil icon at the Tags: section, at the end of the summary at the top
> of the page, right before the first comment.

Found it.

> Thanks again for reporting the bugs and helping us test. There is
> nothing more to do now but wait for the SRU complete. The kernel will be
> released somewhere around the 2nd of December, give or take a few days
> in case any CVEs turn up.

Thank you very much.

Best,

Frank

--
Frank Burkhardt <email address hidden>
IT Dept., Max Planck Institute for Human Cognitive
and Brain Sciences, Leipzig, Germany

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (19.3 KiB)

This bug was fixed in the package linux - 5.0.0-37.40

---------------
linux (5.0.0-37.40) disco; urgency=medium

  * disco/linux: 5.0.0-37.40 -proposed tracker (LP: #1852253)

  * System hangs at early boot (LP: #1851216)
    - x86/timer: Skip PIT initialization on modern chipsets

  * drm/i915: Add support for another CMP-H PCH (LP: #1848491)
    - drm/i915/cml: Add second PCH ID for CMP

  * Some EFI systems fail to boot in efi_init() when booted via maas
    (LP: #1851810)
    - efi: efi_get_memory_map -- increase map headroom

  * seccomp: fix SECCOMP_USER_NOTIF_FLAG_CONTINUE test (LP: #1849281)
    - SAUCE: seccomp: avoid overflow in implicit constant conversion
    - SAUCE: seccomp: rework define for SECCOMP_USER_NOTIF_FLAG_CONTINUE
    - SAUCE: seccomp: fix SECCOMP_USER_NOTIF_FLAG_CONTINUE test

  * dkms artifacts may expire from the pool (LP: #1850958)
    - [Packaging] dkms -- try launchpad librarian for pool downloads
    - [Packaging] dkms -- dkms-build quieten wget verbiage

  * update ENA driver to version 2.1.0 (LP: #1850175)
    - net: ena: fix swapped parameters when calling
      ena_com_indirect_table_fill_entry
    - net: ena: fix: Free napi resources when ena_up() fails
    - net: ena: fix incorrect test of supported hash function
    - net: ena: fix return value of ena_com_config_llq_info()
    - net: ena: improve latency by disabling adaptive interrupt moderation by
      default
    - net: ena: fix ena_com_fill_hash_function() implementation
    - net: ena: add handling of llq max tx burst size
    - net: ena: ethtool: add extra properties retrieval via get_priv_flags
    - net: ena: replace free_tx/rx_ids union with single free_ids field in
      ena_ring
    - net: ena: arrange ena_probe() function variables in reverse christmas tree
    - net: ena: add newline at the end of pr_err prints
    - net: ena: documentation: update ena.txt
    - net: ena: allow automatic fallback to polling mode
    - net: ena: add support for changing max_header_size in LLQ mode
    - net: ena: optimise calculations for CQ doorbell
    - net: ena: add good checksum counter
    - net: ena: use dev_info_once instead of static variable
    - net: ena: add MAX_QUEUES_EXT get feature admin command
    - net: ena: enable negotiating larger Rx ring size
    - net: ena: make ethtool show correct current and max queue sizes
    - net: ena: allow queue allocation backoff when low on memory
    - net: ena: add ethtool function for changing io queue sizes
    - net: ena: remove inline keyword from functions in *.c
    - net: ena: update driver version from 2.0.3 to 2.1.0
    - net: ena: Fix bug where ring allocation backoff stopped too late
    - Revert "net: ena: ethtool: add extra properties retrieval via
      get_priv_flags"
    - net: ena: don't wake up tx queue when down
    - net: ena: clean up indentation issue

  * Add Intel Comet Lake ethernet support (LP: #1848555)
    - SAUCE: e1000e: Add support for Comet Lake

  * Intel Wireless AC 3168 on Eoan complaints FW error in SYNC CMD
    GEO_TX_POWER_LIMIT (LP: #1846016)
    - iwlwifi: exclude GEO SAR support for 3168

  * tsc marked unstable after entered PC10 on Intel CoffeeLake (LP: #1840239...

Changed in linux (Ubuntu Disco):
status: Fix Committed → Fix Released
Revision history for this message
Matthew Ruffell (mruffell) wrote :

Hi Frank,

I have good news. The SRU has finished and the kernel update has been released to -updates. You can "apt update" and "apt upgrade" to install the fixed kernel.

This kernel should be the exact same as the one from -proposed, which is version 5.0.0-37.40~18.04.1, so if you are still running that one, you probably won't have to update to get it.

I hope this fixes everything for you and you have nice stable machines going forward.

Let me know if you have any questions, otherwise I will start moving to close the case in a week or so.

Thanks,
Matthew

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.