[SRU] Memory leak in krb5 version 1.17

[Ponnuvel Palaniyappan]

[ Impact ]

Commit https://github.com/krb5/krb5/commit/1cd2821c19b2b95e39d5fc2f451a035585a40fa5
altered the memory management of krb5_gss_inquire_cred(), introducing defcred to act as
an owner pointer when the function must acquire a default credential.
The commit neglected to update the code to release the default cred
along the successful path. The old code does not trigger because
cred_handle is now reassigned, so the default credential is leaked.

Resulting gradual increase in memory usage (memory leak) and eventual crash.

[ Test Plan ]

Setup 3 VMs:

1. Windows Server act as Domain controller (AD)
2. Windows machine AD Joined with Ostress installed. (Ostress is part of RML utilities https://learn.microsoft.com/en-us/troubleshoot/sql/tools/replay-markup-language-utility)
3. SQL on Linux AD Joined ( configuration steps https://learn.microsoft.com/en-us/sql/linux/sql-server-linux-ad-auth-adutil-tutorial?view=sql-server-ver16)

On the Machine with OStress create a file (name it disconnect.ini) with the following content under the same folder “C:\Program Files\Microsoft Corporation\RMLUtils” where OStress is installed.

disconnect.ini
==============

[Connection Options]
LoginTimeout=30
QuotedIdentifier=Off
AutocommitMode=On
DisconnectPct=100.0
MaxThreadErrors=0

[Query Options]
NoSQLBindCol=Off
NoResultDisplay=Off
PrepareExecute=Off
ExecuteAsync=Off
RollbackOnCancel=Off
QueryTimeout=0
QueryDelay=0
MaxRetries=0
BatchDisconnectPct=0.0
CancelPct=0.00
CancelDelay=0
CancelDelayMin=0
CursorType=
CursorConcurrency=
RowFetchDelay=0

[Replay Options]
Sequencing Options=global sequence
::Sequencing Options=global sequence, dtc replay
DTC Timeout=
DTC Machine=(local)
Playback Coordinator=(local)
StartSeqNum=
StopSeqNum=
TimeoutFactor=1.0

Run the following command to start the load using Ostress, change Server name (-S) accordingly and the number of threads (-n) as needed.

Start 4 different CMD consoles and use the following different commands for each CMD window:
1. ostress.exe -E -S<ServerName/port> -Q"select * from sys.all_objects" -q -cdisconnect.ini -n40 -r9999999 -oc:\temp\log01 -T146
2. ostress.exe -E -S<ServerName/port> -Q"select * from sys.all_views" -q -cdisconnect.ini -n40 -r9999999 -oc:\temp\log02 -T146
3. ostress.exe -E -S<ServerName/port> -Q"select * from sys.all_columns" -q -cdisconnect.ini -n40 -r9999999 -oc:\temp\log03 -T146
4. ostress.exe -E -S<ServerName/port> -Q"select * from sys.all_parameters" -q -cdisconnect.ini -n40 -r9999999 -oc:\temp\log04 -T146

After a run of about 5 hours, the memory usage for this is expected to be around 5G with the fix.
Without the fix, it was observed that it reached around ~22G in 5 hours. Hence the increase in
memory usage can be observed if the ostress.exe programs are let to run longer.

[ Where problems could occur ]

 The fix may not fix the memory leak or could result in releasing the memory
 early in a different code path, and thus resulting in crashes.

 A mitigating fact is that the fix has been in Ubuntu since at least 22.04 and
 they do not exhibit any issues.

 Likewise I've previously provided the fix in a PPA https://launchpad.net/~pponnuvel/+archive/ubuntu/krb5-focal
 to user who's been hit by this issue. They've tested and confirmed it fixes the memory leak.

[ Other Info ]

The commit https://github.com/krb5/krb5/commit/098f874f3b50dd2c46c0a574677324b5f6f3a1a8 fixes the leak.

The fix has been included in newer krb5 releases (Jammy, and Noble have the releases with the fix).

Bionic doesn't have the commit the introduced the memory leak in the first place.
So this will be a Focal-only backport.

[Lucas Kanashiro (lucaskanashiro)]

Hi Ponnuvel,

I see you prepared a debdiff to fix this issue, if you are seeking for a sponsor for you package you could use the Patch Pilot program [1]. Long story short, you could subscribe ~ubuntu-sponsors to this bug and it would go to the Patch Pilot queue.

This is in the Server team queue, and once we have time we will take a look if no one else had already done that. It may take some time.

[1] https://ubuntu.com/community/contribute/ubuntu-development/ubuntu-patch-pilots

[Ponnuvel Palaniyappan (pponnuvel)]

Attaching the debdiff for Focal.

[Heitor Alves de Siqueira (halves)]

Thanks for the debdiff, Pon! Changes look good, it's a cherry-pick from upstream and there have been no follow-up Fixes:. Local autopkgtests have passed and we have confirmation from affected users that have tested these changes in an Azure environment, so I've sponsored this for Focal.

I'd only like to request that when this hits -proposed, it be tested in a long-running session to properly validate the memory leak (e.g. the suggestion of 5 hours from the Test Plan looks like a good start!).

[Andreas Hasenack (ahasenack)]

Hello Ponnuvel, or anyone else affected,

Accepted krb5 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/krb5/1.17-6ubuntu4.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ponnuvel Palaniyappan (pponnuvel)]

1.17-6ubuntu4.5 has been installed from focal-proposed and has been confirmed to fix the leak (memory usage is stable after several hours - following the test procedure). Marking verification done.

[Andreas Hasenack (ahasenack)]

There are many autopkgtest regressions listed at:

https://ubuntu-archive-team.ubuntu.com/proposed-migration/focal/update_excuses.html#krb5

For now I have retried them.

[Mitchell Dzurick (mitchdz)]

There was one more testbed failure for arm64 balsa. I reran that.

[Ponnuvel Palaniyappan (pponnuvel)]

Thanks, Andreas, and Mitchell. All passed now.

[Ponnuvel Palaniyappan (pponnuvel)]

1.17-6ubuntu4.6 has superseded the previous version 1.17-6ubuntu4.5 :(

Uploading a new debdiff on top of 1.17-6ubuntu4.6.

[Andreas Hasenack (ahasenack)]

Same patch as before, just with a new name ;)

[Andreas Hasenack (ahasenack)]

Hello Ponnuvel, or anyone else affected,

Accepted krb5 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/krb5/1.17-6ubuntu4.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted krb5 (1.17-6ubuntu4.7) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

evolution-data-server/unknown (armhf)
freeipa/unknown (s390x)
moonshot-gss-eap/1.0.1-6build1 (arm64, armhf)
nfs-utils/unknown (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#krb5

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ponnuvel Palaniyappan (pponnuvel)]

Verfiication has been repeated with 1.17-6ubuntu4.7 from focal-proposed and has been confirmed to fix the leak (memory usage is stable after several hours - following the test procedure). Marking verification done.

[Dariusz Gadomski (dgadomski)]

I have checked the s390 and armhf - they seem unrelated to the patch, look more like infra hiccup (connection issues). I have restarted them.

[Andreas Hasenack (ahasenack)]

https://autopkgtest.ubuntu.com/packages/postgresql-12/focal/s390x is green now, and https://autopkgtest.ubuntu.com/packages/moonshot-gss-eap/focal/armhf got another tmpfail. I retried that one one more time.

[Andreas Hasenack (ahasenack)]

It passed now https://autopkgtest.ubuntu.com/packages/moonshot-gss-eap/focal/armhf

[Launchpad Janitor (janitor)]

This bug was fixed in the package krb5 - 1.17-6ubuntu4.7

---------------
krb5 (1.17-6ubuntu4.7) focal; urgency=medium

  * Fix a memory leak in krb5_gss_inquire_cred (LP: #2060666)

 -- Ponnuvel Palaniyappan <email address hidden> Thu, 08 Aug 2024 11:06:56 +0100

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for krb5 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.