whoopsie assert failure: double free or corruption (fasttop)

[Alex Murray]

[Impact]
whoopsie is crashing somewhat regularly. There are a couple of crashes in the Error Tracker which look similar:

https://errors.ubuntu.com/problem/5c1f68854a0f3bd5e263f1cd35e4dd944c9e90bd
https://errors.ubuntu.com/problem/5d7e641dc46229c08389420fdd74a7473d2dec98

As of yet (2020-11-22) neither crash is happening with whoopsie version 0.2.73.

[Test Case]
We don't have a manual test case for this crash report but given the number of crashes in the Error Tracker and the number of users of whoopsie the absence of the same crash occurring with the new version of whoopsie should be enough to consider this verified.

[Regression Potential]
Whoopsie will no longer upload crash reports with duplicate keys but this was not something it should have been doing in the first place. That being said we should also test whoopsie with a sample of crashes (apport-test-crashes) and ensure they are received by the staging version of the Ubuntu Error Tracker.

Original Description
--------------------
Apport popped up saying whoopsie had crashed (I assume during a previous crash report upload?)

ProblemType: CrashDistroRelease: Ubuntu 20.10
Package: whoopsie 0.2.72
ProcVersionSignature: Ubuntu 5.8.0-20.21-generic 5.8.10
Uname: Linux 5.8.0-20-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu49
Architecture: amd64
AssertionMessage: double free or corruption (fasttop)
CasperMD5CheckResult: skip
CrashCounter: 1
Date: Fri Oct 9 08:08:56 2020
ExecutablePath: /usr/bin/whoopsie
InstallationDate: Installed on 2019-11-18 (325 days ago)
InstallationMedia: Ubuntu 19.10 "Eoan Ermine" - Release amd64 (20191017)
ProcCmdline: /usr/bin/whoopsie -f
ProcEnviron:
 LANG=en_AU.UTF-8
 LANGUAGE=en_AU:en
 PATH=(custom, no user)
RelatedPackageVersions: apport-noui N/ASignal: 6SourcePackage: whoopsie
StacktraceTop:
 __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f5982159128 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
 malloc_printerr (str=str@entry=0x7f598215b5d8 "double free or corruption (fasttop)") at malloc.c:5389
 _int_free (av=0x7f598218bba0 <main_arena>, p=0x55964a2bf3e0, have_lock=0) at malloc.c:4298
 () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
 g_hash_table_remove_all () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
Title: whoopsie assert failure: double free or corruption (fasttop)
UpgradeStatus: Upgraded to groovy on 2020-10-06 (2 days ago)
UserGroups: N/A
separator:

[Apport retracing service (apport)]

StacktraceTop:
 __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f5982159128 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
 malloc_printerr (str=str@entry=0x7f598215b5d8 "double free or corruption (fasttop)") at malloc.c:5389
 _int_free (av=0x7f598218bba0 <main_arena>, p=0x55964a2bf3e0, have_lock=0) at malloc.c:4298
 g_hash_table_remove_all_nodes.part () from /srv/vms/apport-sandbox-dir/Ubuntu 20.10/amd64/report-sandbox/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6600.0
 g_hash_table_remove_all () from /srv/vms/apport-sandbox-dir/Ubuntu 20.10/amd64/report-sandbox/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6600.0

[Marc Deslauriers (mdeslaur)]

Do you have any additional whoopsie logs in /var/log or in journalctl -u whoopsie.service ?

[Marc Deslauriers (mdeslaur)]

So, I've looked through the code again, and I believe it is mishandling duplicate keys. I'm not sure if duplicate keys can find themselves in crash files, but if so, here's a theory:

value gets malloced here:
533: key = g_malloc ((token_p - p) + 1);

value and key get inserted into the hash table here:
575: g_hash_table_insert (hash_table, key, value ? value : g_strdup(""));

key is then reused here:
505: g_hash_table_insert (hash_table, key, value ? value : g_strdup(""));

If there is a duplicate key and it already exists in the hash table, g_hash_table_insert will free the passed key, which means that a use-after-free is happening on line 505. Later, then the hash table is destroyed here:

776: g_hash_table_destroy (report);

the stale key pointer is being double-freed.

If this code is fixed, we should also make sure the g_hash_table_steal functions on lines 488 and 496 are done _before_ value is realloced on lines 484 and 490. I'm not sure that causes any issue, but it's worth fixing.

[Marc Deslauriers (mdeslaur)]

Here's a proposed fix, not sure if this is the exact cause of the double-free or if duplicate keys are acceptable or not.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Brian Murray (brian-murray)]

I've uploaded this for Hirsute - thanks!

[Launchpad Janitor (janitor)]

This bug was fixed in the package whoopsie - 0.2.73

---------------
whoopsie (0.2.73) hirsute; urgency=medium

  * Attempt to fix double free issue (LP: #1899100)
    - src/whoopsie.c: reject duplicate keys, re-order certain operations.
    - src/tests/data/crash/invalid_key_duplicate,
      src/tests/test_parse_report.c: added test for duplicate keys.

 -- Marc Deslauriers <email address hidden> Mon, 26 Oct 2020 14:40:14 -0400

[Brian Murray (brian-murray)]

Bug 1892713 seems to be a duplicate of this one or will be fixed by the same changes that went into whoopsie 0.2.73.

[Timo Aaltonen (tjaalton)]

Hello Alex, or anyone else affected,

Accepted whoopsie into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/whoopsie/0.2.72.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Setting to verification-done for Groovy given that the buckets linked to in the bug description do not contain whoopsie version number 0.2.72.1 and there are no new buckets with similar signatures.

[Launchpad Janitor (janitor)]

This bug was fixed in the package whoopsie - 0.2.72.1

---------------
whoopsie (0.2.72.1) groovy; urgency=medium

  * Attempt to fix double free issue (LP: #1899100)
    - src/whoopsie.c: reject duplicate keys, re-order certain operations.
    - src/tests/data/crash/invalid_key_duplicate,
      src/tests/test_parse_report.c: added test for duplicate keys.

 -- Marc Deslauriers <email address hidden> Mon, 26 Oct 2020 14:40:14 -0400

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for whoopsie has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Łukasz Zemczak (sil2100)]

Hello Alex, or anyone else affected,

Accepted whoopsie into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/whoopsie/0.2.69ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Alex, or anyone else affected,

Accepted whoopsie into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/whoopsie/0.2.62ubuntu0.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted whoopsie (0.2.69ubuntu0.2) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

indicator-session/17.3.20+19.10.20190921-0ubuntu1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#whoopsie

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted whoopsie (0.2.62ubuntu0.6) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

apport/2.20.9-0ubuntu7.20 (amd64, i386)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#whoopsie

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Brian Murray (brian-murray)]

Looking at the Error Tracker crash buckets linked to from the bug description the version of the package from focal-proposed and bionic-proposed does not appear. Additionally, I was unable to find any similar looking crashes in the Error Tracker so there are no new "double free" buckets with the version of the package from -proposed. Subsequently, I'm setting the tags to verification-done.

[Launchpad Janitor (janitor)]

This bug was fixed in the package whoopsie - 0.2.69ubuntu0.2

---------------
whoopsie (0.2.69ubuntu0.2) focal; urgency=medium

  * Attempt to fix double free issue (LP: #1899100)
    - src/whoopsie.c: reject duplicate keys, re-order certain operations.
    - src/tests/data/crash/invalid_key_duplicate,
      src/tests/test_parse_report.c: added test for duplicate keys.

 -- Brian Murray <email address hidden> Wed, 02 Dec 2020 09:29:08 -0800

[Launchpad Janitor (janitor)]

This bug was fixed in the package whoopsie - 0.2.62ubuntu0.6

---------------
whoopsie (0.2.62ubuntu0.6) bionic; urgency=medium

  * Attempt to fix double free issue (LP: #1899100)
    - src/whoopsie.c: reject duplicate keys, re-order certain operations.
    - src/tests/data/crash/invalid_key_duplicate,
      src/tests/test_parse_report.c: added test for duplicate keys.

 -- Brian Murray <email address hidden> Wed, 02 Dec 2020 09:35:52 -0800