| 2020-10-14 14:24:08 |
Michael Bacarella |
bug |
|
|
added bug |
| 2020-10-14 14:24:08 |
Michael Bacarella |
attachment added |
|
one-line fix from the glibc bugzilla https://bugs.launchpad.net/bugs/1899800/+attachment/5422253/+files/mitigation.patch |
|
| 2020-10-14 16:29:11 |
Ubuntu Foundations Team Bug Bot |
tags |
|
patch |
|
| 2020-10-14 16:29:18 |
Ubuntu Foundations Team Bug Bot |
bug |
|
|
added subscriber Ubuntu Review Team |
| 2020-10-28 20:03:11 |
Balint Reczey |
nominated for series |
|
Ubuntu Groovy |
|
| 2020-10-28 20:03:11 |
Balint Reczey |
bug task added |
|
glibc (Ubuntu Groovy) |
|
| 2020-10-28 20:03:11 |
Balint Reczey |
nominated for series |
|
Ubuntu Bionic |
|
| 2020-10-28 20:03:11 |
Balint Reczey |
bug task added |
|
glibc (Ubuntu Bionic) |
|
| 2020-10-28 20:03:11 |
Balint Reczey |
nominated for series |
|
Ubuntu Focal |
|
| 2020-10-28 20:03:11 |
Balint Reczey |
bug task added |
|
glibc (Ubuntu Focal) |
|
| 2020-10-28 20:06:46 |
Balint Reczey |
glibc (Ubuntu): importance |
Undecided |
Medium |
|
| 2020-10-28 20:06:49 |
Balint Reczey |
glibc (Ubuntu Bionic): importance |
Undecided |
Medium |
|
| 2020-10-28 20:06:52 |
Balint Reczey |
glibc (Ubuntu Focal): importance |
Undecided |
Medium |
|
| 2020-10-28 20:06:55 |
Balint Reczey |
glibc (Ubuntu Groovy): importance |
Undecided |
Medium |
|
| 2020-11-25 09:32:48 |
Dominique Poulain |
bug |
|
|
added subscriber Dominique Poulain |
| 2020-12-01 10:01:23 |
Launchpad Janitor |
glibc (Ubuntu): status |
New |
Fix Released |
|
| 2020-12-16 11:37:38 |
Balint Reczey |
description |
This bug was submitted by Qin Li to glibc bugzilla earlier this year, with a one-line patch, though it hasn't been merged into glibc yet:
https://sourceware.org/bugzilla/show_bug.cgi?id=25847
This bug in pthread conditions will deadlock the OCaml runtime, as well as Python's runtime, and .NET.
The bug was introduced in glibc 2.27, so affects Ubuntu 18.04 onwards. I confirm my OCaml app, as well as the repro from the bugzilla, deadlocks on Ubuntu 20.04 and Ubuntu 18.04. To further strengthen the case that this is because of a bug in glibc, my app and the repro do not deadlock on Ubuntu 16.04.
To rule out kernel issues, I further confirm that no deadlock happens when I copy Ubuntu 16.04's libc to 18.04 and redirect the dynamic linker so my app loads the earlier libc.
I confirm that the one-line patch (available at the above bugzilla) applies cleanly on top of:
* glibc-2.31-0ubuntu9.1 (Ubuntu 20.04 latest)
* glibc-2.28-10 (Debian Buster/10 latest)
* glibc-2.27-3ubuntu1.2 (Ubuntu 18.04 latest)
I confirm that the one-line patch to glibc cures the deadlock issue in my OCaml apps.
On Ubuntu 20.04, I have not been able to get the repro to deadlock in 5 days. My OCaml apps have not deadlocked in 5 days.
On Debian Buster/10, the repro has not deadlocked in about 5 days. This is my desktop box, and I can otherwise use normal applications as usual like the GNOME environment, etc.
On Ubuntu 18.04, the repro takes about 24-48 hours before it triggers a deadlock. Prior to patching glibc, it would take only a few hours. I have not seen my OCaml apps deadlock since applying this patch, however.
On Ubuntu 16.04 I have not been able to get the repro to deadlock ever. My OCaml apps never deadlocked on this platform. This is expected, since this platform runs glibc 2.23, where the bug has not been introduced yet (the bugzilla report claims introduced in 2.27).
As for why 18.04 still deadlocks, I suspect another, unrelated pthread bug was introduced in glibc 2.27 and fixed by 2.28. When applied to glibc 2.27, the one-line patch appears to significantly reduce the deadlocking by an order of magnitude.
Please kindly consider merging the patch into Ubuntu glibc.
More background about this bug, for the sake of future internet searchers:
* https://discuss.ocaml.org/t/is-there-a-known-recent-linux-locking-bug-that-affects-the-ocaml-runtime |
[Impact]
* Various multi-threaded applications using pthread_cond hang.
[Test Case]
* Run the reproducer attached to the upstream bug report (I used a qemu-emulated 8 core machine on a 4 core one):
wget https://sourceware.org/bugzilla/attachment.cgi?id=12480 -O repro-lp1899800.c
gcc -pthread repro-lp1899800.c
./a.out 16
Total Threads Count; 16
RefereeThread - (null) started
LoopCriticalSectionThread - 1 started
...
LoopCriticalSectionThread - 16 started
Monitor - g_counter 411380000, loop_round 3024, threads_finished 13
...
Monitor - g_counter 1920301632, loop_round 266764, threads_finished 0
Monitor - g_counter -1851097664, loop_round 270614, threads_finished 13
Monitor - g_counter -1227241664, loop_round 275201, threads_finished 14
Monitor - g_counter -337385664, loop_round 281744, threads_finished 0
Monitor - g_counter 519822336, loop_round 288047, threads_finished 16
Monitor - g_counter 1401918336, loop_round 294533, threads_finished 0
Monitor - g_counter -1993136960, loop_round 301150, threads_finished 16
Monitor - g_counter -1140185466, loop_round 307422, threads_finished 12
Monitor - g_counter -1063307960, loop_round 307987, threads_finished 15
Monitor - g_counter -1063307960, loop_round 307987, threads_finished 15
Monitor - g_counter -1063307960, loop_round 307987, threads_finished 15
Monitor - g_counter -1063307960, loop_round 307987, threads_finished 15
Monitor - g_counter -1063307960, loop_round 307987, threads_finished 15
...
The lockup is observed as repeating identical lines ^.
* Observe the threads hanging in a few minutes with unfixed libc6 and not hanging for hours with the fixed one.
[Where problems could occur]
* The fix which is rather a workaround in the one-line form is waking up all threads when there is a chance of hitting the deadlock. This causes a slight rare overhead, but the exact amount of the overhead is unknown.
[Original Bug Text]
This bug was submitted by Qin Li to glibc bugzilla earlier this year, with a one-line patch, though it hasn't been merged into glibc yet:
https://sourceware.org/bugzilla/show_bug.cgi?id=25847
This bug in pthread conditions will deadlock the OCaml runtime, as well as Python's runtime, and .NET.
The bug was introduced in glibc 2.27, so affects Ubuntu 18.04 onwards. I confirm my OCaml app, as well as the repro from the bugzilla, deadlocks on Ubuntu 20.04 and Ubuntu 18.04. To further strengthen the case that this is because of a bug in glibc, my app and the repro do not deadlock on Ubuntu 16.04.
To rule out kernel issues, I further confirm that no deadlock happens when I copy Ubuntu 16.04's libc to 18.04 and redirect the dynamic linker so my app loads the earlier libc.
I confirm that the one-line patch (available at the above bugzilla) applies cleanly on top of:
* glibc-2.31-0ubuntu9.1 (Ubuntu 20.04 latest)
* glibc-2.28-10 (Debian Buster/10 latest)
* glibc-2.27-3ubuntu1.2 (Ubuntu 18.04 latest)
I confirm that the one-line patch to glibc cures the deadlock issue in my OCaml apps.
On Ubuntu 20.04, I have not been able to get the repro to deadlock in 5 days. My OCaml apps have not deadlocked in 5 days.
On Debian Buster/10, the repro has not deadlocked in about 5 days. This is my desktop box, and I can otherwise use normal applications as usual like the GNOME environment, etc.
On Ubuntu 18.04, the repro takes about 24-48 hours before it triggers a deadlock. Prior to patching glibc, it would take only a few hours. I have not seen my OCaml apps deadlock since applying this patch, however.
On Ubuntu 16.04 I have not been able to get the repro to deadlock ever. My OCaml apps never deadlocked on this platform. This is expected, since this platform runs glibc 2.23, where the bug has not been introduced yet (the bugzilla report claims introduced in 2.27).
As for why 18.04 still deadlocks, I suspect another, unrelated pthread bug was introduced in glibc 2.27 and fixed by 2.28. When applied to glibc 2.27, the one-line patch appears to significantly reduce the deadlocking by an order of magnitude.
Please kindly consider merging the patch into Ubuntu glibc.
More background about this bug, for the sake of future internet searchers:
* https://discuss.ocaml.org/t/is-there-a-known-recent-linux-locking-bug-that-affects-the-ocaml-runtime |
|
| 2020-12-16 19:51:29 |
Łukasz Zemczak |
glibc (Ubuntu Groovy): status |
New |
Fix Committed |
|
| 2020-12-16 19:51:31 |
Łukasz Zemczak |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2020-12-16 19:51:32 |
Łukasz Zemczak |
bug |
|
|
added subscriber SRU Verification |
| 2020-12-16 19:51:36 |
Łukasz Zemczak |
tags |
patch |
patch verification-needed verification-needed-groovy |
|
| 2021-01-07 07:31:43 |
Balint Reczey |
tags |
patch verification-needed verification-needed-groovy |
patch verification-done verification-done-groovy |
|
| 2021-01-26 19:25:24 |
Brian Murray |
tags |
patch verification-done verification-done-groovy |
block-proposed-groovy patch verification-done verification-done-groovy |
|
| 2021-02-07 15:08:51 |
Slav Ivanyuk |
bug |
|
|
added subscriber Slav Ivanyuk |
| 2021-02-16 06:19:23 |
Launchpad Janitor |
glibc (Ubuntu Bionic): status |
New |
Confirmed |
|
| 2021-02-16 06:19:23 |
Launchpad Janitor |
glibc (Ubuntu Focal): status |
New |
Confirmed |
|
| 2021-04-06 22:11:30 |
Andy Schwerin |
bug |
|
|
added subscriber Andy Schwerin |
| 2021-04-08 09:29:04 |
Łukasz Zemczak |
tags |
block-proposed-groovy patch verification-done verification-done-groovy |
block-proposed-groovy patch verification-needed verification-needed-groovy |
|
| 2021-04-08 09:47:27 |
Łukasz Zemczak |
glibc (Ubuntu Focal): status |
Confirmed |
Fix Committed |
|
| 2021-04-08 09:47:36 |
Łukasz Zemczak |
tags |
block-proposed-groovy patch verification-needed verification-needed-groovy |
block-proposed-groovy patch verification-needed verification-needed-focal verification-needed-groovy |
|
| 2021-04-23 20:57:17 |
Balint Reczey |
tags |
block-proposed-groovy patch verification-needed verification-needed-focal verification-needed-groovy |
block-proposed-groovy patch verification-done-focal verification-needed verification-needed-groovy |
|
| 2021-04-26 16:08:35 |
Launchpad Janitor |
glibc (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2021-04-26 16:08:47 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2021-04-28 13:37:39 |
Balint Reczey |
glibc (Ubuntu Focal): status |
Fix Released |
Fix Committed |
|
| 2021-05-07 04:42:51 |
Lauri Koskela |
bug |
|
|
added subscriber Lauri Koskela |
| 2021-07-28 23:12:54 |
Brian Murray |
glibc (Ubuntu Groovy): status |
Fix Committed |
Won't Fix |
|
| 2021-08-10 09:03:52 |
Evgeny Morozov |
bug watch added |
|
https://github.com/dotnet/runtime/issues/47700 |
|
| 2022-01-25 07:53:09 |
Michael Hudson-Doyle |
tags |
block-proposed-groovy patch verification-done-focal verification-needed verification-needed-groovy |
verification-needed verification-needed-focal |
|
| 2022-03-16 22:03:24 |
Brian Murray |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2022-04-26 00:54:57 |
Michael Hudson-Doyle |
tags |
verification-needed verification-needed-focal |
verification-done-focal |
|
| 2022-05-11 01:47:31 |
Launchpad Janitor |
glibc (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2024-02-17 09:35:12 |
kalvdans |
bug |
|
|
added subscriber kalvdans |
