ThreadSanitizer: unexpected memory mapping with 6.5.0-25.25~22.04.1

[Dimitry Andric]

We updated a number of (amd64) machines from linux-image-6.5.0-21-generic (6.5.0-21.21~22.04.1) to linux-image-6.5.0-25-generic (6.5.0-25.25~22.04.1), and this caused ThreadSanitizer-instrumented programs to immediately exit with an error similar to:

FATAL: ThreadSanitizer: unexpected memory mapping 0x5c4dc2bcd000-0x5c4dc2bed000

Reverting the kernel back to 6.5.0-21.21~22.04.1 and rebooting makes the same executables work again.

There are a few older bugs including https://gcc.gnu.org/bugzilla/show_bug.cgi?id=67308 and https://github.com/google/sanitizers/issues/503 but these are from ~2015, and about gcc 7 and linux 4.1 so it does not seem to be the same problem.

A very small program, lifted from https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual can be used to show the problem:

$ cat simple-race.c
#include <pthread.h>
#include <stdio.h>

int Global;

void *Thread1(void *x) {
  Global++;
  return NULL;
}

void *Thread2(void *x) {
  Global--;
  return NULL;
}

int main() {
  pthread_t t[2];
  pthread_create(&t[0], NULL, Thread1, NULL);
  pthread_create(&t[1], NULL, Thread2, NULL);
  pthread_join(t[0], NULL);
  pthread_join(t[1], NULL);
}

$ cc -fsanitize=thread -fPIE -pie -g simple-race.c -o simple-race

$ ./simple-race
FATAL: ThreadSanitizer: unexpected memory mapping 0x5d161227c000-0x5d161227d000

$ uname -a
Linux buildhost 6.5.0-25-generic #25~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Feb 20 16:09:15 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

When running that exact same binary, on the same machine, but with linux 6.5.0-21:

$ ./simple-race
==================
WARNING: ThreadSanitizer: data race (pid=19373)
  Read of size 4 at 0x560964a2d014 by thread T2:
    #0 Thread2 /home/lxc-unpriv/simple-race.c:12 (simple-race+0x12d1)

  Previous write of size 4 at 0x560964a2d014 by thread T1:
    #0 Thread1 /home/lxc-unpriv/simple-race.c:7 (simple-race+0x128c)

  Location is global 'Global' of size 4 at 0x560964a2d014 (simple-race+0x000000004014)

  Thread T2 (tid=19376, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 main /home/lxc-unpriv/simple-race.c:19 (simple-race+0x1368)

  Thread T1 (tid=19375, finished) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 main /home/lxc-unpriv/simple-race.c:18 (simple-race+0x1347)

SUMMARY: ThreadSanitizer: data race /home/lxc-unpriv/simple-race.c:12 in Thread2
==================
ThreadSanitizer: reported 1 warnings

$ uname -a
Linux buildhost 6.5.0-21-generic #21~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Feb 9 13:32:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

[Dimitry Andric (dimitry.unified-streaming.com)]

Note that similar to the older gcc and upstream sanitizer bugs, this problem may be caused by some kernel API now returning a slightly different result, so it could be debatable whether the issue should be fixed in ThreadSanitizer, or in the kernel itself.

[Dimitry Andric (dimitry.unified-streaming.com)]

One other note, I only get the "unexpected memory mapping" error on amd64. On arm64 machines with 6.5.0-25.25~22.04.1, ThreadSanitizer works without issues.

[Stefan Bader (smb)]

Setting the affected package to the mantic:linux kernel as that is the source for jammy:linux-hwe-6.5.

[Dimitry Andric (dimitry.unified-streaming.com)]

This could be the same thing reported in https://github.com/google/sanitizers/issues/1716 ("Thread Sanitizer FATAL error on kernel version 6.6.6-x").

[Dimitry Andric (dimitry.unified-streaming.com)]

It's indeed the bump of vm.mmap_rnd_bits from 28 to 32, which is part of the changelog in https://launchpad.net/ubuntu/+source/linux/6.5.0-25.25 :

  * test_021_aslr_dapper_libs from ubuntu_qrt_kernel_security failed on K-5.19 /
    J-OEM-6.1 / J-6.2 AMD64 (LP: #1983357)
    - [Config]: set ARCH_MMAP_RND_{COMPAT_, }BITS to the maximum

On my amd64 6.5.0-21-generic machines, the default value is:

vm.mmap_rnd_bits = 28

while on 6.5.0-25-generic, it is:

vm.mmap_rnd_bits = 32

Forcing vm.mmap_rnd_bits back to 28 on 6.5.0-25-generic makes ThreadSanitizer work again.

I'm unsure if this bug should be kept open now, since it is maybe expected that some user space programs break due to the changed randomization settings?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dimitry Andric (dimitry.unified-streaming.com)]

https://github.com/google/sanitizers/issues/1716#issuecomment-2010399341

Summarizing: newer ThreadSanitizer has been updated to be able to work with vm.mmap_rnd_bits = 30, but not higher than that. For other cases, it will now re-execute the binary with ASLR disabled, if possible.

For most people the quick workaround of setting vm.mmap_rnd_bits to 28 might work, but of course you won't get the benefits intended with #1983357 either.

So at some point, it would be nice to have the upstream ThreadSanitizer fixes backported.