libperl5.30 crash (segfault) at Perl__invlist_intersection_maybe_complement_2nd during nginx reload

[Walter]

On Focal, I got this in my kern.log:

  nginx[533]: segfault at 739 ip 00007fadc806d5d9 sp 00007ffc04f5cd50 error 4 in libperl.so.5.30.0[7fadc8005000+166000]

  Code: 00 0f b6 40 30 49 c1 ed 03 49 29 c5 0f 84 17 01 00 00 48 8b 76 10 48 8b 52 10 4c 8d 3c fe 4c 8d 0c c2 84 c9 0f 84 c7 02 00 00 <49> 83 39 00 0f 85 ad 03 00 00 49 83 c1 08 49 83 ed 01 49 8d 74 1d

Looking at IP ( 0x00007fadc806d5d9 - 0x7fadc8005000 ) it appeared to point at 0x685D9 in libperl.so.5.30.0.

  # addr2line -Cfe /usr/lib/x86_64-linux-gnu/libperl.so.5.30 685D9
  Perl_vload_module
  op.c:7752

But when looking at the code, it looks like it's at 0x685D9 + 0x48000 = 0xB05D9:

  # addr2line -Cfe /usr/lib/x86_64-linux-gnu/libperl.so.5.30 B05D9
  Perl__invlist_intersection_maybe_complement_2nd
  regcomp.c:9841

This makes more sense:

  # objdump -d /usr/lib/x86_64-linux-gnu/libperl.so.5.30
  ...
  00000000000b0500 <Perl__invlist_intersection_maybe_complement_2nd@@Base>:
  ...
  b05cd: 4c 8d 0c c2 lea (%rdx,%rax,8),%r9
  b05d1: 84 c9 test %cl,%cl
  b05d3: 0f 84 c7 02 00 00 je b08a0 <Perl__invlist_intersection_maybe_complement_2nd@@Base+0x3a0>

  b05d9: 49 83 39 00 cmpq $0x0,(%r9) <-- here

  b05dd: 0f 85 ad 03 00 00 jne b0990 <Perl__invlist_intersection_maybe_complement_2nd@@Base+0x490>
  b05e3: 49 83 c1 08 add $0x8,%r9
  b05e7: 49 83 ed 01 sub $0x1,%r13

There's a similar segfault:

  nginx[356456]: segfault at 10 ip 00007f4f576785a3 sp 00007ffd0be49220 error 4 in libperl.so.5.30.0[7f4f57610000+166000]

  Code: 48 89 43 10 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 40 00 0f b6 7f 30 48 c1 e8 03 48 29 f8 48 89 c3 74 89 48 8b 02 <4c> 8b 68 10 4d 85 ed 0f 84 28 01 00 00 0f b6 40 30 49 c1 ed 03 49

That is on 0xB05A3, also in Perl__invlist_intersection_maybe_complement_2nd:

  b0598: 48 29 f8 sub %rdi,%rax
  b059b: 48 89 c3 mov %rax,%rbx
  b059e: 74 89 je b0529 <Perl__invlist_intersection_maybe_complement_2nd@@Base+0x29>
  b05a0: 48 8b 02 mov (%rdx),%rax
  b05a3: 4c 8b 68 10 mov 0x10(%rax),%r13 <-- here
  b05a7: 4d 85 ed test %r13,%r13
  b05aa: 0f 84 28 01 00 00 je b06d8 <Perl__invlist_intersection_maybe_complement_2nd@@Base+0x1d8>

On GitHub I found a bug filed for perl 5.30 and this function:

  https://github.com/Perl/perl5/issues/17154

That issue is fixed in perl 5.32.0 and beyond (across multiple commits).

Apparently the bug triggers every now and then, but was not common enough to be noticed. And looking at the timestamps, it is always during an nginx reload.

Cheers,
Walter Doekes
OSSO B.V.

[Walter (wdoekes)]

Forgot to mention the versions:

- libperl5.30:amd64 5.30.0-9ubuntu0.4
- nginx-common 1.18.0-0ubuntu1.4