libperl.so.5.30.0 causes nginx to segfault

[Sami-Pekka Hallikas]

[Steps to reproduce]

1) launch a focal container
$ lxc launch images:ubuntu/focal focal-1897561
2) enter the container
$ lxc shell focal-1897561
3) install libnginx-mod-http-perl
# apt-get install -y nginx-core libnginx-mod-http-perl
4) check nginx journal
# journalctl -fu nginx &
5) reload nginx 2-3 times
# systemctl reload nginx

Eventually you will see this from the journal:

Oct 13 17:13:34 focal-1897561 systemd[1]: Reloaded A high performance web server and a reverse proxy server.
Oct 13 17:13:35 focal-1897561 systemd[1]: nginx.service: Main process exited, code=dumped, status=11/SEGV
Oct 13 17:13:35 focal-1897561 systemd[1]: nginx.service: Killing process 588 (nginx) with signal SIGKILL.
Oct 13 17:13:35 focal-1897561 systemd[1]: nginx.service: Killing process 589 (nginx) with signal SIGKILL.
Oct 13 17:13:35 focal-1897561 systemd[1]: nginx.service: Killing process 588 (nginx) with signal SIGKILL.
Oct 13 17:13:35 focal-1897561 systemd[1]: nginx.service: Killing process 589 (nginx) with signal SIGKILL.
Oct 13 17:13:35 focal-1897561 systemd[1]: nginx.service: Failed with result 'core-dump'.

[Workaround]

If libnginx-mod-http-perl is not needed, purging it and restarting nginx sidesteps the problem.

[Original bug description]

Ubuntu 20.04 LTS

Looks that Perl 5.30 has bug that causes nginx to die, usually occurs after "service nginx reload"

This looks like nginx, but it really is perl issue:
https://github.com/Perl/perl5/issues/17154

Fix done in 5.32 (Perl5 commit ffffbf0) and people are asking backport to 5.30 to get fix also to nginx.

Error:
Sep 28 07:39:43 host kernel: [1340832.811014] nginx[3253005]: segfault at 10 ip 00007fbf3220d593 sp 00007ffd6bba6260 error 4 in libperl.so.5.30.0[7fbf321a5000+166000]

Symptom: Nginx terminates
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu27.9
Architecture: amd64
CasperMD5CheckResult: skip
DistroRelease: Ubuntu 20.04
Package: perl 5.30.0-9build1
PackageArchitecture: amd64
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 5.4.0-42.46-generic 5.4.44
Tags: focal uec-images
Uname: Linux 5.4.0-42-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True

[Ilya Kruchinin (ilyakruchinin)]

Experiencing the same issue with nginx from official Ubuntu repositories in 20.04 failing every 3rd-4th time on reload signal.

[lotuspsychje (lotuspsychje)]

Thank you for reporting this bug and make ubuntu better!

Please run apport-collect 1897561 from a terminal to pull
relevant info into your bug so developers can debug better

[Sami-Pekka Hallikas (semi-hallikas)]

apport information

[Sami-Pekka Hallikas (semi-hallikas)]

apport information

[Victor Rodriguez (vrc-vlm)]

Another workaround is just to delete the symbolic link at /etc/nginx/modules-enabled/50-mod-http-perl.conf and restart Nginx. Now, Nginx wont load the Perl module and will reload correctly.

In my case I'm using "nginx-extras" and can't purge the package libnginx-mod-http-perl due to dependencies.

[David Frendo (davidfrendo)]

I can also confirm having this issue with nginx on Ubuntu 20.04 since it was released.

Confirmed today (4th January 2021) on a fully installation of Ubuntu 20.04.1 LTS Server

perl5 (revision 5 version 30 subversion 0)

Screenshot taken from server (from /var/log/syslog) attached.

Issue appears to happen only when an nginx reload is called.

Unfortunately, this creates quite an issue especially since certbot nginx plugin invokes a reload.

I would be happy to provide any additional data.

Many thanks.

David

[Stefan Zuban (zubanst)]

Having the same issue, Ubuntu 20.04 LTS kernel 5.4.0-58-generic. Happens with a Onlyoffice document server. As a workaround, I scheduled daily service restart. Here is teh kern.log about.

Jan 7 00:00:08 onlyoffice kernel: [107181.395999] nginx[1025]: segfault at 10 ip 00007f51157ae593 sp 00007ffe9d77aab0 error 4 in libperl.so.5.30.0[7f5115746000+166000]
Jan 7 00:00:08 onlyoffice kernel: [107181.396007] Code: 48 89 43 10 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 40 00 0f b6 7f 30 48 c1 e8 03 48 29 f8 48 89 c3 74 89 48 8b 02 <4c> 8b 68 10 4d 85 ed 0f 84 28 01 00 00 0f b6 40 30 49 c1 ed 03 49
Jan 9 00:00:17 onlyoffice kernel: [279985.262561] nginx[122867]: segfault at 20845 ip 00007fe2372765c9 sp 00007ffd8b7b8840 error 4 in libperl.so.5.30.0[7fe23720e000+166000]
Jan 9 00:00:17 onlyoffice kernel: [279985.262567] Code: 00 0f b6 40 30 49 c1 ed 03 49 29 c5 0f 84 17 01 00 00 48 8b 76 10 48 8b 52 10 4c 8d 3c fe 4c 8d 0c c2 84 c9 0f 84 c7 02 00 00 <49> 83 39 00 0f 85 ad 03 00 00 49 83 c1 08 49 83 ed 01 49 8d 74 1d

Looking forward for a fix.
Thank you.

Stefan

[Surya Prakash (suryaelite)]

Hi,

I'm also getting this error while renewal of the SSL certificates from the LetsEncrpt.
Looking forward for a fix.
I can't remove the perl module as it's required for nginx-extras.

Thanks,
Surya

[Nigel (wpcd)]

Just throwing my 2cents in and confirming that this is an issue with Ubuntu 20.04 & NGINX (Extras) when adding a certificate with CERTBOT.

Issue does not occur on Ubuntu 18.04.

Unlike the prior reports that include NGINX, it looks like NGINX actually "hangs" in the background and any attempts to restart it fail because it's already hogging required ports.

[Kraut.Hosting (kraut.hosting)]

Confirming here we found today dead NGINX hanging on certbot renewals:

Feb 09 15:04:50 focal kernel: Code: 48 89 43 10 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 40 00 0f b6 7f 30 48 c1 e8 03 48 29 f8 48 89 c3 74 89 48 8b 02 <4c> 8b 68 10 4d 85 ed 0f 84 28 01 00 00 0f b6 40 30 49 c1 ed 03 49
Feb 09 15:04:50 focal kernel: nginx[937]: segfault at 10 ip 00007fa35cb73593 sp 00007ffcec0a17d0 error 4 in libperl.so.5.30.0[7fa35cb0b000+166000]

Killing off complete and restart helps on the port hogging as @wpcd said.
This feels to drastic so if there's a patch ready let's backport please.

root@focal:~# apt remove libnginx-mod-http-perl
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libluajit-5.1-2 libluajit-5.1-common libnginx-mod-http-cache-purge libnginx-mod-http-fancyindex
  libnginx-mod-http-headers-more-filter libnginx-mod-http-lua libnginx-mod-http-ndk libnginx-mod-http-uploadprogress
  libnginx-mod-nchan
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  libnginx-mod-http-perl

@teward Can we confirm already this one brings remedy or caveats known?

https://github.com/Perl/perl5/issues/17154

[Thomas Ward (teward)]

I have no insights into whether the linked GitHub issue fixes anything. That's a Perl package problem, and not an NGINX one. We could potentially stop requiring the Perl module but a segfault in libperl is a Perl issue, not an nginx one. However, we don't have a *clear* reproduction solution for this that would guarantee testing, and rebuilding Perl and then building the NGINX package against that perl version is a nontrivial process (and I don't have the cycles right now to drive that kind of build test).

[Kraut.Hosting (kraut.hosting)]

@teward Thanks for your thoughts :) Would like to know more about packaging.
Pushing patched Perl packages to PPA and then build NGINX against won't work?
What functions would we loose by just Recommends Perl module instead Depends.

@all We first need a test case for reproducing those NGINX crashes on Perl.
Anyone got such a clear case yet? We just found "random" crashes by certbot.
Plain journalctl & reload/restart doesn't trigger for us on virtualizations.
Together would need to simulate the payload from certbot if that triggers.

[Thomas Ward (teward)]

@Kraut

That would work for *testing* but Perl is interlaced in **a huge portion** of the Ubuntu OS. Which means for any SRU, **all** autopkgtests need to pass, which is a lot more painful if anything fails.

There's more than just NGINX that needs tested with *any* Perl SRU. Which is why I say this is a Perl package issue and not an nginx one.

[Thomas Ward (teward)]

@Kraut Further, the linked patch at the end of the upstream bug is **not** yet part of the Perl codebase, so it's not been accepted upstream, which is another consideration point and probably why this hasn't had much movement on it yet (and why it's marked 'Incomplete' for Perl)

[Sami-Pekka Hallikas (semi-hallikas)]

In my original report I did say that this really is NOT nginx problem, but Perl problem. Problem is in perl 5.30 and it is 5.32 (Perl5 commit ffffbf0) and people are asking backport to 5.30 to get fix also to nginx.

More in https://github.com/Perl/perl5/issues/17154

My biggest issue is that this bug makes nginx quite unstable in LTS version of Ubuntu, because perl module is active by default. Maybe quick fix to get LTS stabile is to disable perl module by default from nginx-extras (it can delivered in it, but if it commented out from default config).

[peacecop kalmer: (peacecop-kalmer)]

The suggested solution for removing "libnginx-mod-http-perl" doesn't probably make the situation better but worse because it would uninstall "onlyoffice-documentserver" which is unwanted:

"kalmer@test:~$ sudo apt remove libnginx-mod-http-perl
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  cabextract libmspack0 libnginx-mod-http-auth-pam libnginx-mod-http-cache-purge libnginx-mod-http-dav-ext libnginx-mod-http-echo
  libnginx-mod-http-fancyindex libnginx-mod-http-geoip libnginx-mod-http-geoip2 libnginx-mod-http-headers-more-filter libnginx-mod-http-lua
  libnginx-mod-http-ndk libnginx-mod-http-subs-filter libnginx-mod-http-uploadprogress libnginx-mod-http-upstream-fair libnginx-mod-nchan
  ttf-mscorefonts-installer
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  nginx-core
Soovituslikud paketid:
  nginx-doc
EEMALDATAKSE järgnevad paketid:
  libnginx-mod-http-perl nginx-extras onlyoffice-documentserver
Paigaldatakse järgnevad UUED paketid:
  nginx-core
0 uuendatud, 1 värskelt paigaldatud, 3 eemaldada ja 0 uuendamata.
On vaja tõmmata 425 kB arhiive.
Pärast seda tegevust vabaneb 949 MB kettaruumi.
Do you want to continue? [Y/n] n
Katkesta."

I disabled the module instead:
"kalmer@test:/etc/nginx/modules-enabled$ sudo rm 50-mod-http-perl.conf"

So far, "nginx" hasn't crashed after multiple reloading and "onlyoffice-documentserver" also seems to work.

[Martinique (martinique)]

This issue affects other applications as well, such as Pidgin with Perl plugins segfaulting.

[Claudio Kuenzler (napsty)]

Can confirm, this bug is still a reality in Ubuntu 20.04 (focal) and is triggered by two reloads in a row.

root@nginx:~# ps aux|grep nginx
root 3356067 0.0 0.1 71600 9788 ? Ss 15:46 0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 3356068 0.0 0.1 73012 15348 ? S 15:46 0:00 nginx: worker process
www-data 3356069 0.0 0.1 73012 15348 ? S 15:46 0:00 nginx: worker process
www-data 3356070 0.0 0.1 73012 15348 ? S 15:46 0:00 nginx: worker process
www-data 3356071 0.0 0.1 73012 15348 ? S 15:46 0:00 nginx: worker process
www-data 3356072 0.0 0.1 73012 15348 ? S 15:46 0:00 nginx: worker process
www-data 3356073 0.0 0.1 73012 15348 ? S 15:46 0:00 nginx: worker process
www-data 3356074 0.0 0.1 73012 15348 ? S 15:46 0:00 nginx: worker process
www-data 3356075 0.0 0.1 73012 15348 ? S 15:46 0:00 nginx: worker process
root 3356102 0.0 0.0 8900 676 pts/0 S+ 15:46 0:00 grep --color=auto nginx

root@nginx:~# service nginx reload

root@nginx:~# ps aux|grep nginx
root 3356067 0.2 0.3 81404 28848 ? Ss 15:46 0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 3356266 0.0 0.2 81644 22608 ? S 15:46 0:00 nginx: worker process
www-data 3356267 0.0 0.2 81644 22608 ? S 15:46 0:00 nginx: worker process
www-data 3356268 0.0 0.2 81644 22608 ? S 15:46 0:00 nginx: worker process
www-data 3356269 0.0 0.2 81644 22608 ? S 15:46 0:00 nginx: worker process
www-data 3356270 0.0 0.2 81644 22608 ? S 15:46 0:00 nginx: worker process
www-data 3356271 0.0 0.2 81644 22608 ? S 15:46 0:00 nginx: worker process
www-data 3356272 0.0 0.2 81644 22608 ? S 15:46 0:00 nginx: worker process
www-data 3356273 0.0 0.2 81644 22608 ? S 15:46 0:00 nginx: worker process
root 3356296 0.0 0.0 17196 7168 ? R 15:46 0:00 /usr/bin/pgrep nginx
root 3356309 0.0 0.0 8900 736 pts/0 S+ 15:46 0:00 grep --color=auto nginx

root@nginx:~# service nginx reload

root@nginx:~# ps aux|grep nginx
root 3356355 0.0 0.0 8900 736 pts/0 S+ 15:46 0:00 grep --color=auto nginx

At the time of the second reload, the following segfault is logged in /var/log/kern.log:

May 4 15:46:50 onl-paylb02-p kernel: [4141537.262674] nginx[3356067]: segfault at 51 ip 00007f6b7d1535c9 sp 00007ffe634334d0 error 4 in libperl.so.5.30.0[7f6b7d0eb000+166000]
May 4 15:46:50 onl-paylb02-p kernel: [4141537.262681] Code: 00 0f b6 40 30 49 c1 ed 03 49 29 c5 0f 84 17 01 00 00 48 8b 76 10 48 8b 52 10 4c 8d 3c fe 4c 8d 0c c2 84 c9 0f 84 c7 02 00 00 <49> 83 39 00 0f 85 ad 03 00 00 49 83 c1 08 49 83 ed 01 49 8d 74 1d

[Enno Ritz (thanathros)]

Hello everyone,

I have the same issue running the following setup:
Ubuntu 20.04.2 LTS
kernel 5.4.0-73-generic
nginx/1.18.0
naxsi 1.3 Latest

Both systemctl reload and the reload through letsencrypt potentially triggers this.
My previous assumption was the usage of naxsi in this, but that's merely an assumption.

Jun 17 23:44:30 localhost kernel: [1893305.871732] show_signal_msg: 20 callbacks suppressed
Jun 17 23:44:30 localhost kernel: [1893305.871753] nginx[209784]: segfault at 41 ip 00007f0ca0922593 sp 00007fff27f6ea60 error 4 in libperl.so.5.30.0[7f0ca08ba000+166000]
Jun 17 23:44:30 localhost kernel: [1893305.871773] Code: 48 89 43 10 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 40 00 0f b6 7f 30 48 c1 e8 03 48 29 f8 48 89 c3 74 89 48 8b 02 <4c> 8b 68 10 4d 85 ed 0f 84 28 01 00 00 0f b6 40 30 49 c1 ed 03 49

Kind regards

[cptX (myrbourfake)]

Hi,
I can confirm that after upgrading ubuntu 18.04 to 20.04.03 when certbot tried to reload the nginx it had issues. After disabling the file 50-mod-http-perl.conf (by renaming it 50-mod-http-perl.conf_bak) the issue was solved!!

[Alexis Lecocq (axtux)]

Hello,

I also found my Nginx server down after several hours. Nowadays certificates have short life time, their issuance is automated and the reloading of the servers is mandatory to update the certificates. In short, an automated action is making the web server crash. In my opinion, the importance of this bug is very high. This seems to be a patch for 5.30.2 https://github.com/Perl/perl5/issues/17154#issuecomment-638452287

Kind regards,

[Xing Li (diegomontoya)]

Confirmed issue still persists with Ubuntu 22.04 and Nginx 1.18.0 with letsencrypt certbot renewal checks.

[3861805.583095] nginx[254589]: segfault at 10 ip 00007fadfd915593 sp 00007ffcca68cc30 error 4 in libperl.so.5.30.0[7fadfd8ad000+166000]

[Timothy Quinn (tim-c-quinn)]

Confirmed with Ubuntu desktop 20.04.1, fully updated. nginx version is 1.18.0 (Ubuntu).

Tried updating nginx using: https://www.linuxbabe.com/security/modsecurity-nginx-debian-ubuntu which brings to 1.23.0 and issue is not resolved.

[wei shuai (cpuwolf)]

same issue here on Ubuntu server 20.04.5 LTS

[Pietro Mingo (skar395)]

I confirm that this problem continues for me with nginx and ubuntu 22.04.
I'm solving by uninstalling libnginx-mod-http-perl replacing nginx-extras with nginx-core.