nginx crashes with dump because of segfault

Bug #1952614 reported by Bernd Lentes

This bug report will be marked for expiration in 22 days if no further activity occurs. (find out why)

6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
nginx (Ubuntu)
Undecided
Unassigned

Bug Description

nginx crashed already twice because of a segfault.
It happened both shortly after logrotate.

Nov 16 00:01:23 documents-oo kernel: [2726316.050697] nginx[2579242]: segfault at 41 ip 00007fbf59548593 sp 00007ffc02313af0 error 4 in libperl.so.5.30.0[7fbf594e0000+166000]
Nov 16 00:01:23 documents-oo kernel: [2726316.050742] Code: 48 89 43 10 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 40 00 0f b6 7f 30 48 c1 e8 03 48 29 f8 48 89 c3 74 89 48 8b 02 <4c> 8b 68 10 4d 85 ed 0f 84 28 01 00 00 0f b6 40 30 49 c1 ed 03 49

root@documents-oo:~# systemctl status nginx.service
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: failed (Result: core-dump) since Tue 2021-11-16 00:03:17 CET; 8h ago
       Docs: man:nginx(8)
    Process: 2994244 ExecReload=/usr/sbin/nginx -g daemon on; master_process on; -s reload (code=exited, status=0/SUCCESS)
   Main PID: 2579242 (code=dumped, signal=SEGV)
      Tasks: 0 (limit: 19097)
     Memory: 19.7M
     CGroup: /system.slice/nginx.service

Nov 16 00:03:17 documents-oo systemd[1]: nginx.service: Killing process 2809495 (nginx) with signal SIGKILL.
Nov 16 00:03:17 documents-oo systemd[1]: nginx.service: Killing process 2809496 (nginx) with signal SIGKILL.
Nov 16 00:03:17 documents-oo systemd[1]: nginx.service: Killing process 2809497 (nginx) with signal SIGKILL.
Nov 16 00:03:17 documents-oo systemd[1]: nginx.service: Killing process 2809498 (nginx) with signal SIGKILL.
Nov 16 00:03:17 documents-oo systemd[1]: nginx.service: Killing process 2809499 (nginx) with signal SIGKILL.
Nov 16 00:03:17 documents-oo systemd[1]: nginx.service: Killing process 2809500 (nginx) with signal SIGKILL.
Nov 16 00:03:17 documents-oo systemd[1]: nginx.service: Killing process 2809501 (nginx) with signal SIGKILL.
Nov 16 00:03:17 documents-oo systemd[1]: nginx.service: Killing process 2809502 (nginx) with signal SIGKILL.
Nov 16 00:03:17 documents-oo systemd[1]: nginx.service: Killing process 2809503 (nginx) with signal SIGKILL.
Nov 16 00:03:17 documents-oo systemd[1]: nginx.service: Failed with result 'core-dump'.

Oct 31 00:02:06 documents-oo kernel: [1340433.341063] nginx[433202]: segfault at 10 ip 00007f6a15e08593 sp 00007ffeef04e220 error 4 in libperl.so.5.30.0[7f6a15da0000+166000]
Oct 31 00:02:06 documents-oo kernel: [1340433.355995] Code: 48 89 43 10 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 40 00 0f b6 7f 30 48 c1 e8 03 48 29 f8 48 89 c3 74 89 48 8b 02 <4c> 8b 68 10 4d 85 ed 0f 84 28 01 00 00 0f b6 40 30 49 c1 ed 03 49

root@documents-oo:~# systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: failed (Result: core-dump) since Sun 2021-10-31 00:04:57 CEST; 45min ago
       Docs: man:nginx(8)
    Process: 1503969 ExecReload=/usr/sbin/nginx -g daemon on; master_process on; -s reload (code=exited, status=0/SUCCESS)
   Main PID: 433202 (code=dumped, signal=SEGV)
      Tasks: 0 (limit: 19097)
     Memory: 6.1M
     CGroup: /system.slice/nginx.service

Okt 31 00:04:57 documents-oo systemd[1]: nginx.service: Killing process 573437 (nginx) with signal SIGKILL.
Okt 31 00:04:57 documents-oo systemd[1]: nginx.service: Killing process 573438 (nginx) with signal SIGKILL.
Okt 31 00:04:57 documents-oo systemd[1]: nginx.service: Killing process 573439 (nginx) with signal SIGKILL.
Okt 31 00:04:57 documents-oo systemd[1]: nginx.service: Killing process 573440 (nginx) with signal SIGKILL.
Okt 31 00:04:57 documents-oo systemd[1]: nginx.service: Killing process 573441 (nginx) with signal SIGKILL.
Okt 31 00:04:57 documents-oo systemd[1]: nginx.service: Killing process 573442 (nginx) with signal SIGKILL.
Okt 31 00:04:57 documents-oo systemd[1]: nginx.service: Killing process 573443 (nginx) with signal SIGKILL.
Okt 31 00:04:57 documents-oo systemd[1]: nginx.service: Killing process 573444 (nginx) with signal SIGKILL.
Okt 31 00:04:57 documents-oo systemd[1]: nginx.service: Killing process 573445 (nginx) with signal SIGKILL.
Okt 31 00:04:57 documents-oo systemd[1]: nginx.service: Failed with result 'core-dump'.

/etc/os-release:
root@documents-oo:/var/crash# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"

apt-cache policy nginx:
root@documents-oo:/var/crash# apt-cache policy nginx
nginx:
  Installiert: 1.18.0-0ubuntu1.2
  Installationskandidat: 1.18.0-0ubuntu1.2
  Versionstabelle:
 *** 1.18.0-0ubuntu1.2 500
        500 http://de.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        500 http://de.archive.ubuntu.com/ubuntu focal-security/main amd64 Packages
        100 /var/lib/dpkg/status
     1.17.10-0ubuntu1 500
        500 http://de.archive.ubuntu.com/ubuntu focal/main amd64 Packages

What i expect to happen:
nginx running without segfault

What happened instead:
nginx crashed with segfault.

This is my first bug-report ever, so please give me advice if i did something wrong.
Thanks.

Bernd

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: nginx 1.18.0-0ubuntu1.2
ProcVersionSignature: Ubuntu 5.4.0-88.99-generic 5.4.140
Uname: Linux 5.4.0-88-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.21
Architecture: amd64
CasperMD5CheckResult: pass
Date: Mon Nov 29 14:26:38 2021
Dependencies:

InstallationDate: Installed on 2021-01-29 (303 days ago)
InstallationMedia: Ubuntu-Server 20.04.1 LTS "Focal Fossa" - Release amd64 (20200731)
PackageArchitecture: all
ProcEnviron:
 TERM=screen
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=de_DE.UTF-8
 SHELL=/bin/bash
SourcePackage: nginx
UpgradeStatus: No upgrade log present (probably fresh install)
mtime.conffile..etc.apport.crashdb.conf: 2021-11-22T18:33:09.012918

CVE References

Revision history for this message
Bernd Lentes (enzoferrari) wrote :
Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

Thank you for taking the time to file a bug report.

Please try to obtain a backtrace following the instructions at http://wiki.ubuntu.com/DebuggingProgramCrash and upload the backtrace (as an attachment) to the bug report. This will greatly help us in tracking down your problem.

Since there is not enough information in your report to begin triage or to
differentiate between a local configuration problem and a bug in Ubuntu, I
am marking this bug as "Incomplete". We would be grateful if you would:
provide a more complete description of the problem, explain why you
believe this is a bug in Ubuntu rather than a problem specific to your
system, and then change the bug status back to "New".

For local configuration issues, you can find assistance here:
http://www.ubuntu.com/support/community

Changed in nginx (Ubuntu):
status: New → Incomplete
Revision history for this message
Bernd Lentes (enzoferrari) wrote :

Hi,
i'm trying hard to create that backtrace, but it doesn't work:

 ...
dynamically loaded /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.10 needs package libxml2, queueing
Installing extra package nginx-core to get ExecutablePath
Get:1 [3029 kB]
Fetched 3029 kB in 0s (0 B/s)
Extracting downloaded debs...
dpkg-source: info: extracting nginx in nginx-1.18.0
dpkg-source: info: unpacking nginx_1.18.0.orig.tar.gz
dpkg-source: info: unpacking nginx_1.18.0-0ubuntu1.2.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying 0002-Make-sure-signature-stays-the-same-in-all-nginx-buil.patch
dpkg-source: info: applying 0003-define_gnu_source-on-other-glibc-based-platforms.patch
dpkg-source: info: applying ubuntu-branding.patch
dpkg-source: info: applying nginx-fix-pidfile.patch
dpkg-source: info: applying CVE-2021-23017-1.patch
dpkg-source: info: applying CVE-2021-23017-2.patch
W: Der Download wird als root und nicht Sandbox-geschützt durchgeführt, da auf die Datei »nginx_1.18.0-0ubuntu1.2.dsc« durch den Benutzer »_apt« nicht zugegriffen werden kann. - pkgAcquire::Run (13: Permission denied)
--- stack trace ---
#0 0x00007fbf59548593 in ?? ()
No symbol table info available.
#1 0x0000000000000000 in ?? ()
No symbol table info available.
--- source code stack trace ---
#0 0x00007fbf59548593 in ?? ()
#1 0x0000000000000000 in ?? ()

It seems the symbols aren't there. Can you help me here with this problem or do i have to ask that anywhere else ?
Sorry, i'm completely new to filing a bug in Ubuntu.

Thanks,

Bernd

Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

Hi Bernd,

In the web page I linked in my first comment [1] you can find a section telling you multiple ways to get the debug symbols in place. Did you try any of the methods listed there?

[1] https://wiki.ubuntu.com/DebuggingProgramCrash

Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

Hi,

Please note that you will need to install libperl's debuginfo as well as nginx's. Thanks.

Revision history for this message
Bernd Lentes (enzoferrari) wrote :

Hi,
i'm trying hard to install the debug symbols. When i succeed i'll come back to you.
And i will install the debug symbols from libperl. Thanks for your help.
Bernd

Revision history for this message
Bernd Lentes (enzoferrari) wrote :

My stack trace still looks poor:
--- stack trace ---
#0 0x00007fbf59548593 in ?? ()
No symbol table info available.
#1 0x0000000000000000 in ?? ()
No symbol table info available
--- source code stack trace ---
#0 0x00007fbf59548593 in ?? ()
#1 0x0000000000000000 in ?? ()

I installed all dbgsym packages via:
find-dbgsym-packages (pidof nginx) 2>/dev/null |xargs aptitude install -y.
It installed all dbgsym packages except one because i had problems with dependencies:
It said it would need libx11-6=2:1.6.9-2ubuntu1.3 for libx11-6-dbgsym, but 2:1.6.9-2ubuntu1.2 is installed.

Additionally i didn't find debug-symbols for libperl5.30:
root@documents-oo:/var/crash/syslog# aptitude search libperl5.30-dbgsym
root@documents-oo:/var/crash/syslog# aptitude search libperl5.30-dbg
no hit.
I installed libperl5i-perl-dbgsym but that didn't help.
I googled for libperl5.30-dbg, but didn't find anything.

Also "find-dbgsym-packages /usr/lib/x86_64-linux-gnu/libperl.so.5.30.0" didn't find anything.

I'm running out of ideas.

Bernd

Revision history for this message
Bernd Lentes (enzoferrari) wrote :

If i understand https://wiki.ubuntu.com/DebuggingProgramCrash correctly,
this sequence should download all necessary debug symbols automatically:
apport-retrace -s -S system -v /var/crash/_usr_sbin_nginx.0.crash -C /var/crash/nginx

But also this creates a poor stacktrace:
--- stack trace ---
#0 0x00007fbf59548593 in ?? ()
No symbol table info available.
#1 0x0000000000000000 in ?? ()
No symbol table info available.
--- source code stack trace ---
#0 0x00007fbf59548593 in ?? ()
#1 0x0000000000000000 in ?? ()

I get this warning:
"W: Der Download wird als root und nicht Sandbox-geschützt durchgeführt, da auf die Datei »nginx_1.18.0-0ubuntu1.2.dsc« durch den Benutzer »_apt« nicht zugegriffen werden kann. - pkgAcquire::Run (13: Permission denied)"
which means translated
"W: Download is performed unsandboxed as root as file »nginx_1.18.0-0ubuntu1.2.dsc' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)"
I googled that and found it's just a warning, no problem.

Bernd

Revision history for this message
Bernd Lentes (enzoferrari) wrote :

"We would be grateful if you would:
provide a more complete description of the problem, explain why you
believe this is a bug in Ubuntu rather than a problem specific to your
system, and then change the bug status back to "New"."

OK.
Nginx created twice a segfault.
/var/log/syslog:
Oct 31 00:02:06 documents-oo kernel: [1340433.341063] nginx[433202]: segfault at 10 ip 00007f6a15e08593 sp 00007ffeef04e220 error 4 in libperl.so.5.30.0[7f6a15da0000+166000]
Oct 31 00:02:06 documents-oo kernel: [1340433.355995] Code: 48 89 43 10 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 40 00 0f b6 7f 30 48 c1 e8 03 48 29 f8 48 89 c3 74 89 48 8b 02 <4c> 8b 68 10 4d 85 ed 0f 84 28 01 00 00 0f b6 40 30 49 c1 ed 03 49
the second segfault looks the same.
I think this is a bug and not a configuration problem because i changed the configuration just a little bit.
And segfaults are normally bugs in the program code.

Bernd

Changed in nginx (Ubuntu):
status: Incomplete → New
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi Bernd,

For your issues with apport-retrace, yes it should download the files it needs, but in your case seems to fail for permission issues. Could you mabe just try it with sudo and rebuild instead of a direct binary path like:
  $ sudo apport-retrace --sandbox system --rebuild-package-info --stdout /var/crash/_usr_sbin_nginx.0.crash
Let us know if that creates a more useful output.

I agree that a crash is a problem either way. But we need to understand and/or recreate it to find if it is in upstream code or any Ubuntu modification/integration to then discuss/work/resolve the problem in the right place.

Therefore even if (hopefully not) your crash continues to refuse to give you proper symbols - did you try the approach of e.g. taking a clean Ubuntu VM, and installing/configuring it the way you did on the system this happened? This might clarify how related (or not) your config changes are and the same for logrotate that you expected to be related.
If you manage to recreate the issue please let us know the steps to reproduce - with those the debugging can happen on our side and everyone can e.g. compare different versions of nginx and/or releases of Ubuntu against this config.

Incomplete again (for now) as just with the addresses and no steps to recreate no one can help yet. Please continue the discussion and the work to get better debug output and (even better) a repro.

Changed in nginx (Ubuntu):
status: New → Incomplete
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I was happy for a second as I thought I found a related case, but this is the very same case on askubuntu [1], so no more insights from there.

[1]: https://askubuntu.com/questions/1377700/ubuntu-20-04-3-nginx-crashes-with-segfault

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers