busybox-static: several network applets segfaulting

[Simon Rettberg]

On a fully up-to-date Ubuntu 17.04, running most applets from busybox-static that are network-related lead to a segfault. Example:

$ busybox nslookup google.com 8.8.8.8
Server: 8.8.8.8
Segmentation fault

$ busybox
BusyBox v1.22.1 (Ubuntu 1:1.22.0-19ubuntu2) multi-call binary.
[...]

$ apt-cache policy busybox
busybox:
  Installed: (none)
  Candidate: 1:1.22.0-19ubuntu2
  Version table:
     1:1.22.0-19ubuntu2 500
        500 http://ftp.fau.de/ubuntu zesty/universe amd64 Packages

$ dpkg -S /bin/busybox
busybox-static: /bin/busybox

$ apt-cache policy busybox-static
busybox-static:
  Installed: 1:1.22.0-19ubuntu2
  Candidate: 1:1.22.0-19ubuntu2
  Version table:
 *** 1:1.22.0-19ubuntu2 500
        500 http://ftp.fau.de/ubuntu zesty/main amd64 Packages
        100 /var/lib/dpkg/status

This even happens when passing invalid domains or DNS servers to use that don't actually run any DNS service. (like "busybox nslookup google.com 1.2.3.4" or "busybox nslookup bar.foof00 8.8.8.8"), so it seems to be early in the network setup.

I could not reproduce the problem on the very same system when compiling stock busybox 1.22.1 (why even? we're at 1.27.x!) myself, neither with a recent git clone. Also two VMs I had at hand running Ubuntu 14.04 and 16.04 didn't show this problem with busybox-static. Installing the package "busybox" (which removes busybox-static) fixes the problem on 17.04.
The only thing that might be related in any way about my system is that it has no IPv6 connectivity (apart from the link-local address).

I also sent a crash report when the apport window popped up, but I have no idea where this ends up and how to add further information, hence this report here. Please let me know if this seemingly trivial bug cannot be reproduced instantly, so I can try to assist with further information.

[Steve Langasek (vorlon)]

$ busybox nslookup google.com 8.8.8.8
Server: 8.8.8.8
Address 1: 8.8.8.8 google-public-dns-a.google.com

Name: google.com
Address 1: 2607:f8b0:400a:809::200e sea15s12-in-x0e.1e100.net
Address 2: 172.217.3.206 sea15s12-in-f14.1e100.net
$

Not reproducible here. Would need a backtrace in order to debug this.

[Simon Rettberg (simon-rettberg)]

Steve, so I just manually gather a core dump and attach it here? I assumed this is what apport is for. I uploaded the crash report using the standard crash report window that pops up when I opened the issue. Do they just go straight to /dev/null instead of the maintainers?

[Steve Langasek (vorlon)]

Thanks. Loading busybox-static-dbgsym and retracing this with gdb, I see the following backtrace:

(gdb) thread apply all bt full

Thread 1 (Thread 0x11bb880 (LWP 31592)):
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#1 0x00007f601bb0af90 in __pthread_initialize_minimal_internal ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#2 0x00007f601bb0a571 in _init () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#3 0x00007f601c358f70 in ?? ()
No symbol table info available.
#4 0x00000000004f147a in call_init.part ()
No symbol table info available.
#5 0x00000000004f1635 in _dl_init ()
No symbol table info available.
#6 0x00000000004e3a46 in dl_open_worker ()
No symbol table info available.
#7 0x00000000004e11f4 in _dl_catch_error ()
No symbol table info available.
#8 0x00000000004e33b9 in _dl_open ()
No symbol table info available.
#9 0x000000000049ff22 in do_dlopen ()
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#10 0x00000000004e11f4 in _dl_catch_error ()
No symbol table info available.
#11 0x00000000004a010e in __libc_dlopen_mode ()
No symbol table info available.
#12 0x0000000000493aa8 in __nss_next2 ()
No symbol table info available.
#13 0x000000000048b310 in gethostbyaddr_r ()
No symbol table info available.
#14 0x000000000048d158 in gni_host_inet_name.isra ()
No symbol table info available.
#15 0x000000000048d752 in getnameinfo ()
No symbol table info available.
#16 0x000000000058c312 in sockaddr2str (sa=0x11bdc50, flags=flags@entry=10)
    at libbb/xconnect.c:439
        host = "8.8.8.8\000\000\334\033\001", '\000' <repeats 20 times>, "\350\003\000\000\000\000\000\000\261tB\000\000\000\000\000\002\000\000\000\000\000\000\000\b\000\000\000\000\000\000\000\240\064\203\230\377\177\000\000\372\316B\000\000\000\000\000\003\000\000\000\000\000\000\000\240\064\203\230\377\177\000\000\220\064\203\230\377\177\000\000\372\235O\000\000\000\000\000\003\000\000\000\000\000\000\000K\303X\000\000\000\000"
        serv = "0\000\000\000\000\000\000\000\000K~\000\000\000\000"
        rc = <optimized out>
        salen = <optimized out>
---Type <return> to continue, or q <return> to quit---
#17 0x000000000058caf0 in xmalloc_sockaddr2hostonly_noport (sa=<optimized out>)
    at libbb/xconnect.c:476
No locals.
#18 0x000000000050eb64 in print_host (hostname=<optimized out>,
    header=header@entry=0x5c187f "Server:") at networking/nslookup.c:92
        dotted = 0x11be080 "8.8.8.8"
        revhost = <optimized out>
        cur = 0x11bdc20
        cnt = 0
        result = 0x11bdc20
        rc = 0
        hint = {ai_flags = 0, ai_family = 0, ai_socktype = 1, ai_protocol = 0,
          ai_addrlen = 0, ai_addr = 0x0, ai_canonname = 0x0, ai_next = 0x0}
#19 0x000000000050ec15 in server_print () at networking/nslookup.c:129
        server = <optimized out>
        sa = <optimized out>
#20 nslookup_main (argc=<optimized out>, argv=0x7fff988337a0)
    at networking/nslookup.c:189
No locals.
#21 0x00000000004f890e in run_applet_no_and_exit (applet_no=<optimized out>,
    argv=argv@entry=0x7fff988337a0) at libbb/appletlib.c:759
        argc = <...

[Simon Rettberg (simon-rettberg)]

Sure, here we go. It might be worth to note that this system was installed as 12.04 or 12.10 and dist-upgraded ever since, so maybe this would explain some oddities, although in general everything works just fine.

libc6:
  Installed: 2.24-9ubuntu2.2
  Candidate: 2.24-9ubuntu2.2
  Version table:
 *** 2.24-9ubuntu2.2 500
        500 http://ftp.fau.de/ubuntu zesty-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu zesty-security/main amd64 Packages
        100 /var/lib/dpkg/status
     2.24-9ubuntu2 500
        500 http://ftp.fau.de/ubuntu zesty/main amd64 Packages

With trial and error I could identify that libnss-resolve seems to be the problem. Removing it from the hosts line in nsswitch fixes the issue. Probably not the best fix but at least it can fall back to the resolver stub via the localhost resolv.conf entry for some caching. Unfortunately I can't upgrade to 17.10 yet until end of January to do additional testing.

[Steve Langasek (vorlon)]

Thanks, adding a systemd task to this bug for libnss-resolve.

It may be that this bug is only reproducible in 17.04 because that was the only release in which libnss-resolve was installed by default. (We dropped this package from ubuntu-standard in 17.10.)

This also may not be fixable on the systemd side, since the failure happens with a static binary calling into an NSS module that calls into pthreads.

And it may not be fixable in busybox without substantial rework of the 'nslookup' applet, which appears to only front onto the glibc resolver apis instead of actually talking to the specified nameserver... which is what nslookup is defined to do.

[Steve Langasek (vorlon)]

btw, the bug description says 'several network applets'. Which besides nslookup do you see this with?

[Simon Rettberg (simon-rettberg)]

Pretty much every single one that would do a hostname lookup it seems. Tried ping, ping6, nc, wget; they all crash.

[Dan Streetman (ddstreet)]

please reopen if this is still an issue