unbound crashes under high query load if num-threads > 1

[Janne Snabb]

$ gdb /usr/sbin/unbound core.28338
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /usr/sbin/unbound...Reading symbols from /usr/lib/debug/usr/sbin/unbound...done.
done.
[New LWP 28345]
[New LWP 28338]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `unbound'.
Program terminated with signal 11, Segmentation fault.
#0 RSA_new_method (engine=<optimized out>) at rsa_lib.c:191
191 rsa_lib.c: No such file or directory.
(gdb) bt
#0 RSA_new_method (engine=<optimized out>) at rsa_lib.c:191
#1 0x00007f7d1a04a0f4 in ldns_key_buf2rsa_raw () from /usr/lib/libldns.so.1
#2 0x0000000000469aea in setup_key_digest (keylen=<optimized out>,
    key=<optimized out>, digest_type=<synthetic pointer>,
    evp_key=<synthetic pointer>, algo=8) at validator/val_sigcrypt.c:1413
#3 verify_canonrrset (reason=0x7f7d177b16d8, keylen=<optimized out>,
    key=<optimized out>, sigblock_len=<optimized out>,
    sigblock=0x7f7d106a6b95 "t$\313\314\032J\002\235y͙'\273\221jj\353\367\222\3550\030e\350RJ\016\355Kb\221\207\071\200\312\344\"\236\232\325G\260\262\241\265\026EC\341\371\302A\376\370==\330\006ݦ\022\062j\033T\255\255\353\231Q\310\v\342\350١AK=e\322\067\335i\373\273\063:7\264\244\035Y\230\226}\030\260\360JX\237\242[\331\327]u\006\351_\003\t\346|E\362l?\364\317+oZ\204P\245`\266", algo=8,
    buf=0x7f7d1056fac0) at validator/val_sigcrypt.c:1506
#4 dnskey_verify_rrset_sig (region=<optimized out>, buf=0x7f7d1056fac0,
    ve=0x18f0310, now=1350983573, rrset=0x7f7d106a67b8,
    dnskey=<optimized out>, dnskey_idx=0, sig_idx=0, sortree=0x7f7d177b14d0,
    buf_canon=0x7f7d177b14dc, reason=0x7f7d177b16d8)
    at validator/val_sigcrypt.c:1683
#5 0x000000000043039c in dnskeyset_verify_rrset_sig (reason=0x7f7d177b16d8,
    sortree=0x7f7d177b14d0, sig_idx=0, dnskey=0x7f7d177b1520,
    rrset=0x7f7d106a67b8, now=1350983573, ve=0x18f0310, env=0x1920298)
    at validator/val_sigcrypt.c:680
#6 dnskeyset_verify_rrset (env=0x1920298, ve=0x18f0310, rrset=0x7f7d106a67b8,
    dnskey=0x7f7d177b1520, sigalg=0x7f7d109102f0 "\b", reason=0x7f7d177b16d8)
    at validator/val_sigcrypt.c:581
#7 0x0000000000432ea4 in val_verify_rrset (reason=0x7f7d177b16d8,
    sigalg=0x7f7d109102f0 "\b", keys=0x7f7d177b1520, rrset=0x7f7d106a67b8,
    ve=0x18f0310, env=0x1920298) at validator/val_utils.c:335
#8 val_verify_rrset_entry (env=0x1920298, ve=0x18f0310, rrset=0x7f7d106a67b8,
    kkey=<optimized out>, reason=0x7f7d177b16d8) at validator/val_utils.c:381
#9 0x000000000042e9f8 in list_is_secure (reason=<optimized out>,
    kkey=<optimized out>, num=<optimized out>, list=<optimized out>,
    ve=<optimized out>, env=<optimized out>) at validator/val_nsec3.c:1313
#10 nsec3_prove_nods (reason=0x7f7d177b16d8, kkey=0x7f7d1090ff20,
    qinfo=0x7f7d106a4190, num=3, list=0x7f7d106a6690, ve=0x18f0310,
    env=0x1920298) at validator/val_nsec3.c:1340
#11 ds_response_to_ke.isra.1 (ke=<synthetic pointer>, qinfo=0x7f7d106a4190,
    rcode=<optimized out>, id=<optimized out>, qstate=<optimized out>)
    at validator/validator.c:2435
#12 process_ds_response.29359 (qstate=<optimized out>, vq=0x7f7d1090fdf8,
    id=<optimized out>, rcode=<optimized out>, msg=<optimized out>,
    qinfo=0x7f7d106a4190, origin=0x7f7d106a6fb8) at validator/validator.c:2552
#13 0x0000000000444d61 in val_inform_super (qstate=0x7f7d106a4190, id=0,
    super=0x7f7d1160e430) at validator/validator.c:2892
#14 0x000000000046191b in mesh_walk_supers (mesh=0x7f7d1055f280,
    mstate=<optimized out>) at services/mesh.c:916
#15 0x0000000000431724 in mesh_continue (ev=<synthetic pointer>,
    s=<optimized out>, mstate=0x7f7d106a4140, mesh=<optimized out>)
    at services/mesh.c:1033
#16 mesh_run (mesh=0x7f7d1055f280, mstate=0x7f7d106a4140, ev=<optimized out>,
    e=0x0) at services/mesh.c:1064
#17 0x0000000000431ed6 in worker_handle_service_reply (c=0x7f7d1033d520,
    arg=0x7f7d106a6600, error=0, reply_info=0x7f7d177b1ca0)
    at daemon/worker.c:283
#18 0x000000000041f34b in serviced_callbacks.39588 (sq=0x7f7d10a8f210,
    error=0, c=0x7f7d1033d520, rep=0x7f7d177b1ca0)
    at services/outside_network.c:1505
#19 0x0000000000428d71 in serviced_udp_callback (c=0x7f7d1033d520,
    arg=0x7f7d10a8f210, error=0, rep=0x7f7d177b1ca0)
    at services/outside_network.c:1776
#20 0x000000000041bbd8 in outnet_udp_cb.part.21 (reply_info=0x7f7d177b1ca0,
    arg=0x7f7d100b2aa0, c=0x7f7d1033d520) at services/outside_network.c:457
#21 outnet_udp_cb (c=0x7f7d1033d520, arg=0x7f7d100b2aa0,
    error=<optimized out>, reply_info=0x7f7d177b1ca0)
    at services/outside_network.c:386
#22 0x0000000000418144 in comm_point_udp_callback (fd=506,
    event=<optimized out>, arg=<optimized out>) at util/netevent.c:626
#23 0x00007f7d19ded94c in event_base_loop () from /usr/lib/libevent-2.0.so.5
#24 0x000000000044db5c in comm_base_dispatch (b=<optimized out>)
    at util/netevent.c:252
#25 0x000000000046944e in worker_work (worker=0x191eeb0)
    at daemon/worker.c:1196
#26 thread_start (arg=0x191eeb0) at daemon/daemon.c:386
#27 0x00007f7d19303e9a in start_thread ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#28 0x00007f7d19030cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#29 0x0000000000000000 in ?? ()
(gdb)

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: unbound 1.4.16-1
ProcVersionSignature: Ubuntu 3.2.0-24.37-generic 3.2.14
Uname: Linux 3.2.0-24-generic x86_64
ApportVersion: 2.0.1-0ubuntu14
Architecture: amd64
Date: Tue Oct 23 09:16:53 2012
InstallationMedia: Ubuntu-Server 11.04 "Natty Narwhal" - Release amd64 (20110426)
ProcEnviron:
 TERM=screen
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: unbound
UpgradeStatus: Upgraded to precise on 2012-04-27 (178 days ago)
mtime.conffile..etc.unbound.unbound.conf: 2011-10-15T09:56:44.494904

[Janne Snabb (snabb)]

Some additional details:

- This bug is triggered only if "num-threads" setting in unbound.conf is higher than 1. Therefore a simple workaround is to reduce the amount of threads to 1 (which is the default).

- A description of the problem and a patch is available at https://unbound.net/pipermail/unbound-users/2012-September/002571.html (see the attachment at the end of that message). The patch does not apply cleanly against Unbound 1.4.16 but it is simple enough to fix it.

- The fix is committed in Unbound SVN repository at r2733. It will be included in the next Unbound version.

I am attaching a patch which applies cleanly against 1.4.16 to this bug.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "unbound-1.4.16-openssl-threads.patch" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

[Scott Kitterman (kitterman)]

This seems like a reasonable patch to include. If you could edit the bug to include the information required in https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template then I can get the fix into precise/quantal-proposed for testing.

[Scott Kitterman (kitterman)]

Fix is already included in raring. Need test case, etc for potential SRU.

[Rolf Leggewie (r0lf)]

quantal has seen the end of its life and is no longer receiving any updates. Marking the quantal task for this ticket as "Won't Fix".

[Steve Langasek (vorlon)]

The Precise Pangolin has reached end of life, so this bug will not be fixed for that release