Assuming the debug symbols I grabbed[1] for my install of bind9 on Xenial match yours (I have bind9 version 1:9.10.3.dfsg.P4-8ubuntu1.7 installed per "apt-cache policy bind9"), I did the following to grab a traceback:
$ sudo apt-get install bind9-dbgsym libdns162-dbgsym libisc160-dbgsym
$ gdb /usr/sbin/named core
(gdb) set pagination off
(gdb) thread apply all bt
... [2] ...
Looking at the backtrace in [2], the interesting parts to me are threads 8, 11 and 20, which are possibly involved in a deadlock[3]. Looks like one of the threads is reloading the configuration (something we would expect MAAS to do), and the other is calling dns_resolver_shutdown() via view_flushanddetach().
[1]: https://wiki.ubuntu.com/Debug%20Symbol%20Packages
[2]: http://paste.ubuntu.com/25292729/
[3]:
Thread 8 (Thread 0x7f95226aa700 (LWP 3203)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f952a351efe in __GI___pthread_mutex_lock (mutex=mutex@entry=0x7f94d4014fe8) at ../nptl/pthread_mutex_lock.c:135
#2 0x00007f952b7a0794 in dns_view_weakdetach (viewp=viewp@entry=0x7f9504389780) at ../../../lib/dns/view.c:597
#3 0x00007f952b7993de in destroy (val=0x7f9504389750) at ../../../lib/dns/validator.c:3891
#4 0x00007f952b79927b in dns_validator_destroy (validatorp=validatorp@entry=0x7f9519462628) at ../../../lib/dns/validator.c:3915
#5 0x00007f952b76b9d1 in validated (task=<optimized out>, event=0x7f95194625d0) at ../../../lib/dns/resolver.c:4722
#6 0x00007f952a9a6360 in dispatch (manager=0x7f952be3b010) at ../../../lib/isc/task.c:1130
#7 run (uap=0x7f952be3b010) at ../../../lib/isc/task.c:1302
#8 0x00007f952a34f6ba in start_thread (arg=0x7f95226aa700) at pthread_create.c:333
#9 0x00007f9529a993dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 11 (Thread 0x7f9520ea7700 (LWP 3206)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f952a9a516b in isc__task_beginexclusive (task0=<optimized out>) at ../../../lib/isc/task.c:1717
#2 0x0000557c34997dc1 in load_configuration (filename=<optimized out>, server=server@entry=0x7f952be44010, first_time=first_time@entry=isc_boolean_false) at ../../../bin/named/server.c:5651
#3 0x0000557c3499a826 in loadconfig (server=0x7f952be44010) at ../../../bin/named/server.c:7162
#4 0x0000557c3499ad48 in reload (server=0x7f952be44010) at ../../../bin/named/server.c:7183
#5 ns_server_reloadcommand (server=0x7f952be44010, args=args@entry=0x7f94fc120af0 "reload", text=text@entry=0x7f9520ea6590) at ../../../bin/named/server.c:7416
#6 0x0000557c34975db5 in ns_control_docommand (message=<optimized out>, text=text@entry=0x7f9520ea6590) at ../../../bin/named/control.c:102
#7 0x0000557c34978b97 in control_recvmessage (task=0x7f952be51010, event=<optimized out>) at ../../../bin/named/controlconf.c:458
#8 0x00007f952a9a6360 in dispatch (manager=0x7f952be3b010) at ../../../lib/isc/task.c:1130
#9 run (uap=0x7f952be3b010) at ../../../lib/isc/task.c:1302
#10 0x00007f952a34f6ba in start_thread (arg=0x7f9520ea7700) at pthread_create.c:333
#11 0x00007f9529a993dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 20 (Thread 0x7f951c69e700 (LWP 3215)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f952a351efe in __GI___pthread_mutex_lock (mutex=0x7f94d43429f8) at ../nptl/pthread_mutex_lock.c:135
#2 0x00007f952b7642a7 in dns_resolver_shutdown (res=0x7f952be56690) at ../../../lib/dns/resolver.c:9035
#3 0x00007f952b7a06a1 in view_flushanddetach (viewp=viewp@entry=0x7f94fc02cc30, flush=flush@entry=isc_boolean_false) at ../../../lib/dns/view.c:508
#4 0x00007f952b7a0757 in dns_view_detach (viewp=viewp@entry=0x7f94fc02cc30) at ../../../lib/dns/view.c:557
#5 0x0000557c3496e55b in ns_client_endrequest (client=0x7f94fc02cbe0) at ../../../bin/named/client.c:694
#6 exit_check (client=0x7f94fc02cbe0) at ../../../bin/named/client.c:382
#7 0x0000557c34970150 in ns_client_detach (clientp=clientp@entry=0x7f951c69ce18) at ../../../bin/named/client.c:2833
#8 0x0000557c34989ee2 in query_find (client=0x0, event=0x0, event@entry=0x7f9518a862b0, qtype=<optimized out>, qtype@entry=0) at ../../../bin/named/query.c:8328
#9 0x0000557c349921cf in query_resume (task=<optimized out>, event=0x7f9518a862b0) at ../../../bin/named/query.c:3826
#10 0x00007f952a9a6360 in dispatch (manager=0x7f952be3b010) at ../../../lib/isc/task.c:1130
#11 run (uap=0x7f952be3b010) at ../../../lib/isc/task.c:1302
#12 0x00007f952a34f6ba in start_thread (arg=0x7f951c69e700) at pthread_create.c:333
#13 0x00007f9529a993dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Assuming the debug symbols I grabbed[1] for my install of bind9 on Xenial match yours (I have bind9 version 1:9.10. 3.dfsg. P4-8ubuntu1. 7 installed per "apt-cache policy bind9"), I did the following to grab a traceback:
$ sudo apt-get install bind9-dbgsym libdns162-dbgsym libisc160-dbgsym
$ gdb /usr/sbin/named core
(gdb) set pagination off
(gdb) thread apply all bt
... [2] ...
Looking at the backtrace in [2], the interesting parts to me are threads 8, 11 and 20, which are possibly involved in a deadlock[3]. Looks like one of the threads is reloading the configuration (something we would expect MAAS to do), and the other is calling dns_resolver_ shutdown( ) via view_flushandde tach().
[1]: https:/ /wiki.ubuntu. com/Debug% 20Symbol% 20Packages paste.ubuntu. com/25292729/ unix/sysv/ linux/x86_ 64/lowlevellock .S:135 pthread_ mutex_lock (mutex= mutex@entry= 0x7f94d4014fe8) at ../nptl/ pthread_ mutex_lock. c:135 viewp@entry= 0x7f9504389780) at ../../. ./lib/dns/ view.c: 597 9750) at ../../. ./lib/dns/ validator. c:3891 destroy (validatorp= validatorp@ entry=0x7f95194 62628) at ../../. ./lib/dns/ validator. c:3915 625d0) at ../../. ./lib/dns/ resolver. c:4722 0x7f952be3b010) at ../../. ./lib/isc/ task.c: 1130 b010) at ../../. ./lib/isc/ task.c: 1302 a700) at pthread_ create. c:333 unix/sysv/ linux/x86_ 64/clone. S:109
[2]: http://
[3]:
Thread 8 (Thread 0x7f95226aa700 (LWP 3203)):
#0 __lll_lock_wait () at ../sysdeps/
#1 0x00007f952a351efe in __GI___
#2 0x00007f952b7a0794 in dns_view_weakdetach (viewp=
#3 0x00007f952b7993de in destroy (val=0x7f950438
#4 0x00007f952b79927b in dns_validator_
#5 0x00007f952b76b9d1 in validated (task=<optimized out>, event=0x7f95194
#6 0x00007f952a9a6360 in dispatch (manager=
#7 run (uap=0x7f952be3
#8 0x00007f952a34f6ba in start_thread (arg=0x7f95226a
#9 0x00007f9529a993dd in clone () at ../sysdeps/
Thread 11 (Thread 0x7f9520ea7700 (LWP 3206)): cond_wait@ @GLIBC_ 2.3.2 () at ../sysdeps/ unix/sysv/ linux/x86_ 64/pthread_ cond_wait. S:185 beginexclusive (task0=<optimized out>) at ../../. ./lib/isc/ task.c: 1717 <optimized out>, server= server@ entry=0x7f952be 44010, first_time= first_time@ entry=isc_ boolean_ false) at ../../. ./bin/named/ server. c:5651 0x7f952be44010) at ../../. ./bin/named/ server. c:7162 0x7f952be44010) at ../../. ./bin/named/ server. c:7183 reloadcommand (server= 0x7f952be44010, args=args@ entry=0x7f94fc1 20af0 "reload", text=text@ entry=0x7f9520e a6590) at ../../. ./bin/named/ server. c:7416 docommand (message=<optimized out>, text=text@ entry=0x7f9520e a6590) at ../../. ./bin/named/ control. c:102 51010, event=<optimized out>) at ../../. ./bin/named/ controlconf. c:458 0x7f952be3b010) at ../../. ./lib/isc/ task.c: 1130 b010) at ../../. ./lib/isc/ task.c: 1302 7700) at pthread_ create. c:333 unix/sysv/ linux/x86_ 64/clone. S:109
#0 pthread_
#1 0x00007f952a9a516b in isc__task_
#2 0x0000557c34997dc1 in load_configuration (filename=
#3 0x0000557c3499a826 in loadconfig (server=
#4 0x0000557c3499ad48 in reload (server=
#5 ns_server_
#6 0x0000557c34975db5 in ns_control_
#7 0x0000557c34978b97 in control_recvmessage (task=0x7f952be
#8 0x00007f952a9a6360 in dispatch (manager=
#9 run (uap=0x7f952be3
#10 0x00007f952a34f6ba in start_thread (arg=0x7f9520ea
#11 0x00007f9529a993dd in clone () at ../sysdeps/
Thread 20 (Thread 0x7f951c69e700 (LWP 3215)): unix/sysv/ linux/x86_ 64/lowlevellock .S:135 pthread_ mutex_lock (mutex= 0x7f94d43429f8) at ../nptl/ pthread_ mutex_lock. c:135 shutdown (res=0x7f952be5 6690) at ../../. ./lib/dns/ resolver. c:9035 viewp@entry= 0x7f94fc02cc30, flush=flush@ entry=isc_ boolean_ false) at ../../. ./lib/dns/ view.c: 508 viewp@entry= 0x7f94fc02cc30) at ../../. ./lib/dns/ view.c: 557 endrequest (client= 0x7f94fc02cbe0) at ../../. ./bin/named/ client. c:694 0x7f94fc02cbe0) at ../../. ./bin/named/ client. c:382 clientp@ entry=0x7f951c6 9ce18) at ../../. ./bin/named/ client. c:2833 0x7f9518a862b0, qtype=<optimized out>, qtype@entry=0) at ../../. ./bin/named/ query.c: 8328 862b0) at ../../. ./bin/named/ query.c: 3826 0x7f952be3b010) at ../../. ./lib/isc/ task.c: 1130 b010) at ../../. ./lib/isc/ task.c: 1302 e700) at pthread_ create. c:333 unix/sysv/ linux/x86_ 64/clone. S:109
#0 __lll_lock_wait () at ../sysdeps/
#1 0x00007f952a351efe in __GI___
#2 0x00007f952b7642a7 in dns_resolver_
#3 0x00007f952b7a06a1 in view_flushanddetach (viewp=
#4 0x00007f952b7a0757 in dns_view_detach (viewp=
#5 0x0000557c3496e55b in ns_client_
#6 exit_check (client=
#7 0x0000557c34970150 in ns_client_detach (clientp=
#8 0x0000557c34989ee2 in query_find (client=0x0, event=0x0, event@entry=
#9 0x0000557c349921cf in query_resume (task=<optimized out>, event=0x7f9518a
#10 0x00007f952a9a6360 in dispatch (manager=
#11 run (uap=0x7f952be3
#12 0x00007f952a34f6ba in start_thread (arg=0x7f951c69
#13 0x00007f9529a993dd in clone () at ../sysdeps/