ntpd crashes when network interface goes down

[Janne Snabb]

ntpd crashes with SIGSEGV when (teredo IPv6) interface goes down after unplugging a network cable:

Oct 22 03:17:03 oil ntpdate[5896]: adjust time server 118.143.17.82 offset 0.005534 sec
Oct 22 03:17:03 oil ntpd[5951]: ntpd 4.2.6p3@1.2290-o Mon Aug 20 15:15:21 UTC 2012 (1)
Oct 22 03:17:03 oil ntpd[5952]: proto: precision = 0.106 usec
Oct 22 03:17:03 oil ntpd[5952]: ntp_io: estimated max descriptors: 2144, initial socket boundary: 16
Oct 22 03:17:03 oil ntpd[5952]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Oct 22 03:17:03 oil ntpd[5952]: Listen and drop on 1 v6wildcard :: UDP 123
Oct 22 03:17:03 oil ntpd[5952]: Listen normally on 2 lo 127.0.0.1 UDP 123
Oct 22 03:17:03 oil ntpd[5952]: Listen normally on 3 eth0 192.168.0.2 UDP 123
Oct 22 03:17:03 oil ntpd[5952]: Listen normally on 4 teredo fe80::ffff:ffff:ffff UDP 123
Oct 22 03:17:03 oil ntpd[5952]: Listen normally on 5 eth0 fe80::16fe:b5ff:fea4:b39f UDP 123
Oct 22 03:17:03 oil ntpd[5952]: Listen normally on 6 teredo 2001:0:53aa:64c:875:7e15:90bc:9593 UDP 123
Oct 22 03:17:03 oil ntpd[5952]: Listen normally on 7 lo ::1 UDP 123
Oct 22 03:17:03 oil ntpd[5952]: peers refreshed
Oct 22 03:17:03 oil ntpd[5952]: Listening on routing socket on fd #24 for interface updates
Oct 22 03:24:45 oil ntpd[5952]: Deleting interface #5 eth0, fe80::16fe:b5ff:fea4:b39f#123, interface stats: received=0, sent=0, dropped=0, active_time=460 secs
Oct 22 03:24:45 oil ntpd[5952]: Deleting interface #3 eth0, 192.168.0.2#123, interface stats: received=40, sent=41, dropped=0, active_time=460 secs
Oct 22 03:24:45 oil ntpd[5952]: 220.130.158.71 interface 192.168.0.2 -> (none)
Oct 22 03:24:45 oil ntpd[5952]: 118.143.17.82 interface 192.168.0.2 -> (none)
Oct 22 03:24:45 oil ntpd[5952]: 14.0.18.136 interface 192.168.0.2 -> (none)
Oct 22 03:24:45 oil ntpd[5952]: 54.251.61.122 interface 192.168.0.2 -> (none)
Oct 22 03:24:45 oil ntpd[5952]: 203.176.128.10 interface 192.168.0.2 -> (none)
Oct 22 03:24:45 oil ntpd[5952]: peers refreshed
Oct 22 03:25:02 oil ntpd[5952]: Deleting interface #6 teredo, 2001:0:53aa:64c:875:7e15:90bc:9593#123, interface stats: received=0, sent=0, dropped=0, active_time=477 secs
Oct 22 03:25:02 oil kernel: [197600.300333] ntpd[5952]: segfault at 8 ip 00007fda05b58321 sp 00007fff380bd540 error 4 in ntpd[7fda05b3b000+96000]

Here is a gdb output:

(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x00007f743f09e321 in remove_interface (ep=0x7f74411771b0) at ntp_io.c:1098
1098 ntp_io.c: No such file or directory.
(gdb) bt
#0 0x00007f743f09e321 in remove_interface (ep=0x7f74411771b0) at ntp_io.c:1098
#1 update_interfaces (port=port@entry=123, receiver=receiver@entry=0x0,
    data=data@entry=0x0) at ntp_io.c:2010
#2 0x00007f743f09fb34 in interface_update (receiver=receiver@entry=0x0,
    data=data@entry=0x0) at ntp_io.c:1617
#3 0x00007f743f0c1215 in timer () at ntp_timer.c:394
#4 0x00007f743f0a489b in ntpdmain (argc=<optimized out>, argv=<optimized out>)
    at ntpd.c:1127
#5 0x00007f743f0954a9 in main (argc=<optimized out>, argv=<optimized out>)
    at ntpd.c:358
(gdb)

I can only reproduce this with the teredo interface. If I disable it (by running "service miredo stop" before testing) I can not re-produce the problem any more.

ProblemType: Bug
DistroRelease: Ubuntu 12.10
Package: ntp 1:4.2.6.p3+dfsg-1ubuntu5
ProcVersionSignature: Ubuntu 3.5.0-17.28-generic 3.5.5
Uname: Linux 3.5.0-17-generic x86_64
ApportVersion: 2.6.1-0ubuntu4
Architecture: amd64
Date: Mon Oct 22 03:42:12 2012
InstallationMedia: Xubuntu 11.10 "Oneiric Ocelot" - Beta amd64 (20110920.2)
KernLog:

MarkForUpload: True
SourcePackage: ntp
UpgradeStatus: Upgraded to quantal on 2012-10-19 (2 days ago)
modified.conffile..etc.ntp.conf: [modified]
mtime.conffile..etc.ntp.conf: 2012-09-02T19:37:38.651830

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Didier Misson (dmlinux)]

Hello, and Happy 2013.

I have same problem with a Kimsufi server to OVH Canada.
IPv6 is configured on this Ubuntu 12.10 server.

When I start the ntpd task, ntpd make a segfault :

Jan 1 17:56:02 ks397789 ntpd[1821]: ntpd 4.2.6p3@1.2290-o Mon Aug 20 15:15:21 UTC 2012 (1)
Jan 1 17:56:02 ks397789 ntpd[1822]: proto: precision = 0.200 usec
Jan 1 17:56:02 ks397789 ntpd[1822]: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16
Jan 1 17:56:02 ks397789 ntpd[1822]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Jan 1 17:56:02 ks397789 ntpd[1822]: Listen and drop on 1 v6wildcard :: UDP 123
Jan 1 17:56:02 ks397789 ntpd[1822]: Listen normally on 2 lo 127.0.0.1 UDP 123
Jan 1 17:56:02 ks397789 ntpd[1822]: Listen normally on 3 eth0 192.95.25.135 UDP 123
Jan 1 17:56:02 ks397789 ntpd[1822]: Listen normally on 4 eth0 fe80::e269:95ff:fed8:6652 UDP 123
Jan 1 17:56:02 ks397789 ntpd[1822]: Listen normally on 5 eth0 2607:5300:60:2987::1 UDP 123
Jan 1 17:56:02 ks397789 ntpd[1822]: Listen normally on 6 lo ::1 UDP 123
Jan 1 17:56:02 ks397789 ntpd[1822]: peers refreshed
Jan 1 17:56:02 ks397789 ntpd[1822]: Listening on routing socket on fd #23 for interface updates
Jan 1 17:56:06 ks397789 ntpd[1822]: Deleting interface #6 lo, ::1#123, interface stats: received=0, sent=0, dropped=0, active_time=2 secs
Jan 1 17:56:06 ks397789 ntpd[1822]: Deleting interface #5 eth0, 2607:5300:60:2987::1#123, interface stats: received=0, sent=0, dropped=0, active_time=2 secs
Jan 1 17:56:06 ks397789 ntpd[1822]: 2605:1b00:0:1::1d interface 2607:5300:60:2987::1 -> (none)
Jan 1 17:56:06 ks397789 kernel: ntpd[1822]: segfault at 8 ip 00007f176a1b5321 sp 00007fffebca98d0 error 4 in ntpd[7f176a198000+96000]
Jan 1 17:56:06 ks397789 kernel: grsec: From 109.129.16.136: Segmentation fault occurred at 0000000000000008 in /usr/sbin/ntpd[ntpd:1822] uid/euid:110/110 gid/egid:118/118, parent /sbin/init[init:1] uid/euid:0/0 gid/egid:0/0

Seem to crash after deleting the IPv6 interface 2607:5300:60:2987::1

The IPv6 :
# ifconfig
eth0 Link encap:Ethernet HWaddr e0:69:95:d8:66:52
          inet addr:192.95.25.135 Bcast:192.95.25.255 Mask:255.255.255.0
          inet6 addr: fe80::e269:95ff:fed8:6652/64 Scope:Link
          inet6 addr: 2607:5300:60:2987::1/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:1140147 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1046438 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:162234138 (162.2 MB) TX bytes:208465599 (208.4 MB)
          Interrupt:44 Base address:0xa000

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:94416 errors:0 dropped:0 overruns:0 frame:0
          TX packets:94416 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:12404004 (12.4 MB) TX bytes:12404004 (12.4 MB)

Thanks.

[Vladimir Panteleev (thecybershadow)]

@dmlinux: I ran into the same problem today (also with OVH's kernel).

The problematic bit of code is at line 1098 in ntpd/ntp_io.c (for ntp-4.2.6.p3+dfsg) - wrapping the UNLINK_SLIST statement into an if(*pmclisthead) block seems to fix the problem. However, I also found that the latest ntp version (ntp-dev-4.2.7p345) doesn't seem to have this problem.

[Marti (intgr)]

An ntpd segfault occurred on 2 of our servers simultaneously, after we disabled IPv6 router advertisement (autoconf) on our network. Seems like the same bug.

[Pali (pali)]

100% reproducable everytime when interface (ipv6 using dhcpv6) goes down. Same backtrace in gdb. Tested on ubuntu precise.

[Paul Gear (paulgear)]

Seeing this or something very similar on xenial s390x:

Dec 6 00:25:48 s0lp3 ntpd[238439]: Deleting interface #163484 tap3c93ac38-8a, fe80::fc16:3eff:fe05:6903%805261#123, interface stats: received=0, sent=0, dropped=0, active_time=206 secs
Dec 6 00:25:53 s0lp3 ntpd[238439]: ./../lib/isc/unix/ifiter_getifaddrs.c:163: INSIST(ifa->ifa_name != ((void *)0)) failed
Dec 6 00:25:53 s0lp3 ntpd[238439]: exiting (due to assertion failure)

[Thomas Cuthbert (tcuthbert)]

Another s390x showing similar symptoms to Paul:

Feb 21 00:56:01 s0lp3 ntpd[253059]: Deleting interface #79102 qvoc1fa7c7e-9e, fe80::1cad:94ff:fe58:5dc7%1315880#123, interface stats: received=0, sent=0, dropped=0, active_time=676 secs
Feb 21 00:56:01 s0lp3 ntpd[253059]: Deleting interface #79103 qvbc1fa7c7e-9e, fe80::470:fbff:fe3b:b540%1315881#123, interface stats: received=0, sent=0, dropped=0, active_time=676 secs
Feb 21 00:56:01 s0lp3 ntpd[253059]: Deleting interface #79119 tapc1fa7c7e-9e, fe80::fc16:3eff:fe87:f3f4%1315899#123, interface stats: received=0, sent=0, dropped=0, active_time=455 secs
Feb 21 00:56:09 s0lp3 ntpd[253059]: Deleting interface #79163 tap59a670ab-70, fe80::fc16:3eff:fe27:b37a%1315951#123, interface stats: received=0, sent=0, dropped=0, active_time=24 secs
Feb 21 00:56:12 s0lp3 ntpd[253059]: ./../lib/isc/unix/ifiter_getifaddrs.c:163: INSIST(ifa->ifa_name != ((void *)0)) failed
Feb 21 00:56:12 s0lp3 ntpd[253059]: exiting (due to assertion failure)

solution is to just restart ntpd.