Segmentation fault in ntpd when system has more than 1134 interface addresses
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| NTP |
Fix Released
|
High
|
|||
| ntp (Ubuntu) |
Fix Released
|
Medium
|
Unassigned | ||
Bug Description
Binary package hint: ntp
$ uname -a
Linux xxxxx 2.6.24-17-generic #1 SMP Thu May 1 13:57:17 UTC 2008 x86_64 GNU/Linux
$ lsb_release -rd
Description: Ubuntu 8.04
Release: 8.04
$ apt-cache policy ntp
ntp:
Installed: 1:4.2.4p4+
Candidate: 1:4.2.4p4+
Version table:
*** 1:4.2.4p4+
500 http://
100 /var/lib/
On a system which has any more than 1134 interface addresses (counting both IPv4 and IPv6 address thus: 'ip addr ls | grep inet | wc -l') ntpd fails with a segmentation fault.
I run a large network for a university (20,000 people), and we routinely operate open-source routers with several thousand interface addresses (every client machine lives in it's own /30 subnet).
Whilst examining the problem I used this script to add interface addresses:
$ cat breakit.sh
#!/bin/sh
set -e
dev="vlan252"
sudo ifdown ${dev} # clears all old addr
naddr=`ip add ls | grep inet | wc -l` # number of addr already on system
echo "Pre-existing addresses on system: ${naddr}"
: ${nbreakit:=1217} # number of addr required to break ntp
sudo ifup ${dev}
total=${naddr}
for I in `seq 1 5` ; do
for J in `seq 0 255` ; do
sudo ip addr add 10.0.${I}.${J}/32 dev ${dev}
total=
if [ ${total} -ge ${nbreakit} ]; then break ; fi
done
if [ ${total} -ge ${nbreakit} ]; then break ; fi
done
echo "Added extra addresses: $((${total}
echo "Total addresses on system now: $(ip add ls | grep inet | wc -l)"
exit 0
$ grep -B1 -A2 vlan252 /etc/network/
#auto vlan252
iface vlan252 inet manual
vlan-raw-device eth0
I could then add the desired number of interfaces by doing:
$ nbreakit=1135 ../breakit.sh
and then run ntpd [1][2] under gdb by doing:
sudo sh -c "ulimit -n 8192 ; gdb --args /usr/sbin/ntpd -n -d -D3 -p /var/run/ntpd.pid -u 115:126 -g"
[1] You can't use '-d -D3' on the standard ubuntu package - debugging is disabled.
[2] It is necessary to raise the number of open files ulimit of 1024 in order to run ntpd with this many addresses.
In order to help me understand what was going wrong with the standard package, I rebuilt it with debugging enabled and symbols not stripped:
$ mkdir work
$ cd work/
$ apt-get source ntp
Reading package lists... Done
Building dependency tree
Reading state information... Done
NOTICE: 'ntp' packaging is maintained in the 'Svn' version control system at:
svn://svn.
Need to get 3120kB of source archives.
Get: 1 http://
Get: 2 http://
Get: 3 http://
Fetched 3120kB in 0s (7433kB/s)
dpkg-source: extracting ntp in ntp-4.2.4p4+dfsg
dpkg-source: unpacking ntp_4.2.
dpkg-source: applying ./ntp_4.
$ cd ntp-4.2.4p4+dfsg/
$ cp -p debian/
$ vi debian/rules
$ diff -u debian/
--- debian/rules.orig 2008-05-28 17:52:21.000000000 +0100
+++ debian/rules 2008-05-28 17:53:21.000000000 +0100
@@ -21,7 +21,7 @@
./configure CFLAGS='$(CFLAGS)' \
- --disable-debugging --sysconfdir=
+ --enable-debugging --sysconfdir=
@@ -104,7 +104,7 @@
dh_perl -a
- dh_strip -a
+ #dh_strip -a
dh_compress -a
dh_fixperms -a
$ dpkg-buildpackage -us -uc
...
dpkg-buildpackage: binary and diff upload (original source NOT included)
$ sudo dpkg -i ../ntp_
Once one has 1135 interface addresses on the system you get this segmentation fault:
0x000000000040c9d2 in update_interfaces (port=<value optimized out>,
receiver=0, data=0x0) at ntp_io.c:769
769 ISC_LIST_
If one increases the number of interface addresses to 1215 you get a different segmentation fault:
update_interfaces (port=<value optimized out>, receiver=0, data=0x0)
at ntp_io.c:1325
1325 if (!(interf->flags & (INT_WILDCARD|
and if one increases it to 1216 or more you finally get this one:
0x000000000040ba7f in add_interface (interface=
756 ISC_LIST_
I had hoped to work around the problem by renaming the devices to 'vlan252:foo', since in ntpd/ntp_
segmentation fault occurs during interface enumeration (building a linked list of all interface+address) this doesn't help.
In the past, earlier versions of ntpd did not use a linked list for this purpose, but rather a fixed-size array of 512. By simply increasing the size of the array I was able to run with large numbers of addresses. It seems to me that, whilst the linked-list code ought to be made to work correctly, it is in fact an unnecessary precaution (and overhead) to bind all addresses on a linux system - only the root user could bind a more specific address than * on port 123, and if one has root the game is over anyway.
I haven't submitted this bug to the upstream package maintainer (http://
I include a full transcript of my tests attached as ntp_bugreport_
| Simon Boggis (s-a-boggis) wrote | #1 |
|
|
#7 |
| Dave Hart (hart-ntp) wrote | #2 |
| Simon Boggis (s-a-boggis) wrote Re: [Bug 235793] Re: Segmentation fault in ntpd when system has more than 1134 interface addresses | #3 |
| Chuck Short (zulcss) wrote | #4 |
| Changed in ntp (Ubuntu): | |
| importance: | Undecided → Medium |
| status: | New → Triaged |
| Changed in ntp: | |
| status: | Unknown → Fix Released |
| Chuck Short (zulcss) wrote | #5 |
| Simon Boggis (s-a-boggis) wrote | #6 |
| Changed in ntp: | |
| importance: | Unknown → High |
| Christian Ehrhardt (paelzer) wrote | #20 |
| Changed in ntp (Ubuntu): | |
| status: | Triaged → Fix Released |
Hi:
We're running into the same problem on a redhat system that was previously
reported against ubuntu.
I don't think I could write a better bug report, so I'll just include/reference
it here:
http://<email address hidden>
$ uname -a
Linux xxxxx 2.6.24-17-generic #1 SMP Thu May 1 13:57:17 UTC 2008 x86_64
GNU/Linux
$ lsb_release -rd
Description: Ubuntu 8.04
Release: 8.04
$ apt-cache policy ntp
ntp:
Installed: 1:4.2.4p4+
Candidate: 1:4.2.4p4+
Version table:
*** 1:4.2.4p4+
500 http://
100 /var/lib/
On a system which has any more than 1134 interface addresses (counting
both IPv4 and IPv6 address thus: 'ip addr ls | grep inet | wc -l') ntpd
fails with a segmentation fault.
I run a large network for a university (20,000 people), and we routinely
operate open-source routers with several thousand interface addresses
(every client machine lives in it's own /30 subnet).
Whilst examining the problem I used this script to add interface addresses:
$ cat breakit.sh
#!/bin/sh
set -e
dev="vlan252"
sudo ifdown ${dev} # clears all old addr
naddr=`ip add ls | grep inet | wc -l` # number of addr already on system
echo "Pre-existing addresses on system: ${naddr}"
: ${nbreakit:=1217} # number of addr required to break ntp
sudo ifup ${dev}
total=${naddr}
for I in `seq 1 5` ; do
for J in `seq 0 255` ; do
sudo ip addr add 10.0.${I}.${J}/32 dev ${dev}
total=
if [ ${total} -ge ${nbreakit} ]; then break ; fi
done
if [ ${total} -ge ${nbreakit} ]; then break ; fi
done
echo "Added extra addresses: $((${total}
echo "Total addresses on system now: $(ip add ls | grep inet | wc -l)"
exit 0
$ grep -B1 -A2 vlan252 /etc/network/
#auto vlan252
iface vlan252 inet manual
vlan-raw-device eth0
I could then add the desired number of interfaces by doing:
$ nbreakit=1135 ../breakit.sh
and then run ntpd [1][2] under gdb by doing:
sudo sh -c "ulimit -n 8192 ; gdb --args /usr/sbin/ntpd -n -d -D3 -p
/var/run/ntpd.pid -u 115:126 -g"
[1] You can't use '-d -D3' on the standard ubuntu package - debugging is
disabled.
[2] It is necessary to raise the number of open files ulimit of 1024 in order
to run ntpd with this many addresses.
In order to help me understand what was going wrong with the standard
package, I rebuilt it with debugging enabled and symbols not stripped:
$ mkdir work
$ cd work/
$ apt-get source ntp
Reading package lists... Done
Building dependency tree
Reading state information... Done
NOTICE: 'ntp' packaging is maintained in the 'Svn' version control system at:
svn://svn.
Need to get 3120kB of source archives.
Get: 1 http://
(dsc) [1034B]
Get: 2 http://
(tar) [2835kB]
Get: 3 http://
(diff) [284kB]
Fetched 3120kB in 0s (7433kB/s)
dpkg-source: extracting ntp in ntp-4.2.4p4+dfsg
dpkg-source: unpacking...