ICMP traceroute does not work properly in Hardy

[Radek Zajic]

Binary package hint: traceroute

Hello Ubuntuers,

I am facing a very strange problem in the traceroute package present in the latest Ubuntu release (Hardy).

Problem description:
Because of lots of systems blocking udp packets, I prefer to use ICMP traceroute (the "traceroute -I" command). However I am unable to use this command in Hardy, because instead of printing the hops properly, it prints the first hop, then prints four lines with stars, fifth line with one star and then continues properly. This happens only with the ICMP tracing and DNS name resolution turned on.

Let me give you an example:
ICMP traceroute, DNS resolution turned on:
rzajic@rzajic-laptop:~$ sudo /usr/bin/traceroute -I 10.128.9.101
traceroute to 10.128.9.101 (10.128.9.101), 30 hops max, 40 byte packets
 1 10.128.6.1 (10.128.6.1) 20.654 ms 35.966 ms *
 2 * * *
 3 * * *
 4 * * *
 5 * * *
 6 * 10.128.9.101 (10.128.9.101) 30.744 ms 81.052 ms
ICMP traceroute, DNS resolution turned off ("-n" parameter):
rzajic@rzajic-laptop:~$ sudo /usr/bin/traceroute -n -I 10.128.9.101
traceroute to 10.128.9.101 (10.128.9.101), 30 hops max, 40 byte packets
 1 10.128.6.1 16.265 ms 35.758 ms 35.788 ms
 2 10.160.0.2 35.835 ms 35.845 ms 35.852 ms
 3 10.160.8.5 35.785 ms 35.794 ms 35.802 ms
 4 10.128.12.11 35.845 ms 35.853 ms 36.047 ms
 5 10.128.9.101 36.059 ms 57.850 ms 57.872 ms
UDP traceroute, name resolution turned on:
rzajic@rzajic-laptop:~$ sudo /usr/bin/traceroute 10.128.9.101
traceroute to 10.128.9.101 (10.128.9.101), 30 hops max, 40 byte packets
 1 10.128.6.1 (10.128.6.1) 16.038 ms 32.945 ms 32.958 ms
 2 10.160.0.2 (10.160.0.2) 32.999 ms 33.002 ms 33.018 ms
 3 10.160.8.5 (10.160.8.5) 32.929 ms 32.932 ms 32.935 ms
 4 klient10.ap-pk.zebetin.brno.czf (10.128.12.11) 32.965 ms 32.982 ms 81.841 ms
 5 10.128.9.101 (10.128.9.101) 81.855 ms 81.860 ms 81.864 ms

To compare, here is also a Windows ICMP traceroute, which works perfectly:
C:\Users\Radek>tracert -d 10.128.9.101

Tracing route to 10.128.9.101 over a maximum of 30 hops

  1 20 ms 17 ms 18 ms 10.128.6.1
  2 20 ms 28 ms 18 ms 10.160.0.2
  3 29 ms 22 ms 42 ms 10.160.8.5
  4 19 ms 26 ms 18 ms 10.128.12.11
  5 40 ms 30 ms 38 ms 10.128.9.101

Trace complete.

And also a Mandriva traceroute (from the other side, the 10.160.8.4 router, which is connected using wired ethernet to 10.160.8.5)

[root@bastila ~]# traceroute -V
Version 1.4a12

[root@bastila ~]# traceroute 10.128.6.136
traceroute to 10.128.6.136 (10.128.6.136), 30 hops max, 38 byte packets
 1 cisco-vlan-008 (10.160.8.1) 2.617 ms 1.692 ms 2.498 ms
 2 openvpn (10.160.0.10) 0.981 ms 0.582 ms 0.640 ms
 3 10.128.6.136 (10.128.6.136) 78.158 ms 102.499 ms 17.792 ms

Possible issue:
You might think of a lossy network - that is not the case. I am experiencing this problem on every network connection in the system (tunelled, physical wired ethernet, physical wireless ethernet), it occurs when tracing the host over a VPN connection, over an internet connections (multiple different providers and speeds, starting 1 Mbit, ending 100 Mbit internet connection line), no matter what I choose, I get this strange behaviour, which already drives me crazy.

You may say, that the routers in the middle do not send ICMP responses. That is not the case also. I have snapped the traffic with tcpdump and attaching it to this bug. You may notice that the hosts _do_ send ICMP responses.

I think that this is a bug in the traceroute 2.0.X package when using some DNS servers - it's waiting for a name resolution for too long and in the meantime it loses the ICMP replies that come over the network.

The version concerned is:

rzajic@rzajic-laptop:~$ traceroute -V
Modern traceroute for Linux, version 2.0.9, Nov 19 2007
Copyright (c) 2006 Dmitry Butskoy, License: GPL

rzajic@rzajic-laptop:~$ dpkg --list | grep traceroute
ii traceroute 2.0.9-3 Traces the route taken by packets over an IP

rzajic@rzajic-laptop:~$ apt-cache show traceroute
Package: traceroute
Priority: optional
Section: net
Installed-Size: 192
Maintainer: Ubuntu Core Developers <email address hidden>
Original-Maintainer: Daniel Baumann <email address hidden>
Architecture: amd64
Version: 2.0.9-3
Depends: libc6 (>= 2.6.1-1)
Conflicts: traceroute-nanog (<< 6.4.2-1), traceproto (<< 1.1.2beta1-3)
Filename: pool/main/t/traceroute/traceroute_2.0.9-3_amd64.deb
Size: 51776
MD5sum: 32a60cea0662ec745d477310a7e92b64
SHA1: f5d743ed8347bae1e8a2036dd40cad78b2e8e154
SHA256: c996606b09be773c5d333d8eab64e89b9e76a2c3c3ee98602ca66c01280708d0
(...)
Bugs: mailto:<email address hidden>
Origin: Ubuntu

How to reproduce the bug:
Simply run sudo traceroute -I www.google.com
(the letter in the traceroute parameter is a capital i fo ICMP traceroute, not small L)

[Radek Zajic (radek-zajic)]

One more comment:
traceroute from two linux boxes on the same network, one running Ubuntu 8.04, the other Mandriva 2008.1:

Mandriva:
[root@localhost opifka]# traceroute -V
Version 1.4a12
Usage: traceroute [-dFInrvx] [-g gateway] [-i iface] [-f first_ttl]
        [-m max_ttl] [ -p port] [-q nqueries] [-s src_addr] [-t tos]
        [-w waittime] [-z pausemsecs] host [packetlen]
[root@localhost opifka]# traceroute -I www.google.com
traceroute: Warning: www.google.com has multiple addresses; using 209.85.129.99
traceroute to www.l.google.com (209.85.129.99), 30 hops max, 38 byte packets
 1 asus (192.168.255.2) 1.501 ms 1.333 ms 1.455 ms
 2 faraon-dg (10.92.6.65) 8.293 ms 4.686 ms 24.652 ms
 3 frb2f (10.92.16.61) 4.297 ms 6.030 ms 4.224 ms
 4 b2frb (10.92.16.58) 7.893 ms 6.119 ms 4.784 ms
 5 g2b (10.92.0.38) 9.331 ms 5.700 ms 5.052 ms
 6 r32g (10.92.0.54) 8.073 ms 8.596 ms 8.552 ms
 7 r3-r2.suchdol.czf (10.19.3.109) 14.231 ms 21.386 ms 10.528 ms
 8 18g-r2-r1.suchdol.czf (10.19.26.1) 10.312 ms 22.786 ms 11.038 ms
 9 r1-r0.trustica.cz (82.208.32.233) 16.565 ms 12.460 ms 17.647 ms
10 81.0.236.1 (81.0.236.1) 18.485 ms 10.968 ms 28.555 ms
11 * * *
12 R3-NE80e-etht0.cas.ip-anywhere.net (217.11.224.17) 12.858 ms 10.082 ms 11.585 ms
13 nix.dialtelecom.cz (194.50.100.9) 62.259 ms 18.921 ms 13.666 ms
14 cz-prg-cr1-sit-10ge2-2.dialtelecom.cz (82.119.245.137) 11.256 ms 12.371 ms 9.815 ms
15 google-gw.dialtelecom.cz (82.119.245.178) 10.891 ms 11.847 ms 10.613 ms
16 209.85.250.228 (209.85.250.228) 12.108 ms 11.546 ms 13.098 ms
17 209.85.250.232 (209.85.250.232) 22.639 ms 18.189 ms 26.728 ms
18 72.14.232.201 (72.14.232.201) 21.384 ms 22.091 ms 72.14.232.203 (72.14.232.203) 46.020 ms
19 72.14.233.206 (72.14.233.206) 58.700 ms 24.022 ms 72.14.233.210 (72.14.233.210) 27.265 ms
20 fk-in-f99.google.com (209.85.129.99) 20.184 ms 19.321 ms 21.651 ms

Ubuntu:
rzajic@rzajic-laptop:~$ traceroute -V
Modern traceroute for Linux, version 2.0.9, Nov 19 2007
Copyright (c) 2006 Dmitry Butskoy, License: GPL
rzajic@rzajic-laptop:~$ sudo traceroute -I www.google.com
traceroute to www.google.com (209.85.129.147), 30 hops max, 40 byte packets
 1 asus.libcice.czf (192.168.255.2) 1.666 ms 2.255 ms 3.466 ms
 2 * * *
 3 * * *
 4 * * *
 5 * * *
 6 * * *
 7 * * *
 8 * * *
 9 r1-r0.trustica.cz (82.208.32.233) 43.916 ms 44.740 ms 45.585 ms
10 81.0.236.1 (81.0.236.1) 47.559 ms 49.233 ms 49.638 ms
11 * * *
12 R3-NE80e-etht0.cas.ip-anywhere.net (217.11.224.17) 55.063 ms 55.436 ms 57.787 ms
13 nix.dialtelecom.cz (194.50.100.9) 55.836 ms 56.575 ms 56.455 ms
14 cz-prg-cr1-sit-10ge2-2.dialtelecom.cz (82.119.245.137) 55.356 ms 55.625 ms 36.154 ms
15 google-gw.dialtelecom.cz (82.119.245.178) 36.336 ms 36.752 ms 36.354 ms
16 209.85.250.228 (209.85.250.228) 37.026 ms 35.466 ms 36.623 ms
17 209.85.250.232 (209.85.250.232) 36.665 ms 47.869 ms 48.066 ms
18 72.14.232.201 (72.14.232.201) 48.304 ms 72.14.232.203 (72.14.232.203) 47.997 ms 47.708 ms
19 72.14.233.210 (72.14.233.210) 49.198 ms 72.14.239.174 (72.14.239.174) 63.419 ms 72.14.233.210 (72.14.233.210) 47.700 ms
20 fk-i...

[Jeruvy (jeruvy)]

Confirmed this does not work. Works from other distros.

[Radek Zajic (radek-zajic)]

The problem seems to be resolved in Jaunty:

root@debpkg:/home/rzajic# traceroute -V
Modern traceroute for Linux, version 2.0.12, Nov 6 2008
Copyright (c) 2008 Dmitry Butskoy, License: GPL v2 or any later

root@debpkg:/home/rzajic# traceroute -I nix.cz
traceroute to nix.cz (195.47.235.3), 30 hops max, 60 byte packets
 1 ns.dalunet.lan (10.160.0.1) 1.202 ms 1.159 ms 1.120 ms
 2 sys0--sys8.dalunet.lan (10.191.255.9) 1.086 ms 1.057 ms 1.018 ms
 3 bbr2--dalunet.dialtelecom.cz (212.80.64.181) 3.263 ms 3.238 ms 3.326 ms
 4 82.119.246.9 (82.119.246.9) 3.171 ms 3.257 ms 3.223 ms
 5 prg-asbr2-kkk-ten1-3.dialtelecom.cz (82.119.246.26) 5.316 ms 5.291 ms 5.250 ms
 6 * * *
 7 nixcz2-cust.tr3.prg.vol.cz (195.122.209.78) 5.026 ms 4.615 ms 4.566 ms
 8 info.nix.cz (195.47.235.3) 4.523 ms 4.704 ms 4.663 ms

root@debpkg:/home/rzajic# traceroute -6 -I nix.cz
traceroute to nix.cz (2a02:38::1001), 30 hops max, 80 byte packets
 1 bb-pl-aval-eth1.ip6.dalunet.cz (2001:4de8:deaf:8000::1) 0.377 ms 0.338 ms 0.306 ms
 2 bb-pl-gate-eth1.ip6.dalunet.cz (2001:4de8:deaf:ffff::1) 0.454 ms 1.383 ms 1.354 ms
 3 2001:4de8:b0ba:deaf::1 (2001:4de8:b0ba:deaf::1) 3.530 ms 3.499 ms 3.467 ms
 4 76sit-te2-4.dialtelecom.cz (2001:4de8:d1a1:1111:d::2) 3.427 ms 3.354 ms 3.323 ms
 5 2001:4de8:d1a1:1111:20::2 (2001:4de8:d1a1:1111:20::2) 4.224 ms 4.195 ms 4.164 ms
 6 2001:4de8:d1a1:1111:21::1 (2001:4de8:d1a1:1111:21::1) 164.187 ms 163.356 ms 163.785 ms
 7 2001:4de8:b0ba:6706:2:0:6:706 (2001:4de8:b0ba:6706:2:0:6:706) 3.796 ms 3.260 ms 3.217 ms
 8 v6-nixcz-cust.tr1.prg2.ipv6.vol.cz (2001:1508:1:18::2) 3.176 ms 3.234 ms 4.145 ms
 9 info.nix.cz (2a02:38::1001) 4.106 ms 3.368 ms 3.325 ms

[Jose Kym S. Alvaro (josekym)]

Having the same problem as well, but only on my Ubuntu install. On Windows, I can tracert and pathping fine from the same PC.

Mine still doesn't work even on a freshly installed Jaunty:

--
root@jksa-laptop:/home/jksa# traceroute -V
Modern traceroute for Linux, version 2.0.12, Nov 5 2008
Copyright (c) 2008 Dmitry Butskoy, License: GPL v2 or any later

root@jksa-laptop:/home/jksa# traceroute -I yahoo.com
traceroute to yahoo.com (209.131.36.159), 30 hops max, 60 byte packets
 1 192.168.20.254 (192.168.20.254) 2.633 ms 2.848 ms *
 2 * * *
 3 * * *
 4 * * *
 5 * * *
 6 * 120.28.0.5 (120.28.0.5) 37.008 ms 39.947 ms
 7 * * *
 8 * * *
 9 * * *
10 * * *
11 * * ae0-p171.msr2.sp1.yahoo.com (216.115.107.83) 218.118 ms
12 te-8-1.bas-a1.sp1.yahoo.com (209.131.32.17) 211.179 ms 213.509 ms te-9-1.bas-a2.sp1.yahoo.com (209.131.32.23) 214.562 ms
13 b1.www.vip.sp1.yahoo.com (209.131.36.159) 215.581 ms 222.002 ms 222.188 ms

root@jksa-laptop:/home/jksa#
--

I also observed that tracepath also fails with the message:

--
root@jksa-laptop:/home/jksa# tracepath google.com
 1: send failed
     Resume: pmtu 65535

root@jksa-laptop:/home/jksa#
--

Furthermore, mtr seems to be working ok.

Any ideas to the cause and how to fix?

[Nizar Kerkeni (nizarus)]

This bug still on ubuntu 12.04.
~$ traceroute -V
Modern traceroute for Linux, version 2.0.18, Oct 18 2011
Copyright (c) 2008 Dmitry Butskoy, License: GPL v2 or any later
~$ traceroute www.ubuntu.com
traceroute to www.ubuntu.com (91.189.90.41), 30 hops max, 60 byte packets
 1 * * *
 2 * * *
 3 * * *
 4 * * *
 5 * * *
 6 * * *
 7 * * *
 8 * * *
 9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Radek Zajic (radek-zajic)]

Confirming the presence in 12.04:

root@router-barrandov:~# traceroute -I 10.128.9.222
traceroute to 10.128.9.222 (10.128.9.222), 30 hops max, 60 byte packets
 1 10.128.6.1 (10.128.6.1) 4.543 ms 4.633 ms *
 2 * * *
 3 * * *
 4 * * *
 5 * * *
 6 * 10.128.9.222 (10.128.9.222) 6.447 ms 7.083 ms

root@router-barrandov:~# traceroute -I 10.128.9.222
traceroute to 10.128.9.222 (10.128.9.222), 30 hops max, 60 byte packets
 1 10.128.6.1 (10.128.6.1) 3.657 ms 3.684 ms *
 2 * * *
 3 * * *
 4 * * *
 5 * * *
 6 * 10.128.9.222 (10.128.9.222) 6.114 ms 6.680 ms

root@router-barrandov:~# traceroute -I -n 10.128.9.222
traceroute to 10.128.9.222 (10.128.9.222), 30 hops max, 60 byte packets
 1 10.128.6.1 3.841 ms 3.856 ms 3.905 ms
 2 10.160.0.2 7.115 ms 7.441 ms 7.965 ms
 3 10.160.8.6 6.924 ms 8.412 ms 8.460 ms
 4 10.128.9.222 7.776 ms 7.944 ms 8.005 ms

root@router-barrandov:~# uname -a
Linux router-barrandov 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

root@router-barrandov:~# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu precise (development branch)"

root@router-barrandov:~# ls -la /usr/sbin/traceroute
lrwxrwxrwx 1 root root 33 Feb 26 10:13 /usr/sbin/traceroute -> /etc/alternatives/traceroute.sbin

root@router-barrandov:~# ls -la /etc/alternatives/traceroute.sbin
lrwxrwxrwx 1 root root 22 Feb 26 10:13 /etc/alternatives/traceroute.sbin -> /usr/bin/traceroute.db

root@router-barrandov:~# ls -la /usr/bin/traceroute.db
-rwxr-xr-x 1 root root 56496 Oct 18 2011 /usr/bin/traceroute.db

root@router-barrandov:~# dpkg -S /usr/bin/traceroute.db
traceroute: /usr/bin/traceroute.db

root@router-barrandov:~# dpkg --list|grep traceroute
ii traceroute 1:2.0.18-1 Traces the route taken by packets over an IPv4/IPv6 network

[era (era)]

Marking as a duplicate of the newer, more succinct bug. Please don't add more "me too" comments.