ipv6 stops working after a while

[swmike]

Binary package hint: linux-image-2.6.27-2-generic

I have configured ipv6 statically in /etc/network/interfaces. Had this running with 8.04 and both 2.6.24 and own-compiled 2.6.26.3, no problems. Now after fresh install of 8.10a4 I am having IPv6 trouble.

After bringing it up after boot, everything works:

Cisco router:

#show ipv6 neighbors
IPv6 Address Age Link-layer Addr State Interface
FE80::240:63FF:FEF5:17C2 0 0040.63f5.17c2 STALE Gi4/0
2A00:801:B::10 0 0040.63f5.17c2 STALE Gi4/0

On linux box:

PING 2a00:801:b::1(2a00:801:b::1) 56 data bytes
64 bytes from 2a00:801:b::1: icmp_seq=1 ttl=64 time=0.658 ms
64 bytes from 2a00:801:b::1: icmp_seq=2 ttl=64 time=0.332 ms
^C
--- 2a00:801:b::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.332/0.495/0.658/0.163 ms
root@via:/mnt/tt/grub# ip -6 n
fe80::20a:42ff:fedd:5470 dev eth0 lladdr 00:0a:42:dd:54:70 DELAY
2a00:801:b::1 dev eth0 lladdr 00:0a:42:dd:54:70 router REACHABLE

Everything is fine, I can download updates via apt-get etc (via ipv6).

After a while the nd goes stale, and cisco router cannot reach the neighbour anymore:

Aug 31 13:15:52.783: ICMPv6-ND: REACH -> STALE: FE80::240:63FF:FEF5:17C2
Aug 31 13:17:43.176: ICMPv6: Sending echo request to 2A00:801:B::10
Aug 31 13:17:43.176: ICMPv6-ND: DELETE -> INCMP: 2A00:801:B::10
Aug 31 13:17:43.176: ICMPv6-ND: Sending NS for 2A00:801:B::10 on GigabitEthernet4/0
Aug 31 13:17:44.176: ICMPv6-ND: Sending NS for 2A00:801:B::10 on GigabitEthernet4/0
Aug 31 13:17:45.176: ICMPv6-ND: Sending NS for 2A00:801:B::10 on GigabitEthernet4/0
Aug 31 13:17:45.176: ICMPv6: Sending echo request to 2A00:801:B::10
Aug 31 13:17:46.176: ICMPv6-ND: INCMP deleted: 2A00:801:B::10
Aug 31 13:17:46.176: ICMPv6-ND: INCMP -> DELETE: 2A00:801:B::10
Aug 31 13:17:47.176: ICMPv6: Sending echo request to 2A00:801:B::10
Aug 31 13:17:47.176: ICMPv6-ND: DELETE -> INCMP: 2A00:801:B::10
Aug 31 13:17:47.176: ICMPv6-ND: Sending NS for 2A00:801:B::10 on GigabitEthernet4/0
Aug 31 13:17:48.176: ICMPv6-ND: Sending NS for 2A00:801:B::10 on GigabitEthernet4/0
Aug 31 13:17:49.176: ICMPv6-ND: Sending NS for 2A00:801:B::10 on GigabitEthernet4/0
Aug 31 13:17:49.176: ICMPv6: Sending echo request to 2A00:801:B::10
Aug 31 13:17:50.176: ICMPv6-ND: INCMP deleted: 2A00:801:B::10
Aug 31 13:17:50.176: ICMPv6-ND: INCMP -> DELETE: 2A00:801:B::10
Aug 31 13:17:51.176: ICMPv6: Sending echo request to 2A00:801:B::10
Aug 31 13:17:51.176: ICMPv6-ND: DELETE -> INCMP: 2A00:801:B::10
Aug 31 13:17:51.176: ICMPv6-ND: Sending NS for 2A00:801:B::10 on GigabitEthernet4/0
Aug 31 13:17:52.176: ICMPv6-ND: Sending NS for 2A00:801:B::10 on GigabitEthernet4/0
Aug 31 13:17:53.176: ICMPv6-ND: Sending NS for 2A00:801:B::10 on GigabitEthernet4/0
Aug 31 13:17:54.176: ICMPv6-ND: INCMP deleted: 2A00:801:B::10

Linux machine also shows itself as being stale, but if I debug ICMP on the cisco router, I can see that the linux machine is sending ICMP packets to the cisco, and it's trying to respond but can't because the nd is stale.

Weird thing is that "tcpdump -p -n proto ipv6" doesn't show any packets, "tcpdump -p -n icmp" doesn't show any ipv6 packets either, so I am having a hard time troubleshooting this. Any ideas?

PS. It might be related to Networkmanager, because I don't have Networkmanager in System->Administration to set up the IPv4 and IPv6 addresses in, so for IPv4 I'm having to kill off dhclient each time I restart the network because networkmanager seems to think I should be running dhcp on the interface. This might have some IPv6 implication as well, I don't know.

[pheeror (pheeror)]

It'd deserve more info from the linux client. Maybe something interesting in logs, ip -6 n sh, ip -6 r sh, ip -6 a sh

As for me, ip6 works fine with the up2date intrepid but I have stateless autoconfiguration in my network (radvd running at a router).

[swmike (ubuntu-s-plass)]

After I restarted networking (networking stop, ifconfig eth0 down, networking start, kill dhclient) ipv6 worked for a few hours. now it doesn't again:

root@via:~# ping6 2a00:801:b::1
PING 2a00:801:b::1(2a00:801:b::1) 56 data bytes
^C
--- 2a00:801:b::1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2013ms

The commands asked for:

root@via:# ip -6 n sh
2a00:801:b::1 dev eth0 lladdr 00:0a:42:dd:54:70 router REACHABLE
root@via:# ip -6 r sh
2a00:801:b::/64 dev eth0 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 4294967295
fe80::/64 dev eth0 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 4294967295
default via 2a00:801:b::1 dev eth0 metric 1 mtu 1500 advmss 1440 hoplimit 4294967295
root@via:# ip -6 a sh
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000
    inet6 2a00:801:b::10/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::240:63ff:fef5:17c2/64 scope link
       valid_lft forever preferred_lft forever

On the cisco:
MikaelA7200Inf3#show ipv6 neighbors
IPv6 Address Age Link-layer Addr State Interface
FE80::240:63FF:FEF5:17C2 1 0040.63f5.17c2 STALE Gi4/0

MikaelA7200Inf3#debug ipv6 icmp
ICMP packet debugging is on
MikaelA7200Inf3#
Aug 31 23:42:34.898: ICMPv6: Received echo request from 2A00:801:B::10
Aug 31 23:42:34.898: ICMPv6: Sending echo reply to 2A00:801:B::10
Aug 31 23:42:35.914: ICMPv6: Received echo request from 2A00:801:B::10
Aug 31 23:42:35.914: ICMPv6: Sending echo reply to 2A00:801:B::10
Aug 31 23:42:36.914: ICMPv6: Received echo request from 2A00:801:B::10
Aug 31 23:42:36.914: ICMPv6: Sending echo reply to 2A00:801:B::10
Aug 31 23:42:37.914: ICMPv6: Received echo request from 2A00:801:B::10
Aug 31 23:42:37.914: ICMPv6: Sending echo reply to 2A00:801:B::10
Aug 31 23:42:39.898: ICMPv6: Received ICMPv6 packet from FE80::240:63FF:FEF5:17C2, type 135
Aug 31 23:42:44.898: ICMPv6: Received ICMPv6 packet from FE80::240:63FF:FEF5:17C2, type 136

MikaelA7200Inf3#debug ipv6 nd
ICMP Neighbor Discovery events debugging is on

Aug 31 23:43:47.158: ICMPv6-ND: DELETE -> INCMP: 2A00:801:B::10
Aug 31 23:43:47.158: ICMPv6-ND: Sending NS for 2A00:801:B::10 on GigabitEthernet4/0
Aug 31 23:43:48.158: ICMPv6-ND: Sending NS for 2A00:801:B::10 on GigabitEthernet4/0
Aug 31 23:43:48.170: ICMPv6: Received echo request from 2A00:801:B::10
Aug 31 23:43:48.170: ICMPv6: Sending echo reply to 2A00:801:B::10
Aug 31 23:43:49.158: ICMPv6-ND: Sending NS for 2A00:801:B::10 on GigabitEthernet4/0
Aug 31 23:43:49.170: ICMPv6: Received echo request from 2A00:801:B::10
Aug 31 23:43:49.170: ICMPv6: Sending echo reply to 2A00:801:B::10
Aug 31 23:43:50.158: ICMPv6-ND: INCMP deleted: 2A00:801:B::10
Aug 31 23:43:50.158: ICMPv6-ND: INCMP -> DELETE: 2A00:801:B::10
Aug 31 23:43:50.170: ICMPv6: Received echo request from 2A00:801:B::10
Aug 31 23:43:50.170: ICMPv6: Sending echo reply to 2A00:801:B::10
Aug 31 23:43:50.170: ICMPv6-ND: DELETE -> INCMP: 2A00:801:B::10
Aug 31 23:43:50.174: ICMPv6-ND: Sending NS for 2A00:801:B::1...

[Leann Ogasawara (leannogasawara)]

Hi swmike,

You commented that you had no issues with a custom compiled 2.6.26.3 kernel. I'm curious if you were ever able to test a 2.6.26 Ubuntu kernel from Intrepid? The reason I ask is I'm wanting to know if this is specifically a regression with the new 2.6.27 Ubuntu kernel of if it existed also with the 2.6.26 Ubuntu kernel. I know the Alpha4 releases of Intrepid contained a 2.6.26 kernel - http://www.ubuntu.com/testing if you'd be interested in testing. Thanks.

[swmike (ubuntu-s-plass)]

I've booted 2.6.26-5-generic now and going to let it run for 24 hours and see what happens.

IPv6 is working right now after reboot anyway.

[swmike (ubuntu-s-plass)]

MikaelA7200Inf3#show ipv6 neighbors
MikaelA7200Inf3#

No neighbours due to no activity for 8+ hours (I rebooted with 2.6.26-5-generic last night, tried it again this morning (4-6 hours without activity, still worked, then I tried it now after 8+ hours of inactivity and it didn't work)) and the status after trying to send pings from the linux box was the following:

MikaelA7200Inf3#show ipv6 neighbors
IPv6 Address Age Link-layer Addr State Interface
FE80::240:63FF:FEF5:17C2 0 0040.63f5.17c2 REACH Gi4/0

swmike@via:~$ ip -6 n sh
2a00:801:b::1 dev eth0 lladdr 00:0a:42:dd:54:70 router STALE
swmike@via:~$ ping6 ipv6
PING ipv6(ipv6.swm.pp.se) 56 data bytes
^C
--- ipv6 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

swmike@via:~$ ip -6 n sh
2a00:801:b::1 dev eth0 lladdr 00:0a:42:dd:54:70 router DELAY
swmike@via:~$ ip -6 n sh
2a00:801:b::1 dev eth0 lladdr 00:0a:42:dd:54:70 router DELAY
swmike@via:~$ ping6 ipv6
PING ipv6(ipv6.swm.pp.se) 56 data bytes
^C
--- ipv6 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1012ms

swmike@via:~$ ip -6 n sh
2a00:801:b::1 dev eth0 lladdr 00:0a:42:dd:54:70 router REACHABLE
swmike@via:~$ ping6 ipv6
PING ipv6(ipv6.swm.pp.se) 56 data bytes
^C
--- ipv6 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms

swmike@via:~$ ip -6 n sh
fe80::20a:42ff:fedd:5470 dev eth0 lladdr 00:0a:42:dd:54:70 DELAY
2a00:801:b::1 dev eth0 lladdr 00:0a:42:dd:54:70 router REACHABLE

So this problem seems to occur after more than 6 hours of inactivity when the ns has completely timeouted, and I'm affected both by 2.6.26-5-generic and 2.6.27-2-generic. I didn't have this problem with a kernel.org 2.6.26.3 that I compiled myself and ran for 38 days on hardy.

Could it be firewall related? I haven't done any configuration on anything like Apparmour or anything like that, so I'm running default settnigs in aspect to any firewall related applications.

[swmike (ubuntu-s-plass)]

I have restarted networking (networking stop && ifconfig eth0 down && networking start && killall dhclient && networking reload)

Now everything works again. I have created a cron script that is run every hour, that pings an ipv6 address. If the problem hasn't occured by tomorrow again, then it must have something to do with neighbour discovery from scratch, whether this is in the kernel or it's some application or firewall that's causing the problem.

[swmike (ubuntu-s-plass)]

I ran the cron script for 14 hours, still worked, then I did the following:

MikaelA7200Inf3#clear ipv6 neighbors
MikaelA7200Inf3#show ipv6 neighbors
MikaelA7200Inf3#

Now ping doesn't work again.

08:01:23.566375 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 64) 2a00:801:b::10 > 2a00:801::f: [icmp6 sum ok] ICMP6, echo request, length 64, seq 213
08:01:24.566375 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 64) 2a00:801:b::10 > 2a00:801::f: [icmp6 sum ok] ICMP6, echo request, length 64, seq 214
08:01:25.566346 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::240:63ff:fef5:17c2 > 2a00:801:b::1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2a00:801:b::1
          source link-address option (1), length 8 (1): 00:40:63:f5:17:c2
            0x0000: 0040 63f5 17c2
08:01:25.566992 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 64) 2a00:801:b::10 > 2a00:801::f: [icmp6 sum ok] ICMP6, echo request, length 64, seq 215
08:01:25.567307 IP6 (class 0xe0, hlim 255, next-header ICMPv6 (58) payload length: 24) 2a00:801:b::1 > fe80::240:63ff:fef5:17c2: [icmp6 sum ok] ICMP6, neighbor advertisement, length 24, tgt is 2a00:801:b::1, Flags [router, solicited]
08:01:26.566375 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 64) 2a00:801:b::10 > 2a00:801::f: [icmp6 sum ok] ICMP6, echo request, length 64, seq 216
08:01:27.566374 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 64) 2a00:801:b::10 > 2a00:801::f: [icmp6 sum ok] ICMP6, echo request, length 64, seq 217
08:01:28.566375 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 64) 2a00:801:b::10 > 2a00:801::f: [icmp6 sum ok] ICMP6, echo request, length 64, seq 218
08:01:29.566375 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 64) 2a00:801:b::10 > 2a00:801::f: [icmp6 sum ok] ICMP6, echo request, length 64, seq 219
08:01:30.564590 IP6 (class 0xe0, hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::20a:42ff:fedd:5470 > fe80::240:63ff:fef5:17c2: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::240:63ff:fef5:17c2
          source link-address option (1), length 8 (1): 00:0a:42:dd:54:70
            0x0000: 000a 42dd 5470
08:01:30.564622 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 24) fe80::240:63ff:fef5:17c2 > fe80::20a:42ff:fedd:5470: [icmp6 sum ok] ICMP6, neighbor advertisement, length 24, tgt is fe80::240:63ff:fef5:17c2, Flags [solicited]
08:01:30.566380 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 64) 2a00:801:b::10 > 2a00:801::f: [icmp6 sum ok] ICMP6, echo request, length 64, seq 220
08:01:31.566377 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 64) 2a00:801:b::10 > 2a00:801::f: [icmp6 sum ok] ICMP6, echo request, length 64, seq 221
08:01:32.566375 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 64) 2a00:801:b::10 > 2a00:801::f: [icmp6 sum ok] ICMP6, echo request, length 64, seq 222

on the cisco router:

Sep 3 08:01:22.567: ICMPv6-ND: Sending NS for 2A00:801:B::10 on GigabitEthernet4/0
Sep 3 08:01:23.567: ICMPv6-ND: Sending NS for 2A00:801:B::10 on GigabitEthernet4/0
Sep 3 08:01:24.567: ICMPv6-ND: INCMP deleted: 2A00:801:B::10
Sep 3 08:01:24.567: ICMPv6-ND: INCMP -> DELE...

[swmike (ubuntu-s-plass)]

I have now tried kernel.org latest kernels of 2.6.25, 2.6.26 and 2.6.27 and they all have the same problem, I also booted a harlddy installation with my kernel.org 2.6.26.3, and it had the same problem also.

The weird thing is that I had ipv6 working perfectly with my old hardy installation (hard drive broke so I had to re-install).

I'm at loss as to what is going on here... No configuration has changed on the cisco router and it's running a very recent IOS so there should be no problem there either. I have disabled IGMP snooping on my switch, didn't change anything either.

[Leann Ogasawara (leannogasawara)]

Hi swmike,

You mentioned you tested an upstream 2.6.27 kernel from kernel.org. Since this is the case this should really be escalated to the upstream kernel developers in a bugzilla.kernel.org report. Care to file this there as well? We can then link this bug report to the upstream one. For more information please refer to https://wiki.ubuntu.com/KernelTeamBugPolicies#Reporting Bugs Upstream . Thanks.

[swmike (ubuntu-s-plass)]

I have been successfully running 9.04 with the ubuntu provided kernels since its release and haven't seen the problem since.

[penalvch (penalvch)]

swmike, this bug report is being closed due to your last comment regarding this being fixed with an update. For future reference you can manage the status of your own bugs by clicking on the current status in the yellow line and then choosing a new status in the revealed drop down box. You can learn more about bug statuses at https://wiki.ubuntu.com/Bugs/Status. Thank you again for taking the time to report this bug and helping to make Ubuntu better. Please submit any future bugs you may find.