Zookeeper listen only to IPv6 interface

[Patrick Hetu]

ii zookeeper 3.3.3+dfsg2-1ubuntu1 High-performance coordination service for distributed applications
ii zookeeperd 3.3.3+dfsg2-1ubuntu1 Init control scripts for zookeeper

I was getting time out from Juju agents and I realised that they try to connect via ipv4
but zookeeper listen only to ipv6:

root@server-54:~# netstat -lt
tcp 0 0 *:ssh *:* LISTEN
tcp6 0 0 [::]:ssh [::]:* LISTEN
tcp6 0 0 [::]:2181 [::]:* LISTEN

After some search I found this flag to force IPv4:

  JAVA_OPTS="-Djava.net.preferIPv4Stack=true"

[James Page (james-page)]

Hi Patrick

Please could you provide a few more details about your IPv4/IPv6 setup? I'm running a dual stack IPv4/6 configuration and don't appear to have the same issue.

[Clint Byrum (clint-fewbar)]

I have the same issue on precise.

I just apt-get installed zookeeper, and I have

tcp6 0 0 :::2181 :::* LISTEN 3425/java

but no corresponding tcp listening socket.

I do not have any sort of explicit IPv6 configuration, just the default networking setup from precise, running DHCP on wifi:

$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
    link/ether 00:23:32:cb:1e:60 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:23:6c:87:2c:2d brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.105/24 brd 192.168.1.255 scope global eth1
    inet6 fe80::223:6cff:fe87:2c2d/64 scope link
       valid_lft forever preferred_lft forever
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    link/ether fe:72:b9:dd:78:47 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0

[James Page (james-page)]

I think that this issue occurs when you only have a 'scope link' ipv6 address; I'm running a full IPv6 dual stack and I don't see this issue:

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether b8:ac:6f:76:db:40 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.50/24 brd 192.168.1.255 scope global eth0
    inet6 2a01:348:2f4:0:baac:6fff:fe76:db40/64 scope global dynamic
       valid_lft 86193sec preferred_lft 14193sec
    inet6 fe80::baac:6fff:fe76:db40/64 scope link
       valid_lft forever preferred_lft forever

I confirmed remote access both over IPv4 and IPv6 to this zookeeper instance.

The IPv6 binding serves both the IPv6 and IPv4 ports; Java by default prefers IPv6 over IPv4 for this reason but it looks like this might be broken when only a 'scope link' address is present. I see this on Java6 and Java7.

[James Page (james-page)]

Hmm - or even not.

Installing on an IPv4 only KVM instance:

ubuntu@ubuntu:~$ netstat -lt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:ssh *:* LISTEN
tcp6 0 0 [::]:2181 [::]:* LISTEN
tcp6 0 0 [::]:37069 [::]:* LISTEN
tcp6 0 0 [::]:ssh [::]:* LISTEN
ubuntu@ubuntu:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f3:77:6c brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.163/24 brd 192.168.122.255 scope global eth0
    inet6 fe80::5054:ff:fef3:776c/64 scope link
       valid_lft forever preferred_lft forever

However I am able to connect remotely to 192.168.122.163:2181.

[James Page (james-page)]

I'm reticent to add JAVA_OPTS="-Djava.net.preferIPv4Stack=true" by default to the zookeeper configuration as this completely disabled IPv6 and I am not able to reproduce an environment where:

tcp6 0 0 [::]:2181 [::]:* LISTEN

results in no IPv4 connections.

[James Page (james-page)]

Clint - could you connect to zookeeper via IPv4 on your test system?

[James Page (james-page)]

For information: http://docs.oracle.com/javase/7/docs/technotes/guides/net/ipv6_guide/index.html

[Clint Byrum (clint-fewbar)]

Excerpts from James Page's message of Fri Jan 06 09:50:52 UTC 2012:
> For information:
> http://docs.oracle.com/javase/7/docs/technotes/guides/net/ipv6_guide/index.html

Interesting. At first, I wasn't able to connect to 127.0.0.1:2181. But
now while testing I am able. I think my issue was actually some weird
permissions in /var/lib/zookeeper, as I've cleared that out and purged/installed
a few times and zookeeper is working flawlessly with only :::2181 listening.

So at this point, I'm not sure this is a bug.

[Launchpad Janitor (janitor)]

[Expired for zookeeper (Ubuntu) because there has been no activity for 60 days.]