Failed to start namenode when private network has IPv6 subnet

Bug #1489433 reported by Kazuki OIKAWA
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Sahara
Fix Released
Medium
Kazuki OIKAWA

Bug Description

When cluster attached private network has IPv6 subnet, namenode failed to start.

2015-08-27 11:10:43,791 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.io.IOException: java.lang.RuntimeException: Could not resolve Kerberos principal name: java.net.UnknownHostException: test-master-v27-0.novalocal: test-master-v27-0.novalocal
        at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:347)
        at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:114)
        at org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:290)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:126)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:752)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:638)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:811)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:795)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
Caused by: java.lang.RuntimeException: Could not resolve Kerberos principal name: java.net.UnknownHostException: test-master-v27-0.novalocal: test-master-v27-0.novalocal
        at org.apache.hadoop.security.AuthenticationFilterInitializer.getFilterConfigMap(AuthenticationFilterInitializer.java:90)
        at org.apache.hadoop.http.HttpServer2.getFilterProperties(HttpServer2.java:454)
        at org.apache.hadoop.http.HttpServer2.constructSecretProvider(HttpServer2.java:445)
        at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:339)
        ... 9 more
Caused by: java.net.UnknownHostException: test-master-v27-0.novalocal: test-master-v27-0.novalocal
        at java.net.InetAddress.getLocalHost(InetAddress.java:1494)
        at org.apache.hadoop.security.SecurityUtil.getLocalHostName(SecurityUtil.java:190)
        at org.apache.hadoop.security.SecurityUtil.replacePattern(SecurityUtil.java:183)
        at org.apache.hadoop.security.SecurityUtil.getServerPrincipal(SecurityUtil.java:138)
        at org.apache.hadoop.security.AuthenticationFilterInitializer.getFilterConfigMap(AuthenticationFilterInitializer.java:87)
        ... 12 more
Caused by: java.net.UnknownHostException: test-master-v27-0.novalocal
        at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
        at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:922)
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1314)
        at java.net.InetAddress.getLocalHost(InetAddress.java:1490)
        ... 16 more
2015-08-27 11:10:43,795 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2015-08-27 11:10:43,797 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:

/etc/hosts writes IPv6 address
ubuntu@test-master-v27-0:~$ cat /etc/hosts
127.0.0.1 localhost
fd28:b3d2:1d2b:0:f816:3eff:fe25:5f06 test-master-v27-0.novalocal test-master-v27-0
fd28:b3d2:1d2b:0:f816:3eff:fef0:8a95 test-slave-v27-0.novalocal test-slave-v27-0

Kazuki OIKAWA (k.oikw)
Changed in sahara:
assignee: nobody → Kazuki OIKAWA (k.oikw)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to sahara (master)

Fix proposed to branch: master
Review: https://review.openstack.org/217669

Changed in sahara:
milestone: none → liberty-rc1
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to sahara (master)

Reviewed: https://review.openstack.org/217669
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=1622710e2bb0a7017abedb8a4dc26d0223583e15
Submitter: Jenkins
Branch: master

commit 1622710e2bb0a7017abedb8a4dc26d0223583e15
Author: Kazuki OIKAWA <email address hidden>
Date: Thu Aug 27 20:48:32 2015 +0900

    Selects IPv4 preferentially for internal_ip

    Hadoop doesn't support IPv6.
    Selects IPv4 preferentially for internal_ip/management_ip
    when private network has IPv4/IPv6 dual subnets.

    Change-Id: Ibc808d5467ea6db081cdc98655c7eb7faffa6ec3
    Closes-Bug: #1489433

Changed in sahara:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in sahara:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in sahara:
milestone: liberty-rc1 → 3.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.