Java JMX refusing connections

[Sorin Sbarnea]

Initially reported here http://stackoverflow.com/questions/21604892/java-jmx-refusing-connections-with-really-weird-case

I do have consistent problems enabling JMX on some servers. It seems that while this works on some, some of them are rejecting connection in a strange way.

The port open and I can telnet to it from both localhost or other hosts.

Here is what I get from jmxterm when I try to connect to one of the faulty ones.

This happens if the port is opened:

$>open 10.80.16.195:8091
#RuntimeIOException: Runtime IO exception: Connection refused to host: 127.0.0.1; nested exception is:
    java.net.ConnectException: Connection refused
THis happens if the port is really closed:

$>open 10.80.16.195:9999
#RuntimeIOException: Runtime IO exception: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.80.16.195; nested exception is:
    java.net.ConnectException: Connection refused]
Here are the parameters used to connect to start the JVM (tomcat): /usr/lib/jvm/java-7-oracle/bin/java -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8091 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -classpath /opt/Confluence.5/bin/bootstrap.jar -Dcatalina.base=/opt/Confluence.5 -Dcatalina.home=/opt/Confluence.5 org.apache.catalina.startup.Bootstrap start

java -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
I observed similar complains from others but nobody provided a clear cause for this. It's less likely to be the JMX client, as I used 3-4 ones and all seem to have similar behaviors, working with some servers and failing on others.

I do know that /etc/hosts may be linked to this because, i remember seeing the 127.0.1.1 reported in some cases and I do know that Debian machines do have this setup.

127.0.0.1 localhost
127.0.0.1 dowa-01.my.domain.com dowa-01
# by default the previous line had 127.0.1.1 instead and this was causing problems with JMX

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
As you probably observed the JMX is binded to the routable address of the machine, that's because that's the only way to allow connections from outside, if you manually specify the IP.

Now what's the problem?