| 2010-09-15 17:23:26 |
C de-Avillez |
description |
Eucalyptus 2.0+brz1239-0ubuntu3 (actually, r1240).
After about 1,300 instances started Eucalyptus suddenly stopped responding to euca-* commands. A brief look at the eucalyptus logs show a series of
23:44:20 DEBUG [AbstractClusterMessageDispatcher:Hashed wheel timer #1] org.jboss.netty.handler.timeout.ReadTimeoutException
org.jboss.netty.handler.timeout.ReadTimeoutException
at org.jboss.netty.handler.timeout.ReadTimeoutHandler.<clinit>(ReadTimeoutHandler.java:59)
at com.eucalyptus.ws.util.ChannelUtil.addPipelineMonitors(ChannelUtil.java:134)
at com.eucalyptus.cluster.handlers.AbstractClusterMessageDispatcher.getPipeline(AbstractClusterMessageDispatcher.java:103)
at com.eucalyptus.ws.client.NioBootstrap.connect(NioBootstrap.java:177)
at com.eucalyptus.ws.client.NioBootstrap.connect(NioBootstrap.java:163)
at com.eucalyptus.cluster.handlers.AbstractClusterMessageDispatcher.write(AbstractClusterMessageDispatcher.java:123)
at com.eucalyptus.cluster.handlers.ClusterCertificateHandler.trigger(ClusterCertificateHandler.java:32)
at com.eucalyptus.cluster.handlers.ClusterCertificateHandler.fireEvent(ClusterCertificateHandler.java:57)
at com.eucalyptus.event.ReentrantListenerRegistry.fireEvent(ReentrantListenerRegistry.java:87)
at com.eucalyptus.event.ReentrantListenerRegistry.fireEvent(ReentrantListenerRegistry.java:68)
at com.eucalyptus.event.ListenerRegistry.fireEvent(ListenerRegistry.java:69)
at com.eucalyptus.cluster.Clusters.start(Clusters.java:135)
at com.eucalyptus.cluster.ClusterBuilder.fireStart(ClusterBuilder.java:93)
at com.eucalyptus.component.Component.startService(Component.java:179)
at com.eucalyptus.ws.ServiceDispatchBootstrapper.start(ServiceDispatchBootstrapper.java:143)
at com.eucalyptus.bootstrap.Bootstrap$Stage.start(Bootstrap.java:112)
at com.eucalyptus.bootstrap.SystemBootstrapper.start(SystemBootstrapper.java:148)
test logs uploaded to lp:~hggdh2/+junk/uec-qa, revision 56. |
Eucalyptus 2.0+brz1239-0ubuntu3.2 (actually, r1240).
After about 1,300 instances started Eucalyptus suddenly stopped responding to euca-* commands. A brief look at the eucalyptus logs show a series of
23:44:20 DEBUG [AbstractClusterMessageDispatcher:Hashed wheel timer #1] org.jboss.netty.handler.timeout.ReadTimeoutException
org.jboss.netty.handler.timeout.ReadTimeoutException
at org.jboss.netty.handler.timeout.ReadTimeoutHandler.<clinit>(ReadTimeoutHandler.java:59)
at com.eucalyptus.ws.util.ChannelUtil.addPipelineMonitors(ChannelUtil.java:134)
at com.eucalyptus.cluster.handlers.AbstractClusterMessageDispatcher.getPipeline(AbstractClusterMessageDispatcher.java:103)
at com.eucalyptus.ws.client.NioBootstrap.connect(NioBootstrap.java:177)
at com.eucalyptus.ws.client.NioBootstrap.connect(NioBootstrap.java:163)
at com.eucalyptus.cluster.handlers.AbstractClusterMessageDispatcher.write(AbstractClusterMessageDispatcher.java:123)
at com.eucalyptus.cluster.handlers.ClusterCertificateHandler.trigger(ClusterCertificateHandler.java:32)
at com.eucalyptus.cluster.handlers.ClusterCertificateHandler.fireEvent(ClusterCertificateHandler.java:57)
at com.eucalyptus.event.ReentrantListenerRegistry.fireEvent(ReentrantListenerRegistry.java:87)
at com.eucalyptus.event.ReentrantListenerRegistry.fireEvent(ReentrantListenerRegistry.java:68)
at com.eucalyptus.event.ListenerRegistry.fireEvent(ListenerRegistry.java:69)
at com.eucalyptus.cluster.Clusters.start(Clusters.java:135)
at com.eucalyptus.cluster.ClusterBuilder.fireStart(ClusterBuilder.java:93)
at com.eucalyptus.component.Component.startService(Component.java:179)
at com.eucalyptus.ws.ServiceDispatchBootstrapper.start(ServiceDispatchBootstrapper.java:143)
at com.eucalyptus.bootstrap.Bootstrap$Stage.start(Bootstrap.java:112)
at com.eucalyptus.bootstrap.SystemBootstrapper.start(SystemBootstrapper.java:148)
test logs uploaded to lp:~hggdh2/+junk/uec-qa, revision 56.
|
|
