Eucalyptus Database hangs

[David Mead]

Ubuntu 11.04
All admin functions of eucalyptus fail

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: eucalyptus-common 2.0.1+bzr1256-0ubuntu4.1
ProcVersionSignature: Ubuntu 2.6.38-8.42-server 2.6.38.2
Uname: Linux 2.6.38-8-server x86_64
.etc.eucalyptus.eucalyptus.cc.conf: CC_NAME="cirruscluster"
Architecture: amd64
Date: Tue Nov 1 16:39:57 2011
InstallationMedia: Ubuntu-Server 11.04 "Natty Narwhal" - Release amd64 (20110426)
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: eucalyptus
UpgradeStatus: No upgrade log present (probably fresh install)

[Leonardo Borda (lborda)]

The eucalyptus cloud had been running without issues for about a month before this.

The error we are getting looks like this:

euca-describe-availability-zones verbose
Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
EC2ResponseError: 400 Bad Request
Failure: 400 Bad Request
javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Cannot open connection

from cloud-error.log:

10:48:05 ERROR [SystemClock:SystemClockTimer] java.lang.RuntimeException: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Cannot open connection
java.lang.RuntimeException: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Cannot open connection
at com.eucalyptus.entities.TxHandle.<init>(TxHandle.java:48)
at com.eucalyptus.entities.EntityWrapper.<init>(EntityWrapper.java:109)
at com.eucalyptus.entities.EntityWrapper.<init>(EntityWrapper.java:92)
at com.eucalyptus.cluster.VmTypes.update(VmTypes.java:119)
at com.eucalyptus.cluster.VmTypes.list(VmTypes.java:148)
at com.eucalyptus.cluster.handlers.ResourceStateHandler.trigger(ResourceStateHandler.java:28)
at com.eucalyptus.cluster.handlers.AbstractClusterMessageDispatcher.fireTimedStatefulTrigger(AbstractClusterMessageDispatcher.java:135)
at com.eucalyptus.cluster.handlers.ResourceStateHandler.fireEvent(ResourceStateHandler.java:36)
at com.eucalyptus.event.ReentrantListenerRegistry.fireEvent(ReentrantListenerRegistry.java:87)
at com.eucalyptus.event.ReentrantListenerRegistry.fireEvent(ReentrantListenerRegistry.java:68)
at com.eucalyptus.event.ListenerRegistry.fireEvent(ListenerRegistry.java:69)
at com.eucalyptus.event.SystemClock.run(SystemClock.java:103)
at java.util.TimerThread.mainLoop(Timer.java:534)
at java.util.TimerThread.run(Timer.java:484)
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Cannot open connection
at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:614)
at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:41)
at com.eucalyptus.entities.TxHandle.<init>(TxHandle.java:42)
... 13 more
Caused by: org.hibernate.exception.JDBCConnectionException: Cannot open connection
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:97)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:449)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142)
at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85)
at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1353)
at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:38)
... 14 more
Caused by: java.sql.SQLException: socket creation error
at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
at org.hsqldb.jdbc.jdbcConnection.<init>(Unknown Source)
at org.hsqldb.jdbcDriver.getCo...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Leonardo Borda (lborda)]

might be related.

http://open.eucalyptus.com/forum/eucalyptus-20-memory-leak

[Leonardo Borda (lborda)]

I am creating a test environment in order to try to reproduce this issue.

[Leonardo Borda (lborda)]

Workaround is to increase memory java heap size to a higher number.

1. vi /etc/init/eucalyptus.conf
change
from:
[ -n "$JVM_MEM" ] || JVM_MEM="512m"
to:
[ -n "$JVM_MEM" ] || JVM_MEM="8192m"

2. save the file

3. restart eucalyptus-cloud service
sudo service eucalyptus-cloud stop
sudo service eucalyptus-cloud start

PS: Check using ps if the eucalyptus-cloud service is using 8192m. like the example below
eg. -eucalyptus-clou,1324 -h / -u eucalyptus --pidfile /var/run/eucalyptus/eucalyptus.pid -l DEBUG -L console-log -Xmx8192m