The eucalyptus cloud had been running without issues for about a month before this.

The error we are getting looks like this:

euca-describe-availability-zones verbose
Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
EC2ResponseError: 400 Bad Request
Failure: 400 Bad Request
javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Cannot open connection

from cloud-error.log:


10:48:05 ERROR [SystemClock:SystemClockTimer] java.lang.RuntimeException: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Cannot open connection
java.lang.RuntimeException: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Cannot open connection
at com.eucalyptus.entities.TxHandle.<init>(TxHandle.java:48)
at com.eucalyptus.entities.EntityWrapper.<init>(EntityWrapper.java:109)
at com.eucalyptus.entities.EntityWrapper.<init>(EntityWrapper.java:92)
at com.eucalyptus.cluster.VmTypes.update(VmTypes.java:119)
at com.eucalyptus.cluster.VmTypes.list(VmTypes.java:148)
at com.eucalyptus.cluster.handlers.ResourceStateHandler.trigger(ResourceStateHandler.java:28)
at com.eucalyptus.cluster.handlers.AbstractClusterMessageDispatcher.fireTimedStatefulTrigger(AbstractClusterMessageDispatcher.java:135)
at com.eucalyptus.cluster.handlers.ResourceStateHandler.fireEvent(ResourceStateHandler.java:36)
at com.eucalyptus.event.ReentrantListenerRegistry.fireEvent(ReentrantListenerRegistry.java:87)
at com.eucalyptus.event.ReentrantListenerRegistry.fireEvent(ReentrantListenerRegistry.java:68)
at com.eucalyptus.event.ListenerRegistry.fireEvent(ListenerRegistry.java:69)
at com.eucalyptus.event.SystemClock.run(SystemClock.java:103)
at java.util.TimerThread.mainLoop(Timer.java:534)
at java.util.TimerThread.run(Timer.java:484)
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Cannot open connection
at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:614)
at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:41)
at com.eucalyptus.entities.TxHandle.<init>(TxHandle.java:42)
... 13 more
Caused by: org.hibernate.exception.JDBCConnectionException: Cannot open connection
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:97)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:449)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142)
at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85)
at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1353)
at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:38)
... 14 more
Caused by: java.sql.SQLException: socket creation error
at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
at org.hsqldb.jdbc.jdbcConnection.<init>(Unknown Source)
at org.hsqldb.jdbcDriver.getConnection(Unknown Source)
at org.hsqldb.jdbcDriver.connect(Unknown Source)
at java.sql.DriverManager.getConnection(DriverManager.java:620)
at java.sql.DriverManager.getConnection(DriverManager.java:169)
at org.logicalcobwebs.proxool.DefaultConnectionBuilder.buildConnection(DefaultConnectionBuilder.java:39)
at org.logicalcobwebs.proxool.Prototyper.buildConnection(Prototyper.java:159)
at org.logicalcobwebs.proxool.ConnectionPool.getConnection(ConnectionPool.java:211)
at org.logicalcobwebs.proxool.ProxoolDriver.connect(ProxoolDriver.java:89)
at java.sql.DriverManager.getConnection(DriverManager.java:620)
at java.sql.DriverManager.getConnection(DriverManager.java:222)
at org.hibernate.connection.ProxoolConnectionProvider.getConnection(ProxoolConnectionProvider.java:75)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
... 19 more