| 2010-01-05 01:41:16 |
Mathias Gug |
bug |
|
|
added bug |
| 2010-01-05 01:41:41 |
Mathias Gug |
eucalyptus (Ubuntu): importance |
Undecided |
High |
|
| 2010-01-05 07:30:53 |
Thierry Carrez |
eucalyptus (Ubuntu): status |
New |
Confirmed |
|
| 2010-01-13 14:00:08 |
Ubuntu QA Website |
tags |
|
iso-testing |
|
| 2010-01-14 07:29:39 |
Thierry Carrez |
eucalyptus (Ubuntu): milestone |
|
lucid-alpha-3 |
|
| 2010-01-14 07:29:39 |
Thierry Carrez |
eucalyptus (Ubuntu): assignee |
|
Dustin Kirkland (kirkland) |
|
| 2010-01-14 07:29:51 |
Thierry Carrez |
nominated for series |
|
Ubuntu Lucid |
|
| 2010-01-14 07:29:51 |
Thierry Carrez |
bug task added |
|
eucalyptus (Ubuntu Lucid) |
|
| 2010-01-29 16:26:13 |
Dustin Kirkland |
summary |
eucalyptus-cloud doesn't reply to requests |
eucalyptus-cloud doesn't reply to requests (due to upstart networking issue) |
|
| 2010-01-29 18:03:12 |
Dustin Kirkland |
summary |
eucalyptus-cloud doesn't reply to requests (due to upstart networking issue) |
eucalyptus-cloud doesn't reply to requests (eucalyptus doesn't work after reboot or services restart issues due to upstart networking behavior) |
|
| 2010-02-03 01:15:35 |
Launchpad Janitor |
branch linked |
|
lp:~ubuntu-core-dev/eucalyptus/ubuntu |
|
| 2010-02-03 01:15:51 |
Launchpad Janitor |
eucalyptus (Ubuntu Lucid): status |
Confirmed |
Fix Released |
|
| 2010-02-03 03:00:30 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/eucalyptus |
|
| 2010-02-10 14:50:40 |
Dustin Kirkland |
nominated for series |
|
Ubuntu Karmic |
|
| 2010-02-10 14:50:40 |
Dustin Kirkland |
bug task added |
|
eucalyptus (Ubuntu Karmic) |
|
| 2010-02-10 14:50:52 |
Dustin Kirkland |
eucalyptus (Ubuntu Karmic): status |
New |
Triaged |
|
| 2010-02-10 14:50:55 |
Dustin Kirkland |
eucalyptus (Ubuntu Karmic): importance |
Undecided |
High |
|
| 2010-02-10 14:51:00 |
Dustin Kirkland |
eucalyptus (Ubuntu Karmic): assignee |
|
Dustin Kirkland (kirkland) |
|
| 2010-02-10 14:51:04 |
Dustin Kirkland |
eucalyptus (Ubuntu Karmic): milestone |
|
karmic-updates |
|
| 2010-02-10 15:21:42 |
Dustin Kirkland |
eucalyptus (Ubuntu Karmic): status |
Triaged |
In Progress |
|
| 2010-02-11 20:42:36 |
Dustin Kirkland |
eucalyptus (Ubuntu Karmic): status |
In Progress |
Fix Committed |
|
| 2010-02-11 20:43:29 |
Dustin Kirkland |
summary |
eucalyptus-cloud doesn't reply to requests (eucalyptus doesn't work after reboot or services restart issues due to upstart networking behavior) |
[SRU] eucalyptus-cloud doesn't reply to requests (eucalyptus doesn't work after reboot or services restart issues due to upstart networking behavior) |
|
| 2010-02-11 20:56:17 |
Dustin Kirkland |
description |
Using 1.6.2~bzr1120-0ubuntu1, I can't get the credentials:
ubuntu@uec-cc:~$ sudo euca_conf --get-credentials mycreds.zip
ERROR: you need to be on the CLC host and the CLC needs to be running.
A wget on the register url also times out:
ubuntu@uec-cc:~$ wget -T 10 -t 1 -O - --no-check-certificate https://127.0.0.1:8443/register
--2010-01-04 20:39:34-- https://127.0.0.1:8443/register
Connecting to 127.0.0.1:8443... connected.
WARNING: cannot verify 127.0.0.1's certificate, issued by `/C=US/ST=CA/L=Santa Barbara/O=db/OU=Eucalyptus/CN=localhost':
Self-signed certificate encountered.
WARNING: certificate common name `localhost' doesn't match requested host name `127.0.0.1'.
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Giving up.
The eucalyptus-cloud process is running a listening on port 8443.
I can see the following errors in /var/log/eucalyptus/cloud-error.log:
0:36:55 [log:653891498@qtp-1693378617-9] ERROR /register
java.lang.RuntimeException: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Cannot open connection
at com.eucalyptus.util.TxHandle.<init>(TxHandle.java:46)
at com.eucalyptus.util.EntityWrapper.<init>(EntityWrapper.java:98)
at com.eucalyptus.util.EntityWrapper.<init>(EntityWrapper.java:91)
at edu.ucsb.eucalyptus.util.EucalyptusProperties.getSystemConfiguration(EucalyptusProperties.java:117)
at edu.ucsb.eucalyptus.admin.server.Registration.getRegistrationId(Registration.java:199)
at edu.ucsb.eucalyptus.admin.server.Registration.doGet(Registration.java:210)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:389)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Cannot open connection
at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:614)
at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:41)
at com.eucalyptus.util.TxHandle.<init>(TxHandle.java:40)
... 24 more
Caused by: org.hibernate.exception.JDBCConnectionException: Cannot open connection
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:97)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:449)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142)
at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85)
at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1353)
at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:38)
... 25 more
Caused by: java.sql.SQLException: Connection is broken: java.net.SocketException: Connection timed out
at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
at org.hsqldb.jdbc.jdbcConnection.getAutoCommit(Unknown Source)
at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.logicalcobwebs.proxool.WrappedConnection.invoke(WrappedConnection.java:162)
at $Proxy27.getAutoCommit(Unknown Source)
at org.hibernate.connection.ProxoolConnectionProvider.getConnection(ProxoolConnectionProvider.java:81)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
... 30 more |
Using 1.6.2~bzr1120-0ubuntu1, I can't get the credentials:
ubuntu@uec-cc:~$ sudo euca_conf --get-credentials mycreds.zip
ERROR: you need to be on the CLC host and the CLC needs to be running.
A wget on the register url also times out:
ubuntu@uec-cc:~$ wget -T 10 -t 1 -O - --no-check-certificate https://127.0.0.1:8443/register
--2010-01-04 20:39:34-- https://127.0.0.1:8443/register
Connecting to 127.0.0.1:8443... connected.
WARNING: cannot verify 127.0.0.1's certificate, issued by `/C=US/ST=CA/L=Santa Barbara/O=db/OU=Eucalyptus/CN=localhost':
Self-signed certificate encountered.
WARNING: certificate common name `localhost' doesn't match requested host name `127.0.0.1'.
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Giving up.
The eucalyptus-cloud process is running a listening on port 8443.
I can see the following errors in /var/log/eucalyptus/cloud-error.log:
0:36:55 [log:653891498@qtp-1693378617-9] ERROR /register
java.lang.RuntimeException: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Cannot open connection
at com.eucalyptus.util.TxHandle.<init>(TxHandle.java:46)
at com.eucalyptus.util.EntityWrapper.<init>(EntityWrapper.java:98)
at com.eucalyptus.util.EntityWrapper.<init>(EntityWrapper.java:91)
at edu.ucsb.eucalyptus.util.EucalyptusProperties.getSystemConfiguration(EucalyptusProperties.java:117)
at edu.ucsb.eucalyptus.admin.server.Registration.getRegistrationId(Registration.java:199)
at edu.ucsb.eucalyptus.admin.server.Registration.doGet(Registration.java:210)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:389)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Cannot open connection
at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:614)
at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:41)
at com.eucalyptus.util.TxHandle.<init>(TxHandle.java:40)
... 24 more
Caused by: org.hibernate.exception.JDBCConnectionException: Cannot open connection
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:97)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:449)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142)
at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85)
at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1353)
at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:38)
... 25 more
Caused by: java.sql.SQLException: Connection is broken: java.net.SocketException: Connection timed out
at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
at org.hsqldb.jdbc.jdbcConnection.getAutoCommit(Unknown Source)
at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.logicalcobwebs.proxool.WrappedConnection.invoke(WrappedConnection.java:162)
at $Proxy27.getAutoCommit(Unknown Source)
at org.hibernate.connection.ProxoolConnectionProvider.getConnection(ProxoolConnectionProvider.java:81)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
... 30 more
== SRU ==
IMPACT: Users of 9.10 UEC will often experience long, non-deterministic delays (10-20 minutes in many cases) when restarting eucalyptus services (which includes reboots, package upgrades, service restarts). This is highly inconvenient, yielding UEC unusable until database network connections reset.
HOW FIXED: The fix consists of two trivial iptables calls being added to the eucalyptus upstart script. Upstream had these calls in there init scripts, but were inadvertently dropped when porting Eucalyptus to upstart. These iptables commands will ensure that the iptables kernel module (and most importantly, the ip connection tracker) is loaded and active before Eucalyptus comes up. WIthout said ip connection tracker, Eucalyptus will often establish a connection to the database, then iptables is loaded and connections are mangled, breaking the connection to the database. The user will see the problem in any one of a number of disguising ways (front end not working, api tools not responding, etc). All of these problems are due to an inaccessible database. After a while (10-20 minutes), Eucalyptus will reset the database connection. With this fix, the above problems should never happen. Eucalyptus should be back up and running within 1-2 minutes of boot (if not immediately).
MINIMAL PATCH:
diff -u eucalyptus-1.6~bzr931/debian/eucalyptus-common.eucalyptus.upstart eucalyptus-1.6~bzr931/debian/eucalyptus-common.eucalyptus.upstart
--- eucalyptus-1.6~bzr931/debian/eucalyptus-common.eucalyptus.upstart
+++ eucalyptus-1.6~bzr931/debian/eucalyptus-common.eucalyptus.upstart
@@ -11,6 +11,10 @@
# Check if installed
[ -f /usr/sbin/euca_conf ] || { stop; exit 0; }
+ # Ensure that the iptables module gets loaded here
+ iptables -t nat -L -n >/dev/null
+ iptables -L -n > /dev/null
+
mkdir -p /var/run/eucalyptus/net
chown eucalyptus:eucalyptus /var/run/eucalyptus /var/run/eucalyptus/net
REPRODUCING THE BUG:
Reboot your UEC (or sudo restart eucalyptus). If restarting eucalyptus takes a *long* time, you are experiencing one symptom of this bug. Once upstart thinks that eucalyptus is up, try: $(sudo wget --no-check-certificate https://localhost:8443) If this takes a long time, or fails to work, you are experiencing a symptom of this bug. Note that the problem is inherently due to a race condition, and therefore may not be immediately reproducible. Try rebooting/restarting a few times, and you will likely hit it.
REGRESSION POTENTIAL:
I cannot see any possible regression potential. The iptables modules will be loaded eventually. This patch just ensures that they get loaded before Eucalyptus tries to start services.
|
|
| 2010-02-17 15:33:40 |
Martin Pitt |
tags |
iso-testing |
iso-testing verification-needed |
|
| 2010-02-17 15:56:46 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/karmic-proposed/eucalyptus |
|
| 2010-02-24 07:41:28 |
Martin Pitt |
tags |
iso-testing verification-needed |
iso-testing verification-done |
|
| 2010-02-24 14:52:28 |
Launchpad Janitor |
eucalyptus (Ubuntu Karmic): status |
Fix Committed |
Fix Released |
|
