[SRU] [scalability] walrus reports java.lang.OutOfMemoryError: Java heap space
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Eucalyptus |
Fix Released
|
High
|
Unassigned | ||
eucalyptus (Ubuntu) |
Fix Released
|
High
|
Dustin Kirkland | ||
Lucid |
Fix Released
|
High
|
Dustin Kirkland |
Bug Description
stress-testing, 400 instances to be started in total. After a while the walrus starts reporting out-of-memory, and all new instances fail.
Ubuntu-server 2010-04-16, UEC 2010-04-16.
ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: eucalyptus-walrus 1.6.2-0ubuntu30
ProcVersionSign
Uname: Linux 2.6.32-21-server x86_64
Architecture: amd64
Date: Fri Apr 16 19:18:49 2010
ProcEnviron:
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: eucalyptus
========
IMPACT:
* This bug affects UEC installs, with the CLC and Walrus on separate machines. After running a few hundred (or thousands) of instances, the Walrus will eventually run out of memory.
ADDRESSED:
* The bug is addressed in two ways. First, a configuration parameter has been added to /etc/eucalyptus
PATCH:
* The patch isn't very minimal. Sorry. But it's what's required to solve the problem. And it's already in the upstream stable branch.
REPRODUCE:
* Reproducing this bug is not easy. We most frequently see this happen when the CLC and Walrus are not co-located on the same machine. And then we run several thousand instances. Eventually, Walrus falls over with OOM.
REGRESSION POTENTIAL:
* There's certainly regression potential in the Walrus itself, as that's the code that's changed. However, we have put this code through a very heavy barrage of scalability testing (which also exercises many of the base features). We believe that we have proactively tested this code thoroughly.
========
Changed in eucalyptus (Ubuntu): | |
status: | New → In Progress |
importance: | Undecided → High |
milestone: | none → ubuntu-10.04 |
assignee: | nobody → Dustin Kirkland (kirkland) |
description: | updated |
summary: |
- walrus reports java.lang.OutOfMemoryError: Java heap space + [scalability] walrus reports java.lang.OutOfMemoryError: Java heap space |
summary: |
- [scalability] walrus reports java.lang.OutOfMemoryError: Java heap space + [SRU] [scalability] walrus reports java.lang.OutOfMemoryError: Java heap + space |
tags: |
added: verification-done removed: verification-needed |
Changed in eucalyptus (Ubuntu Lucid): | |
status: | Fix Released → Fix Committed |
Changed in eucalyptus: | |
status: | Fix Committed → Fix Released |
The suggested fix is to up the jvm's memory to 384M in /etc/init/ eucalyptus. conf:
=== modified file 'debian/ eucalyptus- common. eucalyptus. upstart' eucalyptus- common. eucalyptus. upstart 2010-02-04 01:00:15 +0000 eucalyptus- common. eucalyptus. upstart 2010-04-20 16:59:34 +0000 /eucalyptus. conf
--- debian/
+++ debian/
@@ -36,7 +36,7 @@
script
. /etc/eucalyptus
- opts="-h $EUCALYPTUS -u $EUCA_USER --pidfile /var/run/ eucalyptus/ eucalyptus. pid -l $LOGLEVEL -L console-log" eucalyptus/ eucalyptus. pid -l $LOGLEVEL -L console-log -Xmx384m"
+ opts="-h $EUCALYPTUS -u $EUCA_USER --pidfile /var/run/
services=""
# If the -cloud package is not installed, disable the cloud service