Cloud exhausting Permanent Generation space

[Chris Jones]

I have a cloud which has been running for a couple of weeks under light-moderate load and as of last night it has stopped functioning. cloud-output.log is silent and cloud-debug.log has:

17:53:33 DEBUG [NioServerHandler:Hashed wheel timer #1] Internal Error.
java.lang.OutOfMemoryError: PermGen space

many times over, and eucalyptus-cloud is consuming 100% CPU.

#eucalyptus and some googling indicate that this is java running out of space in a particular buffer, and that -XX:MaxPermSize=256m would help avoid/delay this. Also upstream appears to have made such a change for 2.0.

Is this something we should put into Lucid? Are the benefits/dangers balanced sufficiently? (I have no idea what the drawbacks, if any, are).

[Thierry Carrez (ttx)]

Sounds like something we should consider for a Lucid SRU.

[Dave Walker (davewalker)]

@Ng: Have you set -XX:MaxPermSize=256m on the cloud?

Sounds like a wonderful bug.

I would love to hear if that makes a difference, i'm not too keen on the prospect of an SRU for a 'probably fix'.

Thanks!

[Chris Jones (cmsj)]

Dave: Yes, the cloud in question is now running with that option hacked into the eucalyptus upstart job. It's a bit early to tell if it's helped, I suspect.

[Rolf Leggewie (r0lf)]

lucid has seen the end of its life and is no longer receiving any updates. Marking the lucid task for this ticket as "Won't Fix".