Appserver memory issues with Storm 0.14

[Francis J. Lacoste]

We are seeing a lot more time outs and memory issues on Edge than usual since the Storm 0.14 upgrade landed.

https://devpad.canonical.com/~matsubara/edge-2009-06-22-16:00:00.html

https://lpstats.canonical.com/graphs/AppServer5XXsEdge/

.

[Francis J. Lacoste (flacoste)]

Branch reverting the upgrade is on its way to PQM.

[Francis J. Lacoste (flacoste)]

Coincidently, or perhaps not, all app servers have gone belly up with memory errors.

[Stuart Bishop (stub)]

Prejoin is working fine - you can see the prejoin queries quite clearly in a number of the OOPSes.

The new precache is being used in a single location and is not an issue.

The new Storm does have a new cache, which we wanted to help alleviate the major memory problems a number of our scripts have with the old StupidCache we hacked into old-Storm.

[Stuart Bishop (stub)]

Unassigning from me - what is there to do on this task?

[Stuart Bishop (stub)]

Storm 0.14 will be running shortly on staging, running with GenerationalCache and a cache size of 50000. If staging has memory issues, and edge does not, then Storm is the likely culprit.

[Stuart Bishop (stub)]

Staging starts up fine, but memory usage creeps up and CPU time climbs up to a constant 80%. This happens with both GenerationalCache and our old StupidCache. I suspect a memory leak causing the garbage collector to g nuts.

[Stuart Bishop (stub)]

I'm updating Storm to trunk + lp:~jamesh/storm/bug-217644 (the unmerged patch fixes a regression on trunk).

The branch is lp:~stub/launchpad/update-storm, and it is with ec2test now.

[Stuart Bishop (stub)]

Some test failures: https://pastebin.canonical.com/19146/

These might be regressions from 0.14 even with jamesh's patch, or perhaps we were relying on bad behavior.

I'll have a closer look when I get back on my feet (sick again) unless someone beats me to it. The branch is lp:~stub/launchpad/update-storm.

[Stuart Bishop (stub)]

If anyone works on this, don't merge launchpad/db-devel into the branch as we will need to land this on launchpad/devel once we are satisfied that staging is ok.

[Stuart Bishop (stub)]

Sent off a branch using Storm trunk r290 to ec2test. r290 contains a memory leak fix. r291+ contain changes that break our tests.

[Stuart Bishop (stub)]

Landed Storm 0.14 + trunk r290 on launchpad/db-devel. r8221.

[Stuart Bishop (stub)]

No leaks.