update-pkgcache using too much memory on staging

[Tom Haddon]

I've had to disable the update-pkgcache script on staging as it's using too much memory consistently. Was just seen using 3.5GB of RSS.

[Tom Haddon (mthaddon)]

Er, where "too much" means forcing the server into swap death...

[Stuart Bishop (stub)]

How does this compare to production?

The trigger might have been the change to Storm 0.14, which we might be able to fix by tweaking this script's Storm cache settings.

[Julian Edwards (julian-edwards)]

If that turns out to be the case, we'll need to tweak quite a few other Soyuz scripts before letting them use Storm 0.14 in production.

[Steve McInerney (spm)]

"How does this compare to production?"

the last two runs (15th and 16th of July '09) have been around 1.5Gb RSS each.

[Stuart Bishop (stub)]

Assigning to this milestone - at a minimum we need to confirm this is not a release blocker.

[Julian Edwards (julian-edwards)]

Herb ran a test on staging and with the cache set to 500:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
1000 26950 13.9 11.2 1196480 905636 pts/3 Rl+ 16:40 10:12 /usr/bin/python2.4 cronscripts/update-pkgcache.py

So it's a large improvement.

[Stuart Bishop (stub)]

On Tue, Jul 21, 2009 at 11:56 PM, Julian
Edwards<email address hidden> wrote:
> Herb ran a test on staging and with the cache set to 500:
>
> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> 1000     26950 13.9 11.2 1196480 905636 pts/3  Rl+  16:40  10:12 /usr/bin/python2.4 cronscripts/update-pkgcache.py
>
> So it's a large improvement.

I find that surprising, but it is good news. Are you dealing with any
objects with large text fields or similar?

We should reduce the cache size in the [database] section in
lib/canonical/config/schema-lazr.conf. The appservers already override
this value in the [launchpad] section, so will continue to use the
value we have been testing on edge for the last few weeks. Everything
else will get the reduced cache value, which is good because they have
had less testing and I think it is better to tweak values up for slow
scripts as necessary rather than risk having systems go down because
of high memory use.

--
Stuart Bishop <email address hidden>
http://www.stuartbishop.net/

[Julian Edwards (julian-edwards)]

> I find that surprising, but it is good news. Are you dealing with any
> objects with large text fields or similar?

Yes, descriptions and whatnot.

[Diogo Matsubara (matsubara)]

Fixed in db r8313.