John A Meinel wrote:
> Michael Hudson wrote:
>> John A Meinel wrote:
>>> Obviously the bzr version looks like it includes the latest memory
>>> improvements. Are we sure that all of the extensions got compiled
>>> properly? I wouldn't be surprised if memory consumption was >= before if
>>> _static_tuple_c.c didn't get compiled properly. There were more fixes
>>> than just that, which should have reduced memory consumption.
>> It looks like everything got built correctly:
>> https://pastebin.canonical.com/25447/
> 
> This doesn't look right to me:              vv
> -rwxr-xr-x  1 ...   285 ... _simple_set_pyx.py*
> -rw-r--r--  1 ...   699 ... _simple_set_pyx.pyc
> -rw-r--r--  1 ...   699 ... _simple_set_pyx.pyo
> -rwxr-xr-x  1 ... 99534 ... _simple_set_pyx.so*
> 
> 
> On my system I see:
> $ ls bzrlib/_simple_set_pyx*
> 
> bzrlib/_simple_set_pyx.c
> bzrlib/_simple_set_pyx.pxd
> bzrlib/_simple_set_pyx.pyx
> bzrlib/_simple_set_pyx.h
> bzrlib/_simple_set_pyx.pyd
> bzrlib/_simple_set_pyx_api.h
> 
> 
> Similarly here:
> -rwxr-xr-x  1 ...   285 ... _static_tuple_c.py*
> -rw-r--r--  1 ...   699 ... _static_tuple_c.pyc
> -rw-r--r--  1 ...   699 ... _static_tuple_c.pyo
> -rwxr-xr-x  1 ... 56062 ... _static_tuple_c.so*
> -rwxr-xr-x  1 ...  1557 ... static_tuple.py*
> -rw-r--r--  1 ...  1301 ... static_tuple.pyc
> -rw-r--r--  1 ...  1301 ... static_tuple.pyo
> -rwxr-xr-x  1 ...  2760 ... _static_tuple_py.py*
> -rw-r--r--  1 ...  3529 ... _static_tuple_py.pyc
> -rw-r--r--  1 ...  3529 ... _static_tuple_py.pyo
> 
> 
> I have 0 idea what is creating a 285 byte .py file for a compiled
> extension.

This is what it contains: https://pastebin.canonical.com/25455/  Which
is a bit strange.

> However, it could very easily cause problems using the .so...

I got mbarnett to run lsof -p on an lp-serve process, it looks from
https://pastebin.canonical.com/25454/ like the .so's are loaded.

>>> SInce in python2.4 allocations of
>>> "integer" objects are never released back to the system. And I would
>>> offer that there are probably some other memory allocation improvements
>>> in python 2.5 and newer.
>>>
>>> Also, is there any way to probe what actual command is running on a
>>> process that hits 1GB of RAM? (What branch is it accessing? What action
>>> is it performing, etc.)
>> No :(  Unless bzr serve does some of this.  Is there a server side
>> equivalent of -Dhpss?
> 
> No, and it has been something we wanted to get for the lp guys. I think
> jml was originally wanting more than that. (It is one thing to be able
> to tell that 'get_stream' was requested, it is another to know that this
> is caused by 'bzr branch', versus 'bzr pull', versus ...)

Well, let's not let the perfect get in the way of the good.  I'd just be
happy to know which branch was being accessed.

>>> I wonder if it would be useful to hook up a SIGHUP handler that would
>>> dump the current stack trace into .bzr.log. Do you think it would help
>>> debugging this? (It seems like something that we could do in a small
>>> plugin if it would make it easier.)
>> It would be nice if we could have each lp-serve process not dump
>> everything into the same .bzr.log file I guess, but otherwise that
>> sounds helpful.
> 
>> Cheers,
>> mwh
> 
> 
> It at least sounds better than the current state. :)

Yep.

Cheers,
mwh