Client-initiated backups do not support large databases (95Mb+)

[Ferdinand]

Traceback (most recent call last):
  File "/usr/lib64/python2.5/threading.py", line 486, in __bootstrap_inner
    self.run()
  File "/home/terp/OpenERP/branch/server/5.0/bin/netsvc.py", line 400, in run
    ts.mysend(result)
  File "/home/terp/OpenERP/branch/server/5.0/bin/tiny_socket.py", line 56, in mysend
    sent = self.sock.send(msg[totalsent:])
error: (32, 'Broken pipe')

the error seems to happen during or after the transfer of the dump to the client.

BTW backup of other databases of this server work

in the problematic db some images are stored

[xrg (xrg)]

On Sunday 08 August 2010, you wrote:
> Public bug reported:
>
> Traceback (most recent call last):
> File "/usr/lib64/python2.5/threading.py", line 486, in __bootstrap_inner
> self.run()
> File "/home/terp/OpenERP/branch/server/5.0/bin/netsvc.py", line 400, in
> run ts.mysend(result)
> File "/home/terp/OpenERP/branch/server/5.0/bin/tiny_socket.py", line 56,
> in mysend sent = self.sock.send(msg[totalsent:])
> error: (32, 'Broken pipe')
> the error seems to happen during or after the transfer of the dump to
> the client.
> BTW backup of other databases of this server work
> in the problematic db some images are stored

Do a simple test, please:
using cmdline "pg_dump" try to dump each one of those databases and report
back their size to us... Is it some magic threshold your problematic db
passes?

I guess there is a limitation to the RPC data content length. FYI, we had
discussed in the company about developing a "large file" communications
channel, apart from XML-RPC or Net-RPC, for those chunks of data (including
ir.attachment transfers) that would pass the protocol limit.

[Ferdinand (office-chricar)]

size is 161.950.236 bytes

my adapted postgresql.conf
not sure if this is necessary - found the suggestions somewhere

archive_mode = off
checkpoint_segments = 10
datestyle = 'iso, mdy'
default_text_search_config = 'pg_catalog.english'
effective_cache_size = 256MB
lc_messages = 'en_US.UTF-8'
lc_monetary = 'en_US.UTF-8'
lc_numeric = 'en_US.UTF-8'
lc_time = 'en_US.UTF-8'
listen_addresses = '*'
log_destination = 'stderr'
log_directory = 'pg_log'
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'
logging_collector = on
log_line_prefix = '%t %d %u '
log_min_duration_statement = 0
log_statement = 'all'
log_statement_stats = on
maintenance_work_mem = 128MB
max_connections = 100
max_fsm_pages = 204800
max_stack_depth = 4MB
shared_buffers = 128MB
wal_buffers = 8MB
work_mem = 64MB

[Ferdinand (office-chricar)]

btw - as discussed elsewhere - the dump should stay on the server ..

[Albert Cervera i Areny - http://www.NaN-tic.com (albert-nan)]

A Diumenge, 8 d'agost de 2010, Ferdinand @ ChriCar va escriure:
> btw - as discussed elsewhere - the dump should stay on the server ..

I don't agree. If you want backups on the server, you can create a module for
that, but current functionality is ok for small installs in which users can
take care of backups themselves.

--
Albert Cervera i Areny
http://www.NaN-tic.com
OpenERP Partners
Mòbil: [phone]

http://twitter.com/albertnan
http://albert-nan.blogspot.com

[xrg (xrg)]

On Sunday 08 August 2010, you wrote:
> size is 161.950.236 bytes

hm... even 1.5*that is hardly at the 256MB mark..
Is it slow? could it be a timeout issue?

How large are the databases that /do/ work?

I am afraid, there is little chance that this would be solved now, because it
is most likely a core protocol issue. But its interesting to know our limits,
out of curiosity.

[Ferdinand (office-chricar)]

the others have about 7MB

[xrg (xrg)]

On Sunday 08 August 2010, you wrote:
> the others have about 7MB

There is a note at xml-rpc code that there is some 15MB mark, beyond which
some platforms (what is your client running on?) cannot perform
socket.read()..

http://bugs.python.org/issue792570

I may give that a try a little later.

[Ferdinand (office-chricar)]

OpenSuSE 11.3 - server is 11.0

[Vinay Rana (OpenERP) (vra-openerp)]

We Let the Framework team decide.

Thanks.

[Olivier Dony (Odoo) (odo-openerp)]

Ferdinand,

I could not reproduce this, so we'll need more info if possible.
Can you perhaps provide a dump of the smallest database you have that triggers the problem? (even with all data obfuscated, as long as it occurs)
Where did you get the title of this bug from? Is it some error you noticed on the client side and for which you could provide a detailed trace? Could you post it as well? (All we can see on server trace is that the connection to the client is closed, so perhaps client side can tell more)

Also, can you reproduce this both in XML-RPC and NET-RPC, or only NET-RPC?

The answers to these questions could help us track down the issue...

Thanks!

[Ferdinand (office-chricar)]

the db size (dump) of the db is now about 95MB

nothing in the server log

GTK - client
XMLRPC or NETRPC
Error
"Couldn't backup database."

perhaps this helps
*********************************
trace from koo using netrpc

Could not backup database.
Traceback (most recent call last):
  File "/home/terp/OpenERP/branch/server/5.0/bin/netsvc.py", line 247, in dispatch
    result = LocalService(service_name)(method, *params)
  File "/home/terp/OpenERP/branch/server/5.0/bin/netsvc.py", line 76, in __call__
    return getattr(self, method)(*params)
  File "/home/terp/OpenERP/branch/server/5.0/bin/service/web_services.py", line 200, in dump
    data = stdout.read()
MemoryError

*********************************
** koo using pyro
Could not backup database.
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/Koo/Rpc/Rpc.py", line 156, in call
    result = self.singleCall( obj, method, *args )
  File "/usr/lib/python2.6/site-packages/Koo/Rpc/Rpc.py", line 147, in singleCall
    result = self.proxy.dispatch( obj[1:], method, *encodedArgs )
  File "/usr/lib64/python2.6/site-packages/Pyro/core.py", line 384, in __call__
    return self.__send(self.__name, args, kwargs)
  File "/usr/lib64/python2.6/site-packages/Pyro/core.py", line 459, in _invokePYRO
    return self.adapter.remoteInvocation(name, Pyro.constants.RIF_VarargsAndKeywords, vargs, kargs)
  File "/usr/lib64/python2.6/site-packages/Pyro/protocol.py", line 439, in remoteInvocation
    return self._remoteInvocation(method, flags, *args)
  File "/usr/lib64/python2.6/site-packages/Pyro/protocol.py", line 500, in _remoteInvocation
    answer.raiseEx()
  File "/usr/lib64/python2.6/site-packages/Pyro/errors.py", line 72, in raiseEx
    raise self.excObj
MemoryError
 +--- This exception occured remotely (Pyro) - Remote traceback:
 | Traceback (most recent call last):
 | File "/usr/lib64/python2.5/site-packages/Pyro/protocol.py", line 669, in _handleInvocation2
 | res = obj.Pyro_dyncall(req[1],flags,req[3]) # (method,flags,args)
 | File "/usr/lib64/python2.5/site-packages/Pyro/core.py", line 108, in Pyro_dyncall
 | return getattr(self.delegate or self,method) (*args,**keywords)
 | File "/home/terp/OpenERP/branch/server/5.0/bin/addons/pyro.zip/pyro/pyro.py", line 46, in dispatch
 | result=method(*args)
 | File "/home/terp/OpenERP/branch/server/5.0/bin/service/web_services.py", line 211, in dump
 | return base64.encodestring(data)
 | File "/usr/lib64/python2.5/base64.py", line 316, in encodestring
 | return "".join(pieces)
 | MemoryError
 +--- End of remote traceback

[xrg (xrg)]

On Wednesday 22 December 2010, you wrote:
> the db size (dump) of the db is now about 95MB
>
Well, to me it makes no wonder why the current Net-RPC and XML-RPC
implementations both have some limits on the size of data members. Even if
they manage to transfer 100MB accross the wire, the size of the data would be
too much for the server's memory allocation.

As said a few months ago, I'd suggest to keep this issue in mind, open, and
come up with a "large data" protocol after v6.0 .
You can see in WebDAV that we have chunks-over-http support, already. We could
reuse part of that.

[Ferdinand (office-chricar)]

may be the export code can check for the DB dump file size and give a better hint. ...
it's likely that everyone who stores (larger) images will run into such (now known) problems

[xrg (xrg)]

On Wednesday 22 December 2010, you wrote:
> may be the export code can check for the DB dump file size and give a
> better hint. ... it's likely that everyone who stores (larger) images will
> run into such (now known) problems

Well, I have not figured out what the limit is so far. But I'm sure that all
protocols (Net-RPC, XML-RPC, Pyro ;) ) have some limit, perhaps depending on
other thinks like python version, platform etc.

Even if we allowed dbs < Limit to be dumped, that would be a dangerous
practice, because an admin would setup that and forget to check his backups
the day they cross the limit. It's all dictated by Murphy's Law.

So, my advice: Just don't use the client facility for backups. Use pg_dump or
continuous backup/replication at the server side.

and, by all means: always check your backups! The fact that you had left a
backup script running for months doesn't guarrantee you that your data will be
available, when you will eventually need them!

[Ferdinand (office-chricar)]

On Thursday 23 December 2010 you wrote:
> On Wednesday 22 December 2010, you wrote:
@xrg
thanks - (I know from previous discussions.)
you forgot to mention that attachments are NOT backed up using the client ..
For me it's always the question of (un)fulfilled expectations.
DB-Backup and some other menu items should do what they say .... ( or should
say so (as "uninstall - beta")
suggestion: call it "DB backup of small databases without attachments"

> > may be the export code can check for the DB dump file size and give a
> > better hint. ... it's likely that everyone who stores (larger) images will
> > run into such (now known) problems
>
> Well, I have not figured out what the limit is so far. But I'm sure that all
> protocols (Net-RPC, XML-RPC, Pyro ;) ) have some limit, perhaps depending on
> other thinks like python version, platform etc.
>
> Even if we allowed dbs < Limit to be dumped, that would be a dangerous
> practice, because an admin would setup that and forget to check his backups
> the day they cross the limit. It's all dictated by Murphy's Law.
>
> So, my advice: Just don't use the client facility for backups. Use pg_dump
or
> continuous backup/replication at the server side.
>
> and, by all means: always check your backups! The fact that you had left a
> backup script running for months doesn't guarrantee you that your data will
be
> available, when you will eventually need them!
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/614937
>
> Title:
> backup database unpickling stack underflow
>
> Status in OpenObject Server:
> Incomplete
>
> Bug description:
> Traceback (most recent call last):
> File "/usr/lib64/python2.5/threading.py", line 486, in __bootstrap_inner
> self.run()
> File "/home/terp/OpenERP/branch/server/5.0/bin/netsvc.py", line 400, in
run
> ts.mysend(result)
> File "/home/terp/OpenERP/branch/server/5.0/bin/tiny_socket.py", line 56,
in mysend
> sent = self.sock.send(msg[totalsent:])
> error: (32, 'Broken pipe')
>
> the error seems to happen during or after the transfer of the dump to the
client.
>
> BTW backup of other databases of this server work
>
> in the problematic db some images are stored
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/openobject-server/+bug/614937/+subscribe
>

--
regards
Ferdinand Gassauer
ChriCar Beteiligungs- und Beratungs- GmbH
Official OpenERP Partner

[Olivier Dony (Odoo) (odo-openerp)]

Thanks for the additional feedback!
Marking as confirmed wishlist to be improved in future R&D sprints