Failure to serialize long numbers

[Eugene Morozov]

When trying to find memory leak in my Django application, meliae fails immediately with the following traceback:

ERROR: django.request: Internal Server Error: /play/dump/
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py", line 88, in dispatch
    return handler(request, *args, **kwargs)
  File "/opt/project/apps/play/views.py", line 238, in get
    meliae.scanner.dump_all_objects('/opt/project/heap.dump')
  File "/usr/local/lib/python2.7/site-packages/meliae/scanner.py", line 141, in dump_all_objects
    dump_all_referenced(outf, all_objs, is_pending=True)
  File "/usr/local/lib/python2.7/site-packages/meliae/scanner.py", line 72, in dump_all_referenced
    for ref in get_referents(next):
OverflowError: Python int too large to convert to C long

After compiling python with debug information and stepping through the code in gdb, I found out that this happens because meliae attempts to serialize Python long integers as C long. But Python long has an unlimited range and cannot be cast to C long.

I'm attaching a small patch that attempts to serialize Python long as C long long and returning -1 in case if Python long is too large for C long.

I was able to generate dump successfully after building meliae with this change.

[John A Meinel (jameinel)]

Can you add a test case for this? It should be pretty straight forward to
add a test of dumping a long that is bigger than 32bits and probably also
one for bigger than 64 bits. I'm not really sure how to represent numbers
that are >> 64 bits but having a stable answer at least would be good.

On Sun, May 17, 2020 at 2:35 AM Eugene Morozov <email address hidden>
wrote:

> ** Patch added: "meliae.patch"
>
> https://bugs.launchpad.net/meliae/+bug/1879100/+attachment/5372759/+files/meliae.patch
>
> --
> You received this bug notification because you are a member of Meliae
> Development Team, which is subscribed to Meliae.
> https://bugs.launchpad.net/bugs/1879100
>
> Title:
> Failure to serialize long numbers
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/meliae/+bug/1879100/+subscriptions
>

[Eugene Morozov (emorozov)]

I have added a trivial test case here:
https://github.com/emorozov/meliae/commit/eb50640e2bb14d8661fcff08d4a1e684c16cf3c1

[Eugene Morozov (emorozov)]

Patch with a trivial test.

[Eugene Morozov (emorozov)]

Attached a patch with a trivial test that attempts to serialize large integer.