Memory leak on get_metadata(formatted=True)

[Kota Tsuyuzaki]

pyeclib leaks the memory (in particular the memory for the item in the returning dict) when we call ECDriver.get_metadata(metadata, formatted=True). That is because the method doesn't decrease the reference counter for the items in the returned dict. The fix is very like with the issue [1],

Note that affected calling format is *with formatted=True* so ECDriver.get_metadata(metadata) is safe from the memory leak.

The snippet is attached file.

1: https://bugs.launchpad.net/pyeclib/+bug/1604335

[Kota Tsuyuzaki (tsuyuzaki-kota)]

Fix merged at https://review.openstack.org/#/c/405710/

However, I found the test still failed in my local environment and it's real.

With recent IRC discussion with Tim and Clay [1] and the place we didn't free the alloc-ed memory found.

The fix Clay suggested is [2] so keeping this open until we merged the fix.

1: http://eavesdrop.openstack.org/irclogs/%23openstack-swift/%23openstack-swift.2016-12-02.log.html#t2016-12-02T01:30:11
2: https://gist.github.com/clayg/f3055f975d2288e164cba57b824cc579

[Kota Tsuyuzaki (tsuyuzaki-kota)]

For snippet, the python c api reference is https://docs.python.org/2.7/c-api/arg.html#c.Py_BuildValue and it says as

When memory buffers are passed as parameters to supply data to build objects, as for the s and s# formats, the required data is copied. Buffers provided by the caller are never referenced by the objects created by Py_BuildValue(). In other words, if your code invokes malloc() and passes the allocated memory to Py_BuildValue(), your code is responsible for calling free() for that memory once Py_BuildValue() returns.

[Kota Tsuyuzaki (tsuyuzaki-kota)]

Add small info for this,

Interestingly, we cannot find this bug in xenial we're using in the gate job. However, my testing environment in trusty still get failed.

Looking at python 2.7 version, in trusty, the version is 2.7.6 and 2.7.12 in xenial so that some spec may be changed in the versions? Not sure.

[Kota Tsuyuzaki (tsuyuzaki-kota)]

Fix proposed at https://review.openstack.org/#/c/407302/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407302
Committed: https://git.openstack.org/cgit/openstack/pyeclib/commit/?id=288d1e07de0f3b525ea7ae5b7c7fd67d6187756f
Submitter: Jenkins
Branch: master

commit 288d1e07de0f3b525ea7ae5b7c7fd67d6187756f
Author: Tim Burke <email address hidden>
Date: Tue Dec 6 03:38:27 2016 +0000

    Fix checksum memory leak

    Since we allocate the result buffer in hex_encode_string, we're
    responsible for freeing it later.

    Why didn't we catch it before? Turns out, our tests didn't
    loop enough. I wasn't reliably seeing a test failure at 200k
    iterations, but 400k seems to be enough?

    Closes-Bug: #1634006

    Co-Authored-By: Clay Gerrard <email address hidden>
    Related-Change: I678e10008c3c5bc04640f7f19498334d94cb0cd6
    Change-Id: I0f6e922ba25ad56142f3d095896b9856a436a81c