Double-free in SmartscopesClient

[Michi Henning]

I just got this out of the blue on a test run:

12/37 Test #12: JsonNode .............................. Passed 0.01 sec
      Start 13: SmartScopesClient
*** Error in `/home/michi/src/abstract-reply/build/test/gtest/scopes/internal/smartscopes/SmartScopesClient/SmartScopesClient_test': double free or corruption (fasttop): 0x00002af658002860 ***

I found a core in the test directory but, after looking with gdb, there was no usable information in the core.

I started running the SmartScopesClient_test in a loop and got another failure after about 50 iterations or so. This time a segfault. Again, no useful info in the core. But, from "file core", I get:

core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'python /home/michi/src/abstract-reply/test/gtest/scopes/internal/smartscopes/Sm'

Here is the test output near the failure (earlier output looks normal):

[ RUN ] SmartScopesClientTest.consecutive_searches
127.0.0.1 - - [20/Dec/2013 16:18:25] "GET /smartscopes/v2/search/demo?query=%22stuff%22&session_id=%221234%22&query_id=0&platform=%22%22 HTTP/1.1" 200 324
Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
    self.finish_response()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
    self.write(data)
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 212, in write
    self.send_headers()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 270, in send_headers
    self.send_preamble()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 194, in send_preamble
    'Date: %s\r\n' % format_date_time(time.time())
  File "/usr/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
127.0.0.1 - - [20/Dec/2013 16:18:25] "GET /smartscopes/v2/search/demo?query=%22stuff%22&session_id=%221234%22&query_id=0&platform=%22%22 HTTP/1.1" 500 59
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 651, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 710, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 279, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
127.0.0.1 - - [20/Dec/2013 16:18:25] "GET /smartscopes/v2/search/demo?query=%22stuff%22&session_id=%221234%22&query_id=0&platform=%22%22 HTTP/1.1" 200 324
failed to retrieve search results for session: 1234
error:unity::LogicException: No search for session 1234 is active
[ OK ] SmartScopesClientTest.consecutive_searches (593 ms)
[----------] 3 tests from SmartScopesClientTest (1811 ms total)

[----------] Global test environment tear-down
[==========] 3 tests from 1 test case ran. (1811 ms total)
[ PASSED ] 3 tests.
Running main() from gtest_main.cc
[==========] Running 3 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 3 tests from SmartScopesClientTest
[ RUN ] SmartScopesClientTest.remote_scopes
127.0.0.1 - - [20/Dec/2013 16:18:26] "GET /smartscopes/v2/remote-scopes HTTP/1.1" 200 237
[ OK ] SmartScopesClientTest.remote_scopes (623 ms)
[ RUN ] SmartScopesClientTest.search
127.0.0.1 - - [20/Dec/2013 16:18:26] "GET /smartscopes/v2/search/demo?query=%22stuff%22&session_id=%221234%22&query_id=0&platform=%22%22 HTTP/1.1" 200 324
[ OK ] SmartScopesClientTest.search (594 ms)
[ RUN ] SmartScopesClientTest.consecutive_searches
127.0.0.1 - - [20/Dec/2013 16:18:27] "GET /smartscopes/v2/search/demo?query=%22stuff%22&session_id=%221234%22&query_id=0&platform=%22%22 HTTP/1.1" 200 324
Segmentation fault (core dumped)

[Michi Henning (michihenning)]

I ran this with valgrind too, but didn't get any complaints. It's definitely a race condition of some kind. Possibly related to the issues reported by thread sanitizer?

[PS Jenkins bot (ps-jenkins)]

Fix committed into lp:unity-scopes-api at revision 108, scheduled for release in unity-scopes-api, milestone Unknown

[Launchpad Janitor (janitor)]

This bug was fixed in the package unity-scopes-api - 0.2.0+14.04.20140120-0ubuntu1

---------------
unity-scopes-api (0.2.0+14.04.20140120-0ubuntu1) trusty; urgency=low

  [ Michi Henning ]
  * Added ability for scope to push an exception (as an exception_ptr). On the client side,
    the exception is delivered as the what() string (if the exception is a std::exception) and
    as "unknown exception", otherwise.
  * Disabled running the tests in parallel for CI. They are not written
    to run concurrently because the different tests try to bind to the
    same network endpoints. This should almost certainly fix the failure
    reported here:
    https://launchpadlibrarian.net/159997415/buildlog_ubuntu-saucy-
    i386.unity-scopes-api_0.1.5-0~94~ubuntu13.10.1_FAILEDTOBUILD.txt.gz.
  * Got rid of chatter from helgrind. (Needs helgrind 3.9.0 or later--
    3.8.1 generates bogus errors.) Added public destroy() method so it
    is possible to shut down the reaper explicitly. Fixed race condition
    on setting the self_ weak_ptr.
  * Changed code to match macro change here:
    https://code.launchpad.net/~michihenning/unity-api/noncopyable-
    fix/+merge/200084 Also removed remaining remnants of deprecated
    NonCopyable class.
  * Added CMake option to build with -fsanitize=thread. Fixed a bunch of
    warnings when building with clang.
  * Fixed valgrind error in HttpClient_test.cpp: execv was using
    deallocated memory. Updated CTestCustom.cmake.in to correctly
    suppress the tests that should not be run under valgrind.
  * Fixed shutdown in ObjectAdapter and added proper exception handling
    if the broker thread or a worker thread encounters an exception.
  * Disabled code generation for stand-alone header compilation tests.
    This makes the tests run marginally faster.
  * Added locate() remote method to the Registry, for the scope
    activation logic.
  * Changed ScopeLoader to not run the scope in a separate thread. (This
    is done by scoperunner anyway.) This fixes the problem of an
    exception being thrown by start(), but the scoperunner not realizing
    this. (LP: #1262536)
  * Added suppressions file for thread sanitizer issues in zmq.
  * Fixed race on registry start-up: registry was answering incoming
    requests too early. Other minor cosmetic changes: replaced
    factory()->create() call with factory()->find() because RuntimeImpl
    already instantiates the middleware. Removed redundant #includes.
    (LP: #1267026)
  * Minor fix: got rid of unnecessary unique_ptr. No functional changes.
  * Added formatcode target to CMakeLists.txt. This runs all source and
    header files through a pretty-printer.
  * Updated code for API changes in latest Cap'n Proto. (LP: #1268538)
  * Added check to prevent two servers from binding to the same ipc
    endpoint. Added coverage test for double-bind. Removed attempts to
    deal with servant destructors that throw. It turns out that this
    doesn't work because the destructor is called in the context of the
    map destructor, which calls the shared_ptr destructors, so we end up
    in terminate(). I might theoretically be possible to do this with a
    custom deleter for the...