Not all results recieved when there is a lot of them (>=3000?)

[Paweł Stołowski]

It looks like ZMQ high water-mark limits the number of results that can succesfuly be received by a client to something around 2000; this may depend on the result size and delays between pushes though. With 3000 results in my test code I was receiving approximately 1900, the remainder was dropped on the floor. While these numbers are probably more than we will ever need, it may be worth looking into it. Perhaps ZMQ queue size can be increased, also a meaningful error in the logs about hitting such cases would be nice (if possible at all).

I'll add a pointer to a sample test case from unity-scopes-shell at a later time, please contact me if I forget to do so ;)

[Michi Henning (michihenning)]

Thanks for this! I doubt that we'll be able to log this. I don't think zmq provides the relevant info. The best we can hope for is not blowing the limit (within reason).

[Paweł Stołowski (stolowski)]

<pstolowski> michi, can we count the number of pushes on the sender, and carry this number with finished message? or isn't 'finished' a real message?
<michi> Finished is a real message.
<michi> I’m not sure I follow your reasoning. Can you explain?
<michi> We count the number of pushes from the sender. Easy. Then what?
<michi> Send it with the finished message?
<michi> I guess that would allow the receiver to detect an overrun.
<pstolowski> michi, yes, that's the idea
<michi> Hmmm...
<michi> It would work in principle for error detection.
<michi> I’m not sure how easy it would be to implement down at the zmq level.
<michi> Because it’s stateful but, at the zmq level, we are currently stateless.
<michi> Not impossible, but possibly ugly.
<michi> It’s worth thinking about though.

[Michi Henning (michihenning)]

I had a look at the high water mark for the outgoing socket, and it is at the default of zero, which, according to the zmq doc, means "no limit". This will need a closer look to figure out why messages are being lost.

[Paweł Stołowski (stolowski)]

Here is the branch https://code.launchpad.net/~stolowski/unity-scopes-shell/diff-updates-performance/+merge/284928 which adds a few tests for 2000 results (which works fine). After increasing them to do 3000 you should hopefuly see failures.