We're currently only able to process about three core files per minute with each retracer. We can fire up more retracers and drop some core files at random when we're approaching a high load, both of which mitigate this problem to varying degrees. However, we should also address the problem that our current retracing code is taking far too long to process each core file:
[4:40pm] jjo: ev: basically afaics we are losing the producer|consumer rate battle
[4:40pm] mthaddon: ev: basically the rabbitmq queues are fairly consistently increasing in size, but the server itself is very lightly loaded - should we be firing up more retracers, or is there some way of making existing retracers do more work?
[4:40pm] jjo: ev: while finfolk doesnt get squeezed for load
[4:41pm] ev: the retracers process in serial, so bringing up more of them would be advisable
[4:41pm] jjo: ev: FTR I had fired up to ~7 retracers in parallel , which hover'd loadavg~=8 -> got ~3 oopsen per min
[4:41pm] jjo: ev: but still I couldn't manage to get more than ~3/min