Dimitri and I looked at some log files from when the retracers were dying and noticed that there were python processes OOM'ing on the retracers. We suspect it is the retracer processes themselves and I've added a MemoryLimit option to the retracer-$arch processes on the "regular" retracers. The failed-queue retracers will not have the MemoryLimit set and should be able to retrace anything the regular retracers can not. Additionally, my testing in staging (setting a very low MemoryLimit) indicates that the retracing process doesn't die so I think this'll resolve the cannot restart issue.
Dimitri and I looked at some log files from when the retracers were dying and noticed that there were python processes OOM'ing on the retracers. We suspect it is the retracer processes themselves and I've added a MemoryLimit option to the retracer-$arch processes on the "regular" retracers. The failed-queue retracers will not have the MemoryLimit set and should be able to retrace anything the regular retracers can not. Additionally, my testing in staging (setting a very low MemoryLimit) indicates that the retracing process doesn't die so I think this'll resolve the cannot restart issue.