After failed job redis at 100%, celery queue not draining

[Muharem Hrnjadovic]

Observed this morning on the model facility cluster:

    1 - job with hazard calculation failures aborts and is terminated
    2 - redis load goes up and it uses 100% of cpu time
    3 - the worker machines seem idle
    4 - the remaining task messages of the dead job are *not* drained from the celery queue

A job started subsequently appears hung but redis is not responsive and likely the root cause of the problem.

[Muharem Hrnjadovic (al-maisan)]

The can probably only be reproduced on the model facility cluster.

[Muharem Hrnjadovic (al-maisan)]

I managed to capture an strace log of redis in that situation. Please see attachment.