CNI daemon workers indefinitely waiting for locks

[Yash Gupta]

Rarely, a CNI worker is not able to acquire a lock (For updating/reading the pod registry), and keeps waiting for the lock.
However, the CNI runner times out and kubelet sends more requests to the CNI. These requests spawn more workers which are waiting on the same lock.
This soon results in growing number of workers waiting for the same lock, and exhausting kuryr-cni pod resources (workers, memory, etc)

However, it could not be determined why the deadlock happened in the first place.