If the delay increases over time then mathematically it sounds like the output is consuming bits slower than the input. So /maybe/ you have 48kHz audio being played at 44.1kHz but I would have thought that would be noticeable (sounding distorted - too low). And even if that's not the problem, the delay should hit a limit via some buffering limit, after which it turns into skipping (bug 405294).
Another thought is to see the suggestion in bug 1677202.
If the delay increases over time then mathematically it sounds like the output is consuming bits slower than the input. So /maybe/ you have 48kHz audio being played at 44.1kHz but I would have thought that would be noticeable (sounding distorted - too low). And even if that's not the problem, the delay should hit a limit via some buffering limit, after which it turns into skipping (bug 405294).
Another thought is to see the suggestion in bug 1677202.