Comment 62 for bug 66483

tylerjwilk, basically it's the bug of pulseaudio: https://bugs.launchpad.net/ubuntu/+source/pulseaudio/+bug/190754
I worked around that by setting these values in /etc/pulse/daemon.conf:
default-fragments = 8
default-fragment-size-msec = 5

Another workaround that worked for me is using libsdl1.2debian-esd, but that may increase latency I guess.

For libsdl1.2debian-pulseaudio I filed another bug before I wasn't aware it may be problem of pulseaudio itself: https://bugs.launchpad.net/ubuntu/+source/libsdl1.2/+bug/188986