Comment 11 for bug 812940

libpulse0 depends on libxcb1, which pull other x11 libraries anyway. But if you don't want whole Xorg, then you can run pulseaudio as system mode (/etc/default/pulseaudio), which does not require X11 and dbus user session.