Comment 6 for bug 213149

For Peter Garrett :
I've found at last the solution for using pulseaudio even if X is down.
Well, everything was actually explained in the /etc/default/pulseaudio file :

---------------------------------------------------------------------
# Start the PulseAudio sound server in system mode.
# (enables the pulseaudio init script)
# System mode is not the recommended way to run PulseAudio as it has some
# limitations (such as no shared memory access) and could potentially allow
# users to disconnect or redirect each others audio streams. The
# recommend way to run PulseAudio is as a per-session daemon. For GNOME
# sessions you can install pulseaudio-esound-compat and GNOME will
# automatically start PulseAudio on login (if ESD is enabled in
# System->Preferences->Sound). For other sessions, you can simply start
# PulseAudio with "pulseaudio --daemonize".
# 0 = don't start, 1 = start
PULSEAUDIO_SYSTEM_START=0

# Prevent users from dynamically loading modules into the PulseAudio sound
# server. Dynamic module loading enhances the flexibilty of the PulseAudio
# system, but may pose a security risk.
# 0 = no, 1 = yes
DISALLOW_MODULE_LOADING=1
---------------------------------------------------------------

You have to make PulseAudio act as a Daemon instead as an esd instance of gnome-session.
So change PULSEAUDIO_SYSTEM_START=0 to PULSEAUDIO_SYSTEM_START=1
Everything works fine now ;)