Comment 8 for bug 295136

Well, my guess goes like this:

According to the default "/etc/default/pulseaudio.conf" pulseaudio is only started when somebody has logged in.

If pulseaudio is the default sound sink and you are logged into X, amixer will find the pulseaudio server, get the information it needs and be done with it quickly.

If pulseaudio is the default sound sink and you are NOT logged into X, amixer will search the pulseaudio server but there is none running on the local machine. Therefore, the network will be searched for a pulseaudio server. Most likely there is none on the local network either, and network timeouts are set to high values, so this may take some time and is the symptom we are seeing.

If you uninstall pulseaudio, it is no longer the default sink, and alsa will provide the information for amixer. Alsa does not search the network, so it will be quick to respond.