hi, 
I have a similiar issue but no special config for my ssh:

the ubuntu clients (one is maverick, one is natty) connect via ssh key authentication to libvirtd server.
All servers are fedora, except of one ubuntu (oneiric). The connection to the feodora are all ok for the ubuntu i get:

as normal user:
virsh -c qemu+ssh://<email address hidden>/system list --all 
-> after entering the root pass: connected, the list appears

as root:
virsh -c qemu+ssh://<email address hidden>/system list --all
-> error:
error: Connection reset by peer
error: failed to connect to the hypervisor

debug shows:
15:36:04.233: debug : virInitialize:340 : register drivers
15:36:04.233: debug : virRegisterDriver:928 : registering Test as driver 0
15:36:04.233: debug : virRegisterNetworkDriver:734 : registering Test as network driver 0
15:36:04.233: debug : virRegisterInterfaceDriver:765 : registering Test as interface driver 0
15:36:04.233: debug : virRegisterStorageDriver:796 : registering Test as storage driver 0
15:36:04.233: debug : virRegisterDeviceMonitor:827 : registering Test as device driver 0
15:36:04.233: debug : virRegisterSecretDriver:858 : registering Test as secret driver 0
15:36:04.233: debug : virRegisterNWFilterDriver:889 : registering Test as network filter driver 0
15:36:04.233: debug : virRegisterDriver:928 : registering Xen as driver 1
15:36:04.233: debug : virRegisterDriver:928 : registering OPENVZ as driver 2
15:36:04.233: debug : virRegisterDriver:928 : registering remote as driver 3
15:36:04.233: debug : virRegisterNetworkDriver:734 : registering remote as network driver 1
15:36:04.233: debug : virRegisterInterfaceDriver:765 : registering remote as interface driver 1
15:36:04.233: debug : virRegisterStorageDriver:796 : registering remote as storage driver 1
15:36:04.233: debug : virRegisterDeviceMonitor:827 : registering remote as device driver 1
15:36:04.233: debug : virRegisterSecretDriver:858 : registering remote as secret driver 1
15:36:04.233: debug : virRegisterNWFilterDriver:889 : registering remote as network filter driver 1
15:36:04.233: debug : virConnectOpenAuth:1499 : name=qemu+ssh://<email address hidden>/system, auth=0xe7d778, flags=0
15:36:04.233: debug : do_open:1205 : name "qemu+ssh://<email address hidden>/system" to URI components:
  scheme qemu+ssh
  opaque (null)
  authority (null)
  server kavasir.XXXXXXXXXXX
  user rsgadmin
  port 0
  path /system

15:36:04.233: debug : do_open:1244 : trying driver 0 (Test) ...
15:36:04.233: debug : do_open:1250 : driver 0 Test returned DECLINED
15:36:04.233: debug : do_open:1244 : trying driver 1 (Xen) ...
15:36:04.233: debug : do_open:1250 : driver 1 Xen returned DECLINED
15:36:04.233: debug : do_open:1244 : trying driver 2 (OPENVZ) ...
15:36:04.233: debug : do_open:1250 : driver 2 OPENVZ returned DECLINED
15:36:04.233: debug : do_open:1244 : trying driver 3 (remote) ...
15:36:04.233: debug : doRemoteOpen:565 : proceeding with name = qemu:///system
15:36:04.234: debug : virExecWithHook:712 : ssh -l rsgadmin kavasir.XXXXXX  sh -c 'nc -q 2>&1 | grep -q "requires an argument";if [ $? -eq 0 ] ; then   CMD="nc -q 0 -U /var/run/libvirt/libvirt-sock";else   CMD="nc -U /var/run/libvirt/libvirt-sock";fi;eval "$CMD";'
15:36:04.234: debug : remoteIO:9888 : Do proc=66 serial=0 length=28 wait=(nil)
15:36:04.234: debug : remoteIO:9963 : We have the buck 66 0xb7634008 0xb7634008
15:36:04.310: error : remoteIOReadBuffer:9103 : Connection reset by peer
15:36:04.310: debug : remoteIOEventLoop:9835 : Giving up the buck due to I/O error 66 0xb7634008 (nil)
15:36:04.311: debug : do_open:1250 : driver 3 remote returned ERROR
15:36:04.311: debug : virUnrefConnect:294 : unref connection 0x9c4a918 1
15:36:04.311: debug : virReleaseConnect:249 : release connection 0x9c4a918

my problem:
- my scripts invoke sudo beforehand of virsh since the root user has passwordless login allowed to all servers. This works with all servers, not with ubuntu (see above, a root login issue)

I suspect either libvirtd or apparmor.d on kavasir (the server) to refuse my conection if started as root.

To confirm: my root user has no special config:

root@client:~/.ssh# ls
authorized_keys  id_dsa  id_dsa.pub  known_hosts

my ssh_config:
root@client:~/.ssh# cat /etc/ssh/ssh_config 

# This is the ssh client system-wide configuration file.  See
# ssh_config(5) for more information.  This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.

# Configuration data is parsed as follows:
#  1. command line options
#  2. user-specific file
#  3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.

# Site-wide defaults for some commonly used options.  For a comprehensive
# list of available options, their meanings and defaults, please see the
# ssh_config(5) man page.

Host *
#   ForwardAgent no
#   ForwardX11 no
#   ForwardX11Trusted yes
#   RhostsRSAAuthentication no
#   RSAAuthentication yes
#   PasswordAuthentication yes
#   HostbasedAuthentication no
#   GSSAPIAuthentication no
#   GSSAPIDelegateCredentials no
#   GSSAPIKeyExchange no
#   GSSAPITrustDNS no
#   BatchMode no
#   CheckHostIP yes
#   AddressFamily any
#   ConnectTimeout 0
#   StrictHostKeyChecking ask
#   IdentityFile ~/.ssh/identity
#   IdentityFile ~/.ssh/id_rsa
#   IdentityFile ~/.ssh/id_dsa
#   Port 22
#   Protocol 2,1
#   Cipher 3des
#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc
#   MACs hmac-md5,hmac-sha1,<email address hidden>,hmac-ripemd160
#   EscapeChar ~
#   Tunnel no
#   TunnelDevice any:any
#   PermitLocalCommand no
#   VisualHostKey no
#   ProxyCommand ssh -q -W %h:%p gateway.example.com
    SendEnv LANG LC_*
    HashKnownHosts yes
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials no
root@client:~/.ssh#