Hi, 

I have pretty much this exact same fault with Ubuntu Gusty, as soon as a client (another Ubuntu Gusty machine) connects to the server, Xorg uses near 100% CPU and performance is terrible - the server machine grinds to a halt, and the client machine's remote desktop is sluggish.

The machines are two laptops, the "server" simply grinds to a halt as soon as the "client" connects via vncviewer.

The problem defintely appears related to vino-server : As soon as it starts handling a client request, the Xorg CPU  utilization goes through the roof.

Here is before/after top :
BEFORE :
top - 22:07:37 up  4:04,  3 users,  load average: 0.08, 0.04, 0.15
Tasks: 104 total,   2 running, 102 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    450300k total,   439256k used,    11044k free,     4368k buffers
Swap:   859436k total,    34704k used,   824732k free,   258032k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
    1 root      15   0  2952 1852  532 S  0.0  0.4   0:01.57 init               
    2 root      14  -5     0    0    0 S  0.0  0.0   0:00.00 kthreadd           
    3 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/0        
    4 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0        
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0         
    6 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 events/0           
    7 root      10  -5     0    0    0 S  0.0  0.0   0:00.02 khelper            
   26 root      10  -5     0    0    0 S  0.0  0.0   0:00.12 kblockd/0          
   27 root      10  -5     0    0    0 S  0.0  0.0   0:01.86 kacpid             
   28 root      10  -5     0    0    0 S  0.0  0.0   0:00.03 kacpi_notify       
  111 root      11  -5     0    0    0 S  0.0  0.0   0:00.01 kseriod            
  128 root      15   0     0    0    0 S  0.0  0.0   0:00.04 pdflush            
  129 root      15   0     0    0    0 S  0.0  0.0   0:00.49 pdflush            
  130 root      10  -5     0    0    0 S  0.0  0.0   0:00.78 kswapd0            
  181 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 aio/0              
 2029 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 ksuspend_usbd      
 2030 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 khubd              

AFTER client connection with vncviewer (top run on server)
top - 22:10:06 up  4:07,  3 users,  load average: 0.31, 0.10, 0.15
Tasks: 104 total,   4 running, 100 sleeping,   0 stopped,   0 zombie
Cpu(s): 63.1%us, 14.6%sy,  0.0%ni, 19.6%id,  0.0%wa,  1.0%hi,  1.7%si,  0.0%st
Mem:    450300k total,   436764k used,    13536k free,     4440k buffers
Swap:   859436k total,    34704k used,   824732k free,   258156k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 4982 root      16   0  191m  30m 7232 R 64.8  6.8  54:28.34 Xorg               
15658 steveh    15   0 32736  10m 6112 S 14.3  2.5   1:54.88 vino-server        
16087 steveh    15   0  2368 1148  876 R  0.3  0.3   0:00.09 top                
    1 root      15   0  2952 1852  532 S  0.0  0.4   0:01.57 init               
    2 root      14  -5     0    0    0 S  0.0  0.0   0:00.00 kthreadd           
    3 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/0        
    4 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0        
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0         
    6 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 events/0           
    7 root      10  -5     0    0    0 S  0.0  0.0   0:00.02 khelper            
   26 root      10  -5     0    0    0 S  0.0  0.0   0:00.12 kblockd/0          
   27 root      10  -5     0    0    0 S  0.0  0.0   0:01.87 kacpid             
   28 root      10  -5     0    0    0 S  0.0  0.0   0:00.03 kacpi_notify       
  111 root      11  -5     0    0    0 S  0.0  0.0   0:00.01 kseriod            
  128 root      15   0     0    0    0 S  0.0  0.0   0:00.04 pdflush            
  129 root      15   0     0    0    0 S  0.0  0.0   0:00.49 pdflush            
  130 root      10  -5     0    0    0 S  0.0  0.0   0:00.78 kswapd0            

Can I try anything else to help debug this?