Performance issue with Object Cache

[Bastian Scholz]

For testing I start another sheep in gateway mode on
an additional disk on one of the nodes and enable
Object Cache for this gateway sheep.

But when enabling Object Cache, the rw performance
of random 4k rw drops to ~ 1/10th of the speed without
OC enabled.

Included a fio benchmark on the kvm guest with random
read/writes with 4k blocksize and the startup time of
the guest (node and guest is Debian/Wheezy)

Testcluster with 3 nodes, 3 obj disks each with sheepdog
md and copies set to 3

# sheep --version
Sheepdog daemon version 0.7.7

# qemu --version
QEMU emulator version 1.1.2 (Debian 1.1.2+dfsg-6a)

# virsh --version
0.9.12.3

qemu cache setting is none (managed by libvirt), but
it didnt make any noticeable difference when set to
writeback or writethrough...

# sheep /mnt/soeren/3/ -g -p 7003
# Guest startup time 3-4 seconds
# fio guest results 4k randrw
read : io=11612KB, bw=594415 B/s, iops=145 , runt= 20004msec
write: io=11804KB, bw=604243 B/s, iops=147 , runt= 20004msec

# sheep /mnt/soeren/3/ -g -p 7003 -w size=10G
#Guest startup time 15-17 seconds
# fio guest results 4k randrw
read : io=1172.0KB, bw=59934 B/s, iops=14 , runt= 20024msec
write: io=1256.0KB, bw=64230 B/s, iops=15 , runt= 20024msec

For comparison the fio results of the sheep
cache directory.

read : io=10576KB, bw=541355 B/s, iops=132 , runt= 20005msec
write: io=10712KB, bw=548317 B/s, iops=133 , runt= 20005msec