Re-joing the cluster doesn't remove orphan objects

[sirio81]

Summary:
I'm working with 3 nodes.
A node leave the cluster for x time.
Vdi are deleted on the other node.
When joining back the cluster, the node doesn't remove the orphan objects.
It does it if metadata is remove before joining.

root@test006:~# dog node list
  Id Host:Port V-Nodes Zone
   0 192.168.10.4:7000 127 67807424
   1 192.168.10.5:7000 129 84584640
   2 192.168.10.6:7000 129 101361856

root@test006:~# dog vdi list
  Name Id Size Used Shared Creation time VDI id Copies Tag
  test4 0 5.0 GB 864 MB 0.0 MB 2014-09-10 09:44 fd2de3 3
  test1 0 5.0 GB 864 MB 0.0 MB 2014-09-10 09:43 fd32fc 3
  test3 0 5.0 GB 864 MB 0.0 MB 2014-09-10 09:44 fd3662 3
  test2 0 5.0 GB 864 MB 0.0 MB 2014-09-10 09:43 fd3815 3

root@test006:~# dog node info
Id Size Used Avail Use%
 0 216 GB 3.4 GB 213 GB 1%
 1 220 GB 3.4 GB 216 GB 1%
 2 220 GB 3.4 GB 216 GB 1%
Total 655 GB 10 GB 645 GB 1%
Total virtual image size 20 GB

root@test006:~# df -h /mnt/sheep/0
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-sheep0 220G 3.5G 217G 2% /mnt/sheep/0

(I kill node id 2 and remove the 3 of 4 vdis)

root@test005:~# dog node kill 2

root@test005:~# dog vdi delete test4
root@test005:~# dog vdi delete test3
root@test005:~# dog vdi delete test2

root@test005:~# dog vdi list
  Name Id Size Used Shared Creation time VDI id Copies Tag
  test1 0 5.0 GB 864 MB 0.0 MB 2014-09-10 09:43 fd32fc 3

root@test005:~# dog node info
Id Size Used Avail Use%
 0 216 GB 912 MB 215 GB 0%
 1 220 GB 912 MB 219 GB 0%
Total 436 GB 1.8 GB 434 GB 0%
Total virtual image size 5.0 GB

root@test005:~# df -h /mnt/sheep/0
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-sheepdog 220G 945M 219G 1% /mnt/sheep/0

(I insert back node id 2 and check the used space)

root@test006:~# script/run_sheep.sh

root@test006:~# dog node list
  Id Host:Port V-Nodes Zone
   0 192.168.10.4:7000 127 67807424
   1 192.168.10.5:7000 129 84584640
   2 192.168.10.6:7000 129 101361856

root@test006:~# dog vdi list
  Name Id Size Used Shared Creation time VDI id Copies Tag
  test1 0 5.0 GB 864 MB 0.0 MB 2014-09-10 09:43 fd32fc 3

root@test006:~# dog node info
Id Size Used Avail Use%
 0 216 GB 912 MB 215 GB 0%
 1 220 GB 912 MB 219 GB 0%
 2 218 GB 1.5 GB 216 GB 0%
Total 653 GB 3.3 GB 650 GB 0%

Total virtual image size 5.0 GB
root@test006:~# df -h /mnt/sheep/0
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-sheep0 220G 3.5G 217G 2% /mnt/sheep/0

(Notice the used sapce of vg00-sheep0 didn't vary and node info is showing
1.5GB that is nor 3.5G neither 912M, as it should be).

-------------------------------------

(I repeat the same stepd but this time i remove /var/lib/sheepdog before
re-joining the cluster)

root@test006:~/script# dog node list
  Id Host:Port V-Nodes Zone
   0 192.168.10.4:7000 127 67807424
   1 192.168.10.5:7000 129 84584640
   2 192.168.10.6:7000 129 101361856

root@test006:~/script# dog vdi list
  Name Id Size Used Shared Creation time VDI id Copies Tag
  test4 0 5.0 GB 864 MB 0.0 MB 2014-09-10 09:50 fd2de3 3
  test1 0 5.0 GB 864 MB 0.0 MB 2014-09-10 09:48 fd32fc 3
  test3 0 5.0 GB 864 MB 0.0 MB 2014-09-10 09:49 fd3662 3
  test2 0 5.0 GB 864 MB 0.0 MB 2014-09-10 09:49 fd3815 3

root@test006:~/script# dog node info
Id Size Used Avail Use%
 0 216 GB 3.4 GB 213 GB 1%
 1 220 GB 3.4 GB 216 GB 1%
 2 220 GB 3.4 GB 216 GB 1%
Total 655 GB 10 GB 645 GB 1%

Total virtual image size 20 GB
root@test006:~/script# df -h /mnt/sheep/0
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-sheep0 220G 3.5G 217G 2% /mnt/sheep/0

root@test005:~# dog node kill 2

root@test005:~# dog vdi delete test4
root@test005:~# dog vdi delete test3
root@test005:~# dog vdi delete test2

root@test005:~# dog node info
Id Size Used Avail Use%
 0 216 GB 912 MB 215 GB 0%
 1 220 GB 912 MB 219 GB 0%
Total 436 GB 1.8 GB 434 GB 0%
Total virtual image size 5.0 GB

root@test005:~# df -h /mnt/sheep/0
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-sheepdog 220G 945M 219G 1% /mnt/sheep/0

(re-join the cluster)

root@test006:~/script# rm -r /var/lib/sheepdog/*

root@test006:~/script# ./run_sheep.sh

root@test006:~/script# dog node info
Id Size Used Avail Use%
 0 216 GB 912 MB 215 GB 0%
 1 220 GB 912 MB 219 GB 0%
 2 220 GB 912 MB 219 GB 0%
Total 655 GB 2.7 GB 653 GB 0%
Total virtual image size 5.0 GB

root@test006:~/script# df -h /mnt/sheep/0
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-sheep0 220G 947M 219G 1% /mnt/sheep/0