Using Journal corrupts VDI in case of node lose.
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
sheepdog |
New
|
Undecided
|
Unassigned |
Bug Description
Cluster with 3 nodes (A,B,C), 1 VDI with copies 2:1. Guest running on node A
Killed en started node B and got the following errors in the sheep.log on that node:
Oct 03 16:18:00 NOTICE [main] cluster_
Oct 06 10:54:02 INFO [main] md_add_disk(338) /mnt/sheep/22, vdisk nr 467, total disk 1
Oct 06 10:54:02 ALERT [main] get_vdi_
Oct 06 10:54:02 ALERT [main] get_vdi_
Oct 06 10:54:02 ALERT [main] get_vdi_
....
Oct 06 10:54:02 ALERT [main] get_vdi_
Oct 06 10:54:02 ALERT [main] get_vdi_
Oct 06 10:54:02 ALERT [main] get_vdi_
Oct 06 10:54:02 INFO [main] send_join_
Oct 06 10:54:02 INFO [main] replay_
Oct 06 10:54:02 ERROR [main] replay_
Oct 06 10:54:02 EMERG [main] check_recover_
Oct 06 10:54:02 EMERG [main] crash_handler(267) sheep exits unexpectedly (Aborted).
Oct 06 10:54:03 EMERG [main] sd_backtrace(833) :
Oct 06 10:54:03 EMERG [main] sd_backtrace(833) :
Oct 06 10:54:03 EMERG [main] sd_backtrace(833) :
Oct 06 10:54:03 EMERG [main] sd_backtrace(833) :
On node A, the guest stopped responding and sheep.log gives the following errors:
Oct 03 16:18:00 NOTICE [main] cluster_
Oct 06 10:53:53 ERROR [gway 6564] wait_forward_
Oct 06 10:53:53 ERROR [gway 6567] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused
Oct 06 10:53:53 ERROR [gway 6568] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused
Oct 06 10:53:53 ERROR [gway 6567] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused
Oct 06 10:53:53 ERROR [gway 6568] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused
.......
Oct 06 10:54:00 ERROR [gway 6569] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused
Oct 06 10:54:00 ERROR [gway 6651] wait_forward_
Oct 06 10:54:00 ERROR [rw] read_erasure_
Oct 06 10:54:00 ERROR [rw] read_erasure_
Oct 06 10:54:00 ERROR [gway 6676] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused
Oct 06 10:54:00 ERROR [rw] read_erasure_
.....
Oct 06 10:54:02 ERROR [rw] read_erasure_
Oct 06 10:54:02 ERROR [rw] read_erasure_
Oct 06 10:54:03 ERROR [block] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused
Oct 06 10:54:03 ERROR [block] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused
Oct 06 10:54:03 ALERT [block] do_get_vdis(499) failed to get vdi bitmap from IPv4 ip:10.10.0.22 port:7000
Oct 06 10:54:03 ERROR [main] check_request_
Oct 06 10:54:03 ERROR [main] check_request_
Oct 06 10:54:03 ERROR [rw] sheep_exec_
Oct 06 10:54:03 ERROR [rw] recover_
Oct 06 10:54:03 ERROR [rw] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused
Oct 06 10:54:03 ERROR [rw] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused
Oct 06 10:54:03 ALERT [rw] fetch_object_
Oct 06 10:54:03 ALERT [rw] fetch_object_
Oct 06 10:54:03 ERROR [rw] sheep_exec_
Oct 06 10:54:03 ERROR [rw] sheep_exec_
......
Oct 06 10:54:05 ERROR [rw] sheep_exec_
Oct 06 10:54:05 ERROR [rw] sheep_exec_
Oct 06 10:54:05 INFO [main] recover_
Oct 06 10:54:05 ERROR [rw] sheep_exec_
Oct 06 10:54:05 ERROR [rw] sheep_exec_
......
Oct 06 10:55:07 ERROR [gway 6694] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused
Oct 06 10:55:07 ERROR [gway 6766] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused
Oct 06 10:55:07 INFO [main] recover_
Oct 06 10:55:07 ERROR [gway 6766] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused
Oct 06 10:55:07 ERROR [gway 6653] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused
.....
Oct 06 10:55:08 ERROR [gway 6764] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused
Oct 06 10:55:08 ERROR [gway 6653] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused
Oct 06 10:55:08 ERROR [rw] sheep_exec_
Oct 06 10:55:08 ERROR [rw] sheep_exec_
Oct 06 10:55:08 ERROR [gway 6694] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused
Oct 06 10:55:08 ERROR [gway 6698] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused
And after that continues with:
Oct 06 11:21:38 ERROR [gway 6766] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused
Oct 06 11:21:38 ERROR [gway 6766] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused
and the guest stays not responding. It looks like the object recovery never passes the 99%
And this process corrupted the VDI and can not be repaired with:
dog vdi check micha_test2
object f19e4f0000085d is inconsistent
97.1 % [======
97.2 % [======
97.5 % [======
98.9 % [======
99.4 % [======
100.0 % [======
finish check&repair micha_test2