when a military building containing multiple soldiers is attacked, most soldiers go out to face the enemy. but at least one stay behind. or it should. sometimes, all the soldiers go out to face the enemy, leaivng no garrison behind, so the building can be conquered by the first enemy that passes. which can lead to the funny situation that a citadel with 12 level 10 soldiers is conquered by 13 level 0 soldiers.
This bug has existed a long time, and still it isn't fixed as of 7525. I haven't reported it before beause I didn't have documentation. Now I do, somewhat. As far as I can tell, the bug is not reproducible. you get attacked by lots of soldiers, and sometimes it happens. sometimes not. i atach a saved game where you can see a citadel being empty. I don't know how useful it will be, since the bug is in the moment the soldiers are sent out to intercept the enemy, but it's the best I can do.
I have seen that the soldier left behind is defeated by an enemy at the door. Before this enemy can conquer the building, he is challenged by another soldier, who kills him. Now, the military site is empty.
Your case seems different: All soldiers go far away from the building; there is no fight at the door. I have never seen that before.
A replay would possibly give more information.