The problem is caused by the "official" xfce4 keyboard layout settings manager storing the preferences
in the xfconf registry (you can find them using the Settings Editor under "keyboard-layouts")
while the xkb-panel-plugin stores its preferences in xkb using libxklavier.
The two settings managers are unaware of each other and so they keep destroying each others work.
I think the best solution would be to add the missing options to the official xfce4 keyboard layout settings dialog
(the missing options are layout switching key and layout indicator) and make the xkb-panel-plugin use the same xfconf registry entry and open the official settings manager as its preferences.
I am using this thread as my bug investigation blog in hope somebody will actually fix it eventually...