Comment 3 for bug 1084451

This solved for me:

http://webcache.googleusercontent.com/search?q=cache:8Aa_a9O_FEsJ:www.linux-archive.org/debian-user/519202-xmodmap-settings-forgotten-lost-during-session.html+&cd=4&hl=en&ct=clnk&gl=it&lr=lang_en|lang_it&client=ubuntu

I edited the file which on my machine was found at:
~/.config/xfce4/panel/xkb-plugin-??.rc

and changed the 'never_modify_config' option to true. i then 'kill -9'
the xfce4-xkb-plugin process which respawned and reloaded the conf file.