Comment 70 for bug 173350

It doesn't stick. But, rather than mess around with config files, I've got another work around. Put the command:
xmodmap -e "clear lock" -e "add lock = Caps_Lock"
into the sessions thingy (System-> Preferences -> Sessions -> Add, the name is whatever, the command is the one above etc.).

Also after switching through the GUI, but before running the xmodmap command, ctrl-alt-backspace has to use the 'real' ctrl key. ctrl-alt-arrow (to switch desktops) has to use the 'new' (formally caps) ctrl key.

After running the xmodmap command, the 'new' ctrl key works for everything.