Downgrading to Nvidia driver release 169.04 fixed this problem for me. I was not experiencing this problem under Gutsy, but only started experiencing it under Hardy. I'm pretty confident that it's an Nvidia driver issue if downgrading from 169.12 (Ubuntu's version) to 169.04 (Nvidia's version) fixed the problem.
Downgrading to Nvidia driver release 169.04 fixed this problem for me. I was not experiencing this problem under Gutsy, but only started experiencing it under Hardy. I'm pretty confident that it's an Nvidia driver issue if downgrading from 169.12 (Ubuntu's version) to 169.04 (Nvidia's version) fixed the problem.