He said he was using ext4 on irc.
I also think this is related to sqlite (like this liferea bug: https://bugs.launchpad.net/ubuntu/+source/liferea/+bug/290666)
Jim, could you do something like in that bug report:
# strace -etrace=fsync -o banshee_session.strace banshee 2>&1
When banshee is running try repeating what you usually do that makes banshee slow.
I have a feeling that banshee is writing to banshee.db when you change songs (to update how many times it was played) or when you rate a song, and this slows down your system because of excessive calls to fsync which on ext4 hurts overall performance.