Comment 29 for bug 292588

Hmm, some looking around has made me think this is an error in the qemu-kvm packaging.
Removing qemu-kvm should remove the qemu-kvm upstart job (which actually loads the module).
Qemu can run without kvm, but then it isn't using hardware virtualization. This could be a solution for people only using qemu sporadically.

People who want to keep the option of hardware qemu could modify /etc/init/qemu-kvm.conf
changing
start on runlevel [2345]
to
#start on runlevel [2345]
effectively disabling this line.
Probably running
sudo initctl reload
afterwards to reload the settings.

This should make the kvm module loading manual, with modprobe or with
sudo stop qemu-kvm
and
sudo start qemu-kvm