i7-5820K (also) (4/6 cores (8/12 threads) being passed to VMs)
12GB RAM (also) (8GB being passed to VMs)
MSI X99 SLI Plus (though I don't use SLI)
NVidia GTX 960 2GB pass-thru (also had this problem on a GTX 660 before that died)
GT 740 host card, using nouveau when VMs are running
Oh, I should post my hardware:
i7-5820K (also) (4/6 cores (8/12 threads) being passed to VMs)
12GB RAM (also) (8GB being passed to VMs)
MSI X99 SLI Plus (though I don't use SLI)
NVidia GTX 960 2GB pass-thru (also had this problem on a GTX 660 before that died)
GT 740 host card, using nouveau when VMs are running
We have some pretty similar hardware there.