This does not appear to be specific to OpenStack, nor tempest. I've reproduced with Trusty on Trusty on Trusty, vanilla qemu/kvm.
Simplified reproducer, with an existing MAAS cluster:
@L0 baremetal: - Create a Trusty bare metal host from daily images. - sudo apt-get update -y && sudo apt-get -y install uvtool - sudo uvt-simplestreams-libvirt sync release=trusty arch=amd64 - sudo uvt-simplestreams-libvirt query - ssh-keygen - sudo uvt-kvm create --memory 2048 trusty-vm release=trusty - sudo virsh shutdown trusty-vm - # edit the /etc/libvirt/qemu/trusty-vm.xml to enable serial console dump to file: <serial type='file'> <source path='/tmp/trusty-vm-console.log'/> <target port='0'/> </serial> <console type='file'> <source path='/tmp/trusty-vm-console.log'/> <target type='serial' port='0'/> </console> - sudo virsh define /etc/libvirt/qemu/trusty-vm.xml - sudo virsh start trusty-vm - # confirm console output: - sudo tailf /tmp/trusty-vm-console.log - # take note of the VM's IP: - sudo uvt-kvm ip trusty-vm - # ssh into the new vm.
@L1 "trusty-vm": - sudo apt-get update -y && sudo apt-get -y install uvtool - sudo uvt-simplestreams-libvirt sync release=trusty arch=amd64 - sudo uvt-simplestreams-libvirt query - ssh-keygen - # change .122. to .123. in /etc/libvirt/qemu/networks/default.xml - # make sure default.xml is static linked inside /etc/libvirt/qemu/networks - sudo reboot # for good measure - sudo uvt-kvm create --memory 768 trusty-nest release=trusty - # take note of the nested VM's IP - sudo uvt-kvm ip trusty-vm - # ssh into the new vm.
@L2 "trusty-nest": - sudo apt-get update && sudo apt-get install stress - stress -c 1 -i 1 -m 1 -d 1 -t 600
Now watch the "trusty-vm" console for: [ 496.076004] BUG: soft lockup - CPU#0 stuck for 23s! [ksmd:36]. It happens to me within a couple of minutes. Then, both L1 and L2 become unreachable indefinitely, with two cores on L0 stuck at 100%.
This does not appear to be specific to OpenStack, nor tempest. I've reproduced with Trusty on Trusty on Trusty, vanilla qemu/kvm.
Simplified reproducer, with an existing MAAS cluster:
@L0 baremetal: ms-libvirt sync release=trusty arch=amd64 ms-libvirt query qemu/trusty- vm.xml to enable serial console dump to file: tmp/trusty- vm-console. log'/> tmp/trusty- vm-console. log'/> qemu/trusty- vm.xml vm-console. log
- Create a Trusty bare metal host from daily images.
- sudo apt-get update -y && sudo apt-get -y install uvtool
- sudo uvt-simplestrea
- sudo uvt-simplestrea
- ssh-keygen
- sudo uvt-kvm create --memory 2048 trusty-vm release=trusty
- sudo virsh shutdown trusty-vm
- # edit the /etc/libvirt/
<serial type='file'>
<source path='/
<target port='0'/>
</serial>
<console type='file'>
<source path='/
<target type='serial' port='0'/>
</console>
- sudo virsh define /etc/libvirt/
- sudo virsh start trusty-vm
- # confirm console output:
- sudo tailf /tmp/trusty-
- # take note of the VM's IP:
- sudo uvt-kvm ip trusty-vm
- # ssh into the new vm.
@L1 "trusty-vm": ms-libvirt sync release=trusty arch=amd64 ms-libvirt query qemu/networks/ default. xml qemu/networks
- sudo apt-get update -y && sudo apt-get -y install uvtool
- sudo uvt-simplestrea
- sudo uvt-simplestrea
- ssh-keygen
- # change .122. to .123. in /etc/libvirt/
- # make sure default.xml is static linked inside /etc/libvirt/
- sudo reboot # for good measure
- sudo uvt-kvm create --memory 768 trusty-nest release=trusty
- # take note of the nested VM's IP
- sudo uvt-kvm ip trusty-vm
- # ssh into the new vm.
@L2 "trusty-nest":
- sudo apt-get update && sudo apt-get install stress
- stress -c 1 -i 1 -m 1 -d 1 -t 600
Now watch the "trusty-vm" console for: [ 496.076004] BUG: soft lockup - CPU#0 stuck for 23s! [ksmd:36]. It happens to me within a couple of minutes. Then, both L1 and L2 become unreachable indefinitely, with two cores on L0 stuck at 100%.