Fuel version info (8.0 build #200): http://paste.openstack.org/show/479824/
Nova instance creation fails if KVM is enabled in environment settings, because AppArmor template is missing in libvirt package:
2015-11-23 22:55:21.804 14044 ERROR nova.scheduler.utils [req-ec675d42-e355-403e-8228-aa28c4aa73c6 6ed121a8a6794dc7acb79c47de3c85ad ae864db73c9b4c7ab6de17fefcf57751 - - -] [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] Error from last host: node-2.test.domain.local (node node-2.test.domain.local): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1905, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2057, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance 52f9d275-f6af-4006-87a6-e2d420be9d0b was re-scheduled: internal error: cannot load AppArmor profile 'libvirt-52f9d275-f6af-4006-87a6-e2d420be9d0b'\n"]
2015-11-23 22:55:33.748 14044 ERROR nova.scheduler.utils [req-ec675d42-e355-403e-8228-aa28c4aa73c6 6ed121a8a6794dc7acb79c47de3c85ad ae864db73c9b4c7ab6de17fefcf57751 - - -] [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] Error from last host: node-3.test.domain.local (node node-3.test.domain.local): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1905, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2057, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance 52f9d275-f6af-4006-87a6-e2d420be9d0b was re-scheduled: internal error: cannot load AppArmor profile 'libvirt-52f9d275-f6af-4006-87a6-e2d420be9d0b'\n"]
2015-11-23 22:55:33.801 14044 WARNING nova.scheduler.utils [req-ec675d42-e355-403e-8228-aa28c4aa73c6 6ed121a8a6794dc7acb79c47de3c85ad ae864db73c9b4c7ab6de17fefcf57751 - - -] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [req-ec675d42-e355-403e-8228-aa28c4aa73c6 6ed121a8a6794dc7acb79c47de3c85ad ae864db73c9b4c7ab6de17fefcf57751 - - -] [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] Instance failed to spawn
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] Traceback (most recent call last):
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2155, in _build_resources
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] yield resources
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2009, in _build_and_run_instance
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] block_device_info=block_device_info)
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2444, in spawn
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] block_device_info=block_device_info)
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4519, in _create_domain_and_network
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] xml, pause=pause, power_on=power_on)
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4449, in _create_domain
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] guest.launch(pause=pause)
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 141, in launch
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] self._encoded_xml, errors='ignore')
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] six.reraise(self.type_, self.value, self.tb)
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 136, in launch
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] return self._domain.createWithFlags(flags)
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] result = proxy_call(self._autowrap, f, *args, **kwargs)
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] rv = execute(f, *args, **kwargs)
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] six.reraise(c, e, tb)
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] rv = meth(*args, **kwargs)
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1033, in createWithFlags
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] libvirtError: internal error: cannot load AppArmor profile 'libvirt-52f9d275-f6af-4006-87a6-e2d420be9d0b'
2015-11-23 22:55:19.349 22745 ERROR nova.compute.manager [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b]
2015-11-23 22:55:19.353 22745 INFO nova.compute.manager [req-ec675d42-e355-403e-8228-aa28c4aa73c6 6ed121a8a6794dc7acb79c47de3c85ad ae864db73c9b4c7ab6de17fefcf57751 - - -] [instance: 52f9d275-f6af-4006-87a6-e2d420be9d0b] Terminating instance
2015-11-23 22:58:12.363+0000: 22995: error : virCommandWait:2533 : internal error: Child process (/usr/lib/libvirt/virt-aa-helper -p 0 -c -u libvirt-4868b3c5-22e0-4540-80dc-646c2d8e7c25) unexpected exit status 1: virt-aa-helper: error: template does not exist
virt-aa-helper: error: could not create profile
2015-11-23 22:58:12.363+0000: 22995: error : AppArmorGenSecurityLabel:468 : internal error: cannot load AppArmor profile 'libvirt-4868b3c5-22e0-4540-80dc-646c2d8e7c25'
It's a well known bug in libvirt, see https://peterkieser.com/2014/10/24/libvirt-1-2-9-kvm-qemu-apparmor-support/
Workaround suggested in the article above (ln -s /etc/apparmor.d/libvirt/TEMPLATE.qemu /etc/apparmor.d/libvirt/TEMPLATE.kvm) works fine.
JFYI, looks like AppArmor was enabled for libvirt by the following patch: https://review.openstack.org/#/c/246290/
Diagnostic snapshot is attached.
https:/