Ah it's not a scheduler filter (which is what I thought was related to this), it's a libvirt firewall driver network filter:
http://logs.openstack.org/24/374324/1/gate/gate-tempest-dsvm-cells-ubuntu-xenial/81f45de/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-09-21_21_20_24_446
2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [req-462fe36b-6d85-4b74-9eab-4d2a0ea9c3d5 tempest-ListImageFiltersTestJSON-1517708761 tempest-ListImageFiltersTestJSON-1517708761] [instance: 9103de54-6d83-4877-8075-9807ec099778] Instance failed to spawn 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] Traceback (most recent call last): 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] File "/opt/stack/new/nova/nova/compute/manager.py", line 2078, in _build_resources 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] yield resources 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] File "/opt/stack/new/nova/nova/compute/manager.py", line 1920, in _build_and_run_instance 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] block_device_info=block_device_info) 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2583, in spawn 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] post_xml_callback=gen_confdrive) 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4816, in _create_domain_and_network 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] network_info) 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] File "/opt/stack/new/nova/nova/virt/libvirt/firewall.py", line 339, in setup_basic_filtering 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] self.nwfilter.setup_basic_filtering(instance, network_info) 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] File "/opt/stack/new/nova/nova/virt/libvirt/firewall.py", line 122, in setup_basic_filtering 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] self._ensure_static_filters() 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] File "/opt/stack/new/nova/nova/virt/libvirt/firewall.py", line 225, in _ensure_static_filters 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] self._define_filter(self.nova_no_nd_reflection_filter()) 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] File "/opt/stack/new/nova/nova/virt/libvirt/firewall.py", line 262, in _define_filter 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] self._conn.nwfilterDefineXML(xml) 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] result = proxy_call(self._autowrap, f, *args, **kwargs) 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] rv = execute(f, *args, **kwargs) 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] six.reraise(c, e, tb) 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] rv = meth(*args, **kwargs) 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 4341, in nwfilterDefineXML 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] if ret is None:raise libvirtError('virNWFilterDefineXML() failed', conn=self) 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778] libvirtError: operation failed: filter 'nova-no-nd-reflection' already exists with uuid e740c5ec-c715-4f73-9874-630cc73d4ac2 2016-09-21 21:20:24.446 12207 ERROR nova.compute.manager [instance: 9103de54-6d83-4877-8075-9807ec099778]
Ah it's not a scheduler filter (which is what I thought was related to this), it's a libvirt firewall driver network filter:
http:// logs.openstack. org/24/ 374324/ 1/gate/ gate-tempest- dsvm-cells- ubuntu- xenial/ 81f45de/ logs/screen- n-cpu.txt. gz?level= TRACE#_ 2016-09- 21_21_20_ 24_446
2016-09-21 21:20:24.446 12207 ERROR nova.compute. manager [req-462fe36b- 6d85-4b74- 9eab-4d2a0ea9c3 d5 tempest- ListImageFilter sTestJSON- 1517708761 tempest- ListImageFilter sTestJSON- 1517708761] [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] Instance failed to spawn manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] Traceback (most recent call last): manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] File "/opt/stack/ new/nova/ nova/compute/ manager. py", line 2078, in _build_resources manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] yield resources manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] File "/opt/stack/ new/nova/ nova/compute/ manager. py", line 1920, in _build_ and_run_ instance manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] block_device_ info=block_ device_ info) manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] File "/opt/stack/ new/nova/ nova/virt/ libvirt/ driver. py", line 2583, in spawn manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] post_xml_ callback= gen_confdrive) manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] File "/opt/stack/ new/nova/ nova/virt/ libvirt/ driver. py", line 4816, in _create_ domain_ and_network manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] network_info) manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] File "/opt/stack/ new/nova/ nova/virt/ libvirt/ firewall. py", line 339, in setup_basic_ filtering manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] self.nwfilter. setup_basic_ filtering( instance, network_info) manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] File "/opt/stack/ new/nova/ nova/virt/ libvirt/ firewall. py", line 122, in setup_basic_ filtering manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] self._ensure_ static_ filters( ) manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] File "/opt/stack/ new/nova/ nova/virt/ libvirt/ firewall. py", line 225, in _ensure_ static_ filters manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] self._define_ filter( self.nova_ no_nd_reflectio n_filter( )) manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] File "/opt/stack/ new/nova/ nova/virt/ libvirt/ firewall. py", line 262, in _define_filter manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] self._conn. nwfilterDefineX ML(xml) manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] File "/usr/local/ lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 186, in doit manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] result = proxy_call( self._autowrap, f, *args, **kwargs) manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] File "/usr/local/ lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 144, in proxy_call manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] rv = execute(f, *args, **kwargs) manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] File "/usr/local/ lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 125, in execute manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] six.reraise(c, e, tb) manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] File "/usr/local/ lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 83, in tworker manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] rv = meth(*args, **kwargs) manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] File "/usr/local/ lib/python2. 7/dist- packages/ libvirt. py", line 4341, in nwfilterDefineXML manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] if ret is None:raise libvirtError( 'virNWFilterDef ineXML( ) failed', conn=self) manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78] libvirtError: operation failed: filter 'nova-no- nd-reflection' already exists with uuid e740c5ec- c715-4f73- 9874-630cc73d4a c2 manager [instance: 9103de54- 6d83-4877- 8075-9807ec0997 78]
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.
2016-09-21 21:20:24.446 12207 ERROR nova.compute.