Testing Mitaka in Ubuntu Xenial, rebooting hosts with > 30 instances, we recently hit upon a race condition that seems similar to the one in this issue; maybe we need a wait condition in nova-compute's systemd unit file?
ERROR oslo_service.service [req-34d48ca5-bd93-4d10-a80a-bafad4228467 - - - - -] Error starting thread.
ERROR oslo_service.service Traceback (most recent call last):
ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 680, in run_service
ERROR oslo_service.service service.start()
ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/service.py", line 198, in start
ERROR oslo_service.service self.manager.init_host()
ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1329, in init_host
ERROR oslo_service.service self._init_instance(context, instance)
ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1142, in _init_instance
ERROR oslo_service.service self.driver.plug_vifs(instance, net_info)
ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 880, in plug_vifs
ERROR oslo_service.service self.vif_driver.plug(instance, vif)
ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/vif.py", line 756, in plug
ERROR oslo_service.service func(instance, vif)
ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/vif.py", line 529, in plug_ovs
ERROR oslo_service.service self.plug_ovs_hybrid(instance, vif)
ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/vif.py", line 525, in plug_ovs_hybrid
ERROR oslo_service.service self._plug_bridge_with_port(instance, vif, port='ovs')
ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/vif.py", line 505, in _plug_bridge_with_port
ERROR oslo_service.service linux_net._create_veth_pair(v1_name, v2_name, mtu)
ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 1356, in _create_veth_pair
ERROR oslo_service.service _set_device_mtu(dev, mtu)
ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 1340, in _set_device_mtu
ERROR oslo_service.service check_exit_code=[0, 2, 254])
ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 388, in execute
ERROR oslo_service.service return RootwrapProcessHelper().execute(*cmd, **kwargs)
ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 271, in execute
ERROR oslo_service.service return processutils.execute(*cmd, **kwargs)
ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 389, in execute
ERROR oslo_service.service cmd=sanitized_cmd)
ERROR oslo_service.service ProcessExecutionError: Unexpected error while running command.
ERROR oslo_service.service Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ip link set qvo5ab170bb-a8 mtu 8950
ERROR oslo_service.service Exit code: 1
ERROR oslo_service.service Stdout: u''
ERROR oslo_service.service Stderr: u'Cannot find device "qvo5ab170bb-a8"\n'
ERROR oslo_service.service
Testing Mitaka in Ubuntu Xenial, rebooting hosts with > 30 instances, we recently hit upon a race condition that seems similar to the one in this issue; maybe we need a wait condition in nova-compute's systemd unit file?
ERROR oslo_service. service [req-34d48ca5- bd93-4d10- a80a-bafad42284 67 - - - - -] Error starting thread. service Traceback (most recent call last): service File "/usr/lib/ python2. 7/dist- packages/ oslo_service/ service. py", line 680, in run_service service service.start() service File "/usr/lib/ python2. 7/dist- packages/ nova/service. py", line 198, in start service self.manager. init_host( ) service File "/usr/lib/ python2. 7/dist- packages/ nova/compute/ manager. py", line 1329, in init_host service self._init_ instance( context, instance) service File "/usr/lib/ python2. 7/dist- packages/ nova/compute/ manager. py", line 1142, in _init_instance service self.driver. plug_vifs( instance, net_info) service File "/usr/lib/ python2. 7/dist- packages/ nova/virt/ libvirt/ driver. py", line 880, in plug_vifs service self.vif_ driver. plug(instance, vif) service File "/usr/lib/ python2. 7/dist- packages/ nova/virt/ libvirt/ vif.py" , line 756, in plug service func(instance, vif) service File "/usr/lib/ python2. 7/dist- packages/ nova/virt/ libvirt/ vif.py" , line 529, in plug_ovs service self.plug_ ovs_hybrid( instance, vif) service File "/usr/lib/ python2. 7/dist- packages/ nova/virt/ libvirt/ vif.py" , line 525, in plug_ovs_hybrid service self._plug_ bridge_ with_port( instance, vif, port='ovs') service File "/usr/lib/ python2. 7/dist- packages/ nova/virt/ libvirt/ vif.py" , line 505, in _plug_bridge_ with_port service linux_net. _create_ veth_pair( v1_name, v2_name, mtu) service File "/usr/lib/ python2. 7/dist- packages/ nova/network/ linux_net. py", line 1356, in _create_veth_pair service _set_device_ mtu(dev, mtu) service File "/usr/lib/ python2. 7/dist- packages/ nova/network/ linux_net. py", line 1340, in _set_device_mtu service check_exit_code=[0, 2, 254]) service File "/usr/lib/ python2. 7/dist- packages/ nova/utils. py", line 388, in execute service return RootwrapProcess Helper( ).execute( *cmd, **kwargs) service File "/usr/lib/ python2. 7/dist- packages/ nova/utils. py", line 271, in execute service return processutils. execute( *cmd, **kwargs) service File "/usr/lib/ python2. 7/dist- packages/ oslo_concurrenc y/processutils. py", line 389, in execute service cmd=sanitized_cmd) service ProcessExecutio nError: Unexpected error while running command. service Command: sudo nova-rootwrap /etc/nova/ rootwrap. conf ip link set qvo5ab170bb-a8 mtu 8950 service Exit code: 1 service Stdout: u'' service Stderr: u'Cannot find device "qvo5ab170bb-a8"\n' service
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.
ERROR oslo_service.