All block live migrations are broken when I want nova to calculate live migration type by specifying {'block_migration': 'auto'} in request body. This happens because block_migration and migrate_data.block_migration flags do not have the same value.
In conductor live migrate task we call checks on destination and source that builds up migrate_data in driver and sends them back to conductor:
Then it goes back to conductor and we call compute manager sending both flags - block_migration and migrate_data.block_migration - but we never changed value of block_migration to match migrate_data.block_migration:
Because down in compute manager (and in drivers) we use both flags that have different values (here block_migration=None, migrate_data.block_migration=True), e.g. at this point block_migration=None:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
timer()
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
cb(*args, **kw)
File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 168, in _do_send
waiter.switch(result)
File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
result = function(*args, **kwargs)
File "/opt/stack/nova/nova/utils.py", line 1160, in context_wrapper
return func(*args, **kwargs)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6095, in _live_migration_operation
instance=instance)
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
self.force_reraise()
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6063, in _live_migration_operation
CONF.libvirt.live_migration_bandwidth)
File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
rv = execute(f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
six.reraise(c, e, tb)
File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
rv = meth(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1825, in migrateToURI2
if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
libvirtError: Cannot access storage file '/opt/stack/data/nova/instances/572ad149-b7c5-4b77-85b5-34c1d2d37fcf/disk' (as uid:110, gid:116): No such file or directory
Fast workaround is making sure at compute manager level that block_migration == migrate_data.block_migration, but really we should cleanup all this mess and send only one flag, because it is error-prone and hard to maintain.
All block live migrations are broken when I want nova to calculate live migration type by specifying {'block_migration': 'auto'} in request body. This happens because block_migration and migrate_ data.block_ migration flags do not have the same value.
In conductor live migrate task we call checks on destination and source that builds up migrate_data in driver and sends them back to conductor:
https:/ /github. com/openstack/ nova/blob/ master/ nova/conductor/ tasks/live_ migrate. py#L156
Here we calculate block migration, this is fine:
https:/ /github. com/openstack/ nova/blob/ master/ nova/virt/ libvirt/ driver. py#L5554
Then it goes back to conductor and we call compute manager sending both flags - block_migration and migrate_ data.block_ migration - but we never changed value of block_migration to match migrate_ data.block_ migration:
https:/ /github. com/openstack/ nova/blob/ master/ nova/conductor/ tasks/live_ migrate. py#L68
Because down in compute manager (and in drivers) we use both flags that have different values (here block_migration =None, migrate_ data.block_ migration= True), e.g. at this point block_migration =None:
https:/ /github. com/openstack/ nova/blob/ master/ nova/compute/ manager. py#L5196
We break all block live migrations with:
Traceback (most recent call last): lib/python2. 7/dist- packages/ eventlet/ hubs/hub. py", line 457, in fire_timers lib/python2. 7/dist- packages/ eventlet/ hubs/timer. py", line 58, in __call__ lib/python2. 7/dist- packages/ eventlet/ event.py" , line 168, in _do_send switch( result) lib/python2. 7/dist- packages/ eventlet/ greenthread. py", line 214, in main nova/nova/ utils.py" , line 1160, in context_wrapper nova/nova/ virt/libvirt/ driver. py", line 6095, in _live_migration _operation instance) lib/python2. 7/dist- packages/ oslo_utils/ excutils. py", line 220, in __exit__ force_reraise( ) lib/python2. 7/dist- packages/ oslo_utils/ excutils. py", line 196, in force_reraise reraise( self.type_ , self.value, self.tb) nova/nova/ virt/libvirt/ driver. py", line 6063, in _live_migration _operation libvirt. live_migration_ bandwidth) lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 186, in doit self._autowrap, f, *args, **kwargs) lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 144, in proxy_call lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 125, in execute lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 83, in tworker lib/python2. 7/dist- packages/ libvirt. py", line 1825, in migrateToURI2 ateToURI2( ) failed', dom=self) data/nova/ instances/ 572ad149- b7c5-4b77- 85b5-34c1d2d37f cf/disk' (as uid:110, gid:116): No such file or directory
File "/usr/local/
timer()
File "/usr/local/
cb(*args, **kw)
File "/usr/local/
waiter.
File "/usr/local/
result = function(*args, **kwargs)
File "/opt/stack/
return func(*args, **kwargs)
File "/opt/stack/
instance=
File "/usr/local/
self.
File "/usr/local/
six.
File "/opt/stack/
CONF.
File "/usr/local/
result = proxy_call(
File "/usr/local/
rv = execute(f, *args, **kwargs)
File "/usr/local/
six.reraise(c, e, tb)
File "/usr/local/
rv = meth(*args, **kwargs)
File "/usr/local/
if ret == -1: raise libvirtError ('virDomainMigr
libvirtError: Cannot access storage file '/opt/stack/
Fast workaround is making sure at compute manager level that block_migration == migrate_ data.block_ migration, but really we should cleanup all this mess and send only one flag, because it is error-prone and hard to maintain.