Need a proper message when live migration of an instance with config drive fails

[Anna Babich]

In accordance with https://bugs.launchpad.net/nova/+bug/1246201 it's impossible to complete live migration successfully for some cases with libvirt <= 1.2.17. So, it's expected that some error message appears, when trying to live-migrate an instance created under some of conditions listed in bug-1246201.

The bug reported here has been reproduced on env with 3 controllers, 2 computes (Cinder LVM):
[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 128
cat /etc/fuel_build_number:
 128
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 liberty-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6318.noarch
 rubygem-astute-9.0.0-1.mos730.noarch
 fuel-library9.0-9.0.0-1.mos8207.noarch
 fuelmenu-9.0.0-1.mos263.noarch
 fuel-agent-9.0.0-1.mos269.noarch
 fuel-ui-9.0.0-1.mos2624.noarch
 fuel-migrate-9.0.0-1.mos8207.noarch
 nailgun-mcagents-9.0.0-1.mos730.noarch
 fuel-misc-9.0.0-1.mos8207.noarch
 shotgun-9.0.0-1.mos85.noarch
 python-packetary-9.0.0-1.mos128.noarch
 fuel-bootstrap-cli-9.0.0-1.mos269.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8588.noarch
 fuel-mirror-9.0.0-1.mos128.noarch
 fuel-openstack-metadata-9.0.0-1.mos8588.noarch
 fuel-notify-9.0.0-1.mos8207.noarch
 fuel-setup-9.0.0-1.mos6318.noarch
 python-fuelclient-9.0.0-1.mos297.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-9.0.0-1.mos6318.noarch
 fuel-utils-9.0.0-1.mos8207.noarch
 fuel-nailgun-9.0.0-1.mos8588.noarch
 fuel-ostf-9.0.0-1.mos919.noarch
[root@nailgun ~]#

Steps to reproduce:
1. On every compute node, check that the ‘config_drive_format’ parameter has iso9660 value. Otherwise, modify it and restart nova-api on controllers and nova-compute on computes
2. Boot an ephemeral instance with config drive, e.g.:
nova boot --config-drive true --image Ubuntu --key-name <keyname> --flavor m1.small --nic net-id=$(neutron net-list | awk '/admin_internal_net/{print $2}') --user-data ./script.sh --availability-zone nova:<compute_node_1> vm1 --file rcfile=/root/openrc --meta role=webservers --meta essential=false
3. Against vm’s console, check that configuration drive is available and contains all data carried to it during instance creation:
sudo -i
mkdir -p /mnt/config
mount /dev/$(lsblk | grep rom | awk '{print $1}') /mnt/config
cd /mnt/config/openstack/latest && ls
/mnt/config/openstack/latest/
meta_data.json network_data.json user_data vendor_data.json
4. Initiate block LM of the instance to another compute node:
nova live-migration --block-migrate vm1 <another_compute>

Expected result:
Live-migration is terminated with displaying an error message like 'ERROR (BadRequest): Live migration isn't supported for Config Drive in current libvirt version. (HTTP 400)'

Actual result:
Launched live-migration command doesn't display no error messages, despite live-migration fails in fact:
root@node-1:~# nova live-migration --block-migrate vm1 node-5.test.domain.local
root@node-1:~# nova show vm1 | grep '\( OS-EXT-SRV-ATTR:host \| status \)'
| OS-EXT-SRV-ATTR:host | node-4.test.domain.local |
| status | ACTIVE |

root@node-1:~# nova instance-action-list vm1
+----------------+------------------------------------------+---------+----------------------------+
| Action | Request_ID | Message | Start_Time |
+----------------+------------------------------------------+---------+----------------------------+
| create | req-3166d05d-4cae-4485-8e18-563adc76c666 | - | 2016-03-31T10:40:04.000000 |
| live-migration | req-ce91b564-2a45-4f2b-b476-57e881e85c9f | Error | 2016-03-31T10:58:47.000000 |
+----------------+------------------------------------------+---------+----------------------------+
root@node-1:~# nova instance-action vm1 req-ce91b564-2a45-4f2b-b476-57e881e85c9f
+---------------+----------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+---------------+----------------------------------------------------------------------------------------------------------------------+
| action | live-migration |
| events | [{u'event': u'compute_rollback_live_migration_at_destination', |
| | u'finish_time': u'2016-03-31T10:58:53.000000', |
| | u'result': u'Success', |
| | u'start_time': u'2016-03-31T10:58:52.000000', |
| | u'traceback': None}, |
| | {u'event': u'compute_pre_live_migration', |
| | u'finish_time': u'2016-03-31T10:58:52.000000', |
| | u'result': u'Error', |
| | u'start_time': u'2016-03-31T10:58:48.000000', |
| | u'traceback': u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 409, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 387, in decorated_function |
| | kwargs[\'instance\'], e, sys.exc_info()) |
| | File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ |
| | self.force_reraise() |
| | File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise |
| | six.reraise(self.type_, self.value, self.tb) |
| | File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 375, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5218, in pre_live_migration |
| | migrate_data) |
| | File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 6609, in pre_live_migration |
| | raise exception.NoLiveMigrationForConfigDriveInLibVirt() |
| | '}, |
| | {u'event': u'compute_live_migration', |
| | u'finish_time': u'2016-03-31T10:58:48.000000', |
| | u'result': u'Success', |
| | u'start_time': u'2016-03-31T10:58:48.000000', |
| | u'traceback': None}, |
| | {u'event': u'compute_check_can_live_migrate_source', |
| | u'finish_time': u'2016-03-31T10:58:48.000000', |
| | u'result': u'Success', |
| | u'start_time': u'2016-03-31T10:58:48.000000', |
| | u'traceback': None}, |
| | {u'event': u'compute_check_can_live_migrate_destination', |
| | u'finish_time': u'2016-03-31T10:58:48.000000', |
| | u'result': u'Success', |
| | u'start_time': u'2016-03-31T10:58:47.000000', |
| | u'traceback': None}] |
| instance_uuid | 7150b9ec-1eb1-4332-8bc0-b0ed52fb5490 |
| message | Error |
| project_id | 11e5fbd94b21408ba8e343603adf20c6 |
| request_id | req-ce91b564-2a45-4f2b-b476-57e881e85c9f |
| start_time | 2016-03-31T10:58:47.000000 |
| user_id | ff89041386dd47f995594cb522caa0e1 |
+---------------+----------------------------------------------------------------------------------------------------------------------+
root@node-1:~#

[Dina Belova (dbelova)]

Moved to 10.0

[Roman Podoliaka (rpodolyaka)]

We had a conversation about this in upstream and the general conclusion was that we should make (pre-) migration checks asynchronous and always return a success once the request was accepted. Users should be looking for errors in the instance actions log. Please see https://github.com/openstack/nova-specs/blob/master/specs/newton/approved/async-live-migration-rest-check.rst for details