test live migration fails on multinode job

[Ghanshyam Mann]

Sometimes, test_live_block_migration fails on multinode job (gate-tempest-dsvm-multinode-full).

http://logs.openstack.org/91/202391/5/check/gate-tempest-dsvm-multinode-full/f838b41/logs/testr_results.html.gz

Here migration is aborted with error

2015-07-17 04:10:29.375 ERROR nova.virt.libvirt.driver [req-6c97293d-b191-417e-9e77-d4967b5bfc5d tempest-LiveBlockMigrationTestJSON-824752007 tempest-LiveBlockMigrationTestJSON-153657043] [instance: 6404173e-bb22-43a8-9894-447e2f12591a] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+ssh://stack@devstack-trusty-2-node-hpcloud-b3-3757707/system: Cannot recv data: ssh: Could not resolve hostname devstack-trusty-2-node-hpcloud-b3-3757707: Name or service not known: Connection reset by peer

2015-07-17 04:10:29.812 ERROR nova.virt.libvirt.driver [req-6c97293d-b191-417e-9e77-d4967b5bfc5d tempest-LiveBlockMigrationTestJSON-824752007 tempest-LiveBlockMigrationTestJSON-153657043] [instance: 6404173e-bb22-43a8-9894-447e2f12591a] Migration operation has aborted

And nova put server to active state. After server state is changed from "MIGRATING" to "ACTIVE", Tempest checks target host and fails.

May be Nova behavior of putting server back to Active even migration failed is correct (so that user can still use server on old host) but not sure.

Here we can improve Tempest tests by addition check to list migrations and check the migration status. This will be helpful in debugging.

[Ghanshyam Mann (ghanshyammann)]

More logs-

migration aborted-

2015-07-17 04:10:29.375 ERROR nova.virt.libvirt.driver [req-6c97293d-b191-417e-9e77-d4967b5bfc5d tempest-LiveBlockMigrationTestJSON-824752007 tempest-LiveBlockMigrationTestJSON-153657043] [instance: 6404173e-bb22-43a8-9894-447e2f12591a] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+ssh://stack@devstack-trusty-2-node-hpcloud-b3-3757707/system: Cannot recv data: ssh: Could not resolve hostname devstack-trusty-2-node-hpcloud-b3-3757707: Name or service not known: Connection reset by peer

2015-07-17 04:10:29.812 ERROR nova.virt.libvirt.driver [req-6c97293d-b191-417e-9e77-d4967b5bfc5d tempest-LiveBlockMigrationTestJSON-824752007 tempest-LiveBlockMigrationTestJSON-153657043] [instance: 6404173e-bb22-43a8-9894-447e2f12591a] Migration operation has aborted

Tempest got sever status ACTIVE-

2015-07-17 04:10:30.328 8841 INFO tempest_lib.common.rest_client [req-6a2bc77d-96dc-4693-b0be-b3a9034ea465 ] Request (LiveBlockMigrationTestJSON:test_live_block_migration): 200 GET http://10.0.2.52:8774/v2/77d97fc3f3774db2ae4fa2fa5ae87240/servers/6404173e-bb22-43a8-9894-447e2f12591a 0.110s
2015-07-17 04:10:30.329 8841 DEBUG tempest_lib.common.rest_client [req-6a2bc77d-96dc-4693-b0be-b3a9034ea465 ] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'content-location': 'http://10.0.2.52:8774/v2/77d97fc3f3774db2ae4fa2fa5ae87240/servers/6404173e-bb22-43a8-9894-447e2f12591a', 'content-type': 'application/json', 'x-compute-request-id': 'req-6a2bc77d-96dc-4693-b0be-b3a9034ea465', 'content-length': '1558', 'date': 'Fri, 17 Jul 2015 04:10:30 GMT', 'status': '200', 'connection': 'close'}
        Body: {"server": {"status": "ACTIVE", "updated": "2015-07-17T04:10:29Z", "hostId": "35504644e221f8de9a80e284b12b1c635d28e4009c20968ad11726bd", "addresses": {"private": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:ed:00:14", "version": 4, "addr": "10.1.0.7", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://10.0.2.52:8774/v2/77d97fc3f3774db2ae4fa2fa5ae87240/servers/6404173e-bb22-43a8-9894-447e2f12591a", "rel": "self"}, {"href": "http://10.0.2.52:8774/77d97fc3f3774db2ae4fa2fa5ae87240/servers/6404173e-bb22-43a8-9894-447e2f12591a", "rel": "bookmark"}], "key_name": null, "image": {"id": "9ce5a4ea-b60e-40e9-8cfe-156299600402", "links": [{"href": "http://10.0.2.52:8774/77d97fc3f3774db2ae4fa2fa5ae87240/images/9ce5a4ea-b60e-40e9-8cfe-156299600402", "rel": "bookmark"}]}, "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2015-07-17T04:10:25.000000", "flavor": {"id": "42", "links": [{"href":

Tempest failed for different Host-

    Response - Headers: {'content-location': 'http://10.0.2.52:8774/v2/c7f3b3ffa71c4965a5a7adcd27e1a651/servers/6404173e-bb22-43a8-9894-447e2f12591a', 'content-type': 'application/json', 'x-compute-request-id': 'req-ebc1972a-c847-4f67-b68c-5039877e346b', 'content-length': '1777', 'date': 'Fri, 17 Jul 2015 04:10:30 GMT', 'status': '200', 'connection': 'close'}
        Body: {"server": {"status": "ACTIVE", "updated": "2015-07-17T04:10:29Z", "hostId":...

[Ghanshyam Mann (ghanshyammann)]

Even we should have "gate-tempest-dsvm-multinode-full" for tempest too (non-voting first)

[Ghanshyam Mann (ghanshyammann)]

it will be not possible to get particular migration from migration list as more than 1 migration can present for same server.

So logging all migrations done for particular server if live migration failed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/203949

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/203949
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=bed827642e6e867e8f3fc71a3436dcfbc943eb8b
Submitter: Jenkins
Branch: master

commit bed827642e6e867e8f3fc71a3436dcfbc943eb8b
Author: ghanshyam <email address hidden>
Date: Tue Jul 21 16:03:11 2015 +0900

    Logging migrations for particular server if migration failed

    Sometimes, test_live_block_migration fails on multinode job
    (gate-tempest-dsvm-multinode-full).
    http://logs.openstack.org/91/202391/5/check/gate-tempest-dsvm-multinode-full/f838b41/logs/testr_results.html.gz

    If live migration is failed, server status become Active from Migrating but
    host is not changed. In that case, test_live_block_migration fails for target
    host does not match with server host. But from Tempest failure logs, we do not
    know whether migration is failed or something else happened.

    It will be better for debugging if this tests logs migrations information
    for that server if live migration is failed.

    Closes-Bug: 1476478

    Change-Id: I0f7ba31e44e6d5508483a6552641f59828af20e1