test live migration fails on multinode job
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tempest |
Fix Released
|
Undecided
|
Ghanshyam Mann |
Bug Description
Sometimes, test_live_
Here migration is aborted with error
2015-07-17 04:10:29.375 ERROR nova.virt.
2015-07-17 04:10:29.812 ERROR nova.virt.
And nova put server to active state. After server state is changed from "MIGRATING" to "ACTIVE", Tempest checks target host and fails.
May be Nova behavior of putting server back to Active even migration failed is correct (so that user can still use server on old host) but not sure.
Here we can improve Tempest tests by addition check to list migrations and check the migration status. This will be helpful in debugging.
More logs-
migration aborted-
2015-07-17 04:10:29.375 ERROR nova.virt. libvirt. driver [req-6c97293d- b191-417e- 9e77-d4967b5bfc 5d tempest- LiveBlockMigrat ionTestJSON- 824752007 tempest- LiveBlockMigrat ionTestJSON- 153657043] [instance: 6404173e- bb22-43a8- 9894-447e2f1259 1a] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+ssh: //stack@ devstack- trusty- 2-node- hpcloud- b3-3757707/ system: Cannot recv data: ssh: Could not resolve hostname devstack- trusty- 2-node- hpcloud- b3-3757707: Name or service not known: Connection reset by peer
2015-07-17 04:10:29.812 ERROR nova.virt. libvirt. driver [req-6c97293d- b191-417e- 9e77-d4967b5bfc 5d tempest- LiveBlockMigrat ionTestJSON- 824752007 tempest- LiveBlockMigrat ionTestJSON- 153657043] [instance: 6404173e- bb22-43a8- 9894-447e2f1259 1a] Migration operation has aborted
Tempest got sever status ACTIVE-
2015-07-17 04:10:30.328 8841 INFO tempest_ lib.common. rest_client [req-6a2bc77d- 96dc-4693- b0be-b3a9034ea4 65 ] Request (LiveBlockMigra tionTestJSON: test_live_ block_migration ): 200 GET http:// 10.0.2. 52:8774/ v2/77d97fc3f377 4db2ae4fa2fa5ae 87240/servers/ 6404173e- bb22-43a8- 9894-447e2f1259 1a 0.110s lib.common. rest_client [req-6a2bc77d- 96dc-4693- b0be-b3a9034ea4 65 ] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'} location' : 'http:// 10.0.2. 52:8774/ v2/77d97fc3f377 4db2ae4fa2fa5ae 87240/servers/ 6404173e- bb22-43a8- 9894-447e2f1259 1a', 'content-type': 'application/json', 'x-compute- request- id': 'req-6a2bc77d- 96dc-4693- b0be-b3a9034ea4 65', 'content-length': '1558', 'date': 'Fri, 17 Jul 2015 04:10:30 GMT', 'status': '200', 'connection': 'close'} 17T04:10: 29Z", "hostId": "35504644e221f8 de9a80e284b12b1 c635d28e4009c20 968ad11726bd" , "addresses": {"private": [{"OS-EXT- IPS-MAC: mac_addr" : "fa:16: 3e:ed:00: 14", "version": 4, "addr": "10.1.0.7", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http:// 10.0.2. 52:8774/ v2/77d97fc3f377 4db2ae4fa2fa5ae 87240/servers/ 6404173e- bb22-43a8- 9894-447e2f1259 1a", "rel": "self"}, {"href": "http:// 10.0.2. 52:8774/ 77d97fc3f3774db 2ae4fa2fa5ae872 40/servers/ 6404173e- bb22-43a8- 9894-447e2f1259 1a", "rel": "bookmark"}], "key_name": null, "image": {"id": "9ce5a4ea- b60e-40e9- 8cfe-1562996004 02", "links": [{"href": "http:// 10.0.2. 52:8774/ 77d97fc3f3774db 2ae4fa2fa5ae872 40/images/ 9ce5a4ea- b60e-40e9- 8cfe-1562996004 02", "rel": "bookmark"}]}, "OS-EXT- STS:task_ state": null, "OS-EXT- STS:vm_ state": "active", "OS-SRV- USG:launched_ at": "2015-07- 17T04:10: 25.000000" , "flavor": {"id": "42", "links": [{"href":
2015-07-17 04:10:30.329 8841 DEBUG tempest_
Body: None
Response - Headers: {'content-
Body: {"server": {"status": "ACTIVE", "updated": "2015-07-
Tempest failed for different Host-
Response - Headers: {'content- location' : 'http:// 10.0.2. 52:8774/ v2/c7f3b3ffa71c 4965a5a7adcd27e 1a651/servers/ 6404173e- bb22-43a8- 9894-447e2f1259 1a', 'content-type': 'application/json', 'x-compute- request- id': 'req-ebc1972a- c847-4f67- b68c-5039877e34 6b', 'content-length': '1777', 'date': 'Fri, 17 Jul 2015 04:10:30 GMT', 'status': '200', 'connection': 'close'} 17T04:10: 29Z", "hostId":...
Body: {"server": {"status": "ACTIVE", "updated": "2015-07-