Live migration OSTF test fails with vCenter+Ceph+Mongo

[okosse]

Live migration OSTF test fails in Centos DualHypervisor mode + Ceph + Ceilometer

Step for reproduce:
1. Create environment in wizard with next configuration:
 - CentOS as OS
 - KVM as hypervisor
 - nova network
 - Yes, use ceph storage
 - Install Ceilometer
2. Add 3 controller+cinderVMDK+Mong nodes
3. Add 2 compute + CEPH nodes
4. Deploy cluster
5. Run OSTF

Expected results: all steps are pass

Actual results: Instance live migration failed

This bug does not reproduce without vCenter
 ---------------------fuel-version---------------------------------

api: '1.0'
astute_sha: 0bd72c72369e743376864e8e8dabfe873d40450a
auth_required: true
build_id: 2015-05-25_20-55-26
build_number: '466'
feature_groups:
- mirantis
fuel-library_sha: d7128c27a1b76f4813f3697609f82875c68e85ed
fuel-ostf_sha: 87819878bc0ca572900e1f6933d9b99e666d6f62
fuelmain_sha: 5c8ebddf64ea93000af2de3ccdb4aa8bb766ce93
nailgun_sha: 61ef0edfbfe0c457265a62f0eab05af634ec3b91
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: e19f1b65792f84c4a18b5a9473f85ef3ba172fce
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 0bd72c72369e743376864e8e8dabfe873d40450a
      build_id: 2015-05-25_20-55-26
      build_number: '466'
      feature_groups:
      - mirantis
      fuel-library_sha: d7128c27a1b76f4813f3697609f82875c68e85ed
      fuel-ostf_sha: 87819878bc0ca572900e1f6933d9b99e666d6f62
      fuelmain_sha: 5c8ebddf64ea93000af2de3ccdb4aa8bb766ce93
      nailgun_sha: 61ef0edfbfe0c457265a62f0eab05af634ec3b91
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: e19f1b65792f84c4a18b5a9473f85ef3ba172fce
      release: '6.1'

[Alexander Arzhanov (aarzhanov)]

Please, provide diagnostic snapshot.

[okosse (okosse)]

I reproduced the bug on fuel-6.1-471-2015-05-26_20-59-56.iso (Ubuntu and CentOS)

[Alexander Arzhanov (aarzhanov)]

http://pastebin.com/fLAdZpGP :

urllib3.connectionpool: DEBUG: "GET http://10.109.1.2:8774/v2/c780db7c41394db4b42e7f538852efce/servers/40ce78c6-639b-4653-8a90-db2b9307b3b5 HTTP/1.1" 200 1959
fuel_health.nmanager: DEBUG: Current host is node-4.test.domain.local
urllib3.connectionpool: INFO: Starting new HTTP connection (1): 10.109.0.3
urllib3.connectionpool: DEBUG: Setting read timeout to None
urllib3.connectionpool: DEBUG: "GET http://10.109.1.2:8774/v2/c780db7c41394db4b42e7f538852efce/os-hypervisors/detail HTTP/1.1" 200 3707
fuel_health.tests.smoke.test_live_migration: DEBUG: Free host for migration is domain-c131(Cluster1)
urllib3.connectionpool: INFO: Starting new HTTP connection (1): 10.109.0.3
urllib3.connectionpool: DEBUG: Setting read timeout to None
urllib3.connectionpool: DEBUG: "POST http://10.109.1.2:8774/v2/c780db7c41394db4b42e7f538852efce/servers/40ce78c6-639b-4653-8a90-db2b9307b3b5/action HTTP/1.1" 400 113
fuel_health.common.test_mixins: DEBUG: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/fuel_health/common/test_mixins.py", line 176, in verify
    result = func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/fuel_health/nmanager.py", line 846, in migrate_instance
    instance.live_migrate(host_to)
  File "/usr/lib/python2.6/site-packages/novaclient/v1_1/servers.py", line 344, in live_migrate
    disk_over_commit)
  File "/usr/lib/python2.6/site-packages/novaclient/v1_1/servers.py", line 1124, in live_migrate
    'disk_over_commit': disk_over_commit})
  File "/usr/lib/python2.6/site-packages/novaclient/v1_1/servers.py", line 1240, in _action
    return self.api.client.post(url, body=body)
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 492, in post
    return self._cs_request(url, 'POST', **kwargs)
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 467, in _cs_request
    resp, body = self._time_request(url, method, **kwargs)
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 440, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 433, in request
    raise exceptions.from_response(resp, body, url, method)
BadRequest: Compute service of domain-c131(Cluster1) is unavailable at this time. (HTTP 400) (Request-ID: req-3ff9790d-6a9f-4847-88c3-59e2ad14d5bd)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/186126

[Mike Scherbakov (mihgen)]

Is it an issue in OSTF or OpenStack?

[Alexander Arzhanov (aarzhanov)]

It OSTF issue.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/186126
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=6b4ddbfd3126b77f79759721e86d395bf106b177
Submitter: Jenkins
Branch: master

commit 6b4ddbfd3126b77f79759721e86d395bf106b177
Author: Alexander Arzhanov <email address hidden>
Date: Wed May 27 21:42:44 2015 +0300

    Fix get_free_host for instance live migration test

    -VMware vCenter Server excluded from the list of hypervisors
     because live migration available for nova-compute to nova-compute
     or esxi to esxi.

    Change-Id: Ie3d7ba0e9bd728b7e8bd3174e378be0664d6ab7c
    Closes-bug: #1458916

[okosse (okosse)]

I cheked it on fuel-6.1-478-2015-05-28_20-55-26.iso