Live migration fails on deployment with ceph and enabled ephemeral option

[Tatyanka]

Live Migration fails with message hypervisor can not access to the shared storage

Steps:
1. Create cluster
2. Add 1 node with controller and ceph OSD roles
3. Add 2 node with compute and ceph OSD roles
Storage settings:
 'volumes_ceph': True,
 'images_ceph': True,
 'ephemeral_ceph': True,
 'volumes_lvm': False
4. Deploy the cluster
5. Check ceph status
6. Run ostf tests Live migration
7. ssh on controller - launch instance and run live-migration manually

Expected result:
Instance successfully migrate

Actual:
Migration has not been started with message

node-1.test.domain.local is not on shared storage: Live migration can not be used without shared storage.

ISO: RC2 (522)

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "522"
  build_id: "2015-06-16_13-53-26"
  nailgun_sha: "fa8dec50f3df2626c97f6c38a897cf4e0f80b39d"
  python-fuelclient_sha: "4fc55db0265bbf39c369df398b9dc7d6469ba13b"
  astute_sha: "1ea8017fe8889413706d543a5b9f557f5414beae"
  fuel-library_sha: "3528dddbd0c961290909d5e3e256f55ff75cd2fc"
  fuel-ostf_sha: "8fefcf7c4649370f00847cc309c24f0b62de718d"
  fuelmain_sha: "42020c36d6dec9fedf61faa68aa3674156d41977"

env can be reverted:
http://jenkins-product.srt.mirantis.net:8080/view/6.1_swarm/job/6.1.system_test.centos.ceph_ha_one_controller/18/ and on Ubuntu too

I see that storage option was correctly pushed to hiera
[root@node-2 ~]# hiera storage
{"volumes_ceph"=>true,
 "objects_ceph"=>false,
 "ephemeral_ceph"=>true,
 "iser"=>false,
 "volumes_lvm"=>false,
 "images_vcenter"=>false,
 "pg_num"=>512,
 "osd_pool_size"=>"2",
 "metadata"=>{"label"=>"Storage", "weight"=>60},
 "images_ceph"=>true}
[root@node-2 ~]# exit
logout
Connection to node-2 closed.
 and astute post task was run on computes nodes(1, 2) bit at the same time some option in nova.conf was not enabled:
#
# Options defined in nova.virt.libvirt.imagebackend
#

# VM Images format. Acceptable values are: raw, qcow2, lvm,
# rbd, default. If default is specified, then use_cow_images
# flag is used instead of this one. (string value)
# Deprecated group;name - DEFAULT;libvirt_images_type
#images_type=default

# The RADOS pool in which rbd volumes are stored (string
# value)
# Deprecated group;name - DEFAULT;libvirt_images_rbd_pool
#images_rbd_pool=rbd

if we enable values like:
# Deprecated group;name - DEFAULT;libvirt_images_type
#images_type=default
images_type=rbd

and
# Deprecated group;name - DEFAULT;libvirt_images_rbd_pool
#images_rbd_pool=rbd
images_rbd_pool=compute

It starts to work, I can not find the reason why this values was not updated during post task puppet_manifest: "/etc/puppet/modules/osnailyfacter/modular/astute/ceph_compute.pp"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/192687

[Maksym Strukov (unbelll)]

Repro on 6.1-522

{"build_id": "2015-06-16_13-53-26", "build_number": "522", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-06-16_13-53-26", "build_number": "522", "api": "1.0", "fuel-library_sha": "3528dddbd0c961290909d5e3e256f55ff75cd2fc", "nailgun_sha": "fa8dec50f3df2626c97f6c38a897cf4e0f80b39d", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b", "astute_sha": "1ea8017fe8889413706d543a5b9f557f5414beae", "fuel-ostf_sha": "8fefcf7c4649370f00847cc309c24f0b62de718d", "release": "6.1", "fuelmain_sha": "42020c36d6dec9fedf61faa68aa3674156d41977"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "3528dddbd0c961290909d5e3e256f55ff75cd2fc", "nailgun_sha": "fa8dec50f3df2626c97f6c38a897cf4e0f80b39d", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b", "astute_sha": "1ea8017fe8889413706d543a5b9f557f5414beae", "fuel-ostf_sha": "8fefcf7c4649370f00847cc309c24f0b62de718d", "release": "6.1", "fuelmain_sha": "42020c36d6dec9fedf61faa68aa3674156d41977"}

Repro steps: https://mirantis.testrail.com/index.php?/cases/view/20878

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/192919
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=2e7a08ad9792c700ebf08ce87f4867df36aa9fab
Submitter: Jenkins
Branch: stable/6.1

commit 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
Author: Ryan Moe <email address hidden>
Date: Wed Jun 17 15:29:41 2015 -0700

    Revert "Fix the problem with ceph deployment on scale lab"

    This reverts commit 43b25e4b200c5b994cde81439454d6e2e908a88f.

    Closes-bug: #1466075

    Change-Id: Ib44cf45fb90dca1c8512048d89ceba2b2d47c35b

[OpenStack Infra (hudson-openstack)]

Change abandoned by Stanislav Makar (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/192687
Reason: The patch https://review.openstack.org/#/c/193076/ includes this patch and patch https://review.openstack.org/#/c/190953/ which has been reverted

[Andrey Sledzinskiy (asledzinskiy)]

verified on {

    "build_id": "2015-06-18_17-46-09",
    "build_number": "523",
    "release_versions":

{

    "2014.2.2-6.1":

{

    "VERSION":

{

    "build_id": "2015-06-18_17-46-09",
    "build_number": "523",
    "api": "1.0",
    "fuel-library_sha": "2e7a08ad9792c700ebf08ce87f4867df36aa9fab",
    "nailgun_sha": "126e50e624b1b97043ab87dbf1298312b9771df3",
    "feature_groups":

            [
                "mirantis"
            ],
            "openstack_version": "2014.2.2-6.1",
            "production": "docker",
            "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b",
            "astute_sha": "1ea8017fe8889413706d543a5b9f557f5414beae",
            "fuel-ostf_sha": "8fefcf7c4649370f00847cc309c24f0b62de718d",
            "release": "6.1",
            "fuelmain_sha": "237c6ec1e9c022d69e5f8ab859887352fc5cb74b"
        }
    }

},
"auth_required": true,
"api": "1.0",
"fuel-library_sha": "2e7a08ad9792c700ebf08ce87f4867df36aa9fab",
"nailgun_sha": "126e50e624b1b97043ab87dbf1298312b9771df3",
"feature_groups":

    [
        "mirantis"
    ],
    "openstack_version": "2014.2.2-6.1",
    "production": "docker",
    "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b",
    "astute_sha": "1ea8017fe8889413706d543a5b9f557f5414beae",
    "fuel-ostf_sha": "8fefcf7c4649370f00847cc309c24f0b62de718d",
    "release": "6.1",
    "fuelmain_sha": "237c6ec1e9c022d69e5f8ab859887352fc5cb74b"

}

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitry Borodaenko (<email address hidden>) on branch: master
Review: https://review.openstack.org/193065
Reason: A duplicate revert commit has passed CI and was merged instead: https://review.openstack.org/#/c/195466/

[Alexander Kurenyshev (akurenyshev)]

Repro on

api: '1.0'
astute_sha: 776157f722b13aff5f59bc098cf948793e6498ef
auth_required: true
build_id: 2015-06-24_16-12-23
build_number: '15'
feature_groups:
- mirantis
fuel-library_sha: 7d19bc3783177aebf64fa4c2ae20d845cbd5348f
fuel-ostf_sha: 69e7fa120e8efa7ed74d98efc63079d2f5c69d7b
fuelmain_sha: 3b866d2ff3091a60362327028085fa62fd16c5a0
nailgun_sha: b74f847ec89c4bff1addb830704206dc503125f0
openstack_version: 2014.2.2-7.0
production: docker
python-fuelclient_sha: 1b8574a7c4ea884862763a15c636b066d51f49e7
release: '7.0'
release_versions:
  2014.2.2-7.0:
    VERSION:
      api: '1.0'
      astute_sha: 776157f722b13aff5f59bc098cf948793e6498ef
      build_id: 2015-06-24_16-12-23
      build_number: '15'
      feature_groups:
      - mirantis
      fuel-library_sha: 7d19bc3783177aebf64fa4c2ae20d845cbd5348f
      fuel-ostf_sha: 69e7fa120e8efa7ed74d98efc63079d2f5c69d7b
      fuelmain_sha: 3b866d2ff3091a60362327028085fa62fd16c5a0
      nailgun_sha: b74f847ec89c4bff1addb830704206dc503125f0
      openstack_version: 2014.2.2-7.0
      production: docker
      python-fuelclient_sha: 1b8574a7c4ea884862763a15c636b066d51f49e7
      release: '7.0'

[Sergey Vasilenko (xenolog)]

live migration, at all, works properly on 7.0-51 ISO

[Oleksiy Molchanov (omolchanov)]

Closing as invalid then. Please re-open if reproduced.

[Stanislav Makar (smakar)]

Guys
for 6.1 this is not merged, patch is here https://review.openstack.org/#/c/193076/

For 7.0 the patch is merged https://review.openstack.org/#/c/195468/ that is why all is working on 7.0

[Oleksiy Molchanov (omolchanov)]

I am passing this to sustaining team then

[Tatyanka (tatyana-leontovich)]

Actually for 7.0 issue is partly fixed, according to migration still not finished with success and failed with next messages http://paste.openstack.org/show/395155/ http://jenkins-product.srt.mirantis.net:8080/job/7.0.system_test.ubuntu.upgrade_one_controller/7/

[Mike Scherbakov (mihgen)]

Folks, please assign this bug to Nova or Ceph team if goes into weeds of how particular Nova & Ceph components are configured or work. Please support those teams though as needed.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vladimir Kuklin (<email address hidden>) on branch: master
Review: https://review.openstack.org/192703

[Vladimir Kuklin (vkuklin)]

this issue was reverted in 6.1 and fixed in 7.0

[Tatyanka (tatyana-leontovich)]

verified on 140 iso for 7.0