/var/lib/nova|glance disk drive is not present

[Vladimir Kuklin]

sometimes ubuntu provisioning leads to broken partitioning and particular partiotions not mounted

vgchange -ay shows following:

  WARNING: Duplicate VG name image: Existing xtYzHT-hjc8-SJ4E-fGVG-AmwD-k8jh-7YhqoX (created here) takes precedence over Jtg0w5-B5Jq-1Nao-0f2A-jKya-8QsD-4o1wAK
  WARNING: Duplicate VG name image: Existing Jtg0w5-B5Jq-1Nao-0f2A-jKya-8QsD-4o1wAK (created here) takes precedence over xtYzHT-hjc8-SJ4E-fGVG-AmwD-k8jh-7YhqoX
  Couldn't find device with uuid 2Rm3lv-jUCv-qdAK-OID3-lPAg-zzYa-j8UwdE.
  Refusing activation of partial LV glance. Use --partial to override.
  0 logical volume(s) in volume group "image" now active
  Refusing activation of partial LV glance. Use --partial to override.
  0 logical volume(s) in volume group "image" now active

[Vladimir Kuklin (vkuklin)]

root@node-1:~# pvs
  WARNING: Duplicate VG name image: Existing xtYzHT-hjc8-SJ4E-fGVG-AmwD-k8jh-7YhqoX (created here) takes precedence over Jtg0w5-B5Jq-1Nao-0f2A-jKya-8QsD-4o1wAK
  Couldn't find device with uuid 2Rm3lv-jUCv-qdAK-OID3-lPAg-zzYa-j8UwdE.
  Couldn't find device with uuid jh2Mt8-8PQL-Jn4f-9Wcl-jUyD-HZMm-8MLNWu.
  Couldn't find device with uuid dNBNVQ-dViw-1MiN-FoF6-Vc2t-WliX-wU3QQp.
  PV VG Fmt Attr PSize PFree
  /dev/vda5 image lvm2 a- 31.44g 128.00m
  /dev/vdb2 image lvm2 a- 49.44g 0
  /dev/vdc2 image lvm2 a- 49.44g 0
  unknown device image lvm2 a- 31.44g 128.00m
  unknown device image lvm2 a- 49.44g 0
  unknown device image lvm2 a- 49.44g 0

[Vladimir Kuklin (vkuklin)]

configuration of LVM seems pretty strange

        physical_volumes {

                pv0 {
                        id = "809Ixl-xnm0-qxOr-qqi3-lJB6-edYz-20FF0L"
                        device = "/dev/vda5" # Hint only

                        status = ["ALLOCATABLE"]
                        flags = []
                        dev_size = 65953793 # 31.4492 Gigabytes
                        pe_start = 384
                        pe_count = 1006 # 31.4375 Gigabytes
                }

                pv1 {
                        id = "jh2Mt8-8PQL-Jn4f-9Wcl-jUyD-HZMm-8MLNWu"
                        device = "unknown device" # Hint only

                        status = ["ALLOCATABLE"]
                        flags = ["MISSING"]
                        dev_size = 103702528 # 49.4492 Gigabytes
                        pe_start = 384
                        pe_count = 1582 # 49.4375 Gigabytes
                }

                pv2 {
                        id = "dNBNVQ-dViw-1MiN-FoF6-Vc2t-WliX-wU3QQp"
                        device = "unknown device" # Hint only

                        status = ["ALLOCATABLE"]
                        flags = ["MISSING"]
                        dev_size = 103702528 # 49.4492 Gigabytes
                        pe_start = 384
                        pe_count = 1582 # 49.4375 Gigabytes
                }
        }

        logical_volumes {

                glance {
                        id = "0Pvw0M-cWDG-14ZH-Wg0c-ZEds-78z2-qeX43W"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        segment_count = 3

                        segment1 {
                                start_extent = 0
                                extent_count = 1582 # 49.4375 Gigabytes

                                type = "striped"
                                stripe_count = 1 # linear

                                stripes = [
                                        "pv1", 0
                                ]
                        }
                        segment2 {
                                start_extent = 1582
                                extent_count = 1582 # 49.4375 Gigabytes

                                type = "striped"
                                stripe_count = 1 # linear

                                stripes = [
                                        "pv2", 0
                                ]
                        }
                        segment3 {
                                start_extent = 3164
                                extent_count = 1002 # 31.3125 Gigabytes

                                type = "striped"
                                stripe_count = 1 # linear

                                stripes = [
                                        "pv0", 0
                                ]
                        }
                }
        }
}

[Vladimir Kuklin (vkuklin)]

it seems like some of devices are duplicated by udev

  PV VG Fmt Attr PSize PFree
  /dev/vda5 image lvm2 a- 31.44g 128.00m
  /dev/vdb2 image lvm2 a- 49.44g 0
  /dev/vdc2 image lvm2 a- 49.44g 0
  unknown device image lvm2 a- 31.44g 128.00m
  unknown device image lvm2 a- 49.44g 0
  unknown device image lvm2 a- 49.44g 0

[Vladimir Kozhukalov (kozhukalov)]

Last few patches to pmanager.py make additional logging.
https://review.openstack.org/#/c/91531/
https://review.openstack.org/#/c/92621/
https://review.openstack.org/#/c/92849/

Besides that patch which fixes the issue which appears on HP Smart Array tries to clean lvm metadata right after creating new partitions, so let's make this bug medium and try to reproduce it with those patches. If we can reproduce it we'll be able to see what's going on.

[Dmitry Borodaenko (angdraug)]

Marking Incomplete as we need to reproduce it again as per Vladimir's comment.

[Andrey Sledzinskiy (asledzinskiy)]

I get similar problem on {"build_id": "2014-05-14_01-10-31", "mirantis": "yes", "build_number": "203", "ostf_sha": "ef970b442437072bdfa4ea99a7b2971215b2de18", "nailgun_sha": "155acff248aed9a8295d03d58346daa4851d49b4", "production": "docker", "api": "1.0", "fuelmain_sha": "9df792985f8984063979f16dc94b4df24ef40c2d", "astute_sha": "80e60b66e3cb4e3e61b22c61c4acfa127ba1bf7e", "release": "5.0", "fuellib_sha": "89ccab7ee76980e38c4d9a5fbcdf7df87e35d61f"}

Cluster - Ubuntu, Simple, KVM, Flat Nova-Network, Cinder LVM, 1 controller, 1 compute, 1 cinder
At the end of provisioning get next message on nodes - 'The disk drive for var/lib/glance is not ready yet or not present' (screen is attached)
If skip it with 's' deployment would be succsessful

[Aleksandra Fedorova (bookwar)]

Reproduced on Fuel CI with ISO 203:

https://fuel-jenkins.mirantis.com/job/master_fuellib_review_systest_ubuntu/838/

[Igor Shishkin (teran)]

Got the similar on 205th ISO

[Vladimir Kozhukalov (kozhukalov)]

The original problem was likely fixed with those patches
 https://review.openstack.org/#/c/91531/
https://review.openstack.org/#/c/92621/

It is easy to see in logs that lv is created successfully. Unfortunately, third patch https://review.openstack.org/#/c/92849/ introduced mkfs.xfs option error. This error was fixed in the following 2 patches.
https://review.openstack.org/#/c/93535/
https://review.openstack.org/93726

[Aleksandra Fedorova (bookwar)]

Can not reproduce on Fuel CI with iso 207.

https://fuel-jenkins.mirantis.com/job/master_fuellib_review_systest_ubuntu/846/console

Provisioning was successfull (though deployment failed because of something else)

[Mike Scherbakov (mihgen)]

According to comments of A.Fedorova & V.Kozhukalov, the issue is actually fixed. Closing as Fix Committed. Please reopen and notify the team in #fuel-dev IRC channel if you face this issue ever again.