Errors in cinder-volume log for HA, Ubuntu

[Anastasia Palkina]

"build_id": "2014-02-21_13-53-24",
"mirantis": "yes",
"build_number": "176",
"nailgun_sha": "d528441c490fc53f26a9115fb866d41208727d1a",
"ostf_sha": "e74dcb33134d4334786a4fa27107febf9a497f2b",
"fuelmain_sha": "fe157dcc0c68a237b527cb9b6adf9101c815e374",
"astute_sha": "3d43abeefb60677ce6cae83d31ebbba1ff3cdbe2",
"release": "4.1",
"fuellib_sha": "a964b37f7f96f62a2f92b117c06077b53386a3ae"

1. Create new environment (Ubuntu, HA mode)
2. Choose VLAN segmentation
3. Add 3 controllers, 1 compute and 1 cinder
4. Choose Debug in 'OpenStack Settings' tab
5. Start deployment. It was successful
6. Errors in cinder-volume logs

2014-02-21 11:29:52 ERROR

cinder.volume.manager [req-d4f386a1-924a-4862-8b21-9c7d0bc745bc None None] Bad or unexpected response from the storage volume backend API: Volume Group cinder does not exist

2014-02-21 11:29:52 ERROR

cinder.volume.manager [req-d4f386a1-924a-4862-8b21-9c7d0bc745bc None None] Error encountered during initialization of driver: LVMISCSIDriver

2014-02-21 11:29:52 ERROR

cinder.brick.local_dev.lvm [req-d4f386a1-924a-4862-8b21-9c7d0bc745bc None None] Unable to locate Volume Group cinder

2014-02-21 11:26:20 ERROR

cinder.service [req-131d21fa-6d63-4dbb-ac18-d460d9e0aabb None None] Unhandled exception

[Dmitry Ilyin (idv1985)]

Is there a volume group names cinder?

[Dmitry Borodaenko (angdraug)]

The interesting part is cinder-volume-manager tried to run on controller, which definitely doesn't have cinder VG on account of not having cinder role assigned.

[Dmitry Borodaenko (angdraug)]

There's a faulty logic in this condition:

https://github.com/stackforge/fuel-library/blob/e3ea44c3b607f37401a268a91956c9d222a81bab/deployment/puppet/osnailyfacter/manifests/cluster_ha.pp#L143

The way this is defined, cinder-volume-manager will be set up on every node in the environment when Cinder LVM volume backend is sected in storage settings.

This error is relatively harmless since cinder scheduler is smart enough to ignore cinder-volume-manager instances on nodes that don't have cinder VG and create volumes on cinder nodes. Downgrading to low.

[Dmitry Borodaenko (angdraug)]

Upgrading to Critical since it actually causes breakage with Icehouse.

[Bogdan Dobrelya (bogdando)]

Addressed by: https://review.openstack.org/#/c/88519/

[Dmitry Borodaenko (angdraug)]

Addressed by: https://review.openstack.org/88469

(For some reason, Fuel CI didn't update LP so I'm linking the review manually)

[Bogdan Dobrelya (bogdando)]

Yes, and now due tothe gerrit hook is broken we have 3 patches to address it :-)

One more from Vladimir Kuklin https://review.openstack.org/#/c/88348/

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/88469
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=f5a6c5b03300e8ed929042b55e894b85fe5155bd
Submitter: Jenkins
Branch: master

commit f5a6c5b03300e8ed929042b55e894b85fe5155bd
Author: Dmitry Borodaenko <email address hidden>
Date: Thu Apr 17 16:47:35 2014 -0700

    Do not enable cinder-volume on non-cinder nodes

    Only cinder nodes have "cinder" volume group created during
    provisioning, when cinder-volume service is deployed with Cinder LVM
    backend on a node that doesn't have a "cinder" VG, volume creation on
    such nodes is impossible.

    is_cinder_node variable is no longer used in osnailfyfacter so the logic
    to generate its value can be safely removed.

    Change-Id: I1d4eab2435cc87625733674d26e0c30cc593dd6a
    Closes-bug: #1283035

[Anastasia Palkina (apalkina)]

Verified on ISO #117
"build_id": "2014-04-21_12-12-37",
"mirantis": "yes",
"build_number": "117",
"nailgun_sha": "66395c99412dbd893868d428b0e4748be70e4a61",
"production": "prod",
"ostf_sha": "7d0ea20e1c450a5f571d91bb03141b12c301184b",
"fuelmain_sha": "f0b6f3afe87b0b286d22ee6114b92c1c07fd1ce6",
"astute_sha": "0cb0b26773bd269000a14748e8ba49419462ecaf",
"release": "5.0",
"fuellib_sha": "b0390f0f882b4f1a3969c16f0514761c81691cc2"

[Artem Panchenko (apanchenko-8)]

Reproduced the same issue on ISO # 338 (4.1):

{"build_id": "2014-06-03_00-40-36", "mirantis": "yes", "build_number": "338", "nailgun_sha": "a828d6b7610f872980d5a2113774f1cda6f6810b", "ostf_sha": "c959aa55f83fe2555cf2d382559271c7a9b17467", "fuelmain_sha": "15805b14f36fc0a896f136051a3be10fea4611d9", "astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968", "release": "4.1B", "fuellib_sha": "377f27df4cb008b400b69013dbd606e6225f5c04"}

HA + Ubuntu + NeutronVlan + Ceph for Glance
3 Controllers + 1 Compute + 1 Cinder + 2 Ceph

After successful deployment all controllers have errors in cinder logs:

http://paste.openstack.org/show/82565/

and creating of new volume fails with the following errors:

<158>Jun 3 09:38:20 node-1 cinder-eventlet.wsgi.server INFO: 192.168.0.2 - - [03/Jun/2014 09:38:20] "GET /v1/dfa6efa33ac945d586de917d3b4ae151/limits HTTP/1.1" 200 266 0.009824
<155>Jun 3 09:38:20 node-1 cinder-cinder.scheduler.filters.capacity_filter ERROR: Free capacity not set: volume node info collection broken.
<155>Jun 3 09:38:20 node-1 cinder-cinder.scheduler.filters.capacity_filter ERROR: Free capacity not set: volume node info collection broken.
<155>Jun 3 09:38:20 node-1 cinder-cinder.scheduler.filters.capacity_filter ERROR: Free capacity not set: volume node info collection broken.
<155>Jun 3 09:38:20 node-1 cinder-cinder.volume.flows.create_volume ERROR: Failed to schedule_create_volume: No valid host was found.

I guess we should backport fix for this bug to the 4.1 branch.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/98004

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/98004
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=f28f6a9284e6ecc4d8caa5eab0a3b5275dd18ce0
Submitter: Jenkins
Branch: stable/4.1

commit f28f6a9284e6ecc4d8caa5eab0a3b5275dd18ce0
Author: Dmitry Borodaenko <email address hidden>
Date: Thu Apr 17 16:47:35 2014 -0700

    Do not enable cinder-volume on non-cinder nodes

    Only cinder nodes have "cinder" volume group created during
    provisioning, when cinder-volume service is deployed with Cinder LVM
    backend on a node that doesn't have a "cinder" VG, volume creation on
    such nodes is impossible.

    is_cinder_node variable is no longer used in osnailfyfacter so the logic
    to generate its value can be safely removed.

    Change-Id: I1d4eab2435cc87625733674d26e0c30cc593dd6a
    Closes-bug: #1283035