Errors in cinder-volume log for HA, Ubuntu

Bug #1283035 reported by Anastasia Palkina on 2014-02-21
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Critical
Dmitry Borodaenko
4.1.x
Critical
Dmitry Borodaenko

Bug Description

"build_id": "2014-02-21_13-53-24",
"mirantis": "yes",
"build_number": "176",
"nailgun_sha": "d528441c490fc53f26a9115fb866d41208727d1a",
"ostf_sha": "e74dcb33134d4334786a4fa27107febf9a497f2b",
"fuelmain_sha": "fe157dcc0c68a237b527cb9b6adf9101c815e374",
"astute_sha": "3d43abeefb60677ce6cae83d31ebbba1ff3cdbe2",
"release": "4.1",
"fuellib_sha": "a964b37f7f96f62a2f92b117c06077b53386a3ae"

1. Create new environment (Ubuntu, HA mode)
2. Choose VLAN segmentation
3. Add 3 controllers, 1 compute and 1 cinder
4. Choose Debug in 'OpenStack Settings' tab
5. Start deployment. It was successful
6. Errors in cinder-volume logs

2014-02-21 11:29:52 ERROR

cinder.volume.manager [req-d4f386a1-924a-4862-8b21-9c7d0bc745bc None None] Bad or unexpected response from the storage volume backend API: Volume Group cinder does not exist

2014-02-21 11:29:52 ERROR

cinder.volume.manager [req-d4f386a1-924a-4862-8b21-9c7d0bc745bc None None] Error encountered during initialization of driver: LVMISCSIDriver

2014-02-21 11:29:52 ERROR

cinder.brick.local_dev.lvm [req-d4f386a1-924a-4862-8b21-9c7d0bc745bc None None] Unable to locate Volume Group cinder

2014-02-21 11:26:20 ERROR

cinder.service [req-131d21fa-6d63-4dbb-ac18-d460d9e0aabb None None] Unhandled exception

Anastasia Palkina (apalkina) wrote :
Dmitry Ilyin (idv1985) wrote :

Is there a volume group names cinder?

Dmitry Borodaenko (angdraug) wrote :

The interesting part is cinder-volume-manager tried to run on controller, which definitely doesn't have cinder VG on account of not having cinder role assigned.

Changed in fuel:
status: New → Confirmed
importance: Undecided → Medium
Dmitry Borodaenko (angdraug) wrote :

There's a faulty logic in this condition:

https://github.com/stackforge/fuel-library/blob/e3ea44c3b607f37401a268a91956c9d222a81bab/deployment/puppet/osnailyfacter/manifests/cluster_ha.pp#L143

The way this is defined, cinder-volume-manager will be set up on every node in the environment when Cinder LVM volume backend is sected in storage settings.

This error is relatively harmless since cinder scheduler is smart enough to ignore cinder-volume-manager instances on nodes that don't have cinder VG and create volumes on cinder nodes. Downgrading to low.

Changed in fuel:
importance: Medium → Low
status: Confirmed → Triaged
Mike Scherbakov (mihgen) on 2014-02-22
Changed in fuel:
milestone: 4.1 → 5.0
importance: Low → Medium
Andrew Woodward (xarses) on 2014-04-04
tags: added: low-hanging-fruit
Dmitry Borodaenko (angdraug) wrote :

Upgrading to Critical since it actually causes breakage with Icehouse.

Changed in fuel:
importance: Medium → Critical
Bogdan Dobrelya (bogdando) wrote :
Changed in fuel:
status: Triaged → In Progress
assignee: Fuel Library Team (fuel-library) → Bogdan Dobrelya (bogdando)
Dmitry Borodaenko (angdraug) wrote :

Addressed by: https://review.openstack.org/88469

(For some reason, Fuel CI didn't update LP so I'm linking the review manually)

Bogdan Dobrelya (bogdando) wrote :

Yes, and now due tothe gerrit hook is broken we have 3 patches to address it :-)

One more from Vladimir Kuklin https://review.openstack.org/#/c/88348/

Changed in fuel:
assignee: Bogdan Dobrelya (bogdando) → Dmitry Borodaenko (dborodaenko)

Reviewed: https://review.openstack.org/88469
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=f5a6c5b03300e8ed929042b55e894b85fe5155bd
Submitter: Jenkins
Branch: master

commit f5a6c5b03300e8ed929042b55e894b85fe5155bd
Author: Dmitry Borodaenko <email address hidden>
Date: Thu Apr 17 16:47:35 2014 -0700

    Do not enable cinder-volume on non-cinder nodes

    Only cinder nodes have "cinder" volume group created during
    provisioning, when cinder-volume service is deployed with Cinder LVM
    backend on a node that doesn't have a "cinder" VG, volume creation on
    such nodes is impossible.

    is_cinder_node variable is no longer used in osnailfyfacter so the logic
    to generate its value can be safely removed.

    Change-Id: I1d4eab2435cc87625733674d26e0c30cc593dd6a
    Closes-bug: #1283035

Changed in fuel:
status: In Progress → Fix Committed
Anastasia Palkina (apalkina) wrote :

Verified on ISO #117
"build_id": "2014-04-21_12-12-37",
"mirantis": "yes",
"build_number": "117",
"nailgun_sha": "66395c99412dbd893868d428b0e4748be70e4a61",
"production": "prod",
"ostf_sha": "7d0ea20e1c450a5f571d91bb03141b12c301184b",
"fuelmain_sha": "f0b6f3afe87b0b286d22ee6114b92c1c07fd1ce6",
"astute_sha": "0cb0b26773bd269000a14748e8ba49419462ecaf",
"release": "5.0",
"fuellib_sha": "b0390f0f882b4f1a3969c16f0514761c81691cc2"

Changed in fuel:
status: Fix Committed → Fix Released
Artem Panchenko (apanchenko-8) wrote :

Reproduced the same issue on ISO # 338 (4.1):

{"build_id": "2014-06-03_00-40-36", "mirantis": "yes", "build_number": "338", "nailgun_sha": "a828d6b7610f872980d5a2113774f1cda6f6810b", "ostf_sha": "c959aa55f83fe2555cf2d382559271c7a9b17467", "fuelmain_sha": "15805b14f36fc0a896f136051a3be10fea4611d9", "astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968", "release": "4.1B", "fuellib_sha": "377f27df4cb008b400b69013dbd606e6225f5c04"}

HA + Ubuntu + NeutronVlan + Ceph for Glance
3 Controllers + 1 Compute + 1 Cinder + 2 Ceph

After successful deployment all controllers have errors in cinder logs:

http://paste.openstack.org/show/82565/

and creating of new volume fails with the following errors:

<158>Jun 3 09:38:20 node-1 cinder-eventlet.wsgi.server INFO: 192.168.0.2 - - [03/Jun/2014 09:38:20] "GET /v1/dfa6efa33ac945d586de917d3b4ae151/limits HTTP/1.1" 200 266 0.009824
<155>Jun 3 09:38:20 node-1 cinder-cinder.scheduler.filters.capacity_filter ERROR: Free capacity not set: volume node info collection broken.
<155>Jun 3 09:38:20 node-1 cinder-cinder.scheduler.filters.capacity_filter ERROR: Free capacity not set: volume node info collection broken.
<155>Jun 3 09:38:20 node-1 cinder-cinder.scheduler.filters.capacity_filter ERROR: Free capacity not set: volume node info collection broken.
<155>Jun 3 09:38:20 node-1 cinder-cinder.volume.flows.create_volume ERROR: Failed to schedule_create_volume: No valid host was found.

I guess we should backport fix for this bug to the 4.1 branch.

Artem Panchenko (apanchenko-8) wrote :

Reviewed: https://review.openstack.org/98004
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=f28f6a9284e6ecc4d8caa5eab0a3b5275dd18ce0
Submitter: Jenkins
Branch: stable/4.1

commit f28f6a9284e6ecc4d8caa5eab0a3b5275dd18ce0
Author: Dmitry Borodaenko <email address hidden>
Date: Thu Apr 17 16:47:35 2014 -0700

    Do not enable cinder-volume on non-cinder nodes

    Only cinder nodes have "cinder" volume group created during
    provisioning, when cinder-volume service is deployed with Cinder LVM
    backend on a node that doesn't have a "cinder" VG, volume creation on
    such nodes is impossible.

    is_cinder_node variable is no longer used in osnailfyfacter so the logic
    to generate its value can be safely removed.

    Change-Id: I1d4eab2435cc87625733674d26e0c30cc593dd6a
    Closes-bug: #1283035

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers