in k8s environment vgs return extra line in output

Bug #1619701 reported by Serguei Bezverkhi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Medium
Gorka Eguileor
Newton
Fix Released
Medium
Gorka Eguileor

Bug Description

In k8s environment this command:

sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --binary --noheadings -o name cinder-volumes

returns extra line which prevents cinder-volume to initialize LVM driver properly.

()[root@k8s-5 /]# sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --binary --noheadings -o name cinder-volumes
File descriptor 19 (/dev/urandom) leaked on vgs invocation. Parent PID 74: /var/lib/kolla/venv/bin/python
  cinder-volumes
()[root@k8s-5 /]#

Proposed fix is to ignore "File descriptor" line as it is completely benign.

Changed in cinder:
status: New → Triaged
importance: Undecided → Medium
assignee: nobody → John Griffith (john-griffith)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/365080

Changed in cinder:
status: Triaged → In Progress
Revision history for this message
John Griffith (john-griffith) wrote :

Looks like there is some question as to whether this warning message is actually causing the failure here or not. Seems we'll need to dig a little deeper. Since I can't repro locally might be good if you can gather some more info here.

I also wonder if the next bug you hit is because there's a problem here and we masked it out with the initial patch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (master)

Change abandoned by John Griffith (<email address hidden>) on branch: master
Review: https://review.openstack.org/365080
Reason: Yep, thanks Eric, I'll rework this with the env setting

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/365167

Changed in cinder:
assignee: John Griffith (john-griffith) → Gorka Eguileor (gorka)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/365167
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=055ec1ce73ca55c463481c349b42dee66e5e86d6
Submitter: Jenkins
Branch: master

commit 055ec1ce73ca55c463481c349b42dee66e5e86d6
Author: John Griffith <email address hidden>
Date: Fri Sep 2 23:00:32 2016 +0000

    Provide cfg to ignore lvm descriptor leak warnings

    For some reason the leaded descriptor warning message coming
    from LVM is causing Cinder to fail startup and it appears to be
    masking out the vg response in vgs calls.

    We typically don't hit this, but due to the nature of Kolla and
    I guess going through the different processes via the containers
    this gets logged every time vgs is called. Eric Harney rightly
    pointed out that rather than use exception handling and such
    that we should use the LVM env variable mechanism we already have
    in place in Cinder.

    For now this patch added a new config option to the LVM driver:
        lvm_suppress_fd_warnings=True|False

    This is useful for K8 deployments that have an indirect call to the
    LVM cmds which results in failures.

    For those that are interested, this can also be done outside of
    cinder by setting the silence_logs variable in lvm.conf

    This is made optional as a config flag to avoid any breakage for
    existing deployments during upgrade.

    Change-Id: I85612fa49475beea58d30330c8fe8352a2f91123
    Closes-Bug: #1619701

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/378013

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/newton)

Reviewed: https://review.openstack.org/378013
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=844aa0ac3e8068e25193a680ac0c63d68682de4b
Submitter: Jenkins
Branch: stable/newton

commit 844aa0ac3e8068e25193a680ac0c63d68682de4b
Author: John Griffith <email address hidden>
Date: Fri Sep 2 23:00:32 2016 +0000

    Provide cfg to ignore lvm descriptor leak warnings

    For some reason the leaded descriptor warning message coming
    from LVM is causing Cinder to fail startup and it appears to be
    masking out the vg response in vgs calls.

    We typically don't hit this, but due to the nature of Kolla and
    I guess going through the different processes via the containers
    this gets logged every time vgs is called. Eric Harney rightly
    pointed out that rather than use exception handling and such
    that we should use the LVM env variable mechanism we already have
    in place in Cinder.

    For now this patch added a new config option to the LVM driver:
        lvm_suppress_fd_warnings=True|False

    This is useful for K8 deployments that have an indirect call to the
    LVM cmds which results in failures.

    For those that are interested, this can also be done outside of
    cinder by setting the silence_logs variable in lvm.conf

    This is made optional as a config flag to avoid any breakage for
    existing deployments during upgrade.

    DocImpact: Need to add info regarding config update to docs

    Change-Id: I85612fa49475beea58d30330c8fe8352a2f91123
    Closes-Bug: #1619701
    (cherry picked from commit 055ec1ce73ca55c463481c349b42dee66e5e86d6)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 9.0.0.0rc2

This issue was fixed in the openstack/cinder 9.0.0.0rc2 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 9.0.0

This issue was fixed in the openstack/cinder 9.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 10.0.0.0b1

This issue was fixed in the openstack/cinder 10.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.