Failed to initialize driver due to unrecognized option '--ignoreskippedcluster

[Jianghua Wang]

When start the service of cinder-volume, it failed to initialize driver due to unrecognized option '--ignoreskippedcluster'. And this failure result into many volume relative CI tests.

It looks like the following commit added this option and it's attending to only add this option when lvm is later than 2.02.103. But obviously it's not working as expected: it's using the option of '--ignoreskippedcluster' even the lvm is 2.02.98.

https://review.openstack.org/#/c/396269/

stack@devstack:~/cinder$ sudo vgs --version
  LVM version: 2.02.98(2) (2012-10-15)
  Library version: 1.02.77 (2012-10-15)
  Driver version: 4.27.0

http://dd6b71949550285df7dc-dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/51/396351/11/check/dsvm-tempest-neutron-network/a922416/logs/screen-c-vol.txt.gz

2017-02-01 23:19:15.287 13448 ERROR cinder.volume.manager [req-656a4302-1d88-4853-a3c6-7d37ef382c98 - -] Failed to initialize driver.
2017-02-01 23:19:15.287 13448 ERROR cinder.volume.manager Traceback (most recent call last):
2017-02-01 23:19:15.287 13448 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/manager.py", line 432, in init_host
2017-02-01 23:19:15.287 13448 ERROR cinder.volume.manager self.driver.check_for_setup_error()
2017-02-01 23:19:15.287 13448 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/drivers/lvm.py", line 301, in check_for_setup_error
2017-02-01 23:19:15.287 13448 ERROR cinder.volume.manager self.configuration.lvm_suppress_fd_warnings))
2017-02-01 23:19:15.287 13448 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/brick/local_dev/lvm.py", line 126, in __init__
2017-02-01 23:19:15.287 13448 ERROR cinder.volume.manager self.pv_list = self.get_all_physical_volumes(root_helper, vg_name)
2017-02-01 23:19:15.287 13448 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/brick/local_dev/lvm.py", line 359, in get_all_physical_volumes
2017-02-01 23:19:15.287 13448 ERROR cinder.volume.manager run_as_root=True)
2017-02-01 23:19:15.287 13448 ERROR cinder.volume.manager File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 400, in execute
2017-02-01 23:19:15.287 13448 ERROR cinder.volume.manager cmd=sanitized_cmd)
2017-02-01 23:19:15.287 13448 ERROR cinder.volume.manager ProcessExecutionError: Unexpected error while running command.
2017-02-01 23:19:15.287 13448 ERROR cinder.volume.manager Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C pvs --noheadings --unit=g -o vg_name,name,size,free --separator | --nosuffix --ignoreskippedcluster
2017-02-01 23:19:15.287 13448 ERROR cinder.volume.manager Exit code: 3
2017-02-01 23:19:15.287 13448 ERROR cinder.volume.manager Stdout: u''
2017-02-01 23:19:15.287 13448 ERROR cinder.volume.manager Stderr: u"/sbin/pvs: unrecognized option '--ignoreskippedcluster'\n Error during parsing of command line.\n"

[Sean McGinnis (sean-mcginnis)]

Something is odd here. My latest and greatest Ubuntu 14.04 install has LVM version 2.2.98, which should fail to pass the check and therefore not pass in --ignoreskippedcluster. And if an install does have an LVM version that passes that check, it should include that option. Will have to investigate more.

[Sean McGinnis (sean-mcginnis)]

Output I see on a "typical" Ubuntu 14.04 install:

$ sudo vgs --version
  LVM version: 2.02.98(2) (2012-10-15)
  Library version: 1.02.77 (2012-10-15)
  Driver version: 4.27.0

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"

[Bob Ball (bob-ball)]

I believe the property is never actually calculated in this instance - I added some tracing commands and it appears that LVM.supports_pvs_ignoreskippedcluster is "not None" (i.e. "if LVM.supports_pvs_ignoreskippedcluster" passes) without executing the property method.

I guess it should actually be "self.supports_pvs_ignoreskippedcluster" otherwise it will just check for the existance of the method on LVM

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/428180

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/428189

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bob Ball (<email address hidden>) on branch: master
Review: https://review.openstack.org/428180
Reason: This doesn't work of course because get_all_physical_volumes is static.

Proposed a reversion for consideration at https://review.openstack.org/428180

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bob Ball (<email address hidden>) on branch: master
Review: https://review.openstack.org/428189
Reason: After some thought, realised there is an obvious fix...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/428180
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=c891ca9a7b972aea5bf363add18481f6bf5e4d05
Submitter: Jenkins
Branch: master

commit c891ca9a7b972aea5bf363add18481f6bf5e4d05
Author: Bob Ball <email address hidden>
Date: Thu Feb 2 14:45:43 2017 +0000

    Convert ignoreskippedcluster check to be static

    Using LVM.supports_XXX will check for the existence of the
    property method, rather than attempting to retrieve the property.

    As the caller is a static method, we can't actually use a property
    here, so fix this to use static methods and cache on the class.
    Fixes I89ffea86d0951fe4c80783a612b6cde76c8838b4.

    Change-Id: I7294ccdf9c0c5858d949b01ad55763ad3129d8ec
    Closes-bug: 1661144

[Jianghua Wang (wjh-fresh)]

Bob, thanks for the quick fixing.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 10.0.0.0rc1 release candidate.