volume create fails in /usr/sbin/thin_check

[Scott DAngelo]

volume create fails in /usr/sbin/thin_check: execvp failed: No such file or directory

To reproduce:
Use Ubuntu 16.10
I used Gorka's HA testing local.conf, but reproduced this on master (also repro's on his patches)
curl -o local.conf http://gorka.eguileor.com/files/cinder/manual_ha_aa_local.conf

$ cinder create --name mydisk 1

2016-08-19 15:22:05.936 ERROR cinder.brick.local_dev.lvm [req-9d5bf5c2-52d9-4e51-8442-dd4402f4eb2f 9feac1ca74a84c748f875f64ae5396cf e64bdee31c0747c4b2d2bbe313baf5ed] Error creating Volume
2016-08-19 15:22:05.936 TRACE cinder.brick.local_dev.lvm Traceback (most recent call last):
2016-08-19 15:22:05.936 TRACE cinder.brick.local_dev.lvm File "/opt/stack/cinder/cinder/brick/local_dev/lvm.py", line 534, in create_volume
2016-08-19 15:22:05.936 TRACE cinder.brick.local_dev.lvm run_as_root=True)
2016-08-19 15:22:05.936 TRACE cinder.brick.local_dev.lvm File "/opt/stack/cinder/cinder/utils.py", line 121, in execute
2016-08-19 15:22:05.936 TRACE cinder.brick.local_dev.lvm return processutils.execute(*cmd, **kwargs)
2016-08-19 15:22:05.936 TRACE cinder.brick.local_dev.lvm File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 389, in execute
2016-08-19 15:22:05.936 TRACE cinder.brick.local_dev.lvm cmd=sanitized_cmd)
2016-08-19 15:22:05.936 TRACE cinder.brick.local_dev.lvm ProcessExecutionError: Unexpected error while running command.
2016-08-19 15:22:05.936 TRACE cinder.brick.local_dev.lvm Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C lvcreate -T -V 1g -n volume-add4b37b-6194-4fa4-97f4-d0739ba4e890 stack-volumes-lvmdriver-1/stack-volumes-lvmdriver-1-pool
2016-08-19 15:22:05.936 TRACE cinder.brick.local_dev.lvm Exit code: 5
2016-08-19 15:22:05.936 TRACE cinder.brick.local_dev.lvm Stdout: u''
2016-08-19 15:22:05.936 TRACE cinder.brick.local_dev.lvm Stderr: u' /usr/sbin/thin_check: execvp failed: No such file or directory\n Check of pool stack-volumes-lvmdriver-1/stack-volumes-lvmdriver-1-pool failed (status:2). Manual repair required!\n Aborting. Failed to locally activate thin pool stack-volumes-lvmdriver-1/stack-volumes-lvmdriver-1-pool.\n'
2016-08-19 15:22:05.936 TRACE cinder.brick.local_dev.lvm
2016-08-19 15:22:05.941 ERROR cinder.brick.local_dev.lvm [req-9d5bf5c2-52d9-4e51-8442-dd4402f4eb2f 9feac1ca74a84c748f875f64ae5396cf e64bdee31c0747c4b2d2bbe313baf5ed] Cmd :sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C lvcreate -T -V 1g -n volume-add4b37b-6194-4fa4-97f4-d0739ba4e890 stack-volumes-lvmdriver-1/stack-volumes-lvmdriver-1-pool
2016-08-19 15:22:05.943 ERROR cinder.brick.local_dev.lvm [req-9d5bf5c2-52d9-4e51-8442-dd4402f4eb2f 9feac1ca74a84c748f875f64ae5396cf e64bdee31c0747c4b2d2bbe313baf5ed] StdOut :
2016-08-19 15:22:05.944 ERROR cinder.brick.local_dev.lvm [req-9d5bf5c2-52d9-4e51-8442-dd4402f4eb2f 9feac1ca74a84c748f875f64ae5396cf e64bdee31c0747c4b2d2bbe313baf5ed] StdErr : /usr/sbin/thin_check: execvp failed: No such file or directory
  Check of pool stack-volumes-lvmdriver-1/stack-volumes-lvmdriver-1-pool failed (status:2). Manual repair required!
  Aborting. Failed to locally activate thin pool stack-volumes-lvmdriver-1/stack-volumes-lvmdriver-1-pool.

[Gorka Eguileor (gorka)]

Tested this on Fedora 23 and is working fine, no issues there.

[Eric Harney (eharney)]

Ubuntu's LVM packaging doesn't seem to be setup right to support thin provisioning by default.
/usr/sbin/thin_check: execvp failed: No such file or directory

/usr/sbin/thin_check is in the thin-provisioning-tools package:
  http://packages.ubuntu.com/search?suite=xenial&arch=any&mode=exactfilename&searchon=contents&keywords=thin_check

The lvm2 package only Suggests thin-provisioning-tools rather than requiring it:
  http://packages.ubuntu.com/xenial/lvm2

Installing the thin-provisioning-tools package should fix this.

[Eric Harney (eharney)]

If that fixes things, we should add thin-provisioning-tools as a package to install in devstack files/apts/cinder.

[Scott DAngelo (scott-dangelo)]

Note that this will also repro on Ubuntu 14.04

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/366309

[Scott DAngelo (scott-dangelo)]

Note that this will also repro on Ubuntu 14.04
Code submitted:
https://review.openstack.org/#/c/366309/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/366309
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=f3daa9548efd53e09d141f588342d4d94683b6c8
Submitter: Jenkins
Branch: master

commit f3daa9548efd53e09d141f588342d4d94683b6c8
Author: scottda <email address hidden>
Date: Tue Sep 6 13:45:32 2016 -0600

    Add thin-provisioning-tools for Cinder

    Ubuntu's LVM packaging does not support thin provisioning by
    default:
     /usr/sbin/thin_check: execvp failed: No such file or directory
    This is fixed with install of thin-provisioning-tools.

    Change-Id: I31f572934ea94cae6e2aea27a2c731ee5bca68d3
    Closes-Bug: #1615134

[Gheorghe Ungureanu (gunix)]

This is still a problem in pike. When you install cinder, you don't get thin-provisioning-tools and all volume deployments will fail. It gets fixed by installing the tools.

[Sean McGinnis (sean-mcginnis)]

This is a required system package when using lvm thin provisioning. It is not a component of cinder.