Cinder

Can't get lock for cinder-volumes for simple vgs

Bug #1220275 reported by Dafna Ron on 2013-09-03

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Cinder	Invalid	Undecided	sandeep mane

Bug Description

due to a bug in delete snaoshots I have several delete and create tasks for volme and volume snapshots.

when I try to run a simple vgs , the command get stuck and I have no option other than terminating the command:

[root@opens-vdsb ~(keystone_admin)]# vgs
  /dev/cinder-volumes/_snapshot-69f4bd94-f7df-40bf-94ba-f9e1f6661a3f: read failed after 0 of 4096 at 10737352704: Input/output error
  /dev/cinder-volumes/_snapshot-69f4bd94-f7df-40bf-94ba-f9e1f6661a3f: read failed after 0 of 4096 at 10737410048: Input/output error
  /dev/cinder-volumes/_snapshot-69f4bd94-f7df-40bf-94ba-f9e1f6661a3f: read failed after 0 of 4096 at 0: Input/output error
  /dev/cinder-volumes/_snapshot-69f4bd94-f7df-40bf-94ba-f9e1f6661a3f: read failed after 0 of 4096 at 4096: Input/output error
  /dev/cinder-volumes/_snapshot-24d4e530-5eef-4f4c-8b3b-c1dd51f1b295: read failed after 0 of 4096 at 10737352704: Input/output error
  /dev/cinder-volumes/_snapshot-24d4e530-5eef-4f4c-8b3b-c1dd51f1b295: read failed after 0 of 4096 at 10737410048: Input/output error
  /dev/cinder-volumes/_snapshot-24d4e530-5eef-4f4c-8b3b-c1dd51f1b295: read failed after 0 of 4096 at 0: Input/output error
  /dev/cinder-volumes/_snapshot-24d4e530-5eef-4f4c-8b3b-c1dd51f1b295: read failed after 0 of 4096 at 4096: Input/output error
  /dev/cinder-volumes/_snapshot-0f48b93d-32bb-4078-89be-76ff6468ee97: read failed after 0 of 4096 at 10737352704: Input/output error
  /dev/cinder-volumes/_snapshot-0f48b93d-32bb-4078-89be-76ff6468ee97: read failed after 0 of 4096 at 10737410048: Input/output error
  /dev/cinder-volumes/_snapshot-0f48b93d-32bb-4078-89be-76ff6468ee97: read failed after 0 of 4096 at 0: Input/output error
  /dev/cinder-volumes/_snapshot-0f48b93d-32bb-4078-89be-76ff6468ee97: read failed after 0 of 4096 at 4096: Input/output error
  /dev/cinder-volumes/_snapshot-5a14b4c4-9a14-4de6-abd2-236208bc1429: read failed after 0 of 4096 at 10737352704: Input/output error
  /dev/cinder-volumes/_snapshot-5a14b4c4-9a14-4de6-abd2-236208bc1429: read failed after 0 of 4096 at 10737410048: Input/output error
  /dev/cinder-volumes/_snapshot-5a14b4c4-9a14-4de6-abd2-236208bc1429: read failed after 0 of 4096 at 0: Input/output error
  /dev/cinder-volumes/_snapshot-5a14b4c4-9a14-4de6-abd2-236208bc1429: read failed after 0 of 4096 at 4096: Input/output error
  /dev/cinder-volumes/_snapshot-9a68a5b8-781c-4661-be5c-6a791ff79370: read failed after 0 of 4096 at 10737352704: Input/output error
  /dev/cinder-volumes/_snapshot-9a68a5b8-781c-4661-be5c-6a791ff79370: read failed after 0 of 4096 at 10737410048: Input/output error
  /dev/cinder-volumes/_snapshot-9a68a5b8-781c-4661-be5c-6a791ff79370: read failed after 0 of 4096 at 0: Input/output error
  /dev/cinder-volumes/_snapshot-9a68a5b8-781c-4661-be5c-6a791ff79370: read failed after 0 of 4096 at 4096: Input/output error
^C CTRL-c detected: giving up waiting for lock
  /var/lock/lvm/V_cinder-volumes:aux: flock failed: Interrupted system call
  Can't get lock for cinder-volumes
  VG #PV #LV #SN Attr VSize VFree
  vg0 1 2 0 wz--n- 232.69g 0

this exposes an issue on locks with lvm since we appearantly cannot run any commands at all requiring a lock on the vg if we have a stuck task.

Revision history for this message

Dafna Ron (dron-3) wrote on 2013-09-03:

volumes log Edit (1.0 KiB, application/octet-stream)

Revision history for this message

sandeep mane (sandeep-mane) wrote on 2014-02-18:

hi,

I have followed below steps to reproduce
Wrote 3 scripts to execute cinder commands parallely. ,
                      In script I have created 3 volumes using cinder create commands.
                      Given sleep of 90 seconds so that volumes are “available”
                      Deleted 3 created volumes using cinder delete command.
Executed 1 script on one terminal. So 3 terminals total 9 volumes will be created.

On 4th terminal I have executed Sudo VGS command to check if it give some locking error or not.
I found, VGS is working properly without getting any error

##The same thing I have tried using snapshot create on the volume. With this snapshot-create also I am not getting any issue.

can you please give us exact scenario where you got this error

--Sandeep.

Changed in cinder:
assignee:	nobody → sandeep mane (sandeep-mane)

Revision history for this message

John Griffith (john-griffith) wrote on 2014-02-18:

I suspect you're hitting my favorite issue:
https://bugs.launchpad.net/cinder/+bug/1023755

It's pretty random, and frankly the best solution I've come up with are:
1. use thin volumes
2. set secure_delete to False

It's not a great answer, but I have no luck with the kernel team on this. It is interesting if you're running RHEL or a variant and seeing this as it was always reported by folks that this was an Ubuntu only bug?

Revision history for this message

Dafna Ron (dron-3) wrote on 2014-02-18:

mine was on rhel6

Revision history for this message

sandeep mane (sandeep-mane) wrote on 2014-02-28:

Hi,

Can you please give us more information about exact scenario
if there is any reproducible sequence that will be helpful

Thanks,
Sandeep.

Changed in cinder:
status:	New → Incomplete

Revision history for this message

Dafna Ron (dron-3) wrote on 2014-02-28:

you need local lvs storage to reproduce this.
create/delete a volume snapshots while volume service is down and than start it right away.
do this several times until you have several snapshots stuck and than run lvs

Revision history for this message

Sean McGinnis (sean-mcginnis) wrote on 2015-10-31: Cleanup

Closing stale bug. If this is still an issue please reopen.