Cinder Volume deadlock in quota_reserve and reservation_commit
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cinder |
New
|
Medium
|
Unassigned |
Bug Description
Summary:
Deleting an instance with multiple volumes attached is causing deadlocks in cinder-volume quota_reserve and reservation_commit.
Version: 8.0.0
OS: Ubuntu 14.04
Database: Galera Cluster
Impact:
cinder-volume hangs while deadlocks occur, causing instance creation to fail in production and pre-production environments.
Reproduction:
See attached docs which detail the reproduction and configuration.
Basically, create an instance with a volume-from-image root device and six attached volumes(all backed by ceph RBD). Delete this instance with 'nova delete'
Logs: see attached.
Cinder Config:
[DEFAULT]
glance_api_servers = https:/
glance_api_version = 2
enable_v1_api = True
enable_v2_api = True
enable_v3_api = True
storage_
default_
default_volume_type = b1.standard
volume_
auth_strategy = keystone
enabled_backends = b1.standard
osapi_volume_listen = 0.0.0.0
osapi_volume_
scheduler_
volume_backend_name = DEFAULT
rbd_pool = volumes
rbd_user = volumes
rbd_ceph_conf =/etc/ceph/
rbd_secret_uuid = REDACTED
scheduler_
scheduler_driver = cinder.
nova_catalog_info = compute:Compute Service:publicURL
nova_catalog_
os_region_name = test-1
volume_driver = cinder.
debug = False
verbose = True
log_dir = /var/log/cinder
use_syslog = True
syslog_log_facility = LOG_USER
rpc_backend = rabbit
control_exchange = cinder
api_paste_config = /etc/cinder/
notification_
backend_
[BACKEND]
[BRCD_FABRIC_
[CISCO_
[COORDINATION]
[FC-ZONE-MANAGER]
[KEYMGR]
[cors]
[cors.subdomain]
[database]
connection = mysql:/
idle_timeout = 60
[keystone_
auth_uri = https:/
admin_password=
admin_tenant_
identity_uri=https:/
admin_user=cinder
[matchmaker_redis]
[oslo_concurrency]
lock_path = /var/lock/cinder
[oslo_messaging
[oslo_messaging
[oslo_messaging
amqp_durable_queues = False
rabbit_hosts = REDACTED:
rabbit_use_ssl = False
rabbit_userid = cinder
rabbit_password = REDACTED
rabbit_virtual_host = /
rabbit_ha_queues = True
heartbeat_
heartbeat_rate = 2
[oslo_middleware]
[oslo_policy]
policy_file = /etc/cinder/
[oslo_reports]
[oslo_versioned
[ssl]
ca_file = False
cert_file = /REDACTED
key_file = /REDACTED
[b1.standard]
rbd_user=volumes
volume_
backend_
rbd_ceph_
rbd_secret_
rbd_max_
volume_
rbd_pool=volumes
Changed in cinder: | |
importance: | Undecided → Medium |
Changed in cinder: | |
assignee: | nobody → haobing1 (haobing1) |
Changed in cinder: | |
assignee: | haobing1 (haobing1) → nobody |
attaching cinder-volume logs.