volume stuck in creating/deleting when command is sent when qpid restarts

Bug #1282017 reported by Dafna Ron
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Cinder
Invalid
Medium
Sheel Rana
oslo.messaging
Won't Fix
Undecided
Unassigned

Bug Description

I think that it's basically a race but I slowed it down and it's reproduced 100%

I work on a stand alone cinder server+controller
1. stop qpid on the controller
2. run 'cinder create --display-name <name> <size>'
3. start qpid.

Results:
volume is stuck in creating state

Revision history for this message
ugvddm (271025598-9) wrote :
Revision history for this message
Dafna Ron (dron-3) wrote :

actually I am not sure...
I can't see that the command ever reaches the scheduler or it's not logged.
the only mention of the volume is in the api log

Revision history for this message
Sai Kiran (saikiran) wrote :

I am also facing the same issue, with my environment. Below are the environment details:

Platform: Ubuntu 14.04
AMQP: RabbitMQ Server
OS Version: Juno
Cinder Backend: LVM

Revision history for this message
Sheel Rana (ranasheel2000) wrote :
Download full text (4.0 KiB)

Its not same as https://bugs.launchpad.net/cinder/+bug/1263691.
In this case, command is not reached to message queue itself.

Problem details:
1. stop rabbitMQ service.
root@sheelrana-VirtualBox:~# service rabbitmq-server stop
 * Stopping message broker rabbitmq-server [ OK ]
root@sheelrana-VirtualBox:~#
root@sheelrana-VirtualBox:~# cinder create --display-name=sheel 1

2. Execute Create Volume command
|root@sheelrana-VirtualBox:~# cinder create --display-name=sheel 1
|

After executing this command to create volume, prompt never returns as infinite loop exists to wait for rabbitMQ service.

3. Start rabbitMQ service:
As and when rabbitMQ service is started, prompt returns the same time and displays new volume details but status will be stuck in creating forever.

+---------------------------------------+--------------------------------------+
| Property | Value |
+---------------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2016-01-09T19:58:03.000000 |
| description | None |
| encrypted | False |
| id | 0d21fbc6-7ebc-4208-b93b-8b077ef1dbb1 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | sheel |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 7ad11ebf5d25451daca4916f4bcea30b |
| os-volume-replication:driver_data | None |
| os-volume-replication:extended_status | None |
| replication_status | disabled |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | e3a0c3ac18a44f1e8f5c97a855813a81 |
| volume_type | lvmdrive...

Read more...

Revision history for this message
Sheel Rana (ranasheel2000) wrote :

There is TimetoLive(TTL) config parameter which can handle situation in case rabbitMQ is in running but recipient service is inactive.

But need to check if there is any such configuration or timeout possible to handle situation of returning control to cinder-api in case rabbitMQ itself is not running.

In case no configuration exists, we can modify code with timeout value and exits process after clearing resources like reserved quota etc.
I think clear resource itself exists in task flow of create volume, so need to look for timeout code flow/configuration.

Changed in cinder:
status: New → Confirmed
assignee: nobody → Sheel Rana (ranasheel2000)
Changed in cinder:
importance: Undecided → Medium
Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

This seems like something that would need to be addressed in oslo.messaging to really fix this category of issues. Adding to that project to have it evaluated. Up to that team if there is something that can be cleanly addressed there.

Marking as invalid for Cinder for now.

Changed in cinder:
status: Confirmed → Invalid
Revision history for this message
Mehdi Abaakouk (sileht) wrote :

This driver doesn't exist anymore

Changed in oslo.messaging:
status: New → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.