see "No space left on device" error while running cinder API concurrently

[Harshil]

I have below scenario:
* Create a nova instance
* Associate floating ip
* Create cinder volume of 1 GB
* Attach cinder volume to above instance
* Perform file system operation like mount, unmount etc
* Detach cinder volume
* Delete instance
* Delete volume.

This above scenario works fine when I run it sequentially. Although if I run this scenario in parallel, with two threads doing this scenario at same time I see below error and traceback. I looked around openstack bug tracker and search on web in general. I could not find this being reported by anyone, so opening a bug for this. Note in my test setup I have enough memory to create more volumes and even compute nodes have plenty of memory.

I have attached cinder-all.log and nova-all.log with this bug also.

Below is the traceback:
<155>Sep 17 23:40:24 node-2 cinder-cinder.api.middleware.fault ERROR: Caught error: [Errno 28] No space left on device
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cinder/api/middleware/fault.py", line 75, in __call__
    return req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 582, in __call__
    return self.app(env, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py", line 895, in __call__
    content_type, body, accept)
  File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py", line 943, in _process_stack
    action_result = self.dispatch(meth, request, action_args)
  File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py", line 1019, in dispatch
    return method(req=request, **action_args)
  File "/usr/lib/python2.7/dist-packages/cinder/api/contrib/volume_actions.py", line 195, in _initialize_connection
    connector)
  File "/usr/lib/python2.7/dist-packages/cinder/volume/api.py", line 75, in wrapped
    return func(self, context, target_obj, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinder/volume/api.py", line 423, in initialize_connection
    connector)
  File "/usr/lib/python2.7/dist-packages/cinder/volume/rpcapi.py", line 120, in initialize_connection
    connector=connector)
  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
    wait_for_reply=True, timeout=timeout)
  File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
    timeout=timeout)
  File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 409, in send
    return self._send(target, ctxt, message, wait_for_reply, timeout)
  File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 402, in _send
    raise result
IOError: [Errno 28] No space left on device
<158>Sep 17 23:40:24 node-2 cinder-cinder.api.middleware.fault INFO: http://172.22.191.151:8776/v1/c2038e77062248d6bacf2e703f17f31c/volumes/2f72011e-0180-4e7d-abeb-ea73dbb405c9/action returned with HTTP 500

[Harshil (harsshah)]

nova logs

[Huang Zhiteng (zhiteng-huang)]

Have you checked if the disk in your Cinder nodes is full? RabbitMQ would stop functioning if disk is full. This one likes like it.

[Harshil (harsshah)]

Ok thanks Huang. I think found the issue, one of the instance was getting
spawned on node which I was not aware of and that was low running low in
memory. So that was the root cause of the issue. Thanks for pointing out
the problem.

Thanks,
Harshil

On 9/17/14, 5:34 PM, "Huang Zhiteng" <email address hidden> wrote:

>Have you checked if the disk in your Cinder nodes is full? RabbitMQ
>would stop functioning if disk is full. This one likes like it.
>
>--
>You received this bug notification because you are subscribed to the bug
>report.
>https://bugs.launchpad.net/bugs/1370774
>
>Title:
> see "No space left on device" error while running cinder API
> concurrently
>
>Status in Cinder:
> New
>
>Bug description:
> I have below scenario:
> * Create a nova instance
> * Associate floating ip
> * Create cinder volume of 1 GB
> * Attach cinder volume to above instance
> * Perform file system operation like mount, unmount etc
> * Detach cinder volume
> * Delete instance
> * Delete volume.
>
> This above scenario works fine when I run it sequentially. Although if
> I run this scenario in parallel, with two threads doing this scenario
> at same time I see below error and traceback. I looked around
> openstack bug tracker and search on web in general. I could not find
> this being reported by anyone, so opening a bug for this. Note in my
> test setup I have enough memory to create more volumes and even
> compute nodes have plenty of memory.
>
> I have attached cinder-all.log and nova-all.log with this bug also.
>
> Below is the traceback:
> <155>Sep 17 23:40:24 node-2 cinder-cinder.api.middleware.fault ERROR:
>Caught error: [Errno 28] No space left on device
> Traceback (most recent call last):
> File
>"/usr/lib/python2.7/dist-packages/cinder/api/middleware/fault.py", line
>75, in __call__
> return req.get_response(self.application)
> File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296,
>in send
> application, catch_exc_info=False)
> File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260,
>in call_application
> app_iter = application(self.environ, start_response)
> File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in
>__call__
> return resp(environ, start_response)
> File
>"/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py"
>, line 582, in __call__
> return self.app(env, start_response)
> File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in
>__call__
> return resp(environ, start_response)
> File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in
>__call__
> return resp(environ, start_response)
> File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line
>131, in __call__
> response = self.app(environ, start_response)
> File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in
>__call__
> return resp(environ, start_response)
> File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in
>__call__
> resp = self.call_func(req, *args, **self.kwargs)
> File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in
>call_func
> retu...

[Duncan Thomas (duncan-thomas)]

Marked as invalid given comment #4

Thanks for getting back to us, Harshil