OpenStack Compute (nova)

add volume fails over 26vols and returns 500 API error with libvirt driver

Bug #1770527 reported by Tsuyoshi Nagata
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
melanie witt
Pike
Confirmed
Low
Unassigned
Queens
Confirmed
Low
Unassigned
Rocky
Confirmed
Low
Unassigned

Bug Description

Description
===========
openstack server add volume fails over 26vols

Steps to reproduce
==================
* I did execute this openstack command.

# create a instance with single volume named sles15rc
# openstack server add volume sles15rc vol2
# openstack server add volume sles15rc vol3
     :
# openstack server add volume sles15rc vol26
# openstack server add volume sles15rc vol27
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'NovaException_Remote'> (HTTP 500) (Request-ID: req-d95fea94-31fe-4063-9262-a84088cbaf29)
#

Expected result
===============
This command will success.
# openstack server add volume sles15rc vol27
and instance will had '/dev/vdaa' device volumes.

Actual result
=============
# openstack server add volume sles15rc vol26
 instance will get named /dev/vdz volume.
Next,
# openstack server add volume sles15rc vol27
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'NovaException_Remote'> (HTTP 500) (Request-ID: req-d95fea94-31fe-4063-9262-a84088cbaf29)

I did kvm based command, Its OK.

#virsh attach-device instance-0000001e ~/vol27.xml
#virsh attach-device instance-0000001e ~/vol28.xml

 and instance will had a volumes, vdaa,vdab
It's nova limitation problem.

I made a concept blueprint.
https://blueprints.launchpad.net/nova/+spec/nova-improvement-of-maximum-attach-volumes-more-than-26-vols

Environment
===========

SOC7(Suse Openstack Cloud version 7 openstack-newton)

Logs & Configs
==============
Nova API returned this result.
 NovaException_Remote: No free disk device names for prefix 'vd'

2018-05-11 08:51:11.602 3667 INFO nova.osapi_compute.wsgi.server [req-6cf5fdbf-9681-445b-8117-a61acd4057f4 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
5d680487f6f19 - default default] 10.19.3.70 "POST /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers/6a324dc8-d2ea-4eba-9870-e2ada1cb2bf4/os-volume_attachments HTT
P/1.1" status: 200 len: 522 time: 0.2834640
2018-05-11 08:51:11.716 3668 INFO nova.osapi_compute.wsgi.server [-] 127.0.0.1 "GET / HTTP/1.1" status: 200 len: 491 time: 0.0007210
2018-05-11 08:51:13.880 3668 INFO nova.api.openstack.wsgi [req-c7fc0e7a-13ec-48e7-947c-cd22232aa31f 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b705d68048
7f6f19 - default default] HTTP exception thrown: Instance sles15rc could not be found.
2018-05-11 08:51:13.881 3668 INFO nova.osapi_compute.wsgi.server [req-c7fc0e7a-13ec-48e7-947c-cd22232aa31f 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
5d680487f6f19 - default default] 10.19.3.70 "GET /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers/sles15rc HTTP/1.1" status: 404 len: 432 time: 0.0449822
2018-05-11 08:51:13.887 3668 INFO nova.api.openstack.wsgi [req-7cbf4817-8810-4c30-8e09-8b106f66dee4 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b705d68048
7f6f19 - default default] HTTP exception thrown: Instance sles15rc could not be found.
2018-05-11 08:51:13.889 3668 INFO nova.osapi_compute.wsgi.server [req-7cbf4817-8810-4c30-8e09-8b106f66dee4 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
5d680487f6f19 - default default] 10.19.3.70 "GET /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers/sles15rc HTTP/1.1" status: 404 len: 432 time: 0.0053861
2018-05-11 08:51:13.929 3668 INFO nova.osapi_compute.wsgi.server [req-40067327-b2a6-4194-acc6-17696377d123 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
5d680487f6f19 - default default] 10.19.3.70 "GET /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers?name=sles15rc HTTP/1.1" status: 200 len: 687 time: 0.0372620
2018-05-11 08:51:14.096 3668 INFO nova.osapi_compute.wsgi.server [req-4a17ff34-f457-4e4b-8975-ad9fafa2e635 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
5d680487f6f19 - default default] 10.19.3.70 "GET /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers/6a324dc8-d2ea-4eba-9870-e2ada1cb2bf4 HTTP/1.1" status: 200 len:
 3110 time: 0.1642580
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions [req-d95fea94-31fe-4063-9262-a84088cbaf29 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
5d680487f6f19 - default default] Unexpected exception in API method
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapp
ed
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/volumes.py", line 325, in
create
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions volume_id, device)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 166, in inner
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions return function(self, context, instance, *args, **kwargs)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 147, in inner
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions return f(self, context, instance, *args, **kw)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3518, in attach_volume
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions disk_bus, device_type)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3455, in _attach_volume
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions device_type=device_type)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3437, in _create_volume_bd
m
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions device_type=device_type)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 926, in reserve_block_d
evice_name
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions return cctxt.call(ctxt, 'reserve_block_device_name', **kw)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions retry=self.retry)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions timeout=timeout, retry=retry)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions retry=retry)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions raise result
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions NovaException_Remote: No free disk device names for prefix 'vd'
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions res = self.dispatcher.dispatch(message)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions return self._do_dispatch(endpoint, method, ctxt, args)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions result = func(ctxt, **new_args)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions function_name, call_dict, binary)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions self.force_reraise()
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions return f(self, context, *args, **kw)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions LOG.warning(msg, e, instance=instance)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions self.force_reraise()
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions return function(self, context, *args, **kwargs)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions kwargs['instance'], e, sys.exc_info())
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions self.force_reraise()
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions return function(self, context, *args, **kwargs)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4683, in reserve_block_device_name
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions return do_reserve()
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4677, in do_reserve
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions instance, bdms, new_bdm)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1496, in _get_device_name_for_instance
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions instance, bdms, block_device_obj)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7864, in get_device_name_for_instance
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions block_device_obj, mapping=instance_info['mapping'])
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 394, in get_info_from_bdm
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions device_name = find_disk_dev_for_disk_bus(padded_mapping, bdm_bus)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 197, in find_disk_dev_for_disk_bus
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions dev_prefix)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions NovaException: No free disk device names for prefix 'vd'
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.613 3668 INFO nova.api.openstack.wsgi [req-d95fea94-31fe-4063-9262-a84088cbaf29 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b705d680487f6f19 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'NovaException_Remote'>

See original description
Tags: libvirt
Tsuyoshi Nagata (yukari-papa)
Changed in nova:
assignee: nobody → Tsuyoshi Nagata (yukari-papa)
description: updated
Tsuyoshi Nagata (yukari-papa)
Changed in nova:
status: New → In Progress
Tsuyoshi Nagata (yukari-papa)
description: updated
Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → Wishlist
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/573066

Revision history for this message
melanie witt (melwitt) wrote : Re: openstack server add volume fails over 26vols and returns 500 API error

I'm going to go ahead and use this bug for the 500 API error specifically. The work for being able to increase the maximum number of disk devices allowed to attach to a single instance with the libvirt driver is part of a blueprint: https://blueprints.launchpad.net/nova/+spec/conf-max-attach-volumes

summary: - openstack server add volume fails over 26vols
+ openstack server add volume fails over 26vols and returns 500 API error
Changed in nova:
importance: Wishlist → Medium
melanie witt (melwitt)
tags: added: libvirt
summary: - openstack server add volume fails over 26vols and returns 500 API error
+ add volume fails over 26vols and returns 500 API error with libvirt
+ driver
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/632904

Changed in nova:
assignee: Tsuyoshi Nagata (yukari-papa) → melanie witt (melwitt)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/632904
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6489f2d2b44827d133dad9a3bb52436ee304a934
Submitter: Zuul
Branch: master

commit 6489f2d2b44827d133dad9a3bb52436ee304a934
Author: melanie witt <email address hidden>
Date: Fri Jan 18 16:30:40 2019 +0000

    Raise 403 instead of 500 error from attach volume API

    Currently, the libvirt driver has a limit on the maximum number of
    disk devices allowed to attach to a single instance of 26. If a user
    attempts to attach a volume which would make the total number of
    attached disk devices > 26 for the instance, the user receives a
    500 error from the API.

    This adds a new exception type TooManyDiskDevices and raises it for the
    "No free disk devices names" condition, instead of InternalError, and
    handles it in the attach volume API. We raise TooManyDiskDevices
    directly from the libvirt driver because InternalError is ambiguous and
    can be raised for different error reasons within the same method call.

    Closes-Bug: #1770527

    Change-Id: I1b08ed6826d7eb41ecdfc7102e5e8fcf3d1eb2e1

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 19.0.0.0rc1

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.