nova-compute-lxc limited by available nbd devices to 16 instances

Bug #861504 reported by Scott Moser
28
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Michael Still
nova (Ubuntu)
Fix Released
Low
Unassigned

Bug Description

nova has 'FLAGS.max_nbd_devices' :
   flags.DEFINE_integer('max_nbd_devices', 16,
                     'maximum number of possible nbd devices')

However,
a.) that doesn't really tell you why you care about this number, even though it effectively limits the number of lxc instances you can run at a single time, as each one uses a nbd device.
b.) it does not modify in any way the number that the system can allocate.
   Ie, we would need to change the module parameter 'nbds_max' to nbd.ko.

Perhaps nova-compute-lxc should install a file into /etc/modprobe.d that increases number, ie:
 # /etc/modprobe.d/nova-compute-lxc.conf
 options nbd nbds_max=256

And also set the FLAGS.max_nbd_devices in its nova-compute.conf.

Note, one sucky thing, is that without killing or at least stopping all instances, you wouldn't be able to change the module parameter to increase this number.

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: nova-compute-lxc 2011.3-0ubuntu2
ProcVersionSignature: Ubuntu 3.0.0-12.19-server 3.0.4
Uname: Linux 3.0.0-12-server x86_64
ApportVersion: 1.23-0ubuntu1
Architecture: amd64
Date: Wed Sep 28 11:14:28 2011
NovaConf: Error: [Errno 13] Permission denied: '/etc/nova/nova.conf'
PackageArchitecture: all
ProcEnviron:
 LANGUAGE=en_US:
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: nova
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Scott Moser (smoser) wrote :

Now that i'm thinking a bit more, maybe it would make sense to not have the flag in nova for 'FLAGS.max_nbd_devices'.

Instead, it could just go through /dev/nbd* or /sys/block/nbd* and get a list that way.

Scott Moser (smoser)
summary: - nova-compute-lxc limited by available nbd devices to 16
+ nova-compute-lxc limited by available nbd devices to 16 instances
Changed in nova (Ubuntu):
importance: Undecided → Low
Revision history for this message
Thierry Carrez (ttx) wrote :

Agreed, the parameter seems to be there only to limit the number of nbds Nova will arbitrarily try to grab... and in no way to increase the number the system can handle.

Changed in nova:
importance: Undecided → Medium
status: New → Confirmed
Scott Moser (smoser)
Changed in nova (Ubuntu):
status: New → Confirmed
Revision history for this message
Lawrance (jing) wrote :

i solve the problem by “modprobe nbd nbds_max=30”
and
vi /usr/lib/python2.7/dist-packages/nova/virt/disk/nbd.py
.......
cfg.IntOpt('max_nbd_devices',
               default=30, ##########check it here############
               help='maximum number of possible nbd devices')
......

Revision history for this message
Lorin Hochstein (lorinh) wrote :

Does this bug get resolved by https://review.openstack.org/17886, which removes the max_nbd_devices flag?

Revision history for this message
Michael Still (mikal) wrote :

Yes, if there are more device files than that they will now be used as well.

Changed in nova:
assignee: nobody → Michael Still (mikalstill)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/17886
Committed: http://github.com/openstack/nova/commit/ab3c602ac5456a509057f915a741970afeab0d4b
Submitter: Jenkins
Branch: master

commit ab3c602ac5456a509057f915a741970afeab0d4b
Author: Michael Still <email address hidden>
Date: Mon Dec 10 15:17:12 2012 +1100

    Autodetect nbd devices.

    This means we don't need to the max_nbd_devices flag any more at all.
    This patch also moves the internal representation of the list of
    available nbd devices to not including the leading /dev/, but that is
    not exposed as a change in the interface to the class.

    Resolves bug 861504 and its duplicate 1088339.

    DocImpact.

    Change-Id: I5c6218122ab09781d8a2ecd6d8b76a74be4a6e91

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-2
status: Fix Committed → Fix Released
Chuck Short (zulcss)
Changed in nova (Ubuntu):
status: Confirmed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-2 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.