Please add udev rules for InfiniBand/RDMA device nodes

[Roland Dreier]

Binary package hint: udev

There are InfiniBand/RDMA driver modules shipped with the Ubuntu kernel (ib_uverbs, ib_umad, ib_ucm, rdma_ucm at least) that create device nodes in /dev. Userspace (eg libibverbs1, in Ubuntu universe) expects these nodes to be in /dev/infiniband, and it works better if some of them are owned by group "rdma" (which libibverbs1 will create in its postinst script).

Debian's udev ships with rules to do this by default (although a couple of rules are still pending in Debian bug #427993). Please include similar rules in Ubuntu's udev, such as:

KERNEL=="umad*", NAME="infiniband/%k"
KERNEL=="issm*", NAME="infiniband/%k"
KERNEL=="uverbs*", NAME="infiniband/%k"
KERNEL=="ucm*", NAME="infiniband/%k"
KERNEL=="rdma_ucm", NAME="infiniband/%k"

and permission rules such as:

KERNEL=="uverbs*", GROUP="rdma"
KERNEL=="ucm*", GROUP="rdma"
KERNEL=="rdma_ucm", GROUP="rdma"

[Scott James Remnant (Canonical) (canonical-scott)]

Adding the following rules:

KERNEL=="umad[0-9]*", NAME="infiniband/%k"
KERNEL=="issm[0-9]*, NAME="infiniband/%k"
KERNEL=="uverbs[0-9]*, NAME="infiniband/%k"

Neither the ucm or rdma_ucm are mentioned in devices.txt; they need to appear there before we add rules for them.

Permission rules need to be in libibverbs1, since it creates the group.

[Scott James Remnant (Canonical) (canonical-scott)]

udev (113-0ubuntu2) gutsy; urgency=low

  * Set group of nvram to nvram. LP: #108460.
  * Move udevmonitor to /sbin. LP: #119091.
  * Update shlibs of libvolumeid. LP: #121074.
  * Added rules for infiniband nodes. LP: #124990.
  * Don't seed the initial persistent rules, instead wait for the first
    boot. LP: #123559.

 -- Scott James Remnant <email address hidden> Tue, 10 Jul 2007 17:30:28 +0100

[Roland Dreier (roland.dreier)]

Thanks for integrating some of these rules. A few comments on the rules you decided to leave out:

1) The ucm devices are indeed missing from devices.txt. I'll go through LANANA to get them added to the InfiniBand character major and let you know when they're there.

2) The rdma_ucm device is a misc device created with MISC_DYNAMIC_MINOR, so I don't think there's any way to put it in devices.txt. I notice that you already have a rules such as the one for /dev/mapper/control (which is also a misc device with dynamic minor that does not appear in devices.txt):

    KERNEL=="device-mapper", NAME="mapper/control"

so I think it would make sense to also include the rule

    KERNEL=="rdma_ucm", NAME="infiniband/%k"

3) What is the reasoning behind requiring the permission rules to come from the package that creates the group? If the package is never installed and the group does not exist, the rule is just silently ignored. And the group may exist without being created by the package (NIS for example) -- it's certainly within reason for someone to have minimal cluster nodes running with /usr on NFS, so they don't need to install libibverbs1 on their cluster nodes and hence wouldn't get any udev rules except what udev itself ships.

With that said, I'm not _that_ opposed to putting some udev rules in the libibverbs1 package, except that it really complicates my life by forcing the Ubuntu package to be permanently different from the upstream Debian package, because the README.Debian for udev says:

    MAINTAINERS BEWARE: the use of /etc/udev/rules.d/ by other packages is
    discouraged, except when only RUN rules are added.

and the Debian udev package ships with the InfiniBand group rdma rules already.

[Roland Dreier (roland.dreier)]

Resetting status to incomplete -- I think at least the rdma_ucm rules can go in, and I would like to understand the policy about group ownership rules.

[Scott James Remnant (Canonical) (canonical-scott)]

Added the extra two rules.

The reason for the rules policy is that otherwise udev needs to make every single custom group; you are wrong in that they are ignored, udev will still lookup the group which can cause regressions on LDAP systems, for example.

Our preferred solution in Ubuntu is for programs accessing these devices to do it through HAL, which gives us a much better managed access to them; in the next Ubuntu release we'll be dropping much of the custom groups in favour of user ACLs on the devices.

You quoted the README.Debian for the Debian udev package. Ubuntu and Debian have a vastly different udev package and maintenance policy, which is understandable given that in Ubuntu udev is mandatory and so are HAL, D-BUS, etc.

[Scott James Remnant (Canonical) (canonical-scott)]

udev (113-0ubuntu11) gutsy; urgency=low

  * debian/patches/10-git-linux_raid-1.0-metadata.patch:
    - Upstream patch to fix detection of linux_raid metadata v1.0 and
      produce correct UUIDs for such raid devices. LP: #133773

  * 20-names.rules:
    - Place ucm[0-9]* and rdma_ucm into the infiniband dir. LP: #124990.
  * 80-programs.rules:
    - Fix calling of create_floppy_devices to just $tempnode not
      $root/$tempnode, which is just plain wrong. LP: #132546.

 -- Scott James Remnant <email address hidden> Mon, 24 Sep 2007 13:18:41 +0100