Libmlx4 library contains undefined symbol: ibv_cmd_reg_xrc_rcv_qp

[Tim Wright]

Binary package hint: libmlx4-1

Ubuntu release: 10.04LTS.
Package: libmlx4-1 version 1.0-1

The binary for userspace verbs library for the ConnectX Mellanox cards (libmlx4) appears to be "rogue". It apparently was not built from the checked in source for libmlx4-1. Attempts to use it in a system with a ConnectX card fail with an undefined symbol:

libibverbs: Warning: couldn't load driver 'mlx4':
/usr/lib/libmlx4-rdmav2.so: symbol ibv_cmd_reg_xrc_rcv_qp, version
IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference
libibverbs: Warning: no userspace device-specific driver found for
/sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: couldn't load driver 'mlx4':
/usr/lib/libmlx4-rdmav2.so: symbol ibv_cmd_reg_xrc_rcv_qp, version
IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference
Segmentation fault

However, building the library from the checked-in source and using that version works perfectly, i.e. the binary in the repository absolutely could not have been built from the source in the repository.

[Tim Wright (timw)]

Hmmm, this may be a bad mirror.
The bad library:
-rw-r--r-- 1 root root 29136 Sep 23 2009 /usr/lib/libmlx4-rdmav2.so.BROKEN
And the same library on a different system:
-rw-r--r-- 1 root root 24536 2008-05-12 04:30 /usr/lib/libmlx4-rdmav2.so

[Cody Lee (platinummonkey)]

Confirming this problem as it also affects OpenMPI. Jobs still process, but I'm not sure how much it affects jobs.

nemo:~$ mpirun -np 13 -machinefile mpd.hosts hello
libibverbs: Warning: couldn't load driver 'mlx4': /usr/lib/libmlx4-rdmav2.so: symbol ibv_cmd_reg_xrc_rcv_qp, version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference
... x 13 ...
Process 12 on nemo out of 13
Process 0 on tile-0-0 out of 13
Process 7 on tile-1-3 out of 13
Process 1 on tile-0-1 out of 13
Process 8 on tile-2-0 out of 13
Process 3 on tile-0-3 out of 13
Process 2 on tile-0-2 out of 13
Process 5 on tile-1-1 out of 13
Process 11 on tile-2-3 out of 13
Process 9 on tile-2-1 out of 13
Process 4 on tile-1-0 out of 13
Process 6 on tile-1-2 out of 13
Process 10 on tile-2-2 out of 13