linux-headers-2.6.31-16-server; DRBD module can't be loaded; Unknown symbol cn_add_callback

[Mario Kothe]

Binary package hint: linux-image-2.6.31-16-server

After update from 2.6.31-15 to 2.6.31-16 the drbd module can not be loaded.

Messages from: /var/log/messages

kernel: [ 116.170410] drbd: disagrees about version of symbol cn_add_callback
kernel: [ 116.170419] drbd: Unknown symbol cn_add_callback

Temporary solution: boot up with 2.6.31-15 which works fine.

[Stefan Bader (smb)]

I believe there might be two versions of the module: the in-kernel and a DKMS external one. When booted into the -16 kernel, can you do a "modinfo drbd" and attach the output. While being on it, also do a "dkms status" and also attach that output to the bug report.

[Mario Kothe (mario-kothe)]

Hello,

so I started with the -16 kernel and as requested here are the outputs from modinfo and dkms

#modinfo drbd:

On Mon, 2009-12-14 at 13:21 +0000, Stefan Bader wrote:modinfo drbd
filename: /lib/modules/2.6.31-16-server/updates/dkms/drbd.ko
alias: block-major-147-*
license: GPL
version: 8.3.3
description: drbd - Distributed Replicated Block Device v8.3.3
author: Philipp Reisner <email address hidden>, Lars Ellenberg <email address hidden>
srcversion: 8F005DC598E07E771AA1BA8
depends:
vermagic: 2.6.31-15-server SMP mod_unload modversions
parm: minor_count:Maximum number of drbd devices (1-255) (uint)
parm: disable_sendpage:bool
parm: allow_oos:DONT USE! (bool)
parm: cn_idx:uint
parm: proc_details:int
parm: enable_faults:int
parm: fault_rate:int
parm: fault_count:int
parm: fault_devs:int
parm: usermode_helper:string

#dkms status:

drbd8, 8.3.3, 2.6.31-15-server, x86_64: installed
drbd8, 8.3.3, 2.6.31-16-generic, x86_64: installed
drbd8, 8.3.3, 2.6.31-16-server, x86_64: installed
drbd8, 8.3.3, 2.6.31-14-server, x86_64: installed
drbd8, 8.3.3, 2.6.31-15-generic, x86_64: installed

Well, didn't notice it earlier, but modinfo is showing 2.6.31-15 which it should not. Just clear out the the system is indeed using the -16 kenrel, uname -a output follows:

Linux archangel 2.6.31-16-server #53-Ubuntu SMP Tue Dec 8 05:08:02 UTC 2009 x86_64 GNU/Linux

And the drbd module for 2.6.31-16 does exist. Because a locate drbd.ko returns the following reuslt:

#locate drbd.ko

/lib/modules/2.6.31-11-server/kernel/ubuntu/drbd/drbd.ko
/lib/modules/2.6.31-14-server/kernel/ubuntu/drbd/drbd.ko
/lib/modules/2.6.31-14-server/updates/dkms/drbd.ko
/lib/modules/2.6.31-15-generic/updates/dkms/drbd.ko
/lib/modules/2.6.31-15-server/kernel/ubuntu/drbd/drbd.ko
/lib/modules/2.6.31-15-server/updates/dkms/drbd.ko
/lib/modules/2.6.31-16-generic/updates/dkms/drbd.ko
/lib/modules/2.6.31-16-server/kernel/ubuntu/drbd/drbd.ko
/lib/modules/2.6.31-16-server/updates/dkms/drbd.ko
/var/lib/dkms/drbd8/8.3.3/2.6.31-14-server/x86_64/module/drbd.ko
/var/lib/dkms/drbd8/8.3.3/2.6.31-15-generic/x86_64/module/drbd.ko
/var/lib/dkms/drbd8/8.3.3/2.6.31-15-server/x86_64/module/drbd.ko
/var/lib/dkms/drbd8/8.3.3/2.6.31-16-generic/x86_64/module/drbd.ko
/var/lib/dkms/drbd8/8.3.3/2.6.31-16-server/x86_64/module/drbd.ko

[Mario Kothe (mario-kothe)]

Hello,

so I started with the -16 kernel and as requested here are the outputs
from modinfo and dkms

#modinfo drbd:

On Mon, 2009-12-14 at 13:21 +0000, Stefan Bader wrote:modinfo drbd
filename: /lib/modules/2.6.31-16-server/updates/dkms/drbd.ko
alias: block-major-147-*
license: GPL
version: 8.3.3
description: drbd - Distributed Replicated Block Device v8.3.3
author: Philipp Reisner <email address hidden>, Lars Ellenberg
<email address hidden>
srcversion: 8F005DC598E07E771AA1BA8
depends:
vermagic: 2.6.31-15-server SMP mod_unload modversions
parm: minor_count:Maximum number of drbd devices (1-255)
(uint)
parm: disable_sendpage:bool
parm: allow_oos:DONT USE! (bool)
parm: cn_idx:uint
parm: proc_details:int
parm: enable_faults:int
parm: fault_rate:int
parm: fault_count:int
parm: fault_devs:int
parm: usermode_helper:string

#dkms status:

drbd8, 8.3.3, 2.6.31-15-server, x86_64: installed
drbd8, 8.3.3, 2.6.31-16-generic, x86_64: installed
drbd8, 8.3.3, 2.6.31-16-server, x86_64: installed
drbd8, 8.3.3, 2.6.31-14-server, x86_64: installed
drbd8, 8.3.3, 2.6.31-15-generic, x86_64: installed

Well, didn't notice it earlier, but modinfo is showing 2.6.31-15 which
it should not. Just clear out the the system is indeed using the -16
kenrel, uname -a output follows:

Linux archangel 2.6.31-16-server #53-Ubuntu SMP Tue Dec 8 05:08:02 UTC
2009 x86_64 GNU/Linux

And the drbd module for 2.6.31-16 does exist. Because a locate drbd.ko
returns the following reuslt:

#locate drbd.ko

/lib/modules/2.6.31-11-server/kernel/ubuntu/drbd/drbd.ko
/lib/modules/2.6.31-14-server/kernel/ubuntu/drbd/drbd.ko
/lib/modules/2.6.31-14-server/updates/dkms/drbd.ko
/lib/modules/2.6.31-15-generic/updates/dkms/drbd.ko
/lib/modules/2.6.31-15-server/kernel/ubuntu/drbd/drbd.ko
/lib/modules/2.6.31-15-server/updates/dkms/drbd.ko
/lib/modules/2.6.31-16-generic/updates/dkms/drbd.ko
/lib/modules/2.6.31-16-server/kernel/ubuntu/drbd/drbd.ko
/lib/modules/2.6.31-16-server/updates/dkms/drbd.ko
/var/lib/dkms/drbd8/8.3.3/2.6.31-14-server/x86_64/module/drbd.ko
/var/lib/dkms/drbd8/8.3.3/2.6.31-15-generic/x86_64/module/drbd.ko
/var/lib/dkms/drbd8/8.3.3/2.6.31-15-server/x86_64/module/drbd.ko
/var/lib/dkms/drbd8/8.3.3/2.6.31-16-generic/x86_64/module/drbd.ko
/var/lib/dkms/drbd8/8.3.3/2.6.31-16-server/x86_64/module/drbd.ko

> I believe there might be two versions of the module: the in-kernel and a
> DKMS external one. When booted into the -16 kernel, can you do a
> "modinfo drbd" and attach the output. While being on it, also do a "dkms
> status" and also attach that output to the bug report.
>

[Stefan Bader (smb)]

My educated guess in this case would be that DKMS for whatever
reasons (yet to find out), compiled the -16 module with the -15
kernel headers.

You might try a

sudo dkms remove -m drbd8 -v 8.3.3 -k 2.6.31-16-server
sudo dkms build -m drbd8 -v 8.3.3 -k 2.6.31-16-server

to rebuild the module (probably check also whether the -16
version of linux-headers is installed). If that is successful,
then we have to see whether there are still upgrade logs around
for that failed update and whether this gives a clue what went
wrong.

[Mario Kothe (mario-kothe)]

Well it worked. The system boots up and module was loaded. Problem solved.

One thing that is not clear to me yet.

Why is it that the new drbd.ko is only put in "/lib/modules/2.6.31-16-server/updates/dkms/drbd.ko" and "/var/lib/dkms/drbd8/8.3.3/2.6.31-16-server/x86_64/module/drbd.ko" after I executed the above commands.

The module in the directory /lib/modules/2.6.31-16-server/kernel/ubuntu/drbd/drbd.ko is not updated / upgraded with the process descripted. Is there a reason (explanation) for it or is it the problem which lead to the error ?

[Stefan Bader (smb)]

> Why is it that the new drbd.ko is only put in
> "/lib/modules/2.6.31-16-server/updates/dkms/drbd.ko" and
> "/var/lib/dkms/drbd8/8.3.3/2.6.31-16-server/x86_64/module/drbd.ko" after
> I executed the above commands.
>
> The module in the directory
> /lib/modules/2.6.31-16-server/kernel/ubuntu/drbd/drbd.ko is not updated
> / upgraded with the process descripted. Is there a reason (explanation)
> for it or is it the problem which lead to the error ?

No, actually that is the part that is clear (err, yeah, only to me)
You actually have two modules for drbd. One is supplied by the kernel
package (thats the one under .../kernel/ubuntu/...) and there is another
provided by the drbd8-source package (that is using dkms). The version
of the drbd8-source package is the newer one. And while the one from the
kernel is automatically the right one, the dkms module will need to get
rebuild when the kernel(-headers) package is installed.

I think there was something going wrong in the order there while updating.
And if you find any logs that are related to that update (maybe in
/var/log/apt) you should add those to the bug. Maybe there is a way to
understand what happened there.

[Mario Kothe (mario-kothe)]

So I guess it is also normal that modinfo drbd still reports that its version number is 2.6.31-15-server SMP mod_unload modversions.

I attached the apt logfile. Maybe it helps.

[Brad Figg (brad-figg)]

Resolved as a DKMS package issue.