openvswitch-datapath-dkms kernel module is not getting loaded

[Ridsai]

Distributor ID: Ubuntu
Description: Ubuntu 16.04.4 LTS
Release: 16.04
Codename: xenial

Kernel version: 4.4.0-127-generic
OVS version: 2.7.0

# Issue description
openvswitch-datapath-dkms kernelmodule is not getting loaded

# Error from dmesg
[10863.380564] openvswitch: module verification failed: signature and/or required key missing - tainting kernel
[10863.381804] openvswitch: exports duplicate symbol dst_cache_destroy (owned by kernel)
[10863.517173] openvswitch: exports duplicate symbol dst_cache_destroy (owned by kernel)
[10878.094250] openvswitch: exports duplicate symbol dst_cache_destroy (owned by kernel)

#Error from syslog
May 24 08:37:06 dev65 kernel: [76508.361496] openvswitch: exports duplicate symbol dst_cache_destroy (owned by kernel)
May 24 08:37:06 dev65 openvswitch-switch[18336]: modprobe: ERROR: could not insert 'openvswitch': Exec format error
May 24 08:37:06 dev65 openvswitch-switch[18336]: * Inserting openvswitch module

#installation logs:
Selecting previously unselected package openvswitch-datapath-dkms.
(Reading database ... 84707 files and directories currently installed.)
Preparing to unpack openvswitch-datapath-dkms_2.7.0-1_all.deb ...
Unpacking openvswitch-datapath-dkms (2.7.0-1) ...
Setting up openvswitch-datapath-dkms (2.7.0-1) ...

Creating symlink /var/lib/dkms/openvswitch/2.7.0/source ->
                 /usr/src/openvswitch-2.7.0

DKMS: add completed.

Kernel preparation unnecessary for this kernel. Skipping...

Building module:
cleaning build area....(bad exit status: 2)
./configure --with-linux='/lib/modules/4.4.0-127-generic/build' && make -C datapath/linux......................
cleaning build area....(bad exit status: 2)

DKMS: build completed.

openvswitch:
Running module version sanity check.
 - Original module
 - Installation
   - Installing to /lib/modules/4.4.0-127-generic/updates/dkms/

vport-geneve.ko:
Running module version sanity check.
 - Original module
 - Installation
   - Installing to /lib/modules/4.4.0-127-generic/updates/dkms/

vport-gre.ko:
Running module version sanity check.
 - Original module
 - Installation
   - Installing to /lib/modules/4.4.0-127-generic/updates/dkms/

vport-lisp.ko:
Running module version sanity check.
 - Original module
 - Installation
   - Installing to /lib/modules/4.4.0-127-generic/updates/dkms/

vport-stt.ko:
Running module version sanity check.
 - Original module
 - Installation
   - Installing to /lib/modules/4.4.0-127-generic/updates/dkms/

vport-vxlan.ko:
Running module version sanity check.
 - Original module
 - Installation
   - Installing to /lib/modules/4.4.0-127-generic/updates/dkms/

depmod....

DKMS: install completed.

#Dpkg status
dpkg -l | grep openvswitch
ii openvswitch-common 2.7.0-1 amd64 Open vSwitch common components
ii openvswitch-datapath-dkms 2.7.0-1 all Open vSwitch datapath module source - DKMS version
ii openvswitch-switch 2.7.0-1 amd64 Open vSwitch switch implementations
ii python-openvswitch 2.6.1-0ubuntu5.2~cloud0 all Python bindings for Open vSwitch

#lsmod status
No modules are listing for openvswitch-datapath-dkms

#modprobe error
modprobe openvswitch
modprobe: ERROR: could not insert 'openvswitch': Exec format error
---
AlsaVersion: Advanced Linux Sound Architecture Driver Version k4.4.0-127-generic.
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.20.1-0ubuntu2.17
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/hwC0D0', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/controlC0', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Card0.Amixer.info: Error: [Errno 2] No such file or directory
Card0.Amixer.values: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 16.04
IwConfig: Error: [Errno 2] No such file or directory
Lsusb:
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
 Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: Red Hat KVM
Package: linux (not installed)
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 cirrusdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-127-generic root=UUID=2bd36290-e77b-49ad-9199-c485da6f4f39 ro
ProcVersionSignature: Ubuntu 4.4.0-127.153-generic 4.4.128
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-127-generic N/A
 linux-backports-modules-4.4.0-127-generic N/A
 linux-firmware 1.157.19
RfKill: Error: [Errno 2] No such file or directory
Tags: xenial uec-images
Uname: Linux 4.4.0-127-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 01/01/2007
dmi.bios.vendor: Seabios
dmi.bios.version: 0.5.1
dmi.chassis.type: 1
dmi.chassis.vendor: Red Hat
dmi.modalias: dmi:bvnSeabios:bvr0.5.1:bd01/01/2007:svnRedHat:pnKVM:pvrRHEL6.6.0PC:cvnRedHat:ct1:cvr:
dmi.product.name: KVM
dmi.product.version: RHEL 6.6.0 PC
dmi.sys.vendor: Red Hat

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a kernel version where you were not having this particular problem? This will help determine if the problem you are seeing is the result of a regression, and when this regression was introduced. If this is a regression, we can perform a kernel bisect to identify the commit that introduced the problem.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1773165

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Ridsai (ridsaim)]

This issue is seen on a fresh installation when we tried to install the Openvswitch and its kernel module on a node Also the issue was not seen on kernel version 4.4.0-124-generic. But for kernel version 4.4.0-125 and onwards, we face this issue.

[Ridsai (ridsaim)]

apport information

[Ridsai (ridsaim)]

apport information

[Ridsai (ridsaim)]

apport information

[Ridsai (ridsaim)]

apport information

[Ridsai (ridsaim)]

apport information

[Ridsai (ridsaim)]

apport information

[Ridsai (ridsaim)]

apport information

[Ridsai (ridsaim)]

apport information

[Ridsai (ridsaim)]

apport information

[Ridsai (ridsaim)]

apport information

[Ridsai (ridsaim)]

apport information

[Ridsai (ridsaim)]

apport information

[Joseph Salisbury (jsalisbury)]

@Ridsai,

Are you still see this bug with the latest updates applied?

[Chris Gregan (cgregan)]

@jsalsbury We are still seeing the issue, given ridsaim has not replied lets move forward. I'd like to get some movement on this issue without marking as a Field Critical

[Ridsai (ridsaim)]

Apologies for the late response. I am still seeing the issue with the latest kernel that is (4.4.0-134)

[James Page (james-page)]

Where is the openvswitch-switch-dkms package coming from? I don't recognise the package version as anything that's being shipped in Ubuntu?

[James Page (james-page)]

fwiw the 4.4 intree openvswitch module is recent and up-to-date so use of a ovs supplied dkms module should not be required.

[Joseph Salisbury (jsalisbury)]

I built a Xenial test kernel with a revert of the following commit:
cc62065b735f openvswitch: Delete conntrack entry clashing with an expectation.

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1773165

Can you test this kernel and see if it resolves this bug?

[Joseph Salisbury (jsalisbury)]

Also, do you recall where you got that openvswitch-datapath-dkms from. Rmadison only reports trusty and precise versions for openvswitch-datapath-dkms.

[Ridsai (ridsaim)]

The DKMS package is the one from ovs 2.7.0 with some of our changes to some Nicira extended actions

[Ridsai (ridsaim)]

@Joseph Salisbury

The test kernel is also having the same issue.

Insmode Error:

root@compute2:~/ovs# insmod /lib/modules/4.4.0-138-generic/updates/dkms/openvswitch.ko
insmod: ERROR: could not insert module /lib/modules/4.4.0-138-generic/updates/dkms/openvswitch.ko: Invalid module format

Syslog error:

Oct 24 11:20:24 compute2 systemd[1]: Starting LSB: Open vSwitch switch...
Oct 24 11:20:24 compute2 kernel: [ 165.603193] openvswitch: exports duplicate symbol dst_cache_destroy (owned by kernel)
Oct 24 11:20:24 compute2 openvswitch-switch[3594]: modprobe: ERROR: could not insert 'openvswitch': Exec format error
Oct 24 11:20:24 compute2 openvswitch-switch[3594]: * Inserting openvswitch module
Oct 24 11:20:24 compute2 openvswitch-switch[3594]: * Starting ovsdb-server
Oct 24 11:20:24 compute2 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait -- init -- set Open_vSwitch . db-version=7.14.0
Oct 24 11:20:24 compute2 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . ovs-version=2.7.0 "external-ids:system-id=\"2d3f6b90-9cac-4346-85a8-445448983806\"" "external-ids:hostname=\"compute2.maas\"" "system-type=\"ubuntu\"" "system-version=\"16.04\""
Oct 24 11:20:24 compute2 openvswitch-switch[3594]: * Configuring Open vSwitch system IDs
Oct 24 11:20:24 compute2 kernel: [ 165.700381] openvswitch: exports duplicate symbol dst_cache_destroy (owned by kernel)
Oct 24 11:20:24 compute2 openvswitch-switch[3594]: modprobe: ERROR: could not insert 'openvswitch': Exec format error

[Ridsai (ridsaim)]

After testing it with the test kernel we found that the Linux header contains the Openvswitch.h header files with out any OVS module installed, and these header files are not compatible with OVS2.7, when we tried to build the OVS kernel module"

Files contains in debian package:

dpkg -L linux-headers-4.4.0-138 | grep openvswitch
/usr/src/linux-headers-4.4.0-138/net/openvswitch
/usr/src/linux-headers-4.4.0-138/net/openvswitch/Makefile
/usr/src/linux-headers-4.4.0-138/net/openvswitch/Kconfig
/usr/src/linux-headers-4.4.0-138/include/linux/openvswitch.h
/usr/src/linux-headers-4.4.0-138/include/uapi/linux/openvswitch.h

root@compute2:~/kernel# dpkg -L linux-headers-4.4.0-138-generic | grep openvswitch
/usr/src/linux-headers-4.4.0-138-generic/include/config/openvswitch
/usr/src/linux-headers-4.4.0-138-generic/include/config/openvswitch/gre.h
/usr/src/linux-headers-4.4.0-138-generic/include/config/openvswitch/vxlan.h
/usr/src/linux-headers-4.4.0-138-generic/include/config/openvswitch/geneve.h
/usr/src/linux-headers-4.4.0-138-generic/include/config/openvswitch.h

What is the safe way to remove these header files before Installing the our OVS and compiling the OVS kernel module.

[Ashley Lai (alai)]

According to the Ubuntu package link, the support for openvswitch-datapath-dkms ended in Trusty. It is not available as a DKMS package in Xenial rather it is available as a kernel module.

https://packages.ubuntu.com/search?keywords=openvswitch-datapath-dkms&searchon=names&suite=all&section=all

[Andrei Andone (andrei.andone)]

Hello everybody,

The problem is that we have added functionality to the kernel module.
Since dkms now comes with the kernel itself, how can we add our functionality to it?

Is there a way to gracefully replace the openvswitch kmod with our own, also, in order to build the kernel module against the kernel we would need to at least patch the existing openvswitch.h file that comes with the kernel-headers. Is that an option?

Regards,
Andrei

[Chris Gregan (cgregan)]

@Andrei
Since the driver is now part of the kernel, any changes to the driver, and as a result, the kernel will need to follow the standard upstream kernel patch process found here:

https://www.kernel.org/doc/html/v4.17/process/submitting-patches.html