DKMS brcompat module circular dependency causes broken module

Bug #1125611 reported by Sebastian Marsching on 2013-02-14
44
This bug affects 7 people
Affects Status Importance Assigned to Milestone
openvswitch (Ubuntu)
Critical
Unassigned
Precise
Critical
James Page
Quantal
Critical
James Page

Bug Description

[Impact]
brcompat module is broken

[Test Case]
sudo apt-get install openvswitch-datapath-dkms
sudo apt-get install openvswitch-brcompat
edit /etc/default/openvswitch-switch and enable BRCOMPAT=yes
sudo service openvswitch-switch restart

brcompat daemon will fail to start:
ovs-brcompatd: could not open brcompat socket. Check "brcompat" kernel module.

/var/log/syslog:
brcompat: module license 'unspecified' taints kernel.

Note that the 'force-reload-kmod' may be required after upgrading to force the DKMS modules to be reloaded:

sudo service openvswitch-switch force-reload-kmod

[Regression Potential]
Minimal; the additional cherry picked patch renames the brcompat.c file brcompat_main.c to avoid a circular dependency during DKMS build.

[Original bug report]
On Ubuntu 12.04.2 LTS, after an automatic upgrade from openvswitch 1.4.0-1ubuntu1.3 to openvswith 1.4.0-1ubuntu1.4 the brcompat support does not work any longer.

The brcompat kernel module can be build, installed, and loaded, but openvswitch-brcompatd will not start with the error message: "ovs-brcompatd: could not open brcompat socket. Check "brcompat" kernel module."

After a downgrade back to version 1.4.0-1ubuntu1.3 brcompat works again. I think this is essentially the same problem as described here: https://answers.launchpad.net/ubuntu/+source/openvswitch/+question/221334

It might be unrelated, but after the upgrade "lsmod" does not list the "openvswitch" module as a dependency of the "brcompat" module. After downgrading, the dependency is shown again.

Thiago Martins (martinx) wrote :

Who is responsible for this? Some compiler bot isn't doing its job? lol

The broken Open vSwitch (1.4.0-1ubuntu1.4) `brcompat' package is now available at proposed-updates repository!!

How can a broken package like this pass through SRU (https://wiki.ubuntu.com/StableReleaseUpdates)?

This was supposed to be a LTS version... Right? =P

The module brcompat doesn't work after the latest upgrade.

Versions:

Linux 3.2.0-38-virtual
Open vSwitch 1.4.0-1ubuntu1.4

Note:

brcompat of Open vSwitch 1.4.0-1ubuntu1.3 is fine.

Linux Bridge module working as a fallback. Open vSwitch 1.4.0-1ubuntu1.4 entirely disabled...

You'll need to disable the `-proposed' and `-updates' APT repositories, purge and reinstall openvswitch-* 1.4.0-1ubuntu1.3 packages...

Best,
Thiago

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in openvswitch (Ubuntu):
status: New → Confirmed
Peng Yong (ppyy) wrote :

confirmed! my openstack zone are broken.

how can a so serious bug got into LTS version?

Thiago Martins (martinx) wrote :

Same here... My Openstack with nova-network and Open vSwitch got down until I downgrade the package back to `1.4.0-1ubuntu1.3'.

LTS crash... Something is wrong not only with this openvswitch Ubuntu package but, all future LTS versions must be *-RELOADED... =P

Cheers!
Thiago

Peng Yong (ppyy) wrote :

the bug was in 2 month ago, and backport to LTS now:

https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1088160

how is QA team works in Ubuntu LTS?

Thiago Martins (martinx) wrote :

Apparently, there is no QA Team.

Both -proposed and -updates repository are disabled for now.

100 * (facepalm)

I do not think that it is the same bug as originally described in bug #1088160. This bug was an issue with the build system. However, it might be the same problem that that one comment (https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1088160/comments/2) to that bug described.

James Page (james-page) on 2013-02-16
Changed in openvswitch (Ubuntu):
importance: Undecided → Critical
James Page (james-page) wrote :

Problem identified; circular dependency due to module renaming was causing the brcompat object not to be included in the module (hence the message about tainting the kernel).

Changed in openvswitch (Ubuntu Precise):
status: New → Confirmed
Changed in openvswitch (Ubuntu Quantal):
status: New → Confirmed
Changed in openvswitch (Ubuntu):
status: Confirmed → Fix Released
Changed in openvswitch (Ubuntu Precise):
importance: Undecided → Critical
Changed in openvswitch (Ubuntu Quantal):
importance: Undecided → Critical
Changed in openvswitch (Ubuntu Precise):
assignee: nobody → James Page (james-page)
Changed in openvswitch (Ubuntu Quantal):
assignee: nobody → James Page (james-page)
James Page (james-page) wrote :

This commit was missing from the backport of fixes to support 3.5 kernel:

http://openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=commit;h=bf8ba92e4875d0ca4c26ddd2a4c46fd26f8e401b

summary: - Upgrade to 1.4.0-1ubuntu1.4 breaks brcompat support
+ DKMS brcompat module circular dependency causes broken module
Changed in openvswitch (Ubuntu Precise):
status: Confirmed → In Progress
Changed in openvswitch (Ubuntu Quantal):
status: Confirmed → In Progress
James Page (james-page) wrote :

Fix uploaded to precise-proposed for SRU team review.

I've also uploaded the same fix to https://launchpad.net/~james-page/+archive/openvswitch

description: updated
James Page (james-page) on 2013-02-16
description: updated
description: updated
James Page (james-page) wrote :

Fix also uploaded to quantal-proposed for SRU team review.

I can confirm the the new version (from the PPA) fixes the problem for me.

Thank you very much for the quick solution!

Peng Yong (ppyy) wrote :
Download full text (4.1 KiB)

error in openstack:

# lsmod|grep brcomp
brcompat 13512 0
openvswitch 84038 8 brcompat

# service openvswitch-switch status
ovsdb-server is running with pid 2186
ovs-vswitchd is running with pid 2228
ovs-brcompatd is running with pid 3007

2013-02-17 10:26:14 DEBUG nova.utils [req-1591a4d4-9779-452f-bd87-2801f25ae592 None None] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf brctl addif qbreb2557ea-3b qvbeb2557ea-3b execute /usr/lib/python2.7/dist-packages/nova/utils.py:183
2013-02-17 10:26:14 DEBUG nova.utils [req-1591a4d4-9779-452f-bd87-2801f25ae592 None None] Result was 1 execute /usr/lib/python2.7/dist-packages/nova/utils.py:199
2013-02-17 10:26:14 2046 CRITICAL nova [-] Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf brctl addif qbreb2557ea-3b qvbeb2557ea-3b
Exit code: 1
Stdout: ''
Stderr: "can't add qvbeb2557ea-3b to bridge qbreb2557ea-3b: Invalid argument\n"
2013-02-17 10:26:14 2046 TRACE nova Traceback (most recent call last):
2013-02-17 10:26:14 2046 TRACE nova File "/usr/bin/nova-compute", line 48, in <module>
2013-02-17 10:26:14 2046 TRACE nova service.wait()
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 659, in wait
2013-02-17 10:26:14 2046 TRACE nova _launcher.wait()
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 192, in wait
2013-02-17 10:26:14 2046 TRACE nova super(ServiceLauncher, self).wait()
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 162, in wait
2013-02-17 10:26:14 2046 TRACE nova service.wait()
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
2013-02-17 10:26:14 2046 TRACE nova return self._exit_event.wait()
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2013-02-17 10:26:14 2046 TRACE nova return hubs.get_hub().switch()
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
2013-02-17 10:26:14 2046 TRACE nova return self.greenlet.switch()
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
2013-02-17 10:26:14 2046 TRACE nova result = function(*args, **kwargs)
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 132, in run_server
2013-02-17 10:26:14 2046 TRACE nova server.start()
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 398, in start
2013-02-17 10:26:14 2046 TRACE nova self.manager.init_host()
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 297, in init_host
2013-02-17 10:26:14 2046 TRACE nova self.driver.plug_vifs(instance, legacy_net_info)
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 439, in plug_vifs
2013-02-17 10:26:14 2046...

Read more...

Thiago Martins (martinx) wrote :

The package (1.4.0-1ubuntu1.5~ubuntu12.04.1~ppa1) related on comment #10 (from James PPA) is working Okay.

Problem fixed with James PPA.

Thanks!
Thiago

Thiago Martins (martinx) wrote :

BTW, my Openstack environment, powered by Nova Network with Open vSwitch via brcompat module is working smoothly with openvswitch-1.4.0-1ubuntu1.5~ubuntu12.04.1~ppa1.

Cheers!
Thiago

Peng Yong (ppyy) wrote :

sorry, remove and install again, it works now

Hello Sebastian, or anyone else affected,

Accepted openvswitch into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/openvswitch/1.4.0-1ubuntu1.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in openvswitch (Ubuntu Precise):
status: In Progress → Fix Committed
tags: added: verification-needed
Changed in openvswitch (Ubuntu Quantal):
status: In Progress → Fix Committed
Colin Watson (cjwatson) wrote :

Hello Sebastian, or anyone else affected,

Accepted openvswitch into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/openvswitch/1.4.3-0ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Peng Yong (ppyy) on 2013-02-18
tags: added: verification-done
removed: verification-needed

I tested the following packages from precise-proposed on Ubuntu 12.04 LTS (amd64) and they work fine for me:
openvswitch-brcompat_1.4.0-1ubuntu1.5
openvswitch-common_1.4.0-1ubuntu1.5
openvswitch-controller_1.4.0-1ubuntu1.5
openvswitch-datapath-dkms_1.4.0-1ubuntu1.5
openvswitch-pki_1.4.0-1ubuntu1.5
openvswitch-switch_1.4.0-1ubuntu1.5

Version 1.4.0-1ubuntu1.5 working fine.
Openstack Cloud Computing (Nova Network + brcompat) environment okay.

Thanks!
Thiago

On 18 February 2013 16:55, Sebastian Marsching <email address hidden>wrote:

> I tested the following packages from precise-proposed on Ubuntu 12.04 LTS
> (amd64) and they work fine for me:
> openvswitch-brcompat_1.4.0-1ubuntu1.5
> openvswitch-common_1.4.0-1ubuntu1.5
> openvswitch-controller_1.4.0-1ubuntu1.5
> openvswitch-datapath-dkms_1.4.0-1ubuntu1.5
> openvswitch-pki_1.4.0-1ubuntu1.5
> openvswitch-switch_1.4.0-1ubuntu1.5
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1125611
>
> Title:
> DKMS brcompat module circular dependency causes broken module
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1125611/+subscriptions
>

Colin Watson (cjwatson) wrote :

Waiting the usual seven-day waiting period since this is a regression.

tags: added: regression-update
Colin Watson (cjwatson) wrote :

s/Waiting/Waiving/

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openvswitch - 1.4.0-1ubuntu1.5

---------------
openvswitch (1.4.0-1ubuntu1.5) precise-proposed; urgency=high

  * Fix broken brcompat module due to circular DKMS build dependency
    causing brcompat.o to be excluded from module (LP: #1125611):
    - d/p/0009-datapath-rename-brcompat.c-to-brcompat_main.c.patch:
      Cherry picked fix from upstream VCS which renames brcompat.c ->
      brcompat_main.c to support drop of _mod suffix.
 -- James Page <email address hidden> Sat, 16 Feb 2013 13:12:54 +0000

Changed in openvswitch (Ubuntu Precise):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openvswitch - 1.4.3-0ubuntu2.1

---------------
openvswitch (1.4.3-0ubuntu2.1) quantal-proposed; urgency=high

  * Fix broken brcompat module due to circular DKMS build dependency
    causing brcompat.o to be excluded from module (LP: #1125611):
    - d/p/0009-datapath-rename-brcompat.c-to-brcompat_main.c.patch:
      Cherry picked fix from upstream VCS which renames brcompat.c ->
      brcompat_main.c to support drop of _mod postfix.
  * Fix module-assistant installs of openvswitch-datapath (LP: #1088160):
    - d/rules.modules: Drop _mod postfix when installing modules.
 -- James Page <email address hidden> Sat, 16 Feb 2013 14:02:05 +0000

Changed in openvswitch (Ubuntu Quantal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers