libvirtd looks for ebtables in the wrong location

[Paride Legovini]

Using Train packages on a Bionic system makes libvirt unable to setup a network, as it looks for ebtables at /usr/sbin/ebtables, while the binary is installed as /sbin/ebtables. This is the logged error:

libvirtd[4230]: libvirt version: 5.4.0, package: 0ubuntu5~cloud0 (Openstack Ubuntu Testing Bot <email address hidden> Thu, 05 Sep 2019 11:37:48 +0000)
libvirtd[4230]: hostname: swirlix42
libvirtd[4230]: direct firewall backend requested, but /usr/sbin/ebtables is not available: No such file or directory
libvirtd[4230]: internal error: Failed to initialize a valid firewall backend
libvirtd[4230]: internal error: Failed to initialize a valid firewall backend
[...]

$ ls -l /usr/sbin/ebtables
ls: cannot access '/usr/sbin/ebtables': No such file or directory
$ ls -l /sbin/ebtables
-rwxr-xr-x 1 root root 10088 Jun 28 2018 /sbin/ebtables

$ dpkg -l | grep libvirt
ii libvirt-bin 4.0.0-1ubuntu8.13 arm64 programs for the libvirt library
ii libvirt-clients 5.4.0-0ubuntu5~cloud0 arm64 Programs for the libvirt library
ii libvirt-daemon 5.4.0-0ubuntu5~cloud0 arm64 Virtualization daemon
ii libvirt-daemon-driver-storage-rbd 5.4.0-0ubuntu5~cloud0 arm64 Virtualization daemon RBD storage driver
ii libvirt-daemon-system 5.4.0-0ubuntu5~cloud0 arm64 Libvirt daemon configuration files
ii libvirt0:arm64 5.4.0-0ubuntu5~cloud0 arm64 library for interfacing with different virtualization systems
ii python-libvirt 5.0.0-1~cloud0 arm64 libvirt Python bindings
ii uvtool-libvirt 0~git140-0ubuntu1 all Library and tools for using Ubuntu Cloud Images with libvirt

$ dpkg -l | grep ebtables
ii ebtables 2.0.10.4-3.5ubuntu2.18.04.3 arm64 Ethernet bridge frame table administration

[Paride Legovini (paride)]

It seems that I can't "target to series" this bug to Train.

Anyway the bug can be worked around by converting the system to a "usrmerged" setup, that is by installing the 'usrmerge' package.

Paride

[Christian Ehrhardt  (paelzer)]

FYI: this is correct for libvirt in the respective releases.

$ lxc exec f bash
root@f:~# which ebtables
/usr/sbin/ebtables
root@f:~# exit

$ lxc exec b bash
root@b:~# which ebtables
/sbin/ebtables

But since Train is Focals libvirt on Bionic the path needs to be adapted on the backport.

It comes down to this commit [1] which is correct for Focal and Debian, but not Bionic.
Fortunately it also provides you the ENV-Vars to override the paths as needed.

TBH I have done so on some other Bionic backport that I maintain and I'm pretty sure I IRC pinged someone of the openstack Team on this. This is true since:
$ git tag --contains 76e2cb71d4dd77d66424ad03795c1ade23b747ee
debian/5.0.0-2
debian/5.0.0-3
debian/5.0.0-4
debian/5.1.0-1
debian/5.2.0-1
debian/5.2.0-2
debian/5.6.0-1
debian/5.6.0-2

Therefore it applies to two UCA Trains and Stein as Rocky still was based on 5.0.0-1.

[1]: https://salsa.debian.org/libvirt-team/libvirt/commit/76e2cb71d4dd77d66424ad03795c1ade23b747ee

[Corey Bryant (corey.bryant)]

Thanks Paride and Christian. I've triaged as high as I'm not sure that work-around is ideal (package from universe and could have side-effects?).

[Paride Legovini (paride)]

Thanks Corey. AFAIK installing usrmerge is generally safe, but I would only suggest it in a "be sure to know what you're doing scenario", especially given that after doing the conversion there is no way back. It shouldn't be considered a real fix.

[Corey Bryant (corey.bryant)]

Just a couple of observations. I'm not seeing change [1] in libvirt for disco (stein) so I've dropped that release from affected releases. Also the change in [1] is different from the contents that are in the eoan/focal packages:

        IPTABLES_PATH=/sbin/iptables \
        IP6TABLES_PATH=/sbin/ip6tables \
        EBTABLES_PATH=/usr/sbin/ebtables \

This will be fixed with a patch to the backported package. The update will just focus on changing EBTABLES_PATH to /sbin/ebtables.

[Corey Bryant (corey.bryant)]

Hello Paride, or anyone else affected,

Accepted libvirt 5.4.0-0ubuntu5~cloud1 into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

This will be fixed in Ussuri in libvirt 5.4.0-0ubuntu5~cloud2. The ussuri cloud archive is still pending publication.

[Christian Ehrhardt  (paelzer)]

FYI - this is a dup of my former report in bug 1832297
Not marking as a dup since the processing of this one is rather far already.

[Corey Bryant (corey.bryant)]

The train-proposed fix is included with libvirt 5.4.0-0ubuntu5~cloud1.

[Corey Bryant (corey.bryant)]

@paride, have you been able to verify this fix on train-proposed by any chance?

[Corey Bryant (corey.bryant)]

Note that bionic-train-proposed regression testing was successful this past Friday:

======
Totals
======
Ran: 92 tests in 832.2654 sec.
 - Passed: 84
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 377.7409 sec.

[Guilherme G. Piccoli (gpiccoli)]

Hi @corey.bryant, I've just verified this in -proposed: with the regular Train version (5.4.0-0ubuntu5~cloud0) the problem reproduces. When upgraded to Train-proposed version (5.4.0-0ubuntu5~cloud0, problem is fixed. Hence, marking as verified.

Cheers,

Guilherme

[Corey Bryant (corey.bryant)]

Thanks Guilherme!