firewalld fails to start: FATAL ERROR: No IPv4 and IPv6 firewall: looks for binaries in wrong paths

[Martin Pitt]

In a clean Ubuntu 19.04 (disco) VM installation, firewalld fails to start:

* firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Wed 2019-04-24 06:35:00 EDT; 3min 58s ago
     Docs: man:firewalld(1)
  Process: 516 ExecStart=/usr/sbin/firewalld --nofork --nopid (code=exited, status=0/SUCCESS)
 Main PID: 516 (code=exited, status=0/SUCCESS)

Apr 24 06:34:58 ibm-p8-kvm-03-guest-02 systemd[1]: Starting firewalld - dynamic firewall daemon...
Apr 24 06:34:59 ibm-p8-kvm-03-guest-02 systemd[1]: Started firewalld - dynamic firewall daemon.
Apr 24 06:34:59 ibm-p8-kvm-03-guest-02 firewalld[516]: WARNING: iptables-restore and iptables are missing, disabling IPv4 firewall.
Apr 24 06:34:59 ibm-p8-kvm-03-guest-02 firewalld[516]: WARNING: ip6tables-restore and ip6tables are missing, disabling IPv6 firewall.
Apr 24 06:35:00 ibm-p8-kvm-03-guest-02 firewalld[516]: FATAL ERROR: No IPv4 and IPv6 firewall.
Apr 24 06:35:00 ibm-p8-kvm-03-guest-02 firewalld[516]: ERROR: Raising SystemExit in run_server
Apr 24 06:35:00 ibm-p8-kvm-03-guest-02 systemd[1]: firewalld.service: Succeeded.

This is with the default iptables backend. When switching to FirewallBackend=nftables, firewalld at least starts up, even though it shows warnings:

* firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-04-24 06:40:14 EDT; 2s ago
     Docs: man:firewalld(1)
 Main PID: 501 (firewalld)
    Tasks: 2 (limit: 2306)
   Memory: 33.0M
   CGroup: /system.slice/firewalld.service
           `-501 /usr/bin/python3 /usr/sbin/firewalld --nofork --nopid

Apr 24 06:40:15 ibm-p8-kvm-03-guest-02 firewalld[501]: WARNING: COMMAND_FAILED: UNKNOWN_ERROR: 'ip4tables' backend does not exist
Apr 24 06:40:15 ibm-p8-kvm-03-guest-02 firewalld[501]: WARNING: COMMAND_FAILED: UNKNOWN_ERROR: 'ip4tables' backend does not exist

and doesn't actually work:

# firewall-cmd --state
failed

ProblemType: Bug
DistroRelease: Ubuntu 19.04
Package: firewalld 0.6.3-5ubuntu4
Architecture: amd64

[Martin Pitt (pitti)]

The nftables backend actually does work reasonably well, only firewall-cmd --state seems to be broken.

[Martin Pitt (pitti)]

With the nftables backend, it's not just --state that's broken, it also at least affects --reload:

# firewall-cmd --reload
Error: 'inet firewalld filter_IN_public jump filter_IN_public_allow'
# echo $?
254

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[diehard67 (diehard67)]

same trouble here

iptables backend, doesn't even start
nftables backend, starts but doesn't do anything, though installing nftables package gets firewall-cms --state to work, it says running

some bits of debug console output

diehard@silverbox:~$ sudo firewalld --nofork --debug 10
2019-04-24 03:33:45 DEBUG2: <class 'firewall.core.ipXtables.ip4tables'>: /usr/sbin/iptables-restore will be using option.
2019-04-24 03:33:45 DEBUG2: <class 'firewall.core.ipXtables.ip6tables'>: /usr/sbin/ip6tables-restore will be using option.
2019-04-24 03:33:45 DEBUG2: <class 'firewall.core.ebtables.ebtables'>: /usr/sbin/ebtables-restore /run/firewalld/temp.n5prl97e: 0
2019-04-24 03:33:45 DEBUG1: start()
2019-04-24 03:33:45 DEBUG1: Loading firewalld config file '/etc/firewalld/firewalld.conf'
2019-04-24 03:33:45 DEBUG1: CleanupOnExit is set to 'True'
2019-04-24 03:33:45 DEBUG1: IPv6 rpfilter is enabled
2019-04-24 03:33:45 DEBUG1: LogDenied is set to 'off'
2019-04-24 03:33:45 DEBUG1: AutomaticHelpers is set to 'system'
2019-04-24 03:33:45 DEBUG1: FirewallBackend is set to 'iptables'
2019-04-24 03:33:45 DEBUG2: <class 'firewall.core.ipset.ipset'>: /sbin/ipset list
2019-04-24 03:33:45 DEBUG2: <class 'firewall.core.ipset.ipset'>: /sbin/ipset --help
2019-04-24 03:33:45 WARNING: iptables-restore and iptables are missing, disabling IPv4 firewall.
2019-04-24 03:33:45 WARNING: ip6tables-restore and ip6tables are missing, disabling IPv6 firewall.
2019-04-24 03:33:45 DEBUG1: Conntrack helpers supported by the kernel:
2019-04-24 03:33:45 DEBUG1: nf_conntrack_amanda: amanda
2019-04-24 03:33:45 DEBUG1: nf_conntrack_ftp: ftp
2019-04-24 03:33:45 DEBUG1: nf_conntrack_h323: H.245, Q.931, RAS
2019-04-24 03:33:45 DEBUG1: nf_conntrack_irc: irc
2019-04-24 03:33:45 DEBUG1: nf_conntrack_netbios_ns: netbios-ns
2019-04-24 03:33:45 DEBUG1: nf_conntrack_pptp: pptp
2019-04-24 03:33:45 DEBUG1: nf_conntrack_proto_gre: proto-gre
2019-04-24 03:33:45 DEBUG1: nf_conntrack_sane: sane
2019-04-24 03:33:45 DEBUG1: nf_conntrack_sip: sip
2019-04-24 03:33:45 DEBUG1: nf_conntrack_snmp: snmp
2019-04-24 03:33:45 DEBUG1: nf_conntrack_tftp: tftp
2019-04-24 03:33:45 DEBUG1: NAT helpers supported by the kernel:
2019-04-24 03:33:45 DEBUG1: nf_nat_amanda: amanda
2019-04-24 03:33:45 DEBUG1: nf_nat_ftp: ftp
2019-04-24 03:33:45 DEBUG1: nf_nat_irc: irc
2019-04-24 03:33:45 DEBUG1: nf_nat_sip: sip
2019-04-24 03:33:45 DEBUG1: nf_nat_tftp: tftp
2019-04-24 03:33:45 DEBUG1: nf_nat_h323: h323
2019-04-24 03:33:45 DEBUG1: nf_nat_pptp: pptp

sniped a bunch of loading stuff that doesn't seem relevant

2019-04-24 03:33:45 DEBUG1: Using default zone 'public'
2019-04-24 03:33:45 DEBUG2: <class 'firewall.core.modules.modules'>: /sbin/modprobe nf_conntrack
2019-04-24 03:33:45 DEBUG2: <class 'firewall.core.ebtables.ebtables'>: /usr/sbin/ebtables --concurrent -t nat -L
2019-04-24 03:33:45 DEBUG2: <class 'firewall.core.ebtables.ebtables'>: /usr/sbin/ebtables --concurrent -t filter -L
2019-04-24 03:33:45 FATAL ERROR: No IPv4 and IPv6 firewall.
2019-04-24 03:33:45 ERROR: Raising SystemExit in run_server

I can run iptables command myself so not sure why firewalld can't.

hope this mess helps.

[Martin Pitt (pitti)]

Doing this fixes it:

ln -s /sbin/iptables /usr/sbin/
ln -s /sbin/iptables-restore /usr/sbin/
ln -s /sbin/ip6tables /usr/sbin/
ln -s /sbin/ip6tables-restore /usr/sbin/

Apparently firewalld looks for these binaries in the wrong path, doesn't use $PATH, and assumes a merged /usr system.

[wvengen (wvengen)]

Wow, this is pretty serious! I recently discovered that I'd been running without a firewall for some time (exposing unsecured services, also on public WiFi). Previously it worked, some upgrade must have introduced this issue.

[Martin Pitt (pitti)]

This magically went away on the most recent daily cloud images. @wvengen, does it work for you again as well? If so, we can close this. Thanks!

[Andrej Shadura (andrew.sh)]

This still happens to me.

[Andrej Shadura (andrew.sh)]

Apparently, this happens if iptables <= 1.6.2-1.1, firewalld >= 0.6.3-5. Installing both in their latest versions fixes the issue.