disco: unable to use iptables/enable ufw under -virtual kernel

[James Page]

SRU Justification

Impact: iptables does not work in disco with linux-virtual.

Fix: Add bpfilter to the generic inclusion list.

Regression Potential: There are no code changes and thus minimal potential for regressions.

Test Case: Verify that bpffilter.ko is shipped in linux-modues instead of linux-module-extra and that iptables commands work in disco with the linux-virtual kernel installed.

---

Fresh install of disco:

$ sudo ufw enable
Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
ERROR: problem running ufw-init
iptables-restore v1.6.1: Can't set policy `DROP' on `INPUT' line 3: Bad built-in chain name

iptables-restore: line 22 failed
iptables-restore: line 2 failed
iptables-restore v1.6.1: Couldn't load target `ufw-logging-deny':No such file or directory

Error occurred at line: 30
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
iptables-restore v1.6.1: Couldn't load target `ufw-skip-to-policy-input':No such file or directory

Error occurred at line: 19
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
iptables-restore v1.6.1: Couldn't load target `ufw-user-input':No such file or directory

Error occurred at line: 2
Try `iptables-restore -h' or 'iptables-restore --help' for more information.

Problem running '/etc/ufw/before.rules'
Problem running '/etc/ufw/after.rules'

ProblemType: Bug
DistroRelease: Ubuntu 19.04
Package: ufw 0.36-1ubuntu1
ProcVersionSignature: Ubuntu 5.0.0-8.9-generic 5.0.1
Uname: Linux 5.0.0-8-generic x86_64
ApportVersion: 2.20.10-0ubuntu26
Architecture: amd64
Date: Tue Apr 9 08:49:59 2019
Ec2AMI: ami-000004cf
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.blue
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
PackageArchitecture: all
ProcEnviron:
 TERM=screen-256color
 PATH=(custom, no user)
 LANG=C.UTF-8
 SHELL=/bin/bash
SourcePackage: ufw
UpgradeStatus: No upgrade log present (probably fresh install)
mtime.conffile..etc.default.ufw: 2019-04-09T08:48:31.371301

[Jamie Strandboge (jdstrand)]

I cannot reproduce. I notice that you said 'default install' but at a minimum you have modified /etc/default/ufw to have:

DEFAULT_INPUT_POLICY="ACCEPT"

$ cat /proc/version_signature
Ubuntu 5.0.0-8.9-generic 5.0.1

$ sudo ufw enable
Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
Started bpfilter
Firewall is active and enabled on system startup

Can you give exact steps to verify?

What is the output of 'sudo /usr/share/ufw/check-requirements'? What is the output of that before you run 'ufw enable'?

[James Page (james-page)]

I'm trying to figure out what is modifying /etc/default/ufw - the swift-storage charm does but it just disables IPv6 in some situations.

I reported this from a different cloud instance - so probably worth mentioning that this is a disco image running on openstack from the daily image stream.

[James Page (james-page)]

OK so there is a little modification of the policy prior to enablement:

def initialize_ufw():
    """Initialize the UFW firewall

    Ensure critical ports have explicit allows

    :return: None
    """

    if not config('enable-firewall'):
        log("Firewall has been administratively disabled", "DEBUG")
        return

    # this charm will monitor exclusively the ports used, using 'allow' as
    # default policy enables sharing the machine with other services
    ufw.default_policy('allow', 'incoming')
    ufw.default_policy('allow', 'outgoing')
    ufw.default_policy('allow', 'routed')
    # Rsync manages its own ACLs
    ufw.service('rsync', 'open')
    # Guarantee SSH access
    ufw.service('ssh', 'open')
    # Enable
    ufw.enable(soft_fail=config('allow-ufw-ip6-softfail'))

    # Allow GRE traffic
    add_ufw_gre_rule(os.path.join(UFW_DIR, 'before.rules'))
    ufw.reload()

[James Page (james-page)]

this translates into:

ufw default allow incoming
ufw default allow outgoing
ufw default allow routed

ufw allow rsync
ufw allow ssh

ufw enable

which gets us to the:

ufw.enable(soft_fail=config('allow-ufw-ip6-softfail'))

which is where the failure to enable occurs.

[Jamie Strandboge (jdstrand)]

The usefulness of the policy aside (it is configured wide open), I still can't reproduce with http://cdimage.ubuntu.com/ubuntu-server/daily/current/disco-server-amd64.iso that I just downloaded:

$ sudo ufw default allow incoming
Default incoming policy changed to 'allow'
(be sure to update your rules accordingly)
$ sudo ufw default allow outgoing
Default outgoing policy changed to 'allow'
(be sure to update your rules accordingly)
$ sudo ufw default allow routed
Default routed policy changed to 'allow'
(be sure to update your rules accordingly)
$ sudo ufw allow rsync
Rules updated
Rules updated (v6)
$ sudo ufw allow ssh
Rules updated
Rules updated (v6)
$ sudo ufw enable
Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
Firewall is active and enabled on system startup

$ cat /proc/version_signature
Ubuntu 5.0.0-8.9-generic 5.0.1

[Jamie Strandboge (jdstrand)]

You said that the swift charm is disabling ipv6 in certain situations. Is it updating /etc/default/ufw? Is it disabling it elsewhere? In the failing instance, before trying to setup swift or run ufw or anything, what is the output of 'sudo /usr/share/ufw/check-requirements'?

[Jamie Strandboge (jdstrand)]

FYI, I also tried the above with setting /etc/default/ufw to have IPV6=no and everything works fine (and the v6 rules are not added).

[Jamie Strandboge (jdstrand)]

Another thing that might be useful is attaching a tarball of /lib/ufw, /etc/ufw, and /etc/default/ufw.

[James Page (james-page)]

I can only think this is something cloud-image specific.

FTR the policy is initially configured wide open (as the server may be doing a number of other things the charm deploying does not know about) and then the ports for swift/rsync are restricted to peers.

[Jamie Strandboge (jdstrand)]

"I can only think this is something cloud-image specific."

Can you provide the requested information in the earlier comments?

[James Page (james-page)]

check-requirements from failing swift unit:

$ sudo /usr/share/ufw/check-requirements
Has python: pass (binary: python2.7, version: 2.7.16, py2)
Has iptables: pass
Has ip6tables: pass

Has /proc/net/dev: pass
Has /proc/net/if_inet6: pass

This script will now attempt to create various rules using the iptables
and ip6tables commands. This may result in module autoloading (eg, for
IPv6).
Proceed with checks (Y/n)? y
== IPv4 ==
Creating 'ufw-check-requirements'... done
Inserting RETURN at top of 'ufw-check-requirements'... iptables: No chain/target/match by that name.
ERROR: could insert RETURN rule into 'ufw-check-requirements'. Aborting
FAIL: check your kernel and that you have iptables >= 1.4.0

[James Page (james-page)]

I also tested on a pristine machine:

$ sudo /usr/share/ufw/check-requirements
Has python: pass (binary: python3.7, version: 3.7.3, py3)
Has iptables: pass
Has ip6tables: pass

Has /proc/net/dev: pass
Has /proc/net/if_inet6: pass

This script will now attempt to create various rules using the iptables
and ip6tables commands. This may result in module autoloading (eg, for
IPv6).
Proceed with checks (Y/n)? yes
== IPv4 ==
Creating 'ufw-check-requirements'... done
Inserting RETURN at top of 'ufw-check-requirements'... iptables: No chain/target/match by that name.
ERROR: could insert RETURN rule into 'ufw-check-requirements'. Aborting
FAIL: check your kernel and that you have iptables >= 1.4.0

[James Page (james-page)]

main diff there is py2.7 vs py3 (but that's cause swift still uses python2)

[James Page (james-page)]

OK I think I've figured it out; the cloud images use the -virtual kernel, which does not install linux-modules-extra-*.

If I install the -generic kernel or just the extra modules and reboot, then iptables commands start working again.

[James Page (james-page)]

Dropping ufw from the test case - just running sudo iptables-save against the virtual kernel produces no output.

[James Page (james-page)]

Diff between two identical instances, one with -generic and one with -virtual.

iptables* commands working find with -generic.

[James Page (james-page)]

This might have some implications for fan usage on cloud instances as I see some fan-* rules post switch to generic kernel.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1823862

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Jamie Strandboge (jdstrand)]

Marking the ufw task as Invalid. The kernel doesn't have what is needed to run iptables.

[Jamie Strandboge (jdstrand)]

I can confirm that without linux-modules-extra-*, iptables is broken.

Reduced test case:
$ sudo iptables -L -n
iptables: No chain/target/match by that name.

Full test case:
$ sudo /usr/share/ufw/check-requirements -f
...
ERROR: could not create 'ufw-check-requirements'. Aborting
FAIL: check your kernel and that you have iptables >= 1.4.0

Perhaps the above should be added to image smoke tests?

[Seth Forshee (sforshee)]

All that seems to be missing is bpffilter. I've made a test build with this moved to linux-modules. It appears to be working for check-requirements. Can you test and make sure everything is working as expected? Thanks!

http://people.canonical.com/~sforshee/lp1823862/

[James Page (james-page)]

Tested with the kernel+modules linked in #22

I can confirm I was able to enable ufw and query the installed iptables rules:

$ uname -a
Linux juju-0f0cbd-disco-testing-38 5.0.0-10-generic #11+lp1823862v201904100938 SMP Wed Apr 10 14:40:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ sudo ufw enable
Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
Firewall is active and enabled on system startup

$ sudo iptables-save
# Generated by iptables-save v1.6.1 on Wed Apr 10 16:15:01 2019
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:ufw-after-forward - [0:0]
:ufw-after-input - [0:0]
:ufw-after-logging-forward - [0:0]
:ufw-after-logging-input - [0:0]
:ufw-after-logging-output - [0:0]
:ufw-after-output - [0:0]
:ufw-before-forward - [0:0]
:ufw-before-input - [0:0]
:ufw-before-logging-forward - [0:0]
:ufw-before-logging-input - [0:0]
:ufw-before-logging-output - [0:0]
:ufw-before-output - [0:0]
:ufw-logging-allow - [0:0]
:ufw-logging-deny - [0:0]
:ufw-not-local - [0:0]
:ufw-reject-forward - [0:0]
:ufw-reject-input - [0:0]
:ufw-reject-output - [0:0]
:ufw-skip-to-policy-forward - [0:0]
:ufw-skip-to-policy-input - [0:0]
:ufw-skip-to-policy-output - [0:0]
:ufw-track-forward - [0:0]
:ufw-track-input - [0:0]
:ufw-track-output - [0:0]
:ufw-user-forward - [0:0]
:ufw-user-input - [0:0]
:ufw-user-limit - [0:0]
:ufw-user-limit-accept - [0:0]
:ufw-user-logging-forward - [0:0]
:ufw-user-logging-input - [0:0]
:ufw-user-logging-output - [0:0]
:ufw-user-output - [0:0]
-A INPUT -j ufw-before-logging-input
-A INPUT -j ufw-before-input
-A INPUT -j ufw-after-input
-A INPUT -j ufw-after-logging-input
-A INPUT -j ufw-reject-input
-A INPUT -j ufw-track-input
-A FORWARD -j ufw-before-logging-forward
-A FORWARD -j ufw-before-forward
-A FORWARD -j ufw-after-forward
-A FORWARD -j ufw-after-logging-forward
-A FORWARD -j ufw-reject-forward
-A FORWARD -j ufw-track-forward
-A OUTPUT -j ufw-before-logging-output
-A OUTPUT -j ufw-before-output
-A OUTPUT -j ufw-after-output
-A OUTPUT -j ufw-after-logging-output
-A OUTPUT -j ufw-reject-output
-A OUTPUT -j ufw-track-output
-A ufw-after-input -p udp -m udp --dport 137 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 138 -j ufw-skip-to-policy-input
-A ufw-after-input -p tcp -m tcp --dport 139 -j ufw-skip-to-policy-input
-A ufw-after-input -p tcp -m tcp --dport 445 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 67 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 68 -j ufw-skip-to-policy-input
-A ufw-after-input -m addrtype --dst-type BROADCAST -j ufw-skip-to-policy-input
-A ufw-after-logging-forward -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
-A ufw-after-logging-input -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
-A ufw-before-forward -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 12 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A ufw-before-forward -j ufw-user-forward
...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1823862

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.0.0-11.12

---------------
linux (5.0.0-11.12) disco; urgency=medium

  * linux: 5.0.0-11.12 -proposed tracker (LP: #1824383)

  * hns3: PPU_PF_ABNORMAL_INT_ST over_8bd_no_fe found [error status=0x1]
    (LP: #1824194)
    - net: hns3: fix for not calculating tx bd num correctly

  * disco: unable to use iptables/enable ufw under -virtual kernel
    (LP: #1823862)
    - [Packaging] add bpfilter to linux-modules

  * Make shiftfs a module rather than built-in (LP: #1824354)
    - [Config] CONFIG_SHIFT_FS=m

  * shiftfs: chown sets untranslated ids in lower fs (LP: #1824350)
    - SAUCE: shiftfs: use translated ids when chaning lower fs attrs

  * [Hyper-V] KVP daemon fails to start on first boot of disco VM (LP: #1820063)
    - [Packaging] bind hv_kvp_daemon startup to hv_kvp device

linux (5.0.0-10.11) disco; urgency=medium

  * linux: 5.0.0-10.11 -proposed tracker (LP: #1823936)

  * Apparmor enforcement failure in lxc selftests (LP: #1823379)
    - SAUCE: apparmor: Restore Y/N in /sys for apparmor's "enabled"

  * systemd cause kernel trace "BUG: unable to handle kernel paging request at
    6db23a14" on Cosmic i386 (LP: #1813244)
    - openvswitch: fix flow actions reallocation

linux (5.0.0-9.10) disco; urgency=medium

  * linux: 5.0.0-9.10 -proposed tracker (LP: #1823228)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync git-ubuntu-log
    - [Packaging] update helper scripts
    - [Packaging] resync retpoline extraction

  * Huawei Hi1822 NIC has poor performance (LP: #1820187)
    - net-next/hinic: replace disable_irq_nosync/enable_irq

  * Add uid shifting overlay filesystem (shiftfs) (LP: #1823186)
    - shiftfs: uid/gid shifting bind mount
    - shiftfs: rework and extend
    - shiftfs: support some btrfs ioctls
    - [Config] enable shiftfs

  * Cannot boot or install - have to use nomodeset (LP: #1821820)
    - Revert "drm/i915/fbdev: Actually configure untiled displays"

  * Disco update: v5.0.6 upstream stable release (LP: #1823060)
    - netfilter: nf_tables: fix set double-free in abort path
    - dccp: do not use ipv6 header for ipv4 flow
    - genetlink: Fix a memory leak on error path
    - gtp: change NET_UDP_TUNNEL dependency to select
    - ipv6: make ip6_create_rt_rcu return ip6_null_entry instead of NULL
    - mac8390: Fix mmio access size probe
    - mISDN: hfcpci: Test both vendor & device ID for Digium HFC4S
    - net: aquantia: fix rx checksum offload for UDP/TCP over IPv6
    - net: datagram: fix unbounded loop in __skb_try_recv_datagram()
    - net/packet: Set __GFP_NOWARN upon allocation in alloc_pg_vec
    - net: phy: meson-gxl: fix interrupt support
    - net: rose: fix a possible stack overflow
    - net: stmmac: fix memory corruption with large MTUs
    - net-sysfs: call dev_hold if kobject_init_and_add success
    - net: usb: aqc111: Extend HWID table by QNAP device
    - packets: Always register packet sk in the same order
    - rhashtable: Still do rehash when we get EEXIST
    - sctp: get sctphdr by offset in sctp_compute_cksum
    - sctp: use memdup_user instead of vmemdup_user
    - tcp: do not use ipv6 header for ipv4 flow
    - tipc: allow servic...