Drop rules not being applied correctly until ufw reload

[Bruce Pieterse]

I have a small VPS server running Ubuntu 18.04.2 with ufw 0.35-5 on a Virtuozza platform.

I've noticed that after a restart of the server, some of the rules are not being applied. ufw is configured to DROP incoming by default, but I have to provide drop rules in the user-input chain in order for the DROP rules to work correctly.

After reboot the chain looks like this:

Chain ufw-user-input (1 references)
    pkts bytes target prot opt in out source destination
       0 0 ACCEPT tcp -- * * x.x.x.x 0.0.0.0/0 tcp dpt:443
       0 0 ACCEPT tcp -- * * x.x.x.x 0.0.0.0/0 tcp dpt:80
       0 0 ACCEPT tcp -- * * x.x.x.x 0.0.0.0/0 tcp dpt:25
       0 0 ACCEPT tcp -- * * x.x.x.x 0.0.0.0/0 tcp dpt:22

Running ufw reload changes the chain to:

Chain ufw-user-input (1 references)
    pkts bytes target prot opt in out source destination
       0 0 ACCEPT tcp -- * * x.x.x.x 0.0.0.0/0 tcp dpt:443
       0 0 ACCEPT tcp -- * * x.x.x.x 0.0.0.0/0 tcp dpt:80
       0 0 ACCEPT tcp -- * * x.x.x.x 0.0.0.0/0 tcp dpt:25
       0 0 ACCEPT tcp -- * * x.x.x.x 0.0.0.0/0 tcp dpt:22
       0 0 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
       6 244 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
       0 0 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:25
      87 5220 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22

Firstly, the DROP rules shouldn't be necessary to drop traffic to port 22, since the default is DROP in /etc/default/ufw:

# /etc/default/ufw
#

# Set to yes to apply rules to support IPv6 (no means only IPv6 on loopback
# accepted). You will need to 'disable' and then 'enable' the firewall for
# the changes to take affect.
IPV6=yes

# Set the default input policy to ACCEPT, DROP, or REJECT. Please note that if
# you change this you will most likely want to adjust your rules.
DEFAULT_INPUT_POLICY="DROP"

# Set the default output policy to ACCEPT, DROP, or REJECT. Please note that if
# you change this you will most likely want to adjust your rules.
DEFAULT_OUTPUT_POLICY="ACCEPT"

# Set the default forward policy to ACCEPT, DROP or REJECT. Please note that
# if you change this you will most likely want to adjust your rules
DEFAULT_FORWARD_POLICY="DROP"

# Set the default application policy to ACCEPT, DROP, REJECT or SKIP. Please
# note that setting this to ACCEPT may be a security risk. See 'man ufw' for
# details
DEFAULT_APPLICATION_POLICY="SKIP"

# By default, ufw only touches its own chains. Set this to 'yes' to have ufw
# manage the built-in chains too. Warning: setting this to 'yes' will break
# non-ufw managed firewall rules
MANAGE_BUILTINS=no

#
# IPT backend
#
# only enable if using iptables backend
IPT_SYSCTL=/etc/ufw/sysctl.conf

# Extra connection tracking modules to load. Complete list can be found in
# net/netfilter/Kconfig of your kernel source. Some common modules:
# nf_conntrack_irc, nf_nat_irc: DCC (Direct Client to Client) support
# nf_conntrack_netbios_ns: NetBIOS (samba) client support
# nf_conntrack_pptp, nf_nat_pptp: PPTP over stateful firewall/NAT
# nf_conntrack_ftp, nf_nat_ftp: active FTP support
# nf_conntrack_tftp, nf_nat_tftp: TFTP support (server side)
# nf_conntrack_sane: sane support
IPT_MODULES="nf_conntrack_ftp nf_nat_ftp nf_conntrack_netbios_ns"

But I still get the following logged in the systemd journal:

Aug 17 09:42:45 host kernel: [UFW BLOCK] IN=eth0 OUT= MAC=x:x:x:x:x:x:x SRC=138.59.218.118 DST=x.x.x.x LEN=60 TOS=0x08 PREC=0x20 TTL=52 ID=3788 DF PROTO=TCP SPT=37227 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0
Aug 17 09:42:52 host kernel: [UFW BLOCK] IN=eth0 OUT= MAC=x:x:x:x:x:x:x SRC=41.65.64.36 DST=x.x.x.x LEN=60 TOS=0x08 PREC=0x20 TTL=50 ID=12022 DF PROTO=TCP SPT=46718 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0
Aug 17 09:42:53 host sshd[726]: Connection closed by 138.59.218.118 port 37227 [preauth]
Aug 17 09:42:54 host sshd[728]: Invalid user ubuntu from 41.65.64.36 port 46718
Aug 17 09:42:54 host sshd[728]: pam_unix(sshd:auth): check pass; user unknown
Aug 17 09:42:54 host sshd[728]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=41.65.64.36
Aug 17 09:42:56 host sshd[728]: Failed password for invalid user ubuntu from 41.65.64.36 port 46718 ssh2
Aug 17 09:42:56 host sshd[728]: Received disconnect from 41.65.64.36 port 46718:11: Bye Bye [preauth]
Aug 17 09:42:56 host sshd[728]: Disconnected from invalid user ubuntu 41.65.64.36 port 46718 [preauth]

The above is before issuing ufw reload after a reboot. After a ufw reload traffic is blocked correctly.

I am unsure why extra DROP rules are needed at the end of the chain, so I think this might be a bug.

I will add that I don't see this problem on non-Virtuozzo platforms.

Please let me know if you need additional information.

Thanks

[Jamie Strandboge (jdstrand)]

Can you attach the output of the following command:

$ sudo /usr/share/ufw/check-requirements

[Bruce Pieterse (octoquad)]

$ sudo /usr/share/ufw/check-requirements
Has python: pass (binary: python2.7, version: 2.7.15+, py2)
Has iptables: pass
Has ip6tables: pass

Has /proc/net/dev: pass
Has /proc/net/if_inet6: pass

This script will now attempt to create various rules using the iptables
and ip6tables commands. This may result in module autoloading (eg, for
IPv6).
Proceed with checks (Y/n)? y
== IPv4 ==
Creating 'ufw-check-requirements'... done
Inserting RETURN at top of 'ufw-check-requirements'... done
TCP: pass
UDP: pass
destination port: pass
source port: pass
ACCEPT: pass
DROP: pass
REJECT: pass
LOG: pass
hashlimit: pass
limit: pass
ctstate (NEW): pass
ctstate (RELATED): pass
ctstate (ESTABLISHED): pass
ctstate (INVALID): pass
ctstate (new, recent set): pass
ctstate (new, recent update): pass
ctstate (new, limit): pass
interface (input): pass
interface (output): pass
multiport: pass
comment: pass
addrtype (LOCAL): pass
addrtype (MULTICAST): pass
addrtype (BROADCAST): pass
icmp (destination-unreachable): pass
icmp (source-quench): pass
icmp (time-exceeded): pass
icmp (parameter-problem): pass
icmp (echo-request): pass

== IPv6 ==
Creating 'ufw-check-requirements6'... done
Inserting RETURN at top of 'ufw-check-requirements6'... done
TCP: pass
UDP: pass
destination port: pass
source port: pass
ACCEPT: pass
DROP: pass
REJECT: pass
LOG: pass
hashlimit: pass
limit: pass
ctstate (NEW): pass
ctstate (RELATED): pass
ctstate (ESTABLISHED): pass
ctstate (INVALID): pass
ctstate (new, recent set): pass
ctstate (new, recent update): pass
ctstate (new, limit): pass
interface (input): pass
interface (output): pass
multiport: pass
comment: pass
icmpv6 (destination-unreachable): pass
icmpv6 (packet-too-big): pass
icmpv6 (time-exceeded): pass
icmpv6 (parameter-problem): pass
icmpv6 (echo-request): pass
icmpv6 with hl (neighbor-solicitation): pass
icmpv6 with hl (neighbor-advertisement): pass
icmpv6 with hl (router-solicitation): pass
icmpv6 with hl (router-advertisement): pass
ipv6 rt: pass

All tests passed

[Jamie Strandboge (jdstrand)]

Can you perform the following:

$ mkdir /tmp/ufw
$ sudo ufw show raw > /tmp/ufw/raw
$ sudo tar -zcvf /tmp/1840526.tar.gz /tmp/ufw /etc/default/ufw /etc/ufw /lib/ufw

then attach to this bug /tmp/1840526.tar.gz? (or send to me privately if you prefer)

[Jamie Strandboge (jdstrand)]

Thank you for sending me the requested information privately. I was not able to reproduce this in an up to date Ubuntu 18.04 vm. I performed:

$ cd /
$ sudo tar -zxvf .../1804526.tar.gz
$ sudo reboot

The firewall came up such that 'ufw status' showed everything I expected and correctly blocked incoming connections with the ssh drop rules.

This report sounds like another firewall application is running at the same time, but looking at /tmp/ufw/raw, I don't see anything suspicious, but that may be because you already ran 'ufw reload'. Can you reboot, then attach or send privately the output of:

$ sudo ufw show raw > /tmp/1840526-raw-after-reboot.txt

Thanks!

Btw, you reported this against ufw 0.35-5. ufw 0.36-0ubuntu0.18.04.1 is available in Ubuntu 18.04 and I recommend upgrading (though I tried to reproduce this with both versions).

[Bruce Pieterse (octoquad)]

I upgraded ufw, I now have:

$ apt policy ufw ufw:
  Installed: 0.36-0ubuntu0.18.04.1
  Candidate: 0.36-0ubuntu0.18.04.1
  Version table:
 *** 0.36-0ubuntu0.18.04.1 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     0.35-5 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages

I removed the last four user deny rules and rebooted. As far as I can see, it seems to be working correctly for most traffic, but I'm still seeing this:

Sep 04 18:25:48 host kernel: [UFW BLOCK] IN=eth0 OUT= MAC=x:x:x:x:x:x:x:x:x SRC=103.21.228.3 DST=x.x.x.x LEN=60 TOS=0x08 PREC=0x00 TTL=48 ID=33492 DF PROTO=TCP SPT=34876 DPT=22 WINDOW=14600 RES=0x00 SYN URGP=0
Sep 04 18:25:51 host sshd[1234]: Invalid user fh from 103.21.228.3 port 34876
Sep 04 18:25:51 host sshd[1234]: pam_unix(sshd:auth): check pass; user unknown
Sep 04 18:25:51 host sshd[1234]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.21.228.3
Sep 04 18:25:52 host sshd[1234]: Failed password for invalid user fh from 103.21.228.3 port 34876 ssh2
Sep 04 18:25:53 host sshd[1234]: Received disconnect from 103.21.228.3 port 34876:11: Bye Bye [preauth]
Sep 04 18:25:53 host sshd[1234]: Disconnected from invalid user fh 103.21.228.3 port 34876 [preauth]

After a reboot, I also saw these entries:

Sep 04 18:05:44 host systemd-journald[65]: Journal started
Sep 04 18:05:44 host systemd-journald[65]: Runtime journal (/run/log/journal/e9e1d08bc42c48999865b96c250f40cc) is 3.2M, max 25.6M, 22.3M free.
Sep 04 18:05:44 host systemd[1]: Starting Flush Journal to Persistent Storage...
Sep 04 18:05:44 host systemd-sysctl[69]: Couldn't write '1' to 'kernel/kptr_restrict', ignoring: Permission denied
Sep 04 18:05:44 host systemd-sysctl[69]: Couldn't write '1' to 'fs/protected_hardlinks', ignoring: Permission denied
Sep 04 18:05:44 host systemd-sysctl[69]: Couldn't write '1' to 'fs/protected_symlinks', ignoring: Permission denied
Sep 04 18:05:44 host systemd-sysctl[69]: Couldn't write '176' to 'kernel/sysrq', ignoring: Permission denied
Sep 04 18:05:44 host systemd-sysctl[69]: Couldn't write '1' to 'net/ipv4/tcp_syncookies', ignoring: No such file or directory
Sep 04 18:05:44 host systemd-sysctl[69]: Couldn't write '1' to 'kernel/yama/ptrace_scope', ignoring: Permission denied
Sep 04 18:05:44 host systemd-sysctl[69]: Couldn't write '65536' to 'vm/mmap_min_addr', ignoring: Operation not permitted
Sep 04 18:05:44 host systemd-sysctl[69]: Couldn't write 'fq_codel' to 'net/core/default_qdisc', ignoring: No such file or directory
Sep 04 18:05:44 host systemd[1]: Started Apply Kernel Variables.
Sep 04 18:05:44 host systemd-journald[65]: Time spent on flushing to /var is 924.980ms for 12 entries.
Sep 04 18:05:44 host systemd-journald[65]: System journal (/var/log/journal/e9e1d08bc42c48999865b96c250f40cc) is 1.1G, max 1.0G, 0B free.
Sep 04 18:05:44 host systemd[1]: Started udev Col...

[Jamie Strandboge (jdstrand)]

You mentioned:

"
This one is interesting:

Sep 04 18:05:46 host ifup[194]: iptables-restore: line 151 failed

/etc/network/if-pre-up.d/iptables restores the firewall from /etc/firewall.conf and line 151 is just "COMMIT".

The iptables-restore: line 151 failed issue started around 9th July 2019, which is roughly when I noticed this issue.

I'll send a new tarball shortly.
"

You appear to have an additional firewall package installed and enabled or a cron job for saving/loading the firewall on the system. Please disable this and report back if you are still having issues.

[Bruce Pieterse (octoquad)]

I decided to start over to figure out if this might have been a configuration issue.

I completely purged ufw, and removed all configurations from /etc/ufw and moved /etc/firewall.conf out the way. Reinstalled ufw, reapplied the rules and rebooted. I was still getting the problem.

I had a look inside the /etc/firewall.conf file and it now had more recent timestamps. After every reboot the chains are stored correctly to the file. The last time chains were saved to the file was on Sat Sep 29 10:37:48 2018. Funny thing is, the old one had the same ufw-user-input rules as the newly generated one. ifup No longer throws an error regarding line 151 in the file as well.

I remember there being a Firewall Management option when you mentioned "additional firewall package" in the web interface, but it was disabled and no rules were setup. I also couldn't find any other firewall packages installed via dpkg -l.

This is strange, but if I enable the Firewall via the Firewall Management in the VM panel then ufw works correctly. If I disable it, a few seconds later it would behave exactly as described in this bug report and most traffic is allowed to hit the ports that I've blocked. I've left it running for 30 minutes so far with the Firewall Management enabled with no rules and ufw is working perfectly.

I'm not sure if this issue should be pushed a bit more upstream to Virtuozzo. I see this more of a bug than a feature on the web-based panel in conjunction with ufw in the VM.

Thank you so much for assisting me with this.

I'm going to monitor for another day or two and revert back. So far, so good. :)

[Bruce Pieterse (octoquad)]

Still blocking as it should. This bug can now be closed.

Thanks again.

[Launchpad Janitor (janitor)]

[Expired for ufw because there has been no activity for 60 days.]