ufw breaks boot on network root filesystem

[Mauricio Faria de Oliveira]

[Impact]

A system with rootfs on iSCSI stops booting when ufw.service starts.
The kernel logs iSCSI command/reset timeout until I/O fails and the
root filesystem/journal break.

The issue is that ufw_start() sets the default policy _first_, then
adds rules _later_.

So, a default INPUT policy of DROP (default setting in ufw) prevents
further access to the root filesystem (blocks incoming iSCSI traffic)
thus any rules that could help are not loaded (nor anything else.)

[Fix]

The fix is to set default policy after loading rules in ufw_start().
That seems to be OK as `ip[6]tables-restore -n/--noflush` is used,
and per iptables source, that only sets the chain policy.

This allows the system to boot due to the RELATED,ESTABLISHED rule,
that is introduced by before.rules in INPUT/ufw-before-input chain.

The comparison of `iptables -L` before/after shows no differences
(verified on a local rootfs); `run_tests.sh` has 0 skipped/errors.

[Test Steps]

 * Install Ubuntu on an iSCSI (or other network-based) root filesystem.
   (eg, Oracle Cloud's bare-metal 'BM.Standard1.36' shape.)

 * sudo ufw enable

 * Observed: system may stall immediately if no prior iptables rules.
   (eg, iptables -A INPUT -p tcp -s 169.254.0.2 --sport 3260 -j ACCEPT)

 * Expected: system continues working.

 * sudo reboot

 * Observed: system boot stalls once ufw.service starts (see below.)
 * Expected: system boot should move on.

[Regression Potential]

 * Potential regressions would be observed on ufw start/reload,
   when iptables rules are configured.

 * The resulting iptables configuration has been compared
   before/after the change, with identical rules on both.

[Other Info]

 * Fixed in Debian and Jammy.

[ufw info]

# ufw --version
ufw 0.36
Copyright 2008-2015 Canonical Ltd.

# lsb_release -cs
focal

[Boot Log]

[ 232.168355] iBFT detected.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ...
Setting up software interface enp45s0f0np0
...
[ 254.644505] Loading iSCSI transport class v2.0-870.
[ 254.714938] iscsi: registered transport (tcp)
[ 254.780129] scsi host12: iSCSI Initiator over TCP/IP
...
[ 255.433491] sd 12:0:0:1: [sda] 251658240 512-byte logical blocks: (129 GB/120 GiB)
...
[ 256.379550] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
...
[ 266.620860] systemd[1]: Starting Uncomplicated firewall...
Starting Uncomplicated firewall...
...
[ 298.491560] session1: iscsi_eh_cmd_timed_out scsi cmd 00000000310a6696 timedout
[ 298.580803] session1: iscsi_eh_cmd_timed_out return shutdown or nh
[ 298.656262] session1: iscsi_eh_cmd_timed_out scsi cmd 0000000094ad9246 timedout
[ 298.745237] session1: iscsi_eh_cmd_timed_out return shutdown or nh
[ 298.745270] session1: iscsi_eh_abort aborting sc 00000000310a6696
[ 298.899644] session1: iscsi_eh_abort aborting [sc 00000000310a6696 itt 0x13]
[ 298.985788] session1: iscsi_exec_task_mgmt_fn tmf set timeout
[ 302.075554] session1: iscsi_eh_cmd_timed_out scsi cmd 000000001a9458b5 timedout
[ 302.164786] session1: iscsi_eh_cmd_timed_out return shutdown or nh
[ 314.107541] session1: iscsi_tmf_timedout tmf timedout
[ 314.169797] connection1:0: detected conn error (1021)
[ 314.232266] session1: iscsi_eh_abort abort failed [sc 00000000310a6696 itt 0x13]
[ 314.323531] session1: iscsi_eh_abort aborting sc 0000000094ad9246
[ 314.399640] session1: iscsi_eh_abort sc never reached iscsi layer or it completed.
[ 314.495578] session1: iscsi_eh_abort aborting sc 000000001a9458b5
[ 314.571554] session1: iscsi_eh_abort sc never reached iscsi layer or it completed.
[ 314.664050] session1: iscsi_eh_device_reset LU Reset [sc 00000000310a6696 lun 1]
[ 314.755773] session1: iscsi_eh_device_reset dev reset result = FAILED
[ 314.834736] session1: iscsi_eh_target_reset tgt Reset [sc 00000000310a6696 tgt <...>]
[ 314.954144] session1: iscsi_eh_target_reset tgt <...> reset result = FAILED
[ 315.063456] connection1:0: detected conn error (1021)
[ 315.125743] session1: iscsi_eh_session_reset wait for relogin
[ 398.843556] INFO: task systemd:1 blocked for more than 120 seconds.
...
[ 401.039006] INFO: task jbd2/sda1-8:2522 blocked for more than 123 seconds.
...
[ 402.483917] INFO: task iptables-restor:2648 blocked for more than 124 seconds.
...
[ 435.707549] session1: session recovery timed out after 120 secs
[ 435.780058] session1: iscsi_eh_session_reset failing session reset: Could not log back into <...> [age 0]
[ 435.920710] sd 12:0:0:1: Device offlined - not ready after error recovery
[ 436.003563] sd 12:0:0:1: [sda] tag#105 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK cmd_age=169s
[ 436.015520] sd 12:0:0:1: rejecting I/O to offline device
[ 436.134354] sd 12:0:0:1: [sda] tag#105 CDB: Read(10) 28 00 00 05 8d d8 00 00 08 00
[ 436.198807] blk_update_request: I/O error, dev sda, sector 360816 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 436.198818] blk_update_request: I/O error, dev sda, sector 2324480 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 436.198852] EXT4-fs warning (device sda1): htree_dirblock_to_tree:1004: inode #1398: lblock 0: comm systemd: error -5 reading directory block
[ 436.290259] blk_update_request: I/O error, dev sda, sector 363992 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 436.417093] Buffer I/O error on dev sda1, logical block 262144, lost sync page write
[ 436.417103] blk_update_request: I/O error, dev sda, sector 364040 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 436.417130] JBD2: Error -5 detected when updating journal superblock for sda1-8.
[ 436.417132] Aborting journal on device sda1-8.

[Mauricio Faria de Oliveira (mfo)]

Merge Proposal submitted:
https://code.launchpad.net/~mfo/ufw/+git/ufw/+merge/410091

[Jamie Strandboge (jdstrand)]

I merged the changes into master. Thanks Mauricio!

[Jamie Strandboge (jdstrand)]

For Impish, lets update debian/master, then I'll upload there and sync to Ubuntu.

[Mauricio Faria de Oliveira (mfo)]

MR for debian/master submitted [1].

Since Impish is in Final Freeze as of last week,
this would fit a post-release SRU per [2] IIUIC,
so a sync wouldn't be possible, I think.

Since the devel/J series isn't open yet, perhaps
just an Impish SRU is enough now, as the devel
release will start from its packages in a bit?

I'll check that, and get back to you.

Thanks!

[1] https://code.launchpad.net/~mfo/ufw/+git/ufw/+merge/410152
[2] https://lists.ubuntu.com/archives/ubuntu-devel-announce/2021-October/001301.html

[Jamie Strandboge (jdstrand)]

Ah, I hadn't checked that yet. Yes, please feel free to do the Impish SRU and the 0.36.1-2 that I just uploaded to Debian will float into 'J' after it opens.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ufw - 0.36.1-2

---------------
ufw (0.36.1-2) unstable; urgency=medium

  [ Mauricio Faria de Oliveira ]
  * 0004-set-default-policy-after-load.patch: fix boot stall on iscsi/network
    root filesystem when starting ufw (LP: #1946804)

  [ Jamie Strandboge ]
  * rename python3-versions.diff as 0003-python3-versions.patch
  * debian/upstream/metadata: add Bug-Submit and Bug-Database

 -- Jamie Strandboge <email address hidden> Wed, 13 Oct 2021 19:02:20 +0000

[Mauricio Faria de Oliveira (mfo)]

Verified test packages (ppa:mfo/lp1946804) for
the Impish, Hirsute, Focal, and Bionic releases
on Oracle Cloud's 'BM.Standard1.36' systems.

(Impish/Hirsute: Focal and do-release-upgrade.)

...

Without the patch, the system boot stalls.
With the patch, the system boot continues.

(Note: netfilter-persistent.service needed to
be disabled, otherwise it flushes ufw's rules.)

...

The output of `iptables -L -n` was the same with/without the patch.

# diff iptables.before iptables.after; echo $?
0

# wc -l iptables.before iptables.after
  170 iptables.before
  170 iptables.after
  340 total

...

Versions tested (original/without patch)
I: Version: 0.36.1-1
H: Version: 0.36-7.1
F: Version: 0.36-6
B: Version: 0.36-0ubuntu0.18.04.1

[Mauricio Faria de Oliveira (mfo)]

Uploaded to I/H/F/B.

[Brian Murray (brian-murray)]

Hello Mauricio, or anyone else affected,

Accepted ufw into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ufw/0.36.1-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Mauricio, or anyone else affected,

Accepted ufw into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ufw/0.36-7.1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Mauricio, or anyone else affected,

Accepted ufw into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ufw/0.36-6ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Mauricio, or anyone else affected,

Accepted ufw into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ufw/0.36-0ubuntu0.18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Jamie Strandboge (jdstrand)]

Tested 0.36-6ubuntu1 on focal. apt upgrade succeeded and after reboot the firewall came up with the expected rules in the expected order and I spot-checked allowed and deny traffic. I didn't test on an iSCSI system so won't add verification-done-focal at this time, but I think the testing is probably sufficient for that (I'll let others decide).

[Jamie Strandboge (jdstrand)]

Tested 0.36-0ubuntu0.18.04.2 on bionic. apt upgrade succeeded and after reboot the firewall came up with the expected rules in the expected order and I spot-checked allowed and deny traffic. I didn't test on an iSCSI system so won't add verification-done-focal at this time, but I think the testing is probably sufficient for that (I'll let others decide).

[Mauricio Faria de Oliveira (mfo)]

Tested with Bionic, Focal, Hirsute, and Impish
with the test steps provided, on Oracle Cloud.

All good.

With the packages in -proposed, the system can reboot correctly.
---

bionic
Version: 0.36-0ubuntu0.18.04.2

focal
Version: 0.36-6ubuntu1

hirsute
Version: 0.36-7.1ubuntu1

impish
Version: 0.36.1-1ubuntu1

...

With the packages in -updates, the system stalls on boot
---

bionic
Version: 0.36-0ubuntu0.18.04.1

focal
Version: 0.36-6

hirsute
Version: 0.36-7.1

impish
Version: 0.36.1-1

[Launchpad Janitor (janitor)]

This bug was fixed in the package ufw - 0.36.1-1ubuntu1

---------------
ufw (0.36.1-1ubuntu1) impish; urgency=medium

  * d/p/0004-set-default-policy-after-load.patch: fix boot stall on
    iscsi/network root filesystem when starting ufw (LP: #1946804)

 -- Mauricio Faria de Oliveira <email address hidden> Mon, 25 Oct 2021 14:25:30 -0300

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for ufw has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ufw - 0.36-7.1ubuntu1

---------------
ufw (0.36-7.1ubuntu1) hirsute; urgency=medium

  * d/p/0015-set-default-policy-after-load.patch: fix boot stall on
    iscsi/network root filesystem when starting ufw (LP: #1946804)
  * d/p/0016-unconditionally-reload-with-delete.patch: fix corner
    case of rule deletion with specific/any proto (LP: #1933117)

 -- Mauricio Faria de Oliveira <email address hidden> Mon, 25 Oct 2021 17:58:58 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package ufw - 0.36-6ubuntu1

---------------
ufw (0.36-6ubuntu1) focal; urgency=medium

  * d/p/0012-set-default-policy-after-load.patch: fix boot stall on
    iscsi/network root filesystem when starting ufw (LP: #1946804)
  * d/p/0013-unconditionally-reload-with-delete.patch: fix corner case
    of rule deletion with specific/any proto (LP: #1933117)

 -- Mauricio Faria de Oliveira <email address hidden> Mon, 25 Oct 2021 14:30:14 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package ufw - 0.36-0ubuntu0.18.04.2

---------------
ufw (0.36-0ubuntu0.18.04.2) bionic; urgency=medium

  * d/p/0002-set-default-policy-after-load.patch: fix boot stall on
    iscsi/network root filesystem when starting ufw (LP: #1946804)
  * d/p/0003-unconditionally-reload-with-delete.patch: fix corner case
    of rule deletion with specific/any proto (LP: #1933117)

 -- Mauricio Faria de Oliveira <email address hidden> Mon, 25 Oct 2021 14:30:24 -0300

[Thang Duong (thang-3010)]

Hello folks, sorry for digging the old thread. I'd like to ask whether you have tested the bug on Ubuntu VM? VMware specifically.

Cheers

[Mauricio Faria de Oliveira (mfo)]

Hi. No, I haven't tested on that; but if this helps, the issue is independent of platform/hypervisor as it's in ufw/firewall, thus the fix should help anyway.

[Jamie Strandboge (jdstrand)]

This was fixed in 0.36.2.