ufw doesn't support concurrent updates

[Jeremy]

[Impact]

ufw prior to 0.36 did not support locking or use xtables locking which can lead to problems when using ufw as a dynamic firewall.

[Test Case]

$ sudo apt-get install python-minimal # for the test script
$ sudo ./test_ufw_threads.py # this runs for many seconds

A patched ufw will complete with no errors. With an unpatched ufw, there will be many errors like:
$ sudo ./test_ufw_threads.py
...
ERROR: initcaps
[Errno 2] iptables: Chain already exists.

ERROR: initcaps
[Errno 2] iptables: Chain already exists.

[Regression Potential]

ufw 0.36 implements a global exclusive lock before doing any kernel or file writes. Risk of regression includes the lock file not being released due to coding error. The locking is done via fcntl.lockf and ufw releases it in a finally clause that is careful to always release it.

[Other Info]

This locking code has been in production in the ufw snap for years and is known to work is static environments as well as in dynamic environments (IPS/fail2ban/etc).

= Original description =

On a server under Ubuntu 12.04 I automatically update firewall rules and I unfortunately noticed that if 2 rules are processed at the same time it leads to an inconsistent result between iptables and ufw status.
In fact it can be reproduced using the python script in attachment.

Before executing this script I didn't have any rules matching the ip 192.168.254.1 on my computer.
After a launch iptables takes into acount 464 rules.

sudo iptables -L -n | grep 192.168.254.1 | wc -l
464

It should be 500 rules but the script stressed a lot my system and all rules can't be processed by iptables
sudo ./test_ufw_threads.py
iptables: Resource temporarily unavailable.

iptables: Resource temporarily unavailable.

iptables: Resource temporarily unavailable.
[...]

Anyway, ufw stores only the last updates (in /lib/ufw/user.rules):

sudo ufw status
État : actif

Vers Action Depuis
---- ------ ------
192.168.254.1 1234/tcp ALLOW 192.168.46.9
192.168.254.1 1234/tcp ALLOW 192.168.22.10
192.168.254.1 1234/tcp ALLOW 192.168.32.10
192.168.254.1 1234/tcp ALLOW 192.168.5.10
192.168.254.1 1234/tcp ALLOW 192.168.43.9
192.168.254.1 1234/tcp ALLOW 192.168.40.10
192.168.254.1 1234/tcp ALLOW 192.168.46.10
192.168.254.1 1234/tcp ALLOW 192.168.48.10
192.168.254.1 1234/tcp ALLOW 192.168.42.10
192.168.254.1 1234/tcp ALLOW 192.168.43.10

So I can't delete other rules using ufw, I had to directly use iptables.
In this case I can only delete 10 rules using ufw.
Could you please handle some kind of lock?

[Jamie Strandboge (jdstrand)]

There was a problem with the patch and this will be in the next release of ufw.

[Christopher M Luciano (cmluciano)]

Is there any update on this? I know there is a -w command in iptables with this patch https://patchwork.ozlabs.org/patch/635676/. Is there a way to call through ufw to set this option to increase the waits?

[Jamie Strandboge (jdstrand)]

@Christopher - I implemented this a bit differently but had some troubles with the testsuite. I think '-w' would be much simpler. There isn't a way to pass this into ufw, but you could modify backend_iptables.py towards the end of set_rule() to pass the flag.

[Christopher M Luciano (cmluciano)]

OK great. I will work on this patch.

[Luke (lukepolo)]

Is there a current work around for this, otherwise I have to re-write a bunch of code to go to iptables

[Jamie Strandboge (jdstrand)]

@Luke, you could modify backend_iptables.py as mentioned in comment #5 or you could perhaps use 'man 1 flock' or implement a lock wrapper around your invocation of ufw.

That said, since didn't hear back from Christopher, I'll take a look at implementing this for 0.36.

[Christopher M Luciano (cmluciano)]

Hi all. I posted a question on this last December, but I never received feedback on it. Here is the bug report https://bugs.launchpad.net/ufw/+bug/1652163.

[Luke (lukepolo)]

the flock worked perfectly thank you!

[Jamie Strandboge (jdstrand)]

This is in git master now and in the ufw snap in candidate.

[Jamie Strandboge (jdstrand)]

This is fixed in the new 0.36 release.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ufw - 0.36-1

---------------
ufw (0.36-1) unstable; urgency=medium

  * New upstream release (LP: #1782384, LP: #1664133, LP: #1509725,
    LP: #1695718, LP: #1719211, LP: #1775043, LP: #1204579, LP: #1652163,
    LP: #1377600, Closes: 686248, LP: #1368411, LP: #1586258, Closes: 909163,
    Closes: 884932, LP: #1558068)
    - drop 0002-bug849628.patch (included upstream)
    - drop 0003-use-default-tcp-syncookies.patch (included upstream)
    - drop 0004-lp1633698.patch (included upstream)
  * Remaining changes:
    - 0001-optimize-boot.patch
  * debian/ufw.maintscript: remove /etc/bash_completion.d/ufw on upgrade
    (LP: #1602834)
  * debian/control: remove no longer needed xs-python-version and
    x-python3-version fields
  * update debian/before6.rules.md5sum for file shipped in 0.35-6. While both
    before.rules and before6.rules were updated in this new upstream release,
    0.35-6 mistakenly already had its own md5sum for before.rules, so we don't
    need to add it now.

 -- Jamie Strandboge <email address hidden> Fri, 14 Dec 2018 17:50:47 +0000

[Łukasz Zemczak (sil2100)]

An upload of ufw to cosmic-proposed has been rejected from the upload queue for the following reason: "All bugs mentioned in the .changes file (so therefore also in the new debian/changelog entries) need to comply with SRU standards (test-case, regression potential). Please re-upload after filling out the required info or modify changelog to exclude irrelevant bug numbers.".

[Brian Murray (brian-murray)]

Hello Jeremy, or anyone else affected,

Accepted ufw into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ufw/0.36-0ubuntu0.18.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Jeremy, or anyone else affected,

Accepted ufw into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ufw/0.36-0ubuntu0.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Jamie Strandboge (jdstrand)]

Tested this is fixed in cosmic:

$ apt-cache policy ufw
ufw:
  Installed: 0.36-0ubuntu0.18.10.1
  Candidate: 0.36-0ubuntu0.18.10.1
  Version table:
 *** 0.36-0ubuntu0.18.10.1 500
        500 http://us.archive.ubuntu.com/ubuntu cosmic-proposed/main amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu cosmic-proposed/main i386 Packages
        100 /var/lib/dpkg/status
     0.35-6 500
        500 http://us.archive.ubuntu.com/ubuntu cosmic/main amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu cosmic/main i386 Packages

[Jamie Strandboge (jdstrand)]

Verified this is fixed in bionic:

$ apt-cache policy ufw
ufw:
  Installed: 0.36-0ubuntu0.18.04.1
  Candidate: 0.36-0ubuntu0.18.04.1
  Version table:
 *** 0.36-0ubuntu0.18.04.1 500
        500 http://us.archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu bionic-proposed/main i386 Packages
        100 /var/lib/dpkg/status
     0.35-5 500
        500 http://us.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu bionic/main i386 Packages

[Launchpad Janitor (janitor)]

This bug was fixed in the package ufw - 0.36-0ubuntu0.18.10.1

---------------
ufw (0.36-0ubuntu0.18.10.1) cosmic-proposed; urgency=medium

  * Backport to cosmic addressing the following SRU bugs:
    - LP: #1811129 - master SRU bug
    - LP: #1664133 - before6.rules: echo-reply needs to be before INVALID
    - LP: #1719211 - improve interface name checks
    - LP: #1775043 - shell-completion/bash: adjust for modern bash
    - LP: #1204579 - support concurrent updates
    - LP: #1368411 - add 'prepend' command

 -- Jamie Strandboge <email address hidden> Mon, 25 Mar 2019 20:46:10 +0000

[Robie Basak (racb)]

The verification of the Stable Release Update for ufw has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ufw - 0.36-0ubuntu0.18.04.1

---------------
ufw (0.36-0ubuntu0.18.04.1) bionic-proposed; urgency=medium

  * Backport to bionic addressing the following SRU bugs:
    - LP: #1811129 - master SRU bug
    - LP: #1664133 - before6.rules: echo-reply needs to be before INVALID
    - LP: #1719211 - improve interface name checks
    - LP: #1775043 - shell-completion/bash: adjust for modern bash
    - LP: #1204579 - support concurrent updates
    - LP: #1368411 - add 'prepend' command

 -- Jamie Strandboge <email address hidden> Mon, 25 Mar 2019 21:14:25 +0000

[Raynald de Lahondès (raynald)]

ufw does not support rule removal. For a particular project (a task queue on a cloud adhoc HPC cluster), I have had the occasion to test it in a relatively massive parallel rate (20 concurrent uses). I have experienced:
- nice behaviour when adding rules,
- bad behaviour when removing rules.

I was triggering ufw with ansible parallel playbooks execution.

When removing rules the defect rate was high (as soon as more than two playbooks were launched together, some of the rules were not deleted). In that case, ufw fails silently, which is quite bad.

As a counter measure I used task-spooler (apt install task-spooler) I changed my ansible task from:
```yaml
community.general.ufw:
   from_ip: "{{ hostvars[nodename].ipv4 }}"
   rule: allow
   delete: yes
```

to
```yaml
ansible.builtin.script:
   cmd: /usr/bin/tsp /usr/sbin/ufw delete allow from "{{ hostvars[nodename].ipv4 }}"
   become: true
```

This solved the issue.

[Raynald de Lahondès (raynald)]

I have declared also the bug in Ansible collection repo, however this is clearly a UFW bug:
https://github.com/ansible-collections/community.general/issues/4769