rules disappear at random intervals

[mfellman]

I have a system where every failed login will result in a DENY rule in ufw so I stop further attacks. This results in a lot of rules generated. I found out that sometimes the ALLOW rules (at the end of the user.rules file) disappear sometimes. Just now I had over 2200 rules before and after an update I had only 700+ rules left.

It is version 0.36

I tested the amount of rules with ufw status|grep DENY|wc -l

[Jamie Strandboge (jdstrand)]

Thank you for reporting a bug. What software are you using to add the DENY rules? Note that fail2ban will timeout the deny rules and remove them periodically, but that shouldn't remove the ALLOW rules but also note that the ufw status listing groups things together: incoming, then outgoing then route (forward) rules and within each groups, ipv4 first then ipv6.

Can you (perhaps from backups) provide the following from when you had 2200 rules:

 $ mkdir /tmp/ufw
 $ sudo ufw show raw > /tmp/ufw/raw
 $ sudo tar -zcvf /tmp/1842333.tar.gz /tmp/ufw /etc/default/ufw
 /etc/ufw /lib/ufw

then attach to this bug /tmp/1842333.tar.gz? (or send to me privately if you prefer).

Please do it again with the 700 rules and attach as /tmp/1842333_after.tar.gz

[mfellman (marc-fellman)]

Here the current state. See bug report for comments.

On 3-9-2019 18:24, Jamie Strandboge wrote:
> Thank you for reporting a bug. What software are you using to add the
> DENY rules? Note that fail2ban will timeout the deny rules and remove
> them periodically, but that shouldn't remove the ALLOW rules but also
> note that the ufw status listing groups things together: incoming, then
> outgoing then route (forward) rules and within each groups, ipv4 first
> then ipv6.
>
> Can you (perhaps from backups) provide the following from when you had
> 2200 rules:
>
> $ mkdir /tmp/ufw
> $ sudo ufw show raw > /tmp/ufw/raw
> $ sudo tar -zcvf /tmp/1842333.tar.gz /tmp/ufw /etc/default/ufw
> /etc/ufw /lib/ufw
>
> then attach to this bug /tmp/1842333.tar.gz? (or send to me privately if
> you prefer).
>
> Please do it again with the 700 rules and attach as
> /tmp/1842333_after.tar.gz
>
> ** Changed in: ufw
> Status: New => Incomplete
>

[mfellman (marc-fellman)]

I wanted to move on. It was apparent that for the "production" server it was no longer working for me since it blocked SMTP every now and then and if I did not see it in time could mean loosing mail. I did only manage the DENY rules via UFW and it was/is my suspicion that it is a concurrency issue.

I had a syslogng install starting a script on suspicious entries in the log to block related IP numbers. UIt appears that this is multi threaded where it is sometimes the case that multiple instances are adding DENY rules. In almost all cases it had no issue but in some cases (which I still not know when) it appears to have a corrupted user.rules file and lost lines at the end (The ACCEPT lines of important services).

I switched to using ipset in the after.init (like i found somewhere on the internet). This had a few advantages. One is a smaller ufw status listing and a second is adding an IP number is instant. Not taking almost 10 seconds.

[Jamie Strandboge (jdstrand)]

Thank you for providing additional information, however this seems to be the new configuration you are using with ipset and as such, it doesn't provide the necessary information to reproduce or triage this. You suspected that this might be a concurrency issue, but as of 8387ae082a362b00ad339293ec2ad547338b9209 (in 0.36, available in 12/2018), concurrency is supported so long as you use the 'ufw' command to add rules to the ruleset.

How were you adding the rules and what is the output of 'ufw version'?

[mfellman (marc-fellman)]

If I use ufw version it just tells me:
ufw 0.36
Copyright 2008-2015 Canonical Ltd.

(BTW I run the laytest version of 18.04).

I might try to test it on a seperate instance I have elsewhere. I tried to see if i could install 0.36-1 but could not find a way. The basic command I used to add DENY rules was:

ufw insert 2 deny from <ipnum>

I inserted it after the first basic one just to have an easier way to maintain rules but did not workout as expected. I will try to see what happenes on my second (playground) system this weekend.

[mfellman (marc-fellman)]

Currently I'm running through a list of ip numbers that tried to get into my system on the playground server (Ubuntu 18.10 with 0.36 of ufw). During that process I tried to add ALLOW ports but they get not added for the IPV4 version of the rules. Note this is during adding over 5000 ip numbers with:

ufw insert 2 deny from <ip>

Only after a few tries I was able to add an allow port for ipv4.
root@somehost:~# ufw allow 22
Rule added
Skipping adding existing rule (v6)
root@somehost:~# ufw allow 22
Skipping adding existing rule
Skipping adding existing rule (v6)
root@somehost:~# ufw allow 22
Skipping adding existing rule
Skipping adding existing rule (v6)
root@somehost:~# ufw allow 8822
Skipping adding existing rule
Skipping adding existing rule (v6)
root@somehost:~# ufw allow 2222
Rule added
Skipping adding existing rule (v6)
root@somehost:~# ufw allow 22
Skipping adding existing rule
Skipping adding existing rule (v6)
root@somehost:~# ufw allow 22
Skipping adding existing rule
Skipping adding existing rule (v6)
root@somehost:~# ufw status|grep ALLOW
80 ALLOW Anywhere
8822 ALLOW Anywhere
22 ALLOW Anywhere
80 (v6) ALLOW Anywhere (v6)
443 (v6) ALLOW Anywhere (v6)
22 (v6) ALLOW Anywhere (v6)
2222 (v6) ALLOW Anywhere (v6)
8822 (v6) ALLOW Anywhere (v6)
root@somehost:~# ufw allow 2222
Rule added
Skipping adding existing rule (v6)
root@somehost:~# ufw allow 2222
Rule added
Skipping adding existing rule (v6)
root@somehost:~# ufw allow 2222
Skipping adding existing rule
Skipping adding existing rule (v6)
root@somehost:~#

And now after a some time i decided to try to add the missing 443 for ipv4 (which was there when I started):
root@somehost:~# ufw allow 443
Rule added
Skipping adding existing rule (v6)
root@somehost:~# ufw status|grep ALLOW
WARN: Skipping malformed tuple: deny any any 0.0.0.0/0 any 52.231.25.2411 in

80 ALLOW Anywhere
8822 ALLOW Anywhere
22 ALLOW Anywhere
2222 ALLOW Anywhere
80 (v6) ALLOW Anywhere (v6)
443 (v6) ALLOW Anywhere (v6)
22 (v6) ALLOW Anywhere (v6)
2222 (v6) ALLOW Anywhere (v6)
8822 (v6) ALLOW Anywhere (v6)
root@somehost:~# ufw status|grep ALLOW
80 ALLOW Anywhere
8822 ALLOW Anywhere
22 ALLOW Anywhere
2222 ALLOW Anywhere
80 (v6) ALLOW Anywhere (v6)
443 (v6) ALLOW Anywhere (v6)
22 (v6) ALLOW Anywhere (v6)
2222 (v6) ALLOW Anywhere (v6)
8822 (v6) ALLOW Anywhere (v6)
root@somehost:~#

So it is not there but maybe the error message is telling something?

I retried a few times (synchronously still working on the long list of blocked ip numers):
root@somehost:~# uf...

[mfellman (marc-fellman)]

Some additional testing:

root@somehost:~# ufw allow 443
WARN: Skipping malformed tuple: deny any any 0.0.0.0/0 any 84.193.142.76 n

Rule added
Skipping adding existing rule (v6)
root@somehost:~# ufw allow 443
Rule added
Skipping adding existing rule (v6)
root@somehost:~# ufw allow 443
WARN: Skipping malformed tuple: deny any any 0.0.0.0/0 any 84.217.109.26.213 in

Rule added
Skipping adding existing rule (v6)
root@somehost:~# ufw allow 443
Rule added
Skipping adding existing rule (v6)
root@somehost:~# ufw allow 443
Skipping adding existing rule
Skipping adding existing rule (v6)
root@somehost:~# ufw status|grep ALLOW
80 ALLOW Anywhere
8822 ALLOW Anywhere
22 ALLOW Anywhere
2222 ALLOW Anywhere
80 (v6) ALLOW Anywhere (v6)
443 (v6) ALLOW Anywhere (v6)
22 (v6) ALLOW Anywhere (v6)
2222 (v6) ALLOW Anywhere (v6)
8822 (v6) ALLOW Anywhere (v6)

So it wrongly mentions it has added the rule.

[mfellman (marc-fellman)]

Once the whole list was processed adding allow for port 443 worked without a problem.

[Jamie Strandboge (jdstrand)]

You mentioned:

root@somehost:~# ufw allow 443
WARN: Skipping malformed tuple: deny any any 0.0.0.0/0 any 84.193.142.76 n
...
root@somehost:~# ufw allow 443
WARN: Skipping malformed tuple: deny any any 0.0.0.0/0 any 84.217.109.26.213 in

Can you look in user.rules for these stanzas and paste them here?

Also, what commands did you use to add the rules that ended up with a malformed tuple?

[mfellman (marc-fellman)]

Sorry,

Went away from this solution completely since I needed a something that
was more stable for my mail server. It failed to often with an end
result that I could no longer reach my server in normal ways (rules that
allowed IPV4 SSH access happen to disappear as well when this happened).
Currently blocking over 10000 IP addresses of scriptkiddies at work.

Marc

On 30-10-2019 16:15, Jamie Strandboge wrote:
> You mentioned:
>
> root@somehost:~# ufw allow 443
> WARN: Skipping malformed tuple: deny any any 0.0.0.0/0 any 84.193.142.76 n
> ...
> root@somehost:~# ufw allow 443
> WARN: Skipping malformed tuple: deny any any 0.0.0.0/0 any 84.217.109.26.213 in
>
> Can you look in user.rules for these stanzas and paste them here?
>
> Also, what commands did you use to add the rules that ended up with a
> malformed tuple?
>

[Jamie Strandboge (jdstrand)]

Thanks for getting back. Marking as Invalid then as it looks like the user.rules files were corrupted. If you have more information, please feel free to provide it here or in a new bug.