ufw 0.30.1 make test fail

[Sławomir Nizio]

Tests (make test or ./run_tests.sh -s) fail, both when run as unprivileged user and as root.

what I do (for example as root):
cd ufw-0.30.1
make test

attaching output etc.
PS do I understand correctly that the default set of tests can be run as unprivileged user and root?
Thanks!

specs:
ufw-0.30.1
Sabayon (Gentoo based)
Python 2.7
iptables 1.4.10

[Sławomir Nizio (snizio)]

in case it could be needed, attaching console output from all standard tests (no -s);
apparently tests/testarea has been deleted so I can't attach it again.

[Jamie Strandboge (jdstrand)]

Thank you for using ufw and reporting a bug. './run_tests.sh' can be run without privileges. There are additional 'root' tests that you can run with './run_tests.sh -s root', but be aware in addition to requiring privilege, these tests will modify the running firewall.

As for the tests themselves, this looks curious:
diff -Naur tests/bad/apps/result tests/testarea/tmp/result
make: *** [test] BÅ‚Ä…d 1

It appears you are running this under a non-C local. Can you try with:
LANG=C ./run_tests.sh -s

[Sławomir Nizio (snizio)]

That additional line was from make and it says "Error 1" (I have UTF-8 locale and the characters displayed incorrectly here).
I'm attaching log from the command:
LANG=C make test

not ./run_tests.sh -s as you wanted, because it wouldn't contain that message from make;
also difference between output from those commands is only make's messages - I've checked that:
--- /tmp/output
+++ /tmp/output2
@@ -1,4 +1,3 @@
-./run_tests.sh -s
 Interpreter: python

@@ -21,4 +20,3 @@
 FAILED tests/bad/apps -- result found in tests/testarea/tmp/result
 For more information, see:
 diff -Naur tests/bad/apps/result tests/testarea/tmp/result
-make: *** [test] Error 1
?field.comment=That additional line was from make and it says "Error 1" (I have UTF-8 locale and the characters displayed incorrectly here).
I'm attaching log from the command:
LANG=C make test

not ./run_tests.sh -s as you wanted, because it wouldn't contain that message from make;
also difference between output from those commands is only make's messages - I've checked that:
--- /tmp/output
+++ /tmp/output2
@@ -1,4 +1,3 @@
-./run_tests.sh -s
 Interpreter: python

@@ -21,4 +20,3 @@
 FAILED tests/bad/apps -- result found in tests/testarea/tmp/result
 For more information, see:
 diff -Naur tests/bad/apps/result tests/testarea/tmp/result
-make: *** [test] Error 1

Thanks for your attention.

[Sławomir Nizio (snizio)]

Oops, the comment is a bit "malformed" but everything is here. Please ask if something is unclear.

[Jamie Strandboge (jdstrand)]

Is this still a problem with ufw 0.31.1 or 0.32?

[Sławomir Nizio (snizio)]

Yes, it is.

I took a look at two first failing tests. Let me share my thoughts .

Performing tests 'bad/apps'
- installing
- result:
Command '--dry-run allow to 192.168.0.0/16 app Samba from 192.168.0.0/16 port http' exited with '0', but expected '1'
    ** FAIL **

This command is assumed to fail, but it does not. It fails in the test if I remove this file manually:
rm ./tests/defaults/profiles/samba
and then it'll fail due to non-existent "samba" profile.

Outside of the test environment it executes with no problem, too, provided that I copy 'samba' from examples/ to applications.d.
# ufw allow to 192.168.0.0/16 app Samba from 192.168.0.0/16 port http
Rule added

# echo $?
0

# ufw status numbered
Status: active

     To Action From
     -- ------ ----
[ 1] <snip> DENY OUT Anywhere (out)
[ 2] <snip> DENY OUT Anywhere (log, out)
[ 3] 192.168.0.0/16 Samba ALLOW IN 192.168.0.0/16 80

("<snip>" are two IPs for test purposes; exact IPs aren't relevant here)

# ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing)
New profiles: skip

To Action From
-- ------ ----
192.168.0.0/16 137,138/udp (Samba) ALLOW IN 192.168.0.0/16 80/udp
192.168.0.0/16 139,445/tcp (Samba) ALLOW IN 192.168.0.0/16 80/tcp

<snip> DENY OUT Anywhere
<snip> DENY OUT Anywhere (log)

# ufw delete allow to 192.168.0.0/16 app Samba from 192.168.0.0/16 port http
Rule deleted

(Did I mention ufw has a great UI? :))

Let's take a look at the second failing test.

Performing tests 'bad/args'
- installing
- result:
Command '--dry-run allow smtp/udp' exited with '0', but expected '1'
    ** FAIL **

# ufw allow smtp/udp
Rule added
Rule added (v6)

 # ufw status
Status: active

To Action From
-- ------ ----
25/udp ALLOW Anywhere
25/udp ALLOW Anywhere (v6)

<snip> DENY OUT Anywhere
<snip> DENY OUT Anywhere (log)

# ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing)
New profiles: skip

To Action From
-- ------ ----
25/udp ALLOW IN Anywhere
25/udp ALLOW IN Anywhere (v6)

<snip> DENY OUT Anywhere
<snip> DENY OUT Anywhere (log)

# ufw delete allow smtp/udp
Rule deleted
Rule deleted (v6)

# ufw status
Status: active

To Action From
-- ------ ----
<snip> DENY OUT Anywhere
<snip> DENY OUT Anywhere (log)

----
I don't know if this may be caused by some differences in environment of our distributions or how ufw is installed there, or a bug… I did the test for this evaluation mainly using ./run_t...

[Sławomir Nizio (snizio)]

The problem about 'ufw --dry-run allow smtp/udp' succeeding on my system is that, for whatever reason, there's a difference between the contents of /etc/services on Debian (at least Debian testing) and Gentoo system.
On Debian, smtp is specified with 'tcp' only. On the other one, there are both smtp entries, with 'tcp' and 'udp'. (This is a curiosity by itself!)

This makes get_services_proto() (which uses getservbyname()) from util.py behave differently for those systems. This way it allows setting the smtp/udp rule on my Gentoo based system (so the test fails), and on Debian this happens:
ERROR: Protocol mismatch with specified protocol udp

I haven't investigated the other failures yet.

At least this one is distribution dependent. I understand if you are going to close it as something like WONTFIX then, esp. if it the fix isn't too quick to make. :) Just wanted to make a note on this.

[Jamie Strandboge (jdstrand)]

Ah, this is very helpful-- thanks! There was a similar failure in the past because of Debian changing things around http and www. I will fix this in ufw.

[Sławomir Nizio (snizio)]

Just a note, the "Samba" failure above was also related to the difference ('www' on my system: udp and tcp, get_services_proto("www") returns 'any'; 'www' on Debian: tcp only, function returns there 'tcp'). There are more test failures than the two.
With "services" file from Debian, all standard tests pass on my system. :)