test_network_policies fails on "Reaching out to nginx.netpolicy with restrictions"

[Michael Skalka]

As seen during this test run: https://solutions.qa.canonical.com/testruns/testRun/74ac7049-5b1e-4ced-94e1-b01314b5dd3b
Artifacts here: https://oil-jenkins.canonical.com/artifacts/74ac7049-5b1e-4ced-94e1-b01314b5dd3b/generated/generated/kubernetes/juju-crashdump-kubernetes-2021-01-22-10.38.07.tar.gz

test_network_policies apparently fails to reach out to nginx, however there are no errors in the pod log from the test.

From the crashdump:
less kubernetes-worker_1/pod-logs/netpolicy-nginx-deployment-6bd4c896d6-vqrcn-nginx.log

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
192.168.231.197 - - [22/Jan/2021:10:26:38 +0000] "GET / HTTP/1.1" 200 612 "-" "Wget" "-"
192.168.164.4 - - [22/Jan/2021:10:26:50 +0000] "GET / HTTP/1.1" 200 612 "-" "Wget" "-"
...
192.168.164.4 - - [22/Jan/2021:10:36:52 +0000] "GET / HTTP/1.1" 200 612 "-" "Wget" "-"
192.168.231.197 - - [22/Jan/2021:10:36:58 +0000] "GET / HTTP/1.1" 200 612 "-" "Wget" "-"
192.168.164.4 - - [22/Jan/2021:10:37:09 +0000] "GET / HTTP/1.1" 200 612 "-" "Wget" "-"
192.168.231.197 - - [22/Jan/2021:10:37:15 +0000] "GET / HTTP/1.1" 200 612 "-" "Wget" "-"
EOF

[George Kraft (cynerva)]

The test is failing because Calico isn't properly enforcing network policies. The expectation at this point in the test is that requests from bboxbad (192.168.231.197) should *not* reach nginx, because a NetworkPolicy exists that should deny it.

Calico isn't enforcing network policies because felix is crashing:

2021-01-22 10:39:37.801 [FATAL][35660] int_dataplane.go 1032: Kernel's RPF check is set to 'loose'. This would allow endpoints to spoof their IP address. Calico requires net.ipv4.conf.all.rp_filter to be set to 0 or 1. If you require loose RPF and you are not concerned about spoofing, this check can be disabled by setting the IgnoreLooseRPF configuration parameter to 'true'.

You need to either configure the calico charm with ignore-loose-rpf=true, or update the kubernetes-master and kubernetes-worker sysctl charm config to include setting net.ipv4.conf.all.rp_filter to 0 or 1.