Comment 0 for bug 1853197

Revision history for this message
MIKE OLLIFF (mikeo-symc) wrote :

Ubuntu linux distro, 4.15.0-62 kernel, server platform.
This OS is used as an IPSec VPN gateway. It serves up to several hundred concurrent connections

In an attempt to upgrade from the 4.4 kernel to 4.15, the team noticed that VPN gateway VMs were running out of physical memory after 12-48 hours, depending on load.

Attachments from a server machine in this state in attached leakinfo.txt
output of free -t
output of /proc/meminfo in out of memory condition
output of /slabtop -o -sc
/sys/kernel/debug/page_owner sorted and aggregated after server ran for 12 hrs and ran out of memory
Patches for 4.15 and 5.4

Highlight from page_owner, we can see the leak is a buffer associated with the ipsec impelementation. Each connection leaks 32k of memory via alloc_page with order=3

Page allocated via order 3, mask 0x1085220(GFP_ATOMIC|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP)
 get_page_from_freelist+0xd64/0x1250
 __alloc_pages_nodemask+0x11c/0x2e0
 alloc_pages_current+0x6a/0xe0
 skb_page_frag_refill+0x71/0x100
 esp_output_head+0x265/0x3e0 [esp4]
 esp_output+0xbc/0x180 [esp4]
 xfrm_output_resume+0x179/0x530
 xfrm_output+0x8e/0x230
 xfrm4_output_finish+0x2b/0x30
 __xfrm4_output+0x3a/0x50
 xfrm4_output+0x43/0xc0
 ip_forward_finish+0x51/0x80
 ip_forward+0x38a/0x480
 ip_rcv_finish+0x122/0x410
 ip_rcv+0x292/0x360
 __netif_receive_skb_core+0x815/0xbd0

Patch to fix this issue in 4.15 (tested and verified on same server exhibiting above leak):
diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c
index 728272f..7842f83 100644
--- a/net/xfrm/xfrm_state.c
+++ b/net/xfrm/xfrm_state.c
@@ -451,6 +451,10 @@ static void xfrm_state_gc_destroy(struct xfrm_state *x)
        }
        xfrm_dev_state_free(x);
        security_xfrm_state_free(x);
+
+ if(x->xfrag.page)
+ put_page(x->xfrag.page);
+
        kfree(x);
}

Patch for master branch (5.4 I believe) from Paul Wouters (<email address hidden>)

diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c
index c6f3c4a1bd99..f3423562d933 100644
--- a/net/xfrm/xfrm_state.c
+++ b/net/xfrm/xfrm_state.c
@@ -495,6 +495,8 @@ static void ___xfrm_state_destroy(struct xfrm_state *x)
                                x->type->destructor(x);
                                xfrm_put_type(x->type);
                }
+ if (x->xfrag.page)
+ put_page(x->xfrag.page);
                xfrm_dev_state_free(x);
                security_xfrm_state_free(x);
                xfrm_state_free(x);

Severity: Critical - we are unable to use any kernel later than 4.11, and are sticking with 4.4 in production.