NetworkManager crashed with SIGSEGV in _int_malloc()

[Anders Kaseorg]

I’m seeing NetworkManager crash several times on startup and on resume from suspend. Apport doesn’t catch those for some reason, but this happened when I tried to reproduce by manually running NetworkManager --debug.

(Different crash from bug 1792743? Also apport-retrace didn’t seem to like that one.)

ProblemType: Crash
DistroRelease: Ubuntu 18.10
Package: network-manager 1.12.2-0ubuntu4
ProcVersionSignature: Ubuntu 4.18.0-7.8-generic 4.18.5
Uname: Linux 4.18.0-7-generic x86_64
NonfreeKernelModules: openafs
ApportVersion: 2.20.10-0ubuntu9
Architecture: amd64
Date: Sat Sep 15 14:04:50 2018
EcryptfsInUse: Yes
ExecutablePath: /usr/sbin/NetworkManager
IfupdownConfig: # ifupdown has been replaced by netplan(5) on this system. Do not edit.
InstallationDate: Installed on 2016-02-19 (939 days ago)
InstallationMedia: Ubuntu-GNOME 16.04 LTS "Xenial Xerus" - Alpha amd64 (20160218)
NetworkManager.state:
 [main]
 NetworkingEnabled=true
 WirelessEnabled=true
 WWANEnabled=false
ProcCmdline: NetworkManager --debug
SegvAnalysis:
 Segfault happened at: 0x7f79966ea378 <_int_malloc+3352>: cmp %rcx,0x18(%rdx)
 PC (0x7f79966ea378) ok
 source "%rcx" ok
 destination "0x18(%rdx)" (0x00000018) not located in a known VMA region (needed writable region)!
SegvReason: writing NULL VMA
Signal: 11
SourcePackage: network-manager
StacktraceTop:
 _int_malloc (av=av@entry=0x7f799683ac40 <main_arena>, bytes=bytes@entry=4096) at malloc.c:4014
 __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at malloc.c:3420
 g_malloc0 () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
 nlmsg_alloc_size (len=4096) at src/platform/nm-netlink.c:314
 _nl80211_alloc_msg (id=28, ifindex=2, phy=0, cmd=cmd@entry=32, flags=flags@entry=768) at src/platform/wifi/wifi-utils-nl80211.c:88
Title: NetworkManager crashed with SIGSEGV in _int_malloc()
UpgradeStatus: Upgraded to cosmic on 2018-08-17 (29 days ago)
UserGroups:

nmcli-con: Error: command ['nmcli', '-f', 'all', 'con'] failed with exit code 8: Error: NetworkManager is not running.
nmcli-dev: Error: command ['nmcli', '-f', 'all', 'dev'] failed with exit code 8: Error: NetworkManager is not running.
nmcli-nm: Error: command ['nmcli', '-f', 'all', 'gen'] failed with exit code 8: Error: NetworkManager is not running.

[Apport retracing service (apport)]

StacktraceTop:
 _int_malloc (av=av@entry=0x7f799683ac40 <main_arena>, bytes=bytes@entry=4096) at malloc.c:4014
 __libc_calloc (n=n@entry=1, elem_size=elem_size@entry=4096) at malloc.c:3420
 g_malloc0 (n_bytes=n_bytes@entry=4096) at ../../../../glib/gmem.c:129
 nlmsg_alloc_size (len=4096) at src/platform/nm-netlink.c:314
 _nl80211_alloc_msg (id=28, ifindex=2, phy=0, cmd=cmd@entry=32, flags=flags@entry=768) at src/platform/wifi/wifi-utils-nl80211.c:88

[Anders Kaseorg (andersk)]

Here’s a valgrlind log showing many invalid writes to freed addresses in cb_data_free → curl_multi_remove_handle and nm_connectivity_check_start → curl_multi_add_handle, which could explain the heap corruption.

[Anders Kaseorg (andersk)]

It seems suspicious that cb_data->concheck.curl_ehandle is cleaned up here despite the comment a few lines above warning that it’s unsafe to do so.

https://github.com/NetworkManager/NetworkManager/blob/1.12.2/src/nm-connectivity.c#L190-L213

  /* Contrary to what cURL manual claim it is *not* safe to remove
   * the easy handle "at any moment"; specifically not from the
   * write function. Thus here we just dissociate the cb_data from
   * the easy handle and the easy handle will be cleaned up when the
   * message goes to CURLMSG_DONE in _con_curl_check_connectivity(). */
…
  curl_multi_remove_handle (priv->concheck.curl_mhandle, cb_data->concheck.curl_ehandle);
  curl_easy_cleanup (cb_data->concheck.curl_ehandle);

This cleanup call was added in https://github.com/NetworkManager/NetworkManager/pull/70. The comment was moved by that PR but was present before.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Anders Kaseorg (andersk)]

(Still waiting on sponsorship, but apparently “In Progress” is the right status for that.)

[Anders Kaseorg (andersk)]

1.12.4 is in cosmic-proposed now with the upstream fix.