systemd-resolve crashes fairly often (and reports various assertions)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
systemd (Ubuntu) |
Fix Released
|
Medium
|
Unassigned | ||
Focal |
Incomplete
|
Medium
|
Unassigned | ||
Groovy |
Won't Fix
|
Medium
|
Unassigned | ||
Hirsute |
Fix Released
|
Medium
|
Unassigned |
Bug Description
[impact]
systemd-resolved crashes
[test case]
see original description; I can't reproduce so I'm relying on the reporter(s) to test/verify.
[regression potential]
any regression would likely occur while processing sd_event objects, which are used throughout systemd code; this could result in crashes in almost any part of systemd code. However a more likely regression would be leaks of sd_event objects due to failure to release the final ref for an object.
[scope]
This is needed for f/g/h
This might be fixed by upstream commit f814c871e65df85
[other info]
I believe this is caused by a freed sd_event object that is then processed and calls the on_query_timeout callback with invalid state, leading to failed assertion, which causes resolved to crash; that's what analysis of the crash dump appears to indicate. This may be fixed by the upstream commit referenced in [scope], which takes additional refs during function calls. However I haven't reproduced this myself, so I'm only guessing as to the cause and solution at this point.
I'm unsure why this would not occur in bionic, but per comment 5 it seems it doesn't happen in that release.
[original description]
(Tested on regularly updated Ubuntu 20.04, currently i use systemd 245.4-4ubuntu3.2)
I observe fairly lot of segfaults of systemd-resolve. Frequency vary but … see below.
I have no clue what is the reason. Specific feature of my machine is that apart from normal cable connection (to OpenWRT router) I use OpenVPN for business network (and this submits specific nameserver for myorg.local domain).
~~~~~
$ LC_ALL=C dmesg -T --level=info | grep systemd-resolve
[Sun Nov 29 11:47:37 2020] systemd-
[Sun Nov 29 11:57:27 2020] systemd-
[Sun Nov 29 12:07:37 2020] systemd-
[Sun Nov 29 13:12:26 2020] systemd-
[Sun Nov 29 13:32:57 2020] systemd-
[Sun Nov 29 13:42:37 2020] systemd-
[Sun Nov 29 15:42:26 2020] systemd-
[Sun Nov 29 16:02:36 2020] systemd-
[Sun Nov 29 17:42:35 2020] systemd-
[Sun Nov 29 17:52:35 2020] systemd-
[Sun Nov 29 19:17:55 2020] systemd-
[Sun Nov 29 19:32:35 2020] systemd-
[Sun Nov 29 20:03:54 2020] systemd-
[Sun Nov 29 20:22:24 2020] systemd-
[Sun Nov 29 21:13:10 2020] systemd-
[Sun Nov 29 21:32:34 2020] systemd-
[Sun Nov 29 22:03:20 2020] systemd-
[Sun Nov 29 22:32:44 2020] systemd-
[Sun Nov 29 23:12:23 2020] systemd-
[Sun Nov 29 23:22:34 2020] systemd-
[Mon Nov 30 00:12:23 2020] systemd-
[Mon Nov 30 00:47:54 2020] systemd-
[Mon Nov 30 01:57:53 2020] systemd-
[Mon Nov 30 02:58:08 2020] traps: systemd-
[Mon Nov 30 03:38:08 2020] systemd-
[Mon Nov 30 05:07:22 2020] systemd-
[Mon Nov 30 05:47:43 2020] systemd-
[Mon Nov 30 06:17:42 2020] systemd-
[Mon Nov 30 06:27:21 2020] systemd-
[Mon Nov 30 07:12:21 2020] systemd-
[Mon Nov 30 08:17:21 2020] traps: systemd-
[Mon Nov 30 08:27:31 2020] systemd-
[Mon Nov 30 08:58:25 2020] systemd-
[Mon Nov 30 10:02:20 2020] systemd-
[Mon Nov 30 10:12:20 2020] systemd-
[Mon Nov 30 11:17:30 2020] systemd-
[Mon Nov 30 12:58:18 2020] systemd-
[Mon Nov 30 14:25:41 2020] systemd-
[Mon Nov 30 14:52:39 2020] systemd-
[Mon Nov 30 15:02:29 2020] systemd-
[Mon Nov 30 15:14:18 2020] systemd-
[Mon Nov 30 15:42:18 2020] systemd-
[Mon Nov 30 16:02:39 2020] systemd-
[Mon Nov 30 16:22:28 2020] systemd-
[Mon Nov 30 16:37:18 2020] systemd-
[Mon Nov 30 17:43:02 2020] traps: systemd-
[Mon Nov 30 18:12:48 2020] systemd-
[Mon Nov 30 18:37:38 2020] systemd-
[Mon Nov 30 18:48:02 2020] systemd-
[Mon Nov 30 19:32:28 2020] systemd-
[Mon Nov 30 19:53:01 2020] systemd-
[Mon Nov 30 19:59:17 2020] systemd-
[Mon Nov 30 20:27:17 2020] systemd-
[Mon Nov 30 22:42:37 2020] systemd-
[Mon Nov 30 22:57:36 2020] systemd-
[Tue Dec 1 00:22:16 2020] traps: systemd-
[Tue Dec 1 00:57:16 2020] systemd-
~~~~~
I also noted various interesting assertions in journal, in next comment…
Changed in systemd (Ubuntu): | |
status: | Incomplete → In Progress |
description: | updated |
Changed in systemd (Ubuntu Groovy): | |
status: | New → In Progress |
Changed in systemd (Ubuntu Focal): | |
status: | New → In Progress |
Changed in systemd (Ubuntu Groovy): | |
importance: | Undecided → Medium |
Changed in systemd (Ubuntu Hirsute): | |
importance: | Undecided → Medium |
Changed in systemd (Ubuntu Focal): | |
importance: | Undecided → Medium |
Changed in systemd (Ubuntu Hirsute): | |
assignee: | nobody → Dan Streetman (ddstreet) |
Changed in systemd (Ubuntu Focal): | |
assignee: | nobody → Dan Streetman (ddstreet) |
Changed in systemd (Ubuntu Groovy): | |
assignee: | nobody → Dan Streetman (ddstreet) |
description: | updated |
Changed in systemd (Ubuntu Groovy): | |
assignee: | Dan Streetman (ddstreet) → nobody |
Changed in systemd (Ubuntu Focal): | |
assignee: | Dan Streetman (ddstreet) → nobody |
Changed in systemd (Ubuntu Groovy): | |
status: | In Progress → Incomplete |
Changed in systemd (Ubuntu Focal): | |
status: | In Progress → Incomplete |
Changed in systemd (Ubuntu Groovy): | |
status: | Incomplete → Won't Fix |
Changed in systemd (Ubuntu): | |
assignee: | Dan Streetman (ddstreet) → nobody |
Changed in systemd (Ubuntu Hirsute): | |
assignee: | Dan Streetman (ddstreet) → nobody |
systemd-resolve journal is fairly full of failed assertions. During last 3 days I got 301 of them.
This one is very frequent (I got it 294 times) resolved[ 1590676] : Assertion 'DNS_TRANSACTIO N_IS_LIVE( q->state) ' failed at src/resolve/ resolved- dns-query. c:520, function dns_query_ complete( ). Aborting.
~~~~~
Nov 28 21:10:02 platon systemd-
~~~~~
The rest: resolved[ 1542718] : Assertion '*_head == _item' failed at src/resolve/ resolved- dns-query. c:397, function dns_query_free(). Aborting. resolved[ 1546490] : Assertion 'p->n_ref > 0' failed at src/resolve/ resolved- dns-question. c:33, function dns_question_ unref() . Aborting. resolved[ 1572471] : Assertion '*_head == _item' failed at src/resolve/ resolved- dns-query. c:397, function dns_query_free(). Aborting. resolved[ 1587570] : Assertion '*_head == _item' failed at src/resolve/ resolved- dns-query. c:372, function dns_query_free(). Aborting. resolved[ 1651715] : Assertion 'p->n_ref > 0' failed at src/libsystemd/ sd-event/ sd-event. c:1912, function sd_event_ source_ unref() . Aborting. resolved[ 1710789] : Assertion '*_head == _item' failed at src/resolve/ resolved- dns-query. c:397, function dns_query_free(). Aborting. resolved[ 1726691] : Assertion 'p->n_ref > 0' failed at src/libsystemd/ sd-event/ sd-event. c:1912, function sd_event_ source_ unref() . Aborting.
~~~~~
Nov 28 03:35:02 platon systemd-
Nov 28 04:20:02 platon systemd-
Nov 28 14:20:23 platon systemd-
Nov 28 19:34:52 platon systemd-
Nov 29 18:59:52 platon systemd-
Nov 30 14:35:23 platon systemd-
Nov 30 18:29:52 platon systemd-
~~~~~