networkd crashes when advertising blackhole route thru BGP

Bug #1929122 reported by Matija Vižintin
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Fix Released
Undecided
Unassigned
Bionic
Invalid
Undecided
Unassigned
Focal
Fix Released
Medium
Dan Streetman

Bug Description

[impact]

systemd-networkd crashes due to assertion failure

[test case]

run systemd-networkd and FRR as described in original description below

[regression potential]

and regression would likely involve incorrectly configured routes

[scope]

this is needed only for f

this is fixed upstream by commit 25b831bac8a5e545e1eda5199392c11c7aed4e42 which is first in v246, so this is fixed in groovy and later

the code involved in this isn't present in b so this doesn't apply there

[original description]

lsb_release -rd
Description: Ubuntu 20.04.2 LTS
Release: 20.04

apt-cache policy systemd
systemd:
  Installed: 245.4-4ubuntu3.6

[Impact]
Networkd crashes when a blackhole route (route aggregation is installed as a blackhole route) is advertised via FRR.

[Fix]
networkd should not crash

[Test case]
Bug was reported to systemd (not accepted because systemd version too old) where a test case is described: https://github.com/systemd/systemd/issues/19648 Lower in the comment you can find the full debug log from networkd.

Here you can see the core dump: https://drive.google.com/file/d/1tCKiOSnGkLbx3w9du5_NNepkOQZC_hx7/view?usp=sharing

For convenience I also packed the relevant parts of the prod env in a vagrant box showing the issue: vagrant init matijavizintin/networkd-debug

After some more testing i noticed that immediately after the route is being advertised, networkd detects the change and crashes with "Assertion 'ifindex' failed at src/network/networkd-link.c:757, function link_get(). Aborting."

Steps how it happens:
1. networkd is running
2. frr is started without advertising the aggregated address
3. add aggregated address and reload frr
4. blackhole route is installed and networkd crashes with the error above

Revision history for this message
Dan Streetman (ddstreet) wrote :

Have you tested with the latest upstream code yet to see if the bug still exists there?

You can use the daily builds from this ppa if that helps:
https://code.launchpad.net/~ubuntu-support-team/+archive/ubuntu/systemd

Revision history for this message
Dan Streetman (ddstreet) wrote :

> You can use the daily builds from this ppa

note, please use those builds for *testing* only

Revision history for this message
Dan Streetman (ddstreet) wrote :

never mind, i believe it's fixed with an upstream commit, and a fixed package is building in this ppa:
https://launchpad.net/~ddstreet/+archive/ubuntu/systemd

i'll upload patched systemd later this week; let me know if that doesn't fix this for you

description: updated
Changed in systemd (Ubuntu Bionic):
status: New → Invalid
Changed in systemd (Ubuntu):
status: New → Fix Released
Changed in systemd (Ubuntu Focal):
importance: Undecided → Medium
assignee: nobody → Dan Streetman (ddstreet)
status: New → In Progress
Revision history for this message
Matija Vižintin (matija-vizintin) wrote :

Yes, that's it! Thanks for looking into it and fixing it.

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Matija, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-focal
Revision history for this message
Matija Vižintin (matija-vizintin) wrote :

Hey Brian,

Works like a charm!

root@routing-focal-l3:~# uname -a
Linux routing-focal-l3 5.4.0-52-generic #57-Ubuntu SMP Thu Oct 15 10:57:00 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

root@routing-focal-l3:~# systemd --version
systemd 245 (245.4-4ubuntu3.7)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

The test performed was the one described in the bug report.
Test case:
- add "aggregate-address X.X.X.X/X" to frr
- restart frr and blackhole route is installed

After the blackhole route was installed networkd was still working fine which confirms the bug was fixed.

Thanks!

BR,
Matija

Dan Streetman (ddstreet)
tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Revision history for this message
Matija Vižintin (matija-vizintin) wrote :

Hey Dan, Brian,

When is this patch (245.4-4ubuntu3.7) going to be released to -updates?

Thanks,
Matija

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 245.4-4ubuntu3.7

---------------
systemd (245.4-4ubuntu3.7) focal; urgency=medium

  [ Andy Chi ]
  * debian/patches/lp1926547-hwdb-60-keyboard-Update-Dell-Privacy-Local-Mic-Mute-.patch
    - Apply upstream patch to correct key and device mapping.
      (LP: #1926547)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=62c3ce6d6b2cab762b24aa610d6d135a67bdd76a

  [ Dan Streetman ]
  * d/p/lp1921696/0001-rfkill-improve-error-logging.patch,
    d/p/lp1921696/0002-rfkill-use-short-writes-and-accept-long-reads.patch:
    Handle rfkill api change in kernel 5.10 (LP: #1921696)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=ff0c23ba4fbcfa7f68e98adb6d62798ce54ca1da
  * d/p/lp1929122-network-check-that-received-ifindex-is-valid.patch:
    Check if ifindex is valid (LP: #1929122)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=6378191818bc7d169b657e6f7a2b50cfddb4275e
  * d/p/lp1929560-network-move-set-MAC-and-set-nomaster-operations-out.patch:
    Move link mac and master config out of link_up() (LP: #1929560)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=28cff7ee02a9ebd4ab93026af9fceaa2283725b3
  * d/p/lp1902891-core-mount-mount-command-may-fail-after-adding-the-c.patch:
    Handle failed mount command (LP: #1902891)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b425189a483d7455db870b0ec5b2443c0eea7d76
  * d/p/resolved-Mitigate-DVE-2018-0001-by-retrying-NXDOMAIN-with.patch,
    d/p/lp1880258-log-nxdomain-as-debug.patch,
    d/p/lp1785383-resolved-address-DVE-2018-0001.patch:
    - Use upstream patch for DVE-2018-0001 handling (LP: #1785383)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=ec45ebfee362ad3e429642f7519e8b88f16dc221

  [ Łukasz 'sil2100' Zemczak ]
  * d/p/lp1664844/0001-network-add-ActivationPolicy-configuration-parameter.patch,
    d/p/lp1664844/0002-test-add-ActivationPolicy-unit-tests.patch,
    d/p/lp1664844/0003-save-link-activation-policy-to-state-file-and-displa.patch:
    - add support for configuring the activation policy for an interface
      (LP: #1664844)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=94f7b72d8128c743f35b308101a87d2c53a4074c

 -- Dan Streetman <email address hidden> Thu, 27 May 2021 11:16:17 -0400

Changed in systemd (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.