systemd-networkd - RoutingPolicyRule does not apply correctly

[Ioanna Alifieraki]

[Impact]

When attaching multiple IPs on secondary interface the routing policy rules do not
apply for all IPs.
The rules that are applied after restarting networkd are random.

The issue has been reported and addressed upstream.

[Test Case]

Attach multiple IPs to secondary interface, config file:

$ cat /etc/systemd/network/ens8.network
[Match]
Name=ens8

[Network]
Address=172.31.28.195/20
Address=172.31.24.151/20
Address=172.31.24.152/20
Address=172.31.24.153/20

[Route]
Destination=0.0.0.0/0
Gateway=172.31.16.1
Table=1000

[Route]
Destination=172.31.24.153
Gateway=0.0.0.0
Scope=link
Table=1000

[Route]
Destination=172.31.28.195
Gateway=0.0.0.0
Scope=link
Table=1000

[Route]
Destination=172.31.24.151
Gateway=0.0.0.0
Scope=link
Table=1000

[Route]
Destination=172.31.24.152
Gateway=0.0.0.0
Scope=link
Table=1000

[RoutingPolicyRule]
From=172.31.24.153
Table=1000

[RoutingPolicyRule]
From=172.31.28.195
Table=1000

[RoutingPolicyRule]
From=172.31.24.152
Table=1000

[RoutingPolicyRule]
From=172.31.24.151
Table=1000

$ ip rule show
0: from all lookup local
0: from 172.31.24.151 lookup 1000
32766: from all lookup main
32767: from all lookup default

- restart networkd

$ sudo systemctl restart systemd-networkd

$ ip rule show
0: from all lookup local
0: from 172.31.24.151 lookup 1000
0: from 172.31.24.153 lookup 1000
0: from 172.31.28.195 lookup 1000
32766: from all lookup main
32767: from all lookup default

- restart again

$ sudo systemctl restart systemd-networkd

$ ip rule show
0: from all lookup local
0: from 172.31.24.151 lookup 1000
0: from 172.31.24.152 lookup 1000
32766: from all lookup main
32767: from all lookup default

The behaviour is random

[Regression Potential]

May impact the systemd-networkd service functionality.

[Other]

Affects Disco Bionic and Cosmic.

Upstream bug : https://github.com/systemd/systemd/issues/11280
Upstream fix : https://github.com/systemd/systemd/pull/11795/commits

$ lsb_release -rd
Description: Ubuntu Disco Dingo (development branch)
Release: 19.04

Systemd version :
systemd:
  Installed: 240-6ubuntu1
  Candidate: 240-6ubuntu1
  Version table:
 *** 240-6ubuntu1 500
        500 http://gb.archive.ubuntu.com/ubuntu disco/main amd64 Packages
        100 /var/lib/dpkg/status

[Ioanna Alifieraki (joalif)]

debdiff for Disco

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "lp1818282_disco.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Eric Desrochers (slashd)]

Sponsored for Disco, considering that after a discussion with joalif, she committed to file a debian bug and submit the patchset to debian as well for stay consistent in the chain Upstream -> Debian -> Ubuntu.

- Eric

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 240-6ubuntu2

---------------
systemd (240-6ubuntu2) disco; urgency=medium

  * d/p/network-remove-routing-policy-rule-from-foreign-rule.patch
  * d/p/network-do-not-remove-rule-when-it-is-requested-by-e.patch
    - Fix RoutingPolicyRule does not apply correctly (LP: #1818282)

 -- Ioanna Alifieraki <email address hidden> Mon, 04 Mar 2019 10:32:19 +0000

[Ioanna Alifieraki (joalif)]

Debdiff for cosmic.

[Ioanna Alifieraki (joalif)]

Debdiff for Bionic.

[Łukasz Zemczak (sil2100)]

Hello Ioanna, or anyone else affected,

Accepted systemd into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/239-7ubuntu10.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Ioanna, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.20 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ioanna Alifieraki (joalif)]

VERIFICATION Bionic

- Using configuration file from description

$ dpkg -l | grep systemd
ii systemd 237-3ubuntu10.19 amd64 system and service manager

$ ip rule
0: from all lookup local
0: from 172.31.24.153 lookup 1000
0: from 172.31.28.195 lookup 1000
0: from 172.31.24.151 lookup 1000
32766: from all lookup main
32767: from all lookup default

-- Install systemd from -proposed

$ dpkg -l | grep systemd
ii systemd 237-3ubuntu10.20 amd64 system and service manager

$ ip rule
0: from all lookup local
0: from 172.31.24.153 lookup 1000
0: from 172.31.28.195 lookup 1000
0: from 172.31.24.151 lookup 1000
0: from 172.31.24.152 lookup 1000
32766: from all lookup main
32767: from all lookup default

$ sudo systemctl restart systemd-networkd

$ ip rule
0: from all lookup local
0: from 172.31.24.153 lookup 1000
0: from 172.31.28.195 lookup 1000
0: from 172.31.24.152 lookup 1000
0: from 172.31.24.151 lookup 1000
32766: from all lookup main
32767: from all lookup default

[Ioanna Alifieraki (joalif)]

VERIFICATION Cosmic

- Using configuration file from description

$ dpkg -l | grep systemd
ii systemd 239-7ubuntu10.12 amd64 system and service manager

$ ip rule
0: from all lookup local
0: from 172.31.24.153 lookup 1000
0: from 172.31.28.195 lookup 1000
32766: from all lookup main
32767: from all lookup default

-- Install systemd from -proposed

$ dpkg -l | grep systemd
ii systemd 239-7ubuntu10.13 amd64 system and service manager

$ ip rule
0: from all lookup local
0: from 172.31.24.153 lookup 1000
0: from 172.31.24.152 lookup 1000
0: from 172.31.28.195 lookup 1000
0: from 172.31.24.151 lookup 1000
32766: from all lookup main
32767: from all lookup default

$ sudo systemctl restart systemd-networkd

$ ip rule
0: from all lookup local
0: from 172.31.24.153 lookup 1000
0: from 172.31.24.152 lookup 1000
0: from 172.31.28.195 lookup 1000
0: from 172.31.24.151 lookup 1000
32766: from all lookup main
32767: from all lookup default

[Dan Streetman (ddstreet)]

autopkgtest regressions, bionic:

re-running:

apt (i386) : re-running; test seems flaky (random failures in previous runs)
dovecot (arm64) : re-running, test seems flaky (same failure in previous runs)

ignore:

systemd (ppc64el) : existing failure bug 1821625
linux (various) : tests very flaky, fail more often than pass
snapd (various) : test seems to install lots from github,
                  no idea what it's unhappy about;
                  started failing before our package was uploaded
mariadb (various) : always failed.
linux-aws-edge (amd64) : always failed.

[Dan Streetman (ddstreet)]

autopkgtest regressions, cosmic:

re-running:

systemd (various) : some of the archs fail frequently, but i'll try re-running again

ignore:

linux (various) : tests very flaky, fail more often than pass
docker.io (various) : snapd appears to have broken this test
snapd (various) : snapd appears to have also broken this test
(snapd in bionic is also likely to be failing due to snapd problems)

[Dan Streetman (ddstreet)]

autopkgtest regressions:

bionic:

apt passed on re-run

dovecot (armhf) failed again; i'm re-running it again, but i think we can ignore it as it failed in the same way for the openssl upload before we uploaded this.

snapd i opened bug 1824237, although that might be limited to only cosmic, not bionic

all other bionic tests ignorable as explained in previous comments

cosmic:

snapd as with bionic, bug 1824237
docker.io tests fail also due to bug 1824237 (because they use snaps in their tests, and the core snap is broken)

systemd test are failing, i think, also because of bug 1824237. since the problem is the cosmic 'core' snap is broken, all other snaps fail, and the failing test for all systemd archs is 'boot-and-services' which checks that no services failed. it appears some of the services, like lxd, because they could not start due to broken core snap; i'm testing locally to verify this.

all other cosmic tests ignorable as explained in previous comments

[Dan Streetman (ddstreet)]

autopkgtest regressions:

bionic:

dovecot (armhf) re-run passed, hurray!

this leaves only test failures that i think we can ignore:
systemd (ppc64el): bug 1821625
linux (various): very flaky tests
snapd (various) might be bug 1824237, in any case they are all very flaky
mariadb-10.1 (various): bug 1824335
linux-aws-edge (amd64): bug 1723223

cosmic:

also only ignorable test failures:
linux (various): very flaky tests
docker.io (various): bug 1824237
systemd (various): bug 1824237
snapd (various): definitely bug 1824237

so i believe all remaining autopkgtest regressions are ignorable for this upload.

[Łukasz Zemczak (sil2100)]

Thanks for the autopkgtest check! Looks good. For cosmic (and partially bionic too), I will not be hinting the test failures that we think LP: #182423 is responsible for. Since apparently those should go away with the new snapd - if not, we'll at least know what's up.

Few others I will hint before releasing the package.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 239-7ubuntu10.13

---------------
systemd (239-7ubuntu10.13) cosmic; urgency=medium

  [ Ioanna Alifieraki ]
  * d/p/backport_network-fix-return-value-of-routing_policy_rule_get.patch,
    d/p/backport_network-remove-routing-policy-rule-from-foreign.patch,
    d/p/backport_network-do-not-remove-rule-when-requested-by-existing-links.patch:
      - Fix RoutingPolicyRule does not apply correctly (LP: #1818282)

  [ Dan Streetman ]
  * d/p/networkd-Track-address-configuration.patch,
    d/p/networkd-Use-only-a-generic-CONFIGURING-state.patch,
    d/p/networkd-don-t-remove-route.patch,
    d/p/networkd-don-t-remove-ip-address.patch,
    d/p/Move-link_check_ready-to-later-in-the-file.patch,
    d/p/network-set-_configured-flags-to-false-before-reques.patch,
    d/p/Install-routes-after-addresses-are-ready.patch:
      - PreferredSource not working in *.network files (LP: #1812760)
  * Specify Ubuntu's Vcs-Git

 -- Dan Streetman <email address hidden> Thu, 04 Apr 2019 07:29:38 -0400

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 237-3ubuntu10.20

---------------
systemd (237-3ubuntu10.20) bionic; urgency=medium

  [ Ioanna Alifieraki ]
  * d/p/backport_network-fix-return-value-of-routing_policy_rule_get.patch,
    d/p/backport_network-remove-routing-policy-rule-from-foreign.patch,
    d/p/backport_network-do-not-remove-rule-when-requested-by-existing-links.patch:
    - Fix RoutingPolicyRule does not apply correctly (LP: #1818282)

  [ Dan Streetman ]
  * d/p/fix-test-22.patch
    - fix TEST-22 failures
  * d/p/networkd-Track-address-configuration.patch,
    d/p/networkd-Use-only-a-generic-CONFIGURING-state.patch,
    d/p/networkd-don-t-remove-route.patch,
    d/p/networkd-don-t-remove-ip-address.patch,
    d/p/Move-link_check_ready-to-later-in-the-file.patch,
    d/p/network-set-_configured-flags-to-false-before-reques.patch,
    d/p/Install-routes-after-addresses-are-ready.patch:
    - PreferredSource not working in *.network files (LP: #1812760)

  [ Dimitri John Ledkov ]
  * Specify Ubuntu's Vcs-Git

 -- Dan Streetman <email address hidden> Thu, 04 Apr 2019 07:29:38 -0400