Network Manager will not remove Netplan YAMLs when connections are deleted

[Danilo Egea Gondolfo]

[ Impact ]

Desktop users, or any users with YAML files in /usr/lib/netplan, can't delete
Network Manager connections persistently. That means that, when the connection is
deliberately deleted by the user, it will re-appear when the system is rebooted or
netplan apply is executed.

This is happening because the systemd service unit is setting the property "ProtectSystem"
to true. Because of that, /usr is being presented to the Network Manager daemon as read-only.
When connections are deleted, libnetplan will try to open its YAML files with writing permissions
and will fail for files from /usr/lib/netplan. Even if the user hasn't added any files there manually,
the file /usr/lib/netplan/00-network-manager-all.yaml will be installed by the package ubuntu-settings.

This issue is fixed by allow-listing /usr/lib/netplan with ReadWritePaths=/usr/lib/netplan in systemd
so the Network Manager's daemon will be able to write to that directory.

This upload also improves the autopkgtests related to Netplan. Network Manager will be
started by systemd, which ensures we are testing in the same environment conditions
used by a desktop installation. It also adds a few more instances of connections deletions so
we can test a bit more that YAML files are being removed. It also adds all the dependencies
required by the test script (which sadly was causing the nm_netplan.py tests to be skipped).

[ Test Plan ]

Launch a new Mantic VM:

$ lxc launch ubuntu:mantic --vm

Install network-manager and ubuntu-settings:

# apt install network-manager ubuntu-settings

Run Netplan

# netplan apply

Create a dummy connection via nmcli:

# nmcli con add type dummy connection.interface-name dummy0

Check a new YAML will be created in /etc/netplan

Delete the connection with nmcli

# nmcli con del dummy-dummy0

Check the YAML WAS NOT removed from /etc/netplan

You will see the error below in the NetworkManager's journal

netplan_delete_connection: Cannot write output state: Read-only file system

Add the PPA containing the fix and run the same test described above

# add-apt-repository ppa:danilogondolfo/network-manager
# apt update
# apt upgrade

Check that the YAML will be created when the connection is added and deleted and the connection is removed.

[ Where problems could occur ]

As the only change is a relaxation of the restrictions applied by systemd on the environment where Network Manager
runs, we are not expecting any regression.

As for the changes in the autopkgtest related to Netplan, they are passing on all architectures.

Autopkgtests

amd64 - https://autopkgtest.ubuntu.com/results/autopkgtest-mantic-danilogondolfo-network-manager/mantic/amd64/n/network-manager/20231023_175203_b2798@/log.gz
ppc64 - https://autopkgtest.ubuntu.com/results/autopkgtest-mantic-danilogondolfo-network-manager/mantic/ppc64el/n/network-manager/20231023_182332_f0497@/log.gz
s390x - https://autopkgtest.ubuntu.com/results/autopkgtest-mantic-danilogondolfo-network-manager/mantic/s390x/n/network-manager/20231023_190810_ced8d@/log.gz
arm64 - https://autopkgtest.ubuntu.com/results/autopkgtest-mantic-danilogondolfo-network-manager/mantic/arm64/n/network-manager/20231024_084542_ac017@/log.gz
armhf - https://autopkgtest.ubuntu.com/results/autopkgtest-mantic-danilogondolfo-network-manager/mantic/armhf/n/network-manager/20231024_083545_ac017@/log.gz

[ Other Info ]

--- Original description ---

When a connection is deleted using any NM facility, libnetplan is failing to delete the YAML file. Because of that, the connection will be recreated when "netplan generate" runs again.

This is probably being caused by a combination of two things. First, the NM's systemd unit has this setting "ProtectSystem=true", which will mount /usr as read-only for NM. Second, we migrated the default "00-network-manager-all.yaml" file to, /usr/lib/netplan recently [1]. When libnetplan tries to open this file for writing, the open system fails with EROFS:

---
22517 openat(AT_FDCWD, "/lib/netplan/00-network-manager-all.yaml", O_WRONLY|O_CREAT|O_TRUNC, 0600) = -1 EROFS (Read-only file system)
22517 write(2, "netplan_delete_connection: Canno"..., 76) = 76
---

[1] - https://launchpad.net/ubuntu/+source/ubuntu-settings/23.10.1

[陈金平 (virtual163)]

2023-10-24T17:37:01.668964+08:00 MrChen-T14 NetworkManager[780]: <info> [1698140221.6688] keyfile: deleting netplan connection: NM-10cb8fd3-2680-4977-b422-1edfe625344f
2023-10-24T17:37:01.669081+08:00 MrChen-T14 NetworkManager[780]: Permissions for /etc/netplan/00-installer-config.yaml are too open. Netplan configuration should NOT be accessible by others.
2023-10-24T17:37:01.669839+08:00 MrChen-T14 NetworkManager[780]: Permissions for /etc/netplan/00-installer-config.yaml are too open. Netplan configuration should NOT be accessible by others.
2023-10-24T17:37:01.671549+08:00 MrChen-T14 NetworkManager[780]: netplan_delete_connection: Cannot write output state: Read-only file system
2023-10-24T17:37:01.770621+08:00 MrChen-T14 NetworkManager[780]: <info> [1698140221.7705] audit: op="connection-delete" uuid="10cb8fd3-2680-4977-b422-1edfe625344f" name="新元申直播" pid=37966 uid=1000 result="success"

[Lukas Märdian (slyon)]

I staged the changes for 'noble' in the 'ubuntu/master' branch:
https://git.launchpad.net/network-manager/log/

And uploaded the contents of the 'ubuntu-mantic' branch as an SRU:
https://git.launchpad.net/network-manager/log/?h=ubuntu-mantic
https://launchpad.net/ubuntu/mantic/+queue?queue_state=1

[Brian Murray (brian-murray)]

Hello Danilo, or anyone else affected,

Accepted network-manager into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/network-manager/1.44.2-1ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted network-manager (1.44.2-1ubuntu1.2) for mantic have finished running.
The following regressions have been reported in tests triggered by the package:

network-manager/1.44.2-1ubuntu1.2 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/mantic/update_excuses.html#network-manager

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Danilo Egea Gondolfo (danilogondolfo)]

The autopkgtest failures on arm64 are not related to these changes and it's happening for a while now. So they are not regressions.

I'm running network-manager from proposed on my main machine and the problem is resolved.

I was running 1.44.2-1ubuntu1.1 and upgraded to 1.44.2-1ubuntu1.2:

2023-10-25 09:52:16 upgrade network-manager:amd64 1.44.2-1ubuntu1.1 1.44.2-1ubuntu1.2

I tested it by simply removing a bunch of Network Manager connections from my system and checking all the YAML files were also removed.

It can be easily verified with a desktop VM in LXD:
 - Add a connection using the Network Manager GUI
- Remove the connection you just added
- Run netplan apply
- Check Network Manager again and observe the connection is back
- You'll see the following error in NetworkManager's journal: netplan_delete_connection: Cannot write output state: Read-only file system
- Upgrade network-manager from -proposed and run the same steps. All the deleted connection should result in deleted YAML files.

[Lukas Märdian (slyon)]

I can confirm this fixes the issue for me, too, using network-manager 1.44.2-1ubuntu1.2 from mantic-proposed.

$ sudo apt install -t mantic-proposed network-manager
[...]
$ LC_ALL=C apt list -i network-manager
Listing... Done
network-manager/mantic-proposed,now 1.44.2-1ubuntu1.2 amd64 [installed]

$ nmcli con add type dummy connection.interface-name dummy0
$ sudo cat /etc/netplan/90-NM-4189c63a-ecee-4a39-90f3-fffad2b96d0b.yaml
network:
  version: 2
  dummy-devices:
    NM-4189c63a-ecee-4a39-90f3-fffad2b96d0b:
      renderer: NetworkManager
      networkmanager:
        uuid: "4189c63a-ecee-4a39-90f3-fffad2b96d0b"
        name: "dummy-dummy0"
        passthrough:
          connection.interface-name: "dummy0"
          dummy._: ""
          ipv4.method: "disabled"
          ipv6.addr-gen-mode: "default"
          ipv6.method: "disabled"
          ipv6.ip6-privacy: "-1"
          proxy._: ""
$ nmcli d show dummy0
GENERAL.DEVICE: dummy0
GENERAL.TYPE: dummy
GENERAL.HWADDR: 32:89:DC:E0:74:2F
GENERAL.MTU: 1500
GENERAL.STATE: 100 (verbunden)
GENERAL.CONNECTION: dummy-dummy0
GENERAL.CON-PATH: /org/freedesktop/NetworkManager/ActiveConnection/10
IP4.GATEWAY: --
IP6.GATEWAY: --

$ nmcli con del dummy-dummy0
$ LC_ALL=C nmcli d show dummy0
Error: Device 'dummy0' not found.
$ sudo LC_ALL=C cat /etc/netplan/90-NM-4189c63a-ecee-4a39-90f3-fffad2b96d0b.yaml
cat: /etc/netplan/90-NM-4189c63a-ecee-4a39-90f3-fffad2b96d0b.yaml: No such file or directory
$ journalctl -u NetworkManager -e # I cannot spot any errors in here

[Launchpad Janitor (janitor)]

This bug was fixed in the package network-manager - 1.44.2-1ubuntu1.2

---------------
network-manager (1.44.2-1ubuntu1.2) mantic; urgency=medium

  * debian/tests/nm_netplan.py:
    Start Network Manager via systemd. The .service unit file sets
    ProtectSystem to true and we want to run the Netplan tests with this
    restriction enabled.
  * d/p/netplan/0003-Allow-the-NetworkManager-daemon-to-write-to-lib-netp.patch:
    Allow-list /usr/lib/netplan so libnetplan can open files from that
    directory with writing permission. See LP: #2040153
  * debian/tests/control.
    Add all the dependencies required by the nm_netplan.py tests.

 -- Danilo Egea Gondolfo <email address hidden> Mon, 23 Oct 2023 16:29:46 +0100

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for network-manager has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package network-manager - 1.44.2-1ubuntu2

---------------
network-manager (1.44.2-1ubuntu2) noble; urgency=medium

  [ Lukas Märdian ]
  * network-manager.postinst: Skip unknown connection profiles (LP: #2039503)
  * d/network-manager.postinst: Drop reboot notification (LP: #2040292)

  [ Danilo Egea Gondolfo ]
  * debian/tests/nm_netplan.py
    Start Network Manager via systemd. The .service unit file sets
    ProtectSystem to true and we want to run the Netplan tests with this
    restriction enabled.
  * d/p/netplan/0003-Allow-the-NetworkManager-daemon-to-write-to-lib-netp.patch
    Allow-list /usr/lib/netplan so libnetplan can open files from that
    directory with writing permission. See LP: #2040153
  * debian/tests/control
    Add all the dependencies required by the nm_netplan.py tests.
  * debian/tests/network_test_base.py.
    Increase the waiting time between creating a veth pair and reading their
    MAC addresses. On arm64, the system is taking longer to change the MAC
    after creation, leading to failures due to differences in the expected
    and current MAC addresses. See LP: #2023183

 -- Lukas Märdian <email address hidden> Thu, 26 Oct 2023 11:48:18 +0200