misleading error messages when /etc/NetworkManager/system-connections is a cross-filesystem symlink

[Andrea Ieri]

If the /etc/NetworkManager/system-connections directory is a cross-filesystem symlink, activating connections will fail with the unrelated messages:

* "failed to create file ...: Read-only file system" (TRACE level)

* "failure adding connection: settings plugin does not support adding connections" (DEBUG level and above)

* no error message (Gnome UI)

I suspect moving the systems-connections directory onto a separate filesystem is not supported because network manager tries to atomically move a connection temp file into place and fails, but this is really hard to figure out since the error messages are completely unrelated.

Versions:

Ubuntu 20.10
network-manager 1.26.2-1ubuntu1
network-manager-gnome 1.18.0-1ubuntu2

How to reproduce:

sudo mv /etc/NetworkManager/system-connections /some/other/mountpoint/system-connections
sudo ln -s /some/other/mountpoint/system-connections /etc/NetworkManager/system-connections

Then:
* select any wifi network via the Gnome UI, or
* connect to a wifi network via nmcli, or
* connect to a wifi network via nmtui

Symptoms:

* "failed to create file ...: Read-only file system" (TRACE level, journalctl)

* "failure adding connection: settings plugin does not support adding connections" (DEBUG level and above, journalctl and nmtui/nmcli output)

* no error message (Gnome UI)

Relevant sample logs:

Feb 01 20:56:04 laptop NetworkManager[1368510]: <trace> [1612230964.4038] active-connection[0x56274f6f0a60]: creating
Feb 01 20:56:04 laptop NetworkManager[1368510]: <debug> [1612230964.4039] active-connection[0x56274f6f0a60]: set device "wlp61s0" [0x56274f883f00]
Feb 01 20:56:04 laptop NetworkManager[1368510]: <debug> [1612230964.4039] device[d62304f9da70d783] (wlp61s0): add_pending_action (1): 'activation-5'
Feb 01 20:56:04 laptop NetworkManager[1368510]: <debug> [1612230964.4040] active-connection[0x56274f6f0a60]: constructed (NMActRequest, version-id 5, type managed)
Feb 01 20:56:04 laptop NetworkManager[1368510]: <trace> [1612230964.4040] auth: call[37]: CheckAuthorization(org.freedesktop.NetworkManager.network-control), subject=unix-process[pid=1369566, uid=0, start=18908113] (succeeding for root)
Feb 01 20:56:04 laptop NetworkManager[1368510]: <trace> [1612230964.4041] auth: call[37]: completed: authorized=1, challenge=0 (simulated)
Feb 01 20:56:04 laptop NetworkManager[1368510]: <trace> [1612230964.4041] auth: call[38]: CheckAuthorization(org.freedesktop.NetworkManager.settings.modify.system), subject=unix-process[pid=1369566, uid=0, start=18908113] (succeeding for root)
Feb 01 20:56:04 laptop NetworkManager[1368510]: <trace> [1612230964.4042] auth: call[38]: completed: authorized=1, challenge=0 (simulated)
Feb 01 20:56:04 laptop NetworkManager[1368510]: <trace> [1612230964.4042] settings: add-connection: failed to add a4e6f563-eb2a-41ff-bc23-986ee3438ed9/'mywifi_2G': settings plugin does not support adding connections
Feb 01 20:56:04 laptop NetworkManager[1368510]: <trace> [1612230964.4061] keyfile: commit: a4e6f563-eb2a-41ff-bc23-986ee3438ed9 (mywifi_2G) failed to add: error writing to file '/etc/NetworkManager/system-connections/mywifi_2G.nmconnection': failed to create file /etc/NetworkManager/system-connections/mywifi_2G.nmconnection.ZE5DY0: Read-only file system
Feb 01 20:56:04 laptop NetworkManager[1368510]: <trace> [1612230964.4062] settings: add-connection: failed to add a4e6f563-eb2a-41ff-bc23-986ee3438ed9/'mywifi_2G': error writing to file '/etc/NetworkManager/system-connections/mywifi_2G.nmconnection': failed to create file /etc/NetworkManager/system-connections/mywifi_2G.nmconnection.ZE5DY0: Read-only file system
Feb 01 20:56:04 laptop NetworkManager[1368510]: <debug> [1612230964.4062] active-connection[0x56274f6f0a60]: Failed to activate '(null)': failure adding connection: settings plugin does not support adding connections
Feb 01 20:56:04 laptop NetworkManager[1368510]: <debug> [1612230964.4062] active-connection[0x56274f6f0a60]: set state deactivated (was unknown)
Feb 01 20:56:04 laptop NetworkManager[1368510]: <debug> [1612230964.4063] active-connection[0x56274f6f0a60]: check-master-ready: not signalling (state deactivated, no master)
Feb 01 20:56:04 laptop NetworkManager[1368510]: <debug> [1612230964.4063] device[d62304f9da70d783] (wlp61s0): remove_pending_action (0): 'activation-5'
Feb 01 20:56:04 laptop NetworkManager[1368510]: <debug> [1612230964.4066] audit: socket created
Feb 01 20:56:04 laptop NetworkManager[1368510]: <debug> [1612230964.4067] create NMAuditManager singleton (a031d34f795a73ab)
Feb 01 20:56:04 laptop NetworkManager[1368510]: <info> [1612230964.4069] audit: op="connection-add-activate" pid=1369566 uid=0 result="fail" reason="failure adding connection: settings plugin does not support adding connections"
Feb 01 20:56:04 laptop NetworkManager[1368510]: <debug> [1612230964.4069] active-connection[0x56274f6f0a60]: disposing

[U Aberg (c-u1)]

I see this behaviour also when /etc is still on the rootfs, but fstab has a "ro" for rootfs.

on Ubuntu 18.04 LTS and 20.04 LTS (arm versions for odroids, but not unlikely generally), I get a

# nmcli con add type ethernet ifname eth0 con-name XXXXX
"Error: Failed to add 'XXXXX' connection: failure adding connection: settings plugin does not support adding connections"

Changing fstab to rw and rebooting removes the error; added or modified connections, including static ip addresses, Wifi connections etc, will persist and work as expected when rebooting later with ro for rootfs in fstab.

for me, any fix would ideally enable functionality after a simple
# mount -o remount,rw /dev/mmcblk0p2 /
instead of having to change fstab and reboot.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Matt Culler (mrculler)]

I think there are multiple bugs in play here. My system-connections dir is a symlink within the same filesystem, and whenever I try to save a new connection I get the "settings plugin does not support adding connections" message.

Aug 18 17:11:08 workhorse4 NetworkManager[4092654]: <info> [1660857068.1365] audit: op="connection-add" pid=4124192 uid=1000 result="fail" reason="failure adding connection: settings plugin does not support adding connections"

The symlink target dir has the same ownership/permissions as the original system-connections dir - 0755, root:root.