Bad character encoding with NetworkManager backend

[Glandos]

Ubuntu 23.10
netplan 0.107-5ubuntu0.3
network-manager 1.44.2-1ubuntu1.2

My WiFi network has a Latin1 character in it. It's managed by hostapd (by myself), so the character is UTF-8.
Previously, this was managed by NetworkManager just fine.

Now in netplan, the following happens:
- the password is saved with latin1 encoding, resulting in \xe9 in YAML files
- this encoding is not understood by NetworkManager, resulting in double encoding withing password field
- I have to re-enter the password each time the laptop goes out of sleep.

[Danilo Egea Gondolfo (danilogondolfo)]

Hello, thank you for your bug report.

I managed to reproduce the issue and I'm testing a fix for that.

For our own reference, I'll add more details below:

The libyaml's emitter is converting valid utf-8 characters to latin1. In this case it's happening here https://github.com/yaml/libyaml/blob/0.2.5/src/emitter.c#L2051.

Example:

The keyfile below:

---
[connection]
id=Wi-Fi connection 1
uuid=f486322b-81e0-4f59-99c4-6d55348d8c46
type=wifi
interface-name=wlan0

[wifi]
mode=infrastructure
ssid=195;161;195;161;195;161;195;169;195;169;195;173;195;173;195;179;195;179;195;186;195;186;195;129;195;137;195;141;195;147;195;154;

[wifi-security]
key-mgmt=wpa-psk
psk=ZáááéééíííóóóÚÚÁÉÍÓÚ

[ipv4]
method=auto

[ipv6]
addr-gen-mode=stable-privacy
method=auto

[proxy]
---

Will emit the YAML below:

---
network:
  version: 2
  wifis:
    NM-f486322b-81e0-4f59-99c4-6d55348d8c46:
      renderer: NetworkManager
      match:
        name: "wlan0"
      dhcp4: true
      dhcp6: true
      ipv6-address-generation: "stable-privacy"
      access-points:
        "195;161;195;161;195;161;195;169;195;169;195;173;195;173;195;179;195;179;195;186;195;186;195;129;195;137;195;141;195;147;195;154;":
          auth:
            key-management: "psk"
            password: "Z\xE1\xE1\xE1\xE9\xE9\xE9\xED\xED\xED\xF3\xF3\xF3\xDA\xDA\xC1\xC9\xCD\xD3\xDA"
          networkmanager:
            uuid: "f486322b-81e0-4f59-99c4-6d55348d8c46"
            name: "Wi-Fi connection 1"
            passthrough:
              ipv6.ip6-privacy: "-1"
              proxy._: ""
      networkmanager:
        uuid: "f486322b-81e0-4f59-99c4-6d55348d8c46"
        name: "Wi-Fi connection 1"
---

The string in the "psk" field is valid latin1 encoded text:

>>> b"Z\xE1\xE1\xE1\xE9\xE9\xE9\xED\xED\xED\xF3\xF3\xF3\xDA\xDA\xC1\xC9\xCD\xD3\xDA".decode('latin1')
'ZáááéééíííóóóÚÚÁÉÍÓÚ'

Checking the libyaml function closely, one of the things it checks is if emitter->unicode is set.
This variable is documented as: /** Allow unescaped non-ASCII characters? */

The libyaml API has a function to set it: yaml_emitter_set_unicode(emitter_ptr, 1);

After using it in libnetplan, the YAML below is generated:

---
network:
  version: 2
  wifis:
    NM-f486322b-81e0-4f59-99c4-6d55348d8c46:
      renderer: NetworkManager
      match:
        name: "wlan0"
      dhcp4: true
      dhcp6: true
      ipv6-address-generation: "stable-privacy"
      access-points:
        "195;161;195;161;195;161;195;169;195;169;195;173;195;173;195;179;195;179;195;186;195;186;195;129;195;137;195;141;195;147;195;154;":
          auth:
            key-management: "psk"
            password: "ZáááéééíííóóóÚÚÁÉÍÓÚ"
          networkmanager:
            uuid: "f486322b-81e0-4f59-99c4-6d55348d8c46"
            name: "Wi-Fi connection 1"
            passthrough:
              ipv6.ip6-privacy: "-1"
              proxy._: ""
      networkmanager:
        uuid: "f486322b-81e0-4f59-99
---

I still need to prepare a patch and investigate if something will break with this change.

[Lukas Märdian (slyon)]

https://github.com/canonical/netplan/pull/485

[Lukas Märdian (slyon)]

fixed in https://github.com/canonical/netplan/releases/tag/1.0.1

[Danilo Egea Gondolfo (danilogondolfo)]

Hello,

A fix for this problem is available in the new netplan.io 1.0.1. At the moment it's available for Noble in the -proposed pocket.

Would it be possible for you to install it from -proposed (if you are running Ubuntu 24.04) and check if it fixes the problem for you?

https://wiki.ubuntu.com/Testing/EnableProposed

[Glandos (glandos)]

For now, my laptop is still using 23.10, but I plan to upgrade to 24.04 soon. I'll let you informed then.

Anyway, thank for the quick resolution of the issue.

[Glandos (glandos)]

I was able to upgrade to 24.04, it's now working, thanks a lot.