[network-manager] temporary dns lookup failure after updating

[Paul Larson]

When updating a lot of snaps at the same time, we sometimes see all of them after network manager fail with something like this: (note, it's not just the snap from the example below that fails, sometimes it's others like core)

error: cannot install "checkbox-plano": Get https://search.apps.ubuntu.com/api/v1/snaps/details/checkbox-plano?channel=stable&confinement=strict: dial tcp: lookup search.apps.ubuntu.com on [::1]:53: read udp [::1]:39768->[::1]:53: read: connection refused

I noticed that it seems to only happen after the network manager snap updates, so I tried reverting it to stable and doing something like 'sudo snap refresh network-manager && sudo snap install <foo>'. The first time it worked, the second time, I was able to reproduce the error above.

The system this happened on was connected via ethernet.

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

Thanks for the report! What do you see in /etc/netplan/00-snapd-config.yaml ?

[Paul Larson (pwlars)]

# This is the initial network config.
# It can be overwritten by cloud-init or console-conf.
network:
    version: 2
    ethernets:
        all-en:
            match:
                name: "en*"
            dhcp4: true
        all-eth:
            match:
                name: "eth*"
            dhcp4: true

[Tony Espy (awe)]

@Paul

Why do you have network-manager installed if the system is ethernet only? Also in general, if you're going to use the NM snap, it's best to let it take full control over all networking devices. We do this on the Plano device by dropping the a file containing the following into /etc/netplan:

network:
  renderer: NetworkManager

Also, the error you're seeing doesn't look to me like a DNS error, it looks like the store's endpoint is refusing the connection from snapd (hence permission denied).

The symptom you're seeing could be a side-effect of interactions between NM and networkd, and the fact the NM is restarted on upgrade...

Can you please report the version/revision of the core and network-manager snap's on your system? Also does the problem require a certain number so snaps to be refreshed in addition to network-manager, or can you get it to happen with just one additional snap? You might also want to attach a syslog covering the time period when the problem occurs.

[Paul Larson (pwlars)]

This actually is on a plano device (stlouis) where I'm hitting the problem, and it does have that in /etc/netplan/00-default-nm-renderer.yaml. It does not seem to depend on a lot of snaps being refreshed at once, as mentioned before, I can reproduce it with just 'sudo snap refresh network-manager --ignore-validation && sudo snap install <something>" Anything can be used for the install snap, even hello. As a quick way of reproducing, something like this script should get you there (in this case, I'm flipping between edge and stable since there are different versions of nm in each - it ran about all the way through about twice for me before hitting the error):
#!/bin/bash
set -ex
while [ 1 ]; do
    sudo snap refresh --edge network-manager --ignore-validation && sudo snap install hello && sudo snap remove hello
    [ "$?" != "0" ] && break
    sleep 1

    sudo snap refresh --stable network-manager --ignore-validation && sudo snap install hello && sudo snap remove hello
    [ "$?" != "0" ] && break
    sleep 1
done

Here are the last 100 lines of syslog from when I was running the script above and saw it reproduced: https://paste.ubuntu.com/24866493/