Ubuntu 18.04 networking issue (connection drops after approx. 15 minutes)

[Robert Roos]

I'm currently troubleshooting some network issues on Ubuntu 18.04 LTS Bionic. After checking a few logs I noticed that the networkd-dispatcher is generating errors. I'm unsure if this is a bug but I'd like to report it over here just to be sure:

```
Jun 04 18:14:21 [servername] systemd-networkd-wait-online[2079]: managing: br2
Jun 04 18:14:22 [servername] networkd-dispatcher[1890]: WARNING:Unknown index 7 seen, reloading interface list
Jun 04 18:14:23 [servername] networkd-dispatcher[1890]: WARNING:Unknown index 8 seen, reloading interface list
Jun 04 18:14:23 [servername] networkd-dispatcher[1890]: ERROR:Unknown interface index 8 seen even after reload
Jun 04 18:14:23 [servername] networkd-dispatcher[1890]: WARNING:Unknown index 8 seen, reloading interface list
Jun 04 18:14:23 [servername] networkd-dispatcher[1890]: ERROR:Unknown interface index 8 seen even after reload
```

These issues occur consistently on several servers. The problem is that I lose the connection specifically towards the internet after approx. 15 minutes. Internal facing interfaces (same hardware and driver) seem to behave differently and remain properly connected to the internal network. These network-dispatcher errors seem to occur simultanously when the connection on the external interface permanently drops.

The physical network links still appears to be online when the connection is lost.

I'm still also able to ping the default gateway from my provider to access the Internet. But all IP addresses beyond this gateway on the Internet seem to be unreachable.

I have disabled TCP offloading and firewalls to eliminate any influences. But that didn't work.
The ARP table shows that my gateway is a STALE record after the connection drops. So it seems related to the ARP-table or something likewise.

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1775131/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Julian Andres Klode (juliank)]

The networkd-dispatcher errors are not causing your connection issues - networkd dispatcher just starts scripts after network changes. I wonder if the connection issues are causing the networkd-dispatcher issues.

There needs to be some more information. Could you
(1) add -vv to networkd_dispatcher_args= in /etc/default/networkd-dispatcher and restart it?
(2) include the output of networkctl list

Thanks!

[Martin Goik (goik)]

Hello,

feeling very unhappy possibly joining the club. We have a current 18.04.1 x86_64 desktop system. Being an educational institution we clone our master installation PC's image to ~80 client machines sharing identical hardware. The subsequently described problems appear both on our master and many (if not all) client PC's.

Though times vary in between 5 and 15 minutes the network connection disappears completely. In case I'm lucky having an open shell on the victim »ifconfig« shows the external network interface's IP unassigned. So the desktop freezes completely.

I've attached an excerpt from /var/log/syslog that might lead to a clue. We've current BIOS on all machines.

Despite the attached excerpt full archived /var/log is available at https://cloud.mi.hdm-stuttgart.de/index.php/s/FNgwQs9Pwb64yCR.

[Martin Goik (goik)]

Hard to admit but we had a totally different culprit: A conflict between hardware clock and NTP sync caused dhclient dropping the interface probably »believing« lease timeout being reached.

[Launchpad Janitor (janitor)]

[Expired for networkd-dispatcher (Ubuntu) because there has been no activity for 60 days.]

[Dragon Cloud (dragon-cloud-deactivatedaccount)]

@Julian - As no one seems to be doing that, I'll take the initiative and give you the information instead.

(1) add -vv to networkd_dispatcher_args= in /etc/default/networkd-dispatcher and restart it?
(2) include the output of networkctl list

Output of networkctl list:

$ networkctl list
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 eth0 ether routable configured
  3 eth1 ether routable configured
  4 docker0 ether no-carrier unmanaged

4 links listed.

All log items surrounding the incident with -vv enabled:

Jan 09 19:10:04 lb-plentiful-reservoir networkd-dispatcher: DEBUG:Signal: typ=dbus.String('org.freedesktop.network1.Link'), data=dbus.Dictionary({dbus.String('OperationalState'): dbus.String('no-carrier', variant_level=1)}, signature=dbus.Signature('sv')), path=dbus.ObjectPath('/org/freedesktop/network1/link/_370')
Jan 09 19:10:04 lb-plentiful-reservoir networkd-dispatcher: WARNING:Unknown index 70 seen, reloading interface list
Jan 09 19:10:04 lb-plentiful-reservoir networkd-dispatcher: DEBUG:Performed interface scan; state: <Dispatcher({'iface_names_by_idx': {1: 'lo', 2: 'eth0', 3: 'eth1', 4: 'docker0', 69: 'veth3a1259f', 70: 'vethb90b319'}, 'ifaces_by_name': {'lo': NetworkctlListState(idx=1, name='lo', type='loopback', operational='carrier', administrative='unmanaged'), 'eth0': NetworkctlListState(idx=2, name='eth0', type='ether', operational='routable', administrative='configured'), 'eth1': NetworkctlListState(idx=3, name='eth1', type='ether', operational='routable', administrative='configured'), 'docker0': NetworkctlListState(idx=4, name='docker0', type='ether', operational='no-carrier', administrative='unmanaged'), 'veth3a1259f': NetworkctlListState(idx=69, name='veth3a1259f', type='ether', operational='off', administrative='unmanaged'), 'vethb90b319': NetworkctlListState(idx=70, name='vethb90b319', type='ether', operational='no-carrier', administrative='unmanaged')}, 'script_dir': '/etc/networkd-dispatcher:/usr/lib/networkd-dispatcher'})>
Jan 09 19:10:04 lb-plentiful-reservoir networkd-dispatcher: DEBUG:No change represented by operational state 'no-carrier' for interface 'vethb90b319'
Jan 09 19:10:04 lb-plentiful-reservoir networkd-dispatcher: DEBUG:Signal: typ=dbus.String('org.freedesktop.network1.Link'), data=dbus.Dictionary({dbus.String('AdministrativeState'): dbus.String('unmanaged', variant_level=1)}, signature=dbus.Signature('sv')), path=dbus.ObjectPath('/org/freedesktop/network1/link/_369')
Jan 09 19:10:04 lb-plentiful-reservoir networkd-dispatcher: DEBUG:No change represented by administrative state 'unmanaged' for interface 'veth3a1259f'
Jan 09 19:10:04 lb-plentiful-reservoir networkd-dispatcher: DEBUG:Signal: typ=dbus.String('org.freedesktop.network1.Link'), data=dbus.Dictionary({dbus.String('AdministrativeState'): dbus.String('unmanaged', variant_level=1)}, signature=dbus.Signature('sv')), path=dbus.ObjectPath('/org/freedesktop/network1/link/_370')
Jan 09 19:10:04 lb-plentiful-reservoir networkd-dispatcher: DEBUG:No change represented by administrative st...

[clayton craft (craftyguy)]

networkd-dispatcher, on its own, does not configure or control network interfaces. It listens on d-bus for events from systemd-networkd. So this is almost certainly a symptom rather than the cause of your interface stability issues, *unless* you have given networkd-dispatcher some script to execute that controls interfaces. If that's the case, you should probably not do that, since it would cause all sorts of hard-to-debug issues :)

[Naveen Kumar Goyal (naveengoyal)]

Ubuntu:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.1 LTS
Release: 18.04
Codename: bionic

------------------------------------------
root@u18_svr_madsz:/icxmSim/dist# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 10.206.49.229 netmask 255.255.255.0 broadcast 10.206.49.255
        inet6 fe80::20c:29ff:fe1e:a792 prefixlen 64 scopeid 0x20<link>
        ether 00:0c:29:1e:a7:92 txqueuelen 1000 (Ethernet)
        RX packets 3732692 bytes 539980595 (539.9 MB)
        RX errors 0 dropped 911281 overruns 0 frame 0
        TX packets 230447 bytes 18384025 (18.3 MB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ens34: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 192.168.49.229 netmask 255.255.0.0 broadcast 192.168.255.255
        inet6 3001:10::49:229 prefixlen 64 scopeid 0x0<global>
        inet6 fe80::20c:29ff:fe1e:a79c prefixlen 64 scopeid 0x20<link>
        ether 00:0c:29:1e:a7:9c txqueuelen 1000 (Ethernet)
        RX packets 634317126 bytes 109589158528 (109.5 GB)
        RX errors 0 dropped 3113 overruns 0 frame 0
        TX packets 519166 bytes 50492097 (50.4 MB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

icx5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet6 fe80::6438:8fff:fed6:4a54 prefixlen 64 scopeid 0x20<link>
        ether 66:38:8f:d6:4a:54 txqueuelen 1000 (Ethernet)
        RX packets 1287609 bytes 78787666 (78.7 MB)
        RX errors 0 dropped 2805 overruns 0 frame 0
        TX packets 5 bytes 446 (446.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
        inet 127.0.0.1 netmask 255.0.0.0
        inet6 ::1 prefixlen 128 scopeid 0x10<host>
        loop txqueuelen 1000 (Local Loopback)
        RX packets 1457 bytes 261910 (261.9 KB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 1457 bytes 261910 (261.9 KB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

------------------------------------------
Operation:
create 2 virtual interface of mac vlan type, as 2icx0, 2icx1,
always the last created interface is availble, older one get deleted due to

Apr 16 10:56:28 u18_svr_madsz networkd-dispatcher[824]: WARNING:Unknown index 72 seen, reloading interface list
Apr 16 10:56:28 u18_svr_madsz systemd-networkd[22168]: 2icx0: Lost carrier

Same feature macvlan work in 16.04

ip link add link ens34 2icx0 type macvlan
ifconfig 2icx0 up
ip addr add 192.168.49.230/16 dev 2icx0

ip link add link ens34 2icx1 type macvlan
ifconfig 2icx1 up
ip addr add 192.168.49.231/16 dev 2icx1

------------------------------------------

syslog:
pr 16 10:56:27 u18_svr_madsz systemd-udevd[30948]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 16 10:56:27 u18_svr_madsz systemd-udevd[30948]: Could not generate persistent MAC address for 2icx0: No such file or directory
Apr 16 10:56:27 u18_svr_madsz systemd-networkd[22168]: 2icx0: Gained carrier
Apr 16 10:56:27 u18_svr_madsz networkd-dispatcher[824]: WARNING:Unknown ind...

[Naveen Kumar Goyal (naveengoyal)]

this scenario is: i need to create 1000 of virtual interface , this is scale environment, where each new interface causing restart of networkd-dispatcher, losing out on older interface, if there creation is not completed successfully.

[Naveen Kumar Goyal (naveengoyal)]

Failed Logs:
------
Apr 16 10:56:27 u18_svr_madsz systemd-udevd[30948]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 16 10:56:27 u18_svr_madsz systemd-udevd[30948]: Could not generate persistent MAC address for 2icx0: No such file or directory
Apr 16 10:56:27 u18_svr_madsz systemd-networkd[22168]: 2icx0: Gained carrier
Apr 16 10:56:27 u18_svr_madsz networkd-dispatcher[824]: WARNING:Unknown index 71 seen, reloading interface list
Apr 16 10:56:27 u18_svr_madsz systemd-timesyncd[22214]: Network configuration changed, trying to establish connection.
Apr 16 10:56:27 u18_svr_madsz systemd-udevd[30960]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 16 10:56:27 u18_svr_madsz systemd-udevd[30960]: Could not generate persistent MAC address for 2icx1: No such file or directory
Apr 16 10:56:28 u18_svr_madsz systemd-networkd[22168]: 2icx1: Gained carrier
Apr 16 10:56:28 u18_svr_madsz networkd-dispatcher[824]: WARNING:Unknown index 72 seen, reloading interface list
Apr 16 10:56:28 u18_svr_madsz systemd-networkd[22168]: 2icx0: Lost carrier
Apr 16 10:56:28 u18_svr_madsz systemd-timesyncd[22214]: Synchronized to time server 91.189.89.199:123 (ntp.ubuntu.com).
Apr 16 10:56:29 u18_svr_madsz systemd-networkd[22168]: 2icx1: Gained IPv6LL

Successful Logs:
------
Apr 16 12:10:38 u18_svr_madsz systemd-udevd[31348]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 16 12:10:38 u18_svr_madsz systemd-networkd[22168]: 2icx0: Gained carrier
Apr 16 12:10:38 u18_svr_madsz systemd-timesyncd[22214]: Network configuration changed, trying to establish connection.
Apr 16 12:10:38 u18_svr_madsz networkd-dispatcher[824]: WARNING:Unknown index 75 seen, reloading interface list
Apr 16 12:10:38 u18_svr_madsz systemd-udevd[31348]: Could not generate persistent MAC address for 2icx0: No such file or directory
Apr 16 12:10:38 u18_svr_madsz systemd-udevd[31358]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 16 12:10:38 u18_svr_madsz systemd-udevd[31358]: Could not generate persistent MAC address for 2icx1: No such file or directory
Apr 16 12:10:38 u18_svr_madsz systemd-networkd[22168]: 2icx1: Gained carrier
Apr 16 12:10:38 u18_svr_madsz networkd-dispatcher[824]: WARNING:Unknown index 76 seen, reloading interface list
Apr 16 12:10:38 u18_svr_madsz systemd-timesyncd[22214]: Synchronized to time server 91.189.91.157:123 (ntp.ubuntu.com).
Apr 16 12:10:39 u18_svr_madsz systemd-networkd[22168]: 2icx0: Gained IPv6LL
Apr 16 12:10:40 u18_svr_madsz systemd-networkd[22168]: 2icx1: Gained IPv6LL

[Julian Andres Klode (juliank)]

Your logs do not show network dispatcher restarting, but it is reloading it's list of interfaces as it has to figure out which interface it got notified about.

[Julian Andres Klode (juliank)]

Anyhow, none of this is a bug I nthe dispatcher, and I don't know what else, so I'm going to reassign it to the kernel - if there are broken ARP tables and whatnot, what else could it be.

[Naveen Kumar Goyal (naveengoyal)]

problem occurs when both interface created using multiprocessing module of python
----------------
Process 1 execution ->
ip link add link ens34 2icx0 type macvlan
ifconfig 2icx0 up
ip addr add 192.168.49.230/16 dev 2icx0

Process 2 execution ->
ip link add link ens34 2icx1 type macvlan
ifconfig 2icx1 up
ip addr add 192.168.49.231/16 dev 2icx1

----------------

however this works when run in a order like:
----------------------------------------------------------------
ip link add link ens34 2icx0 type macvlan
ifconfig 2icx0 up
ip addr add 192.168.49.230/16 dev 2icx0
ip link add link ens34 2icx1 type macvlan
ifconfig 2icx1 up
ip addr add 192.168.49.231/16 dev 2icx1

[Naveen Kumar Goyal (naveengoyal)]

Summing UP all Above comments:
----------------------------

Ubuntu:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.1 LTS
Release: 18.04
Codename: bionic

this scenario is: i need to create 1000 of virtual interface , this is scale environment, where each new interface causing restart of networkd-dispatcher, losing out on older interface, if there creation is not completed successfully.

-------<wbr>-------<wbr>-------<wbr>-------<wbr>-------<wbr>-------
Operation:
create 2 virtual interface of mac vlan type, as 2icx0, 2icx1,
always the last created interface is availble, older one get deleted due to

Apr 16 10:56:28 u18_svr_madsz networkd-<wbr>dispatcher[<wbr>824]: WARNING:Unknown index 72 seen, reloading interface list
Apr 16 10:56:28 u18_svr_madsz systemd-<wbr>networkd[<wbr>22168]: 2icx0: Lost carrier

Same feature macvlan work in 16.04

<wbr>-------<wbr>-------<wbr>-------<wbr>-------<wbr>-------

problem occurs when both interface created using multiprocessing module of python
----------------
Process 1 execution ->
ip link add link ens34 2icx0 type macvlan
ifconfig 2icx0 up
ip addr add 192.168.49.230/16 dev 2icx0

Process 2 execution ->
ip link add link ens34 2icx1 type macvlan
ifconfig 2icx1 up
ip addr add 192.168.49.231/16 dev 2icx1

----------------

however this works when run in a order like:
-------<wbr>-------<wbr>-------<wbr>-------<wbr>-------<wbr>-------<wbr>-------<wbr>-------<wbr>-------<wbr>-
ip link add link ens34 2icx0 type macvlan
ifconfig 2icx0 up
ip addr add 192.168.49.230/16 dev 2icx0
ip link add link ens34 2icx1 type macvlan
ifconfig 2icx1 up
ip addr add 192.168.49.231/16 dev 2icx1

Failed Logs:
------
Apr 16 10:56:27 u18_svr_madsz systemd-<wbr>udevd[30948]<wbr>: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 16 10:56:27 u18_svr_madsz systemd-<wbr>udevd[30948]<wbr>: Could not generate persistent MAC address for 2icx0: No such file or directory
Apr 16 10:56:27 u18_svr_madsz systemd-<wbr>networkd[<wbr>22168]: 2icx0: Gained carrier
Apr 16 10:56:27 u18_svr_madsz networkd-<wbr>dispatcher[<wbr>824]: WARNING:Unknown index 71 seen, reloading interface list
Apr 16 10:56:27 u18_svr_madsz systemd-<wbr>timesyncd[<wbr>22214]: Network configuration changed, trying to establish connection.
Apr 16 10:56:27 u18_svr_madsz systemd-<wbr>udevd[30960]<wbr>: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 16 10:56:27 u18_svr_madsz systemd-<wbr>udevd[30960]<wbr>: Could not generate persistent MAC address for 2icx1: No such file or directory
Apr 16 10:56:28 u18_svr_madsz systemd-<wbr>networkd[<wbr>22168]: 2icx1: Gained carrier
Apr 16 10:56:28 u18_svr_madsz networkd-<wbr>dispatcher[<wbr>824]: WARNING:Unknown index 72 seen, reloading interface list
Apr 16 10:56:28 u18_svr_madsz systemd-<wbr>networkd[<wbr>22168]: 2icx0: Lost carrier
Apr 16 10:56:28 u18_svr_madsz systemd-<wbr>timesyncd[<wbr>22214]: Synchronized to time server 91.189.89.199:123 (ntp.ubuntu.com).
Apr 16 10:56:29 u18_svr_madsz systemd-<wbr>networkd[<wbr>22168]: 2icx1: Gained IPv6LL

Successful Logs:
------
Apr 16 12:10:38 u18_svr_madsz systemd-...

[Stan Scholtz (launcjpadnet)]

i have the same problem over here, i posted more details over here:
https://github.com/mailcow/mailcow-dockerized/issues/2918

i believe to reproduce the problem you can:
- set up multiple subnets under the same interface name
- make sure systemd-networkd is the only system to handle public interface (no NetworkManager or netplan)
- install somesoftware which uses networkd-dispatcher (like mailcow on docker)

- wait a week to see 1 minute downtime upon hitting the concurrency described above
OR
- my interfaces also restart (causing disconnets) upon systemd updates (via apt upgrade), or systemd re-install (with chances of 4 to 5 of hitting disconnect)

pls understand that to hit the concurrency problem you probably need to have lot's of IPs or subnets in your network configuration file in /etc/systemd/network/enpXXX.network (i got 7 ipv4 addresses there)

best regards, stan

[Florian (florian-ke)]

Also happening on 20.04

[Tony (tony-sarajarvi)]

I second that. Seeing this on 20.04 for some reason.

[Christopher Trahey (ctrahey91)]

Also observing this in 20.04.2 (kernel 5.4.0-70).
I will try swapping out physical transceivers on both ends of the link and a new switch port...

Of particular interest to me, netplan does not appear to re-run when this happens. I have a device renaming rule that matches on name, sets IP, and sets name. This device never comes back up unless I manually run netplan apply (which is always successful, but obvs. untenable). The root device (eno2) is not even listed in `ip link` output during the incident.

Please advise what logging/information I can provide to help track down culprit.

[Christopher Trahey (ctrahey91)]

Not sure if this will overlap with anyone else, but my experience was a combination of 2 things:

1. Reboots were happening, caused by unattended-upgrade
2. The device causing trouble was a renamed device.

Turns out it was trying to get renamed twice: Once very early in boot from eth1 to eno2, then again based on my neplan from eno2 to the name I wanted. This was failing, and leaving the device down.

See my comment #117 on this issue for more details: https://bugs.launchpad.net/netplan/+bug/1770082