| 2017-10-04 10:19:13 |
Nicolino Curalli |
bug |
|
|
added bug |
| 2017-10-04 10:21:38 |
Oliver Grawert |
summary |
Networkd fail to set ip address between leases if ip address changes |
Networkd fail to set ip address between leases if ip address changes on UbuntuCore |
|
| 2017-10-04 10:22:29 |
Oliver Grawert |
nominated for series |
|
Ubuntu Xenial |
|
| 2017-10-04 10:22:29 |
Oliver Grawert |
bug task added |
|
systemd (Ubuntu Xenial) |
|
| 2017-10-04 10:24:14 |
Oliver Grawert |
bug task added |
|
snappy |
|
| 2017-10-06 15:33:59 |
Michael Vogt |
bug watch added |
|
https://github.com/systemd/systemd/issues/5625 |
|
| 2017-10-07 10:18:41 |
Oliver Grawert |
snappy: status |
New |
Fix Committed |
|
| 2017-10-07 10:18:44 |
Oliver Grawert |
snappy: importance |
Undecided |
High |
|
| 2017-10-07 10:18:54 |
Oliver Grawert |
snappy: assignee |
|
Michael Vogt (mvo) |
|
| 2017-10-09 10:38:19 |
Dimitri John Ledkov |
nominated for series |
|
Ubuntu Artful |
|
| 2017-10-09 10:38:19 |
Dimitri John Ledkov |
bug task added |
|
systemd (Ubuntu Artful) |
|
| 2017-10-09 10:38:19 |
Dimitri John Ledkov |
nominated for series |
|
Ubuntu Zesty |
|
| 2017-10-09 10:38:19 |
Dimitri John Ledkov |
bug task added |
|
systemd (Ubuntu Zesty) |
|
| 2017-10-11 16:02:18 |
Dimitri John Ledkov |
description |
Hi there,
we found a replicable issue that involves the Ubuntu Core networking and causes complete loss of connectivity.
We run a custom board with ubuntu core: the architecure is amrhf.
We replicated this issue with an official Ubuntu Core image on a Raspberry Pi: other platform was been tested.
It shows that it is a snap core problem which interests networkd: we use the default network stack based on networkd + netplan.
Below steps to replicate the issue.
1)Setup a dhcp server for lease of about some minutes (i.e 10 minutes).
2)Boot the board and wait for get an ip from dhcp server
3)Before the lease expires, set a reservation for a different ip address
Depending on lease duration before the lease expires( for 10 minute we have 2 minutes before ), networkd configure the new address in addition to the previous one.
When the lease expire both ip address ( the prevoius and the new one ) disappear from the interested network interface.
Depending on lease duration before the second lease expires ( for 10 minure we have 2 minutes before ) networkd configure only the new ip address on the network interface and the ping toward an outside host work properly.
During the test the dhcp server records correctly leases and their duration.
We check directly from console the network interface setting with the tool ip, checking continuously the value for ip address and valid_lft fields for the interested network interface.
Please note that if the ip address setting are the same between leases the problem doesn’t jump out.
Please note that if the ip address setting are different between lease the problem jumps out and it is very bad:
Typically the lease time on consumer router are about some day, then a board that change ip between lease loses the network connectivity for some day without a direct action.
Please, after a confirmation from your side of the issue, could you fix the problem or escalate the issue at the upstream project?
We are available for further testing.
Below syslog annotated with ip address state:
Oct 4 09:48:06 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:49:36 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:49:25 localhost systemd-timesyncd[996]: Network configuration changed, trying to establish connection.
Oct 4 09:49:26 localhost systemd-timesyncd[996]: Synchronized to time server 91.189.89.199:123 (ntp.ubuntu.com).
Oct 4 09:49:26 localhost systemd[1]: Starting Update resolvconf for networkd DNS...
Oct 4 09:49:26 localhost systemd[1]: Started Update resolvconf for networkd DNS.
Oct 4 09:49:37 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:51:07 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:51:09 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:52:39 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:52:40 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:54:10 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:54:11 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:55:41 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:54:23 localhost systemd-timesyncd[996]: Network configuration changed, trying to establish connection.
Oct 4 09:54:23 localhost systemd[1]: Starting Update resolvconf for networkd DNS...
Oct 4 09:54:23 localhost systemd-timesyncd[996]: Synchronized to time server 91.189.89.199:123 (ntp.ubuntu.com).
Oct 4 09:54:23 localhost systemd[1]: Started Update resolvconf for networkd DNS.
Oct 4 09:55:43 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:57:13 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:57:14 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:58:44 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:58:46 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:00:16 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:00:17 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:01:47 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:01:48 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:03:18 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:03:05 localhost systemd-networkd[623]: eth0: DHCPv4 address 192.168.5.124/24 via 192.168.5.1 -----> here begin two ip address state
Oct 4 10:03:05 localhost systemd-timesyncd[996]: Network configuration changed, trying to establish connection.
Oct 4 10:03:05 localhost systemd[1]: Starting Update resolvconf for networkd DNS...
Oct 4 10:03:05 localhost systemd[1]: Started Update resolvconf for networkd DNS.
Oct 4 10:03:15 localhost systemd-timesyncd[996]: Timed out waiting for reply from 91.189.89.199:123 (ntp.ubuntu.com).
Oct 4 10:03:16 localhost systemd-timesyncd[996]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
Oct 4 10:03:20 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:04:50 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:04:23 localhost systemd-timesyncd[996]: Network configuration changed, trying to establish connection. ----> disappear any address
Oct 4 10:04:23 localhost systemd[1]: Starting Update resolvconf for networkd DNS...
Oct 4 10:04:23 localhost systemd[1]: Started Update resolvconf for networkd DNS.
Oct 4 10:04:51 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:06:21 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:06:23 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:07:53 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:07:54 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:09:24 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:09:24 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:10:54 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:10:55 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4
10:12:25 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ] ------> before this line interface eth0 don't have address, after this line the interfaces get 192.168.5.124/24( the second lise adddress)
Oct 4 10:11:49 localhost systemd[1]: Starting Update resolvconf for networkd DNS...
Oct 4 10:11:49 localhost systemd[1]: Started Update resolvconf for networkd DNS.
Oct 4 10:12:25 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:13:55 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:13:18 localhost systemd[1]: Started Session 23 of user domotz.
Cheers,
Nicolino |
[Impact]
* networkd fails to renew a lease, specifically it fails to change IPv4 address via DHCP renew/rebind.
* networkd relies on a kernel feature to promote secondary IPv4 address to primary, upon primary address lease expiry.
* this sysctl tunable was not enabled by default in systemd.
[Test Case]
Add a device, and assign two IPv4 addresses. First one, with a short lease time. Second one, with a different ip and a longer lease time. Second one should be treated as secondary ip address, and upon expiry of the first one, should be promoted and become primary ip address. The below scripted instructions simulate this:
sudo ip link add name testleases type dummy
sudo ip address add 192.0.2.10/27 dev testleases \
valid_lft 5 preferred_lft 5
sudo ip address add 192.0.2.11/27 dev testleases \
valid_lft 11 preferred_lft 11
ip address list dev testleases | \
grep -q 'inet 192.0.2.10/27 scope global dynamic testleases' \
&& echo ok || echo not ok
ip address list dev testleases | \
grep -q 'inet 192.0.2.11/27 scope global secondary dynamic testleases' \
&& echo ok || echo not ok
sleep 6
ip address list dev testleases | \
grep -q 'inet 192.0.2.11/27 scope global dynamic testleases' \
&& echo ok || echo not ok
sudo ip link del dev testleases
[Regression Potential]
* This changes the default kernel behaviour, previously upon expiry of the primary address, secondary addresses were removed as well. Which is imho silly.
* comparing networkd renewal with isc-dhcp renewal the semantics are quite different. Upon acquiring new ip address, isc-dhcp would instantly flush existing ip address, and add a new one. Networkd add the new address as secondary, and waits for old one to expire first before promoting / switching to using the new ip address. IMHO kernel should have an API to promote secondary ip address to a primary one.
* This update also applies other safe-looking options, which are currently also already applied via sysctls shipped in other packages
# Source route verification
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
# Enable hard and soft link protection
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* This update also applies the following upstream/bufferbloat.net recommended setting
# Fair Queue CoDel packet scheduler to fight bufferbloat
net.core.default_qdisc = fq_codel
[Other Info]
* Original bug report
Hi there,
we found a replicable issue that involves the Ubuntu Core networking and causes complete loss of connectivity.
We run a custom board with ubuntu core: the architecure is amrhf.
We replicated this issue with an official Ubuntu Core image on a Raspberry Pi: other platform was been tested.
It shows that it is a snap core problem which interests networkd: we use the default network stack based on networkd + netplan.
Below steps to replicate the issue.
1)Setup a dhcp server for lease of about some minutes (i.e 10 minutes).
2)Boot the board and wait for get an ip from dhcp server
3)Before the lease expires, set a reservation for a different ip address
Depending on lease duration before the lease expires( for 10 minute we have 2 minutes before ), networkd configure the new address in addition to the previous one.
When the lease expire both ip address ( the prevoius and the new one ) disappear from the interested network interface.
Depending on lease duration before the second lease expires ( for 10 minure we have 2 minutes before ) networkd configure only the new ip address on the network interface and the ping toward an outside host work properly.
During the test the dhcp server records correctly leases and their duration.
We check directly from console the network interface setting with the tool ip, checking continuously the value for ip address and valid_lft fields for the interested network interface.
Please note that if the ip address setting are the same between leases the problem doesn’t jump out.
Please note that if the ip address setting are different between lease the problem jumps out and it is very bad:
Typically the lease time on consumer router are about some day, then a board that change ip between lease loses the network connectivity for some day without a direct action.
Please, after a confirmation from your side of the issue, could you fix the problem or escalate the issue at the upstream project?
We are available for further testing.
Below syslog annotated with ip address state:
Oct 4 09:48:06 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:49:36 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:49:25 localhost systemd-timesyncd[996]: Network configuration changed, trying to establish connection.
Oct 4 09:49:26 localhost systemd-timesyncd[996]: Synchronized to time server 91.189.89.199:123 (ntp.ubuntu.com).
Oct 4 09:49:26 localhost systemd[1]: Starting Update resolvconf for networkd DNS...
Oct 4 09:49:26 localhost systemd[1]: Started Update resolvconf for networkd DNS.
Oct 4 09:49:37 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:51:07 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:51:09 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:52:39 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:52:40 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:54:10 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:54:11 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:55:41 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:54:23 localhost systemd-timesyncd[996]: Network configuration changed, trying to establish connection.
Oct 4 09:54:23 localhost systemd[1]: Starting Update resolvconf for networkd DNS...
Oct 4 09:54:23 localhost systemd-timesyncd[996]: Synchronized to time server 91.189.89.199:123 (ntp.ubuntu.com).
Oct 4 09:54:23 localhost systemd[1]: Started Update resolvconf for networkd DNS.
Oct 4 09:55:43 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:57:13 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:57:14 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:58:44 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:58:46 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:00:16 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:00:17 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:01:47 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:01:48 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:03:18 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:03:05 localhost systemd-networkd[623]: eth0: DHCPv4 address 192.168.5.124/24 via 192.168.5.1 -----> here begin two ip address state
Oct 4 10:03:05 localhost systemd-timesyncd[996]: Network configuration changed, trying to establish connection.
Oct 4 10:03:05 localhost systemd[1]: Starting Update resolvconf for networkd DNS...
Oct 4 10:03:05 localhost systemd[1]: Started Update resolvconf for networkd DNS.
Oct 4 10:03:15 localhost systemd-timesyncd[996]: Timed out waiting for reply from 91.189.89.199:123 (ntp.ubuntu.com).
Oct 4 10:03:16 localhost systemd-timesyncd[996]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
Oct 4 10:03:20 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:04:50 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:04:23 localhost systemd-timesyncd[996]: Network configuration changed, trying to establish connection. ----> disappear any address
Oct 4 10:04:23 localhost systemd[1]: Starting Update resolvconf for networkd DNS...
Oct 4 10:04:23 localhost systemd[1]: Started Update resolvconf for networkd DNS.
Oct 4 10:04:51 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:06:21 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:06:23 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:07:53 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:07:54 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:09:24 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:09:24 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:10:54 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:10:55 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4
10:12:25 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ] ------> before this line interface eth0 don't have address, after this line the interfaces get 192.168.5.124/24( the second lise adddress)
Oct 4 10:11:49 localhost systemd[1]: Starting Update resolvconf for networkd DNS...
Oct 4 10:11:49 localhost systemd[1]: Started Update resolvconf for networkd DNS.
Oct 4 10:12:25 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:13:55 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:13:18 localhost systemd[1]: Started Session 23 of user domotz.
Cheers,
Nicolino |
|
| 2017-10-11 16:02:34 |
Dimitri John Ledkov |
systemd (Ubuntu Artful): status |
New |
In Progress |
|
| 2017-10-11 16:02:40 |
Dimitri John Ledkov |
systemd (Ubuntu Artful): importance |
Undecided |
High |
|
| 2017-10-11 16:06:05 |
Dimitri John Ledkov |
bug |
|
|
added subscriber Ubuntu Security Team |
| 2017-10-11 16:06:10 |
Dimitri John Ledkov |
bug |
|
|
added subscriber Ubuntu Kernel Repositories |
| 2017-10-11 16:06:16 |
Dimitri John Ledkov |
bug |
|
|
added subscriber Ubuntu Server Team |
| 2017-10-12 12:16:18 |
Francis Ginther |
tags |
|
id-59de49a6bfa10c370d70d043 |
|
| 2017-10-12 13:52:25 |
Robie Basak |
bug |
|
|
added subscriber Robie Basak |
| 2017-10-12 16:30:30 |
Robie Basak |
description |
[Impact]
* networkd fails to renew a lease, specifically it fails to change IPv4 address via DHCP renew/rebind.
* networkd relies on a kernel feature to promote secondary IPv4 address to primary, upon primary address lease expiry.
* this sysctl tunable was not enabled by default in systemd.
[Test Case]
Add a device, and assign two IPv4 addresses. First one, with a short lease time. Second one, with a different ip and a longer lease time. Second one should be treated as secondary ip address, and upon expiry of the first one, should be promoted and become primary ip address. The below scripted instructions simulate this:
sudo ip link add name testleases type dummy
sudo ip address add 192.0.2.10/27 dev testleases \
valid_lft 5 preferred_lft 5
sudo ip address add 192.0.2.11/27 dev testleases \
valid_lft 11 preferred_lft 11
ip address list dev testleases | \
grep -q 'inet 192.0.2.10/27 scope global dynamic testleases' \
&& echo ok || echo not ok
ip address list dev testleases | \
grep -q 'inet 192.0.2.11/27 scope global secondary dynamic testleases' \
&& echo ok || echo not ok
sleep 6
ip address list dev testleases | \
grep -q 'inet 192.0.2.11/27 scope global dynamic testleases' \
&& echo ok || echo not ok
sudo ip link del dev testleases
[Regression Potential]
* This changes the default kernel behaviour, previously upon expiry of the primary address, secondary addresses were removed as well. Which is imho silly.
* comparing networkd renewal with isc-dhcp renewal the semantics are quite different. Upon acquiring new ip address, isc-dhcp would instantly flush existing ip address, and add a new one. Networkd add the new address as secondary, and waits for old one to expire first before promoting / switching to using the new ip address. IMHO kernel should have an API to promote secondary ip address to a primary one.
* This update also applies other safe-looking options, which are currently also already applied via sysctls shipped in other packages
# Source route verification
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
# Enable hard and soft link protection
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* This update also applies the following upstream/bufferbloat.net recommended setting
# Fair Queue CoDel packet scheduler to fight bufferbloat
net.core.default_qdisc = fq_codel
[Other Info]
* Original bug report
Hi there,
we found a replicable issue that involves the Ubuntu Core networking and causes complete loss of connectivity.
We run a custom board with ubuntu core: the architecure is amrhf.
We replicated this issue with an official Ubuntu Core image on a Raspberry Pi: other platform was been tested.
It shows that it is a snap core problem which interests networkd: we use the default network stack based on networkd + netplan.
Below steps to replicate the issue.
1)Setup a dhcp server for lease of about some minutes (i.e 10 minutes).
2)Boot the board and wait for get an ip from dhcp server
3)Before the lease expires, set a reservation for a different ip address
Depending on lease duration before the lease expires( for 10 minute we have 2 minutes before ), networkd configure the new address in addition to the previous one.
When the lease expire both ip address ( the prevoius and the new one ) disappear from the interested network interface.
Depending on lease duration before the second lease expires ( for 10 minure we have 2 minutes before ) networkd configure only the new ip address on the network interface and the ping toward an outside host work properly.
During the test the dhcp server records correctly leases and their duration.
We check directly from console the network interface setting with the tool ip, checking continuously the value for ip address and valid_lft fields for the interested network interface.
Please note that if the ip address setting are the same between leases the problem doesn’t jump out.
Please note that if the ip address setting are different between lease the problem jumps out and it is very bad:
Typically the lease time on consumer router are about some day, then a board that change ip between lease loses the network connectivity for some day without a direct action.
Please, after a confirmation from your side of the issue, could you fix the problem or escalate the issue at the upstream project?
We are available for further testing.
Below syslog annotated with ip address state:
Oct 4 09:48:06 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:49:36 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:49:25 localhost systemd-timesyncd[996]: Network configuration changed, trying to establish connection.
Oct 4 09:49:26 localhost systemd-timesyncd[996]: Synchronized to time server 91.189.89.199:123 (ntp.ubuntu.com).
Oct 4 09:49:26 localhost systemd[1]: Starting Update resolvconf for networkd DNS...
Oct 4 09:49:26 localhost systemd[1]: Started Update resolvconf for networkd DNS.
Oct 4 09:49:37 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:51:07 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:51:09 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:52:39 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:52:40 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:54:10 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:54:11 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:55:41 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:54:23 localhost systemd-timesyncd[996]: Network configuration changed, trying to establish connection.
Oct 4 09:54:23 localhost systemd[1]: Starting Update resolvconf for networkd DNS...
Oct 4 09:54:23 localhost systemd-timesyncd[996]: Synchronized to time server 91.189.89.199:123 (ntp.ubuntu.com).
Oct 4 09:54:23 localhost systemd[1]: Started Update resolvconf for networkd DNS.
Oct 4 09:55:43 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:57:13 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:57:14 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:58:44 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:58:46 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:00:16 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:00:17 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:01:47 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:01:48 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:03:18 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:03:05 localhost systemd-networkd[623]: eth0: DHCPv4 address 192.168.5.124/24 via 192.168.5.1 -----> here begin two ip address state
Oct 4 10:03:05 localhost systemd-timesyncd[996]: Network configuration changed, trying to establish connection.
Oct 4 10:03:05 localhost systemd[1]: Starting Update resolvconf for networkd DNS...
Oct 4 10:03:05 localhost systemd[1]: Started Update resolvconf for networkd DNS.
Oct 4 10:03:15 localhost systemd-timesyncd[996]: Timed out waiting for reply from 91.189.89.199:123 (ntp.ubuntu.com).
Oct 4 10:03:16 localhost systemd-timesyncd[996]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
Oct 4 10:03:20 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:04:50 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:04:23 localhost systemd-timesyncd[996]: Network configuration changed, trying to establish connection. ----> disappear any address
Oct 4 10:04:23 localhost systemd[1]: Starting Update resolvconf for networkd DNS...
Oct 4 10:04:23 localhost systemd[1]: Started Update resolvconf for networkd DNS.
Oct 4 10:04:51 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:06:21 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:06:23 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:07:53 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:07:54 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:09:24 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:09:24 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:10:54 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:10:55 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4
10:12:25 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ] ------> before this line interface eth0 don't have address, after this line the interfaces get 192.168.5.124/24( the second lise adddress)
Oct 4 10:11:49 localhost systemd[1]: Starting Update resolvconf for networkd DNS...
Oct 4 10:11:49 localhost systemd[1]: Started Update resolvconf for networkd DNS.
Oct 4 10:12:25 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:13:55 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:13:18 localhost systemd[1]: Started Session 23 of user domotz.
Cheers,
Nicolino |
[Impact]
* networkd fails to renew a lease, specifically it fails to change IPv4 address via DHCP renew/rebind.
* networkd relies on a kernel feature to promote secondary IPv4 address to primary, upon primary address lease expiry.
* this sysctl tunable was not enabled by default in systemd.
[Test Case]
Add a device, and assign two IPv4 addresses. First one, with a short lease time. Second one, with a different ip and a longer lease time. Second one should be treated as secondary ip address, and upon expiry of the first one, should be promoted and become primary ip address. The below scripted instructions simulate this:
sudo ip link add name testleases type dummy
sudo ip address add 192.0.2.10/27 dev testleases \
valid_lft 5 preferred_lft 5
sudo ip address add 192.0.2.11/27 dev testleases \
valid_lft 11 preferred_lft 11
ip address list dev testleases | \
grep -q 'inet 192.0.2.10/27 scope global dynamic testleases' \
&& echo ok || echo not ok
ip address list dev testleases | \
grep -q 'inet 192.0.2.11/27 scope global secondary dynamic testleases' \
&& echo ok || echo not ok
sleep 6
ip address list dev testleases | \
grep -q 'inet 192.0.2.11/27 scope global dynamic testleases' \
&& echo ok || echo not ok
sudo ip link del dev testleases
[Regression Potential]
* This changes the default kernel behaviour, previously upon expiry of the primary address, secondary addresses were removed as well. Which is imho silly.
* comparing networkd renewal with isc-dhcp renewal the semantics are quite different. Upon acquiring new ip address, isc-dhcp would instantly flush existing ip address, and add a new one. Networkd add the new address as secondary, and waits for old one to expire first before promoting / switching to using the new ip address. IMHO kernel should have an API to promote secondary ip address to a primary one.
* This update also applies other safe-looking options, which are currently also already applied via sysctls shipped in other packages
# Source route verification
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
# Enable hard and soft link protection
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* This update also applies the following upstream/bufferbloat.net recommended setting
# Fair Queue CoDel packet scheduler to fight bufferbloat
net.core.default_qdisc = fq_codel
* [~racb] There are complex network setups out there, such as HA with corosync/pacemaker, OpenStack Neutron, and that kind of thing. If this fix were SRU'd, will all of these things in the wild cope with this sysctl change?
[Other Info]
* Original bug report
Hi there,
we found a replicable issue that involves the Ubuntu Core networking and causes complete loss of connectivity.
We run a custom board with ubuntu core: the architecure is amrhf.
We replicated this issue with an official Ubuntu Core image on a Raspberry Pi: other platform was been tested.
It shows that it is a snap core problem which interests networkd: we use the default network stack based on networkd + netplan.
Below steps to replicate the issue.
1)Setup a dhcp server for lease of about some minutes (i.e 10 minutes).
2)Boot the board and wait for get an ip from dhcp server
3)Before the lease expires, set a reservation for a different ip address
Depending on lease duration before the lease expires( for 10 minute we have 2 minutes before ), networkd configure the new address in addition to the previous one.
When the lease expire both ip address ( the prevoius and the new one ) disappear from the interested network interface.
Depending on lease duration before the second lease expires ( for 10 minure we have 2 minutes before ) networkd configure only the new ip address on the network interface and the ping toward an outside host work properly.
During the test the dhcp server records correctly leases and their duration.
We check directly from console the network interface setting with the tool ip, checking continuously the value for ip address and valid_lft fields for the interested network interface.
Please note that if the ip address setting are the same between leases the problem doesn’t jump out.
Please note that if the ip address setting are different between lease the problem jumps out and it is very bad:
Typically the lease time on consumer router are about some day, then a board that change ip between lease loses the network connectivity for some day without a direct action.
Please, after a confirmation from your side of the issue, could you fix the problem or escalate the issue at the upstream project?
We are available for further testing.
Below syslog annotated with ip address state:
Oct 4 09:48:06 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:49:36 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:49:25 localhost systemd-timesyncd[996]: Network configuration changed, trying to establish connection.
Oct 4 09:49:26 localhost systemd-timesyncd[996]: Synchronized to time server 91.189.89.199:123 (ntp.ubuntu.com).
Oct 4 09:49:26 localhost systemd[1]: Starting Update resolvconf for networkd DNS...
Oct 4 09:49:26 localhost systemd[1]: Started Update resolvconf for networkd DNS.
Oct 4 09:49:37 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:51:07 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:51:09 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:52:39 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:52:40 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:54:10 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:54:11 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:55:41 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:54:23 localhost systemd-timesyncd[996]: Network configuration changed, trying to establish connection.
Oct 4 09:54:23 localhost systemd[1]: Starting Update resolvconf for networkd DNS...
Oct 4 09:54:23 localhost systemd-timesyncd[996]: Synchronized to time server 91.189.89.199:123 (ntp.ubuntu.com).
Oct 4 09:54:23 localhost systemd[1]: Started Update resolvconf for networkd DNS.
Oct 4 09:55:43 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:57:13 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:57:14 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 09:58:44 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 09:58:46 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:00:16 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:00:17 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:01:47 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:01:48 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:03:18 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:03:05 localhost systemd-networkd[623]: eth0: DHCPv4 address 192.168.5.124/24 via 192.168.5.1 -----> here begin two ip address state
Oct 4 10:03:05 localhost systemd-timesyncd[996]: Network configuration changed, trying to establish connection.
Oct 4 10:03:05 localhost systemd[1]: Starting Update resolvconf for networkd DNS...
Oct 4 10:03:05 localhost systemd[1]: Started Update resolvconf for networkd DNS.
Oct 4 10:03:15 localhost systemd-timesyncd[996]: Timed out waiting for reply from 91.189.89.199:123 (ntp.ubuntu.com).
Oct 4 10:03:16 localhost systemd-timesyncd[996]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
Oct 4 10:03:20 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:04:50 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:04:23 localhost systemd-timesyncd[996]: Network configuration changed, trying to establish connection. ----> disappear any address
Oct 4 10:04:23 localhost systemd[1]: Starting Update resolvconf for networkd DNS...
Oct 4 10:04:23 localhost systemd[1]: Started Update resolvconf for networkd DNS.
Oct 4 10:04:51 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:06:21 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:06:23 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:07:53 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:07:54 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:09:24 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:09:24 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:10:54 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:10:55 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4
10:12:25 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ] ------> before this line interface eth0 don't have address, after this line the interfaces get 192.168.5.124/24( the second lise adddress)
Oct 4 10:11:49 localhost systemd[1]: Starting Update resolvconf for networkd DNS...
Oct 4 10:11:49 localhost systemd[1]: Started Update resolvconf for networkd DNS.
Oct 4 10:12:25 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Wed Oct 4 10:13:55 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Oct 4 10:13:18 localhost systemd[1]: Started Session 23 of user domotz.
Cheers,
Nicolino |
|
| 2017-10-23 10:10:28 |
Michael Vogt |
bug watch added |
|
https://github.com/systemd/systemd/issues/7163 |
|
| 2017-11-18 08:58:37 |
Launchpad Janitor |
systemd (Ubuntu): status |
In Progress |
Fix Released |
|
| 2018-02-20 15:40:24 |
Dimitri John Ledkov |
systemd (Ubuntu Zesty): status |
New |
Won't Fix |
|
| 2018-02-20 15:40:27 |
Dimitri John Ledkov |
systemd (Ubuntu Xenial): status |
New |
In Progress |
|
| 2018-02-22 18:36:13 |
Brian Murray |
systemd (Ubuntu Artful): status |
In Progress |
Fix Committed |
|
| 2018-02-22 18:36:15 |
Brian Murray |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2018-02-22 18:36:19 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2018-02-22 18:36:23 |
Brian Murray |
tags |
id-59de49a6bfa10c370d70d043 |
id-59de49a6bfa10c370d70d043 verification-needed verification-needed-artful |
|
| 2018-03-08 17:32:49 |
Brian Murray |
systemd (Ubuntu Xenial): status |
In Progress |
Fix Committed |
|
| 2018-03-08 17:32:56 |
Brian Murray |
tags |
id-59de49a6bfa10c370d70d043 verification-needed verification-needed-artful |
id-59de49a6bfa10c370d70d043 verification-needed verification-needed-artful verification-needed-xenial |
|
| 2018-03-19 14:22:09 |
Dimitri John Ledkov |
tags |
id-59de49a6bfa10c370d70d043 verification-needed verification-needed-artful verification-needed-xenial |
id-59de49a6bfa10c370d70d043 verification-done-artful verification-needed verification-needed-xenial |
|
| 2018-03-19 14:31:29 |
Dimitri John Ledkov |
tags |
id-59de49a6bfa10c370d70d043 verification-done-artful verification-needed verification-needed-xenial |
id-59de49a6bfa10c370d70d043 verification-done verification-done-artful verification-done-xenial |
|
| 2018-03-21 11:42:35 |
Launchpad Janitor |
systemd (Ubuntu Artful): status |
Fix Committed |
Fix Released |
|
| 2018-03-21 11:42:35 |
Launchpad Janitor |
cve linked |
|
2017-15908 |
|
| 2018-03-21 11:42:58 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2018-03-21 11:53:33 |
Launchpad Janitor |
systemd (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
| 2018-12-18 04:10:07 |
maria smith |
attachment added |
|
printer.png https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1721223/+attachment/5223123/+files/printer.png |
|
| 2018-12-18 14:08:41 |
Colin Watson |
attachment removed |
printer.png https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1721223/+attachment/5223123/+files/printer.png |
|
|
| 2019-02-26 10:00:11 |
john smith |
attachment added |
|
job-vs-career-og.png https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1721223/+attachment/5241750/+files/job-vs-career-og.png |
|
| 2019-02-27 00:31:21 |
William Grant |
attachment removed |
job-vs-career-og.png https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1721223/+attachment/5241750/+files/job-vs-career-og.png |
|
|
| 2023-09-20 11:38:40 |
Michael Vogt |
affects |
snappy |
snapd |
|
